Why should either type of fans concern themselves with the other?
The casual only reads the story when they come around to it, then dip to enjoy some other content. They don't come into contact with the hardcore fanbase, as they rarely frequent forums outside of the surface discussions upon the release of a chapter.
The hardcore obsess over the story weekly, sometimes even daily. They know more than the casual does or can, getting deeper into the minutiae of the story. Even comparing the wealth of knowledge alone, the casual can't come near the hardcore, so there's no debate between them other than personal preferences in terms of characters of plot points.
If the casual enjoys One Piece on a casual basis, that's their mode of consuming the story.
If the hardcore enjoys dissecting One Piece to its very atoms, that's their mode of consuming the story.
They're two separate worlds.