Well, you don't have to. You can keep it closed, and you can pick and choose what you want to listen to sometimes too. But if someone's talking publicly, there's not a problem in listening in, unless you think they deserve privacy. Besides, filters are there for that.

And just because you carry it with you doesn't mean you have to talk if you don't want to.