There seems to be a steriotype that 'the annie mays' are a think that 'the kids' watch. Kids who aren't even into anime seem to have picked this up--from parents and grandparents, which weirds me out no end. It leads to some weird situations.

But yeah, you totally have the right to like whatever you want; there are adult-only anime cons, I think that's pretty clear cultural ok on liking it as an adult.