We live in a social world. Our world is about people and relationships. While it may be true that what you think of me is none of my business on some level, there is another level where, if I want to continue to be a compassionate person with meaningful relationships, I need to be able to examine how my own behavior affects others.
Your buttons are your buttons. They are made up of your own experiences and interpretations of life. They are your story. That's true, and it's important for us to realize that when our buttons are pushed, we were the ones that made them buttons in the first place. But if I know what your buttons are, no matter what I think of them, I don't need to push them - and I can take responsibility for my part in things when I do push them, whether I push them intentionally or not.
Relationships are complicated. They would be a lot simpler if we would just be willing to step outside ourselves a bit more and take things personally a bit less. If I have hurt someone unintentionally - even if I think it is unreasonable that they are hurt - what does it harm me to just apologize? Apologizing doesn't mean I'm taking the blame; it's an act of compassion. I don't have to take responsibility for someone's feeling hurt (that's their own responsibility), but I can certainly say that I never had the intention of being hurtful and that I'm sorry that's what happened. And then at least there's room for a conversation.
I think sometimes it's easy to take things too literally, and the end result of that is dangerous. All one needs to do is look at many of the word's religions to see what results from taking things literally. But the same goes for New Age wisdom (which, of course, isn't new at all), and I think it's really important for us to remember that.
So, while it's true that what you think of me is none of my business, what I think of you is my business, and I can choose to love you or take things personally. I can choose to understand and have compassion for what you think (which includes your reactions), or be hurt and callus. Love and hatred are both two-way streets, and they each start with us. It's good to remember that.
The wisest words ever spoken were "Do unto others as you would have them do unto you." Why? Because they are the litmus test for wisdom. Measure these words against anything, and you will always be guided back to a place of love.