Several years ago I met a fellow designer who pretty much hated Forsythia. There was nothing good about it and no place for it in the garden design world. No exceptions. (This became a topic for many jokes between us).
This was the first time I had come across plant prejudice and since then, have experienced several instances where distain for a particular plant were indeed held - and found myself absolutely cringing at the thought of certain plants (think Picea glauca conica, or Dusty Miller). It seems that many times it comes down to color, but sometimes habit, form or scent create the basis for our opinions. Or maybe the plant has been overused, is dated, or has a stigma attached to it. Brings back negative childhood memories? I've heard that one too.
Now I don't think that all plants belong in all places, or in all gardens. I have specifically not used a plant for a certain site or in a certain area, simply because it would be happier somewhere else. And happier isn't referring to soil, sun or water requirements (though sometimes) - but rather, what the plant wants as far as atmosphere, companions, and even it's caretakers. And sometimes a site needs a certain energy, which some plants can't provide.
Forsythia!
Over time, I have come to the opinion that every plant, even a tumble weed, has a garden or a landscape waiting for it. It's just finding that right site and many times the right combination of plants for it to grow along side. Some plants are tricker than others to place indeed, but they still have a garden to which they belong. Even a shrub you generally struggle with can surprise you with how well it fits into a particular space. So many times it comes down to context.
In the end, it's not so much about what doesn't belong in a specific garden as what does.
And I like to think that all plants have a place they belong.