I live in a culture of success. I'm being constantly told that if I put my mind to it I can achieve anything. "Yes we can!" It's great - it gives me motivation when I need it. When things get tough I am reminded that if I truly believe in my dream I can move mountains. I would not be where I am now without this push. It's awesome, but not for all dreams.
I call those other dreams dreamitts. They are bad dreams, but unlike nightmares, they will not scare you away. Quite the opposite, they are like bright light that lures you in. It's a candle shining in the dark and you are the hopeless moth - when you get too close you will just burn. Dreamitts will make you suffer while trying to achieve them. You will fall. A lot. You will hear "You can do it!" stand up just to fall again. And since dreamitts are so unattainable they seem much more valuable than they really are.
Our lives are short, our energy is limited. Some dreams are just not worth pursuing. Sometimes it is just better to let it go: to admit it's too hard and too painful and to channel your creativity, passion and energy into something that is actually worth it.
But don't think about it as a failure. It's a terribly difficult decision to make - even if painful, status quo is less risky than a change. I would not call it giving up either: it is putting your awesome potential into something better suited.