I think you have the right perspective, you have a lot of your career left so keep pursuing fields that interest you.
I can relate to both sides. ID is a very competitive field if you are looking for jobs that give the designer creative freedom and understand the product development process. You have to be very talented. The good thing is that with dedication and passion you can get your skills and knowledge there. But, if you are not in the top caliber group you’ll likely be in your current situation or out of the field.
I’ve managed to keep learning and remain determined to work in the “cool” design jobs. Eventhough i’m not completely there yet, I see my dedication paying off. If you are serious about it and make design your lifestyle, it’s possible.