I know the hardest part for me wouldn’t be school, but it would be when I get my first job that pays $35k and I have some 30 year old lecturing me on how business works. I would suggest doing some soul searching on if you are ready to accept 2-5 years of doing technical drawings or renderings of someone else’s design (the reality out of school) before ditching your current career.
My uncle told me when I was 10-12 years old that I should go into something to make money and then find a way to do design on the side. Maybe that could be an alternative for you? Is there a product that you would like to develop yourself? You mention shoes. You can find shoe factories pretty easily online (Moreover, I’m sure some people here could help you find a good one). I’ve seen low production runs that could be done for a small investment. If you can try to make it a business, but can live with it being a hobby, maybe that’s right for you.
Best of luck whatever your decision.