It seems like there are pros and cons to starting from scratch with a bachelor’s versus master’s. It might help to contact the admissions dept at schools you are considering and ask what they would recommend.
I’m having the exact problem too and it feels like it depends on the program. Some like IIT welcome non-designers to their Master of Design program where others might expect you to have more of a design foundation. It also helps to have the work experience (which I do not have). It seems like going the masters’ route might be cheaper and save time but does it provide the skills to become a designer?? Is it the same experience?