It depends on what “few things” you are getting tired of.
4 years should have yielded some valuable experiences by now like: seeing product shipped, working with other departments, budgeting, hiring staff, expanding your departments capability and hiring outside resources. There are also the other more unpleasant sides of design management: firing staff, legal issues, knowing the task and schedule at hand and not having the resources to do the job the way you visualize or to your standards of design quality and so on.
I suggest this…Take some time to get away from the office and put together a plan for your current company for the next five years. Write it up, and present it to the right people at the right time. See what you can get and what the company is willing to give you if you can show them what you want and how you can deliver on what you promise. Doing this will let you know how much potential your future has with your current company. If they balk, you know that you must take your ideas and plans else where.
Good luck…