The subject matter of athletics / sport is irrelevant. Everyone likes to spend their ‘free’ time doing something whether it be reading, watching movies, collecting sweet Elvis memorabilia, etc. The root cause of the issue is that you have a career that you are passionate about and find yourself striving to improve and advance in even while outside of work. This is a good thing. If you were working a job you had zero interest in and treated it strictly as a source of income I bet you would find you had ‘more’ time to dedicate to other interests.
Just wait until you have kids. That’s when the real trade-offs happen.