Yep, I hear ya. Life is also about tradeoffs. Which you allude to. If one piece of pie is humongous, then the other pieces will be smaller, and some eliminated.
I recall talking to someone years ago about some book or books I was reading. She said "I don't have time to read." She watched a lot of TV.
I know you are very active. Me, not so much. But I am active. I believe in trying to do something structured 30 minutes a day. I lift 3 times a week. The other 4 days, yoga, a stretch, or maybe tai chi or some joint mobility work. Walk the dogs 2 times a day minimum. My point is that this time is a trade off. It cuts into TV watching, or Internet surfing.
I've read about workaholics, who spend most of their time working. I don't think that this is an enjoyable way to live. Maybe I'm wrong .. but to me a balance is important. Sure there are days, maybe weeks where something is going to consume you. If a family member is in the hospital, if there is a work project, if you decide to go on a "bootcamp" kick ..