Asking people what they do for fun

When I ask people what they do for fun, most people either instantly reply or awkwardly try to think of something. There's no right answer; I'm just curious. The fact that you enjoy watching / playing sports, reading, playing board games, or DIY electronics helps me to get to know you better and quickly establishes if we share interests.

Recently, I was told by someone, who I don't view as a workaholic, that this seemed like a loaded question because it felt like I was casting judgement on people who don't have hobbies. While that wasn't my intention, it's made me wonder why there are people who don't prioritize having fun.

I suppose some view being busy as a proxy for being important / powerful. So, they wear the fact that they don't have time for fun as a "badge of honor".