That's the point of contention. It boils down to the fact that the boss is paying you to work on his stuff. Instead you work on your own stuff.

It's a different story if the boss says spend your free work time doing whatever you want, or develop some skills of your own choosing. In my experience that's usually not the case.