I'm just starting out, and my plan is to figure out how many hours I want to have to spend caring for my trees and then limit my collection accordingly.
At present I want to need to do something with my trees at least a couple of times a week, but I don't want to be overwhelmed by their needs. I think somewhere between 10-12 trees plus my various propagation efforts might be enough for me.
If I go on holidays, I would consider taking my most vulnerable trees with me! Or set up some kind of drip watering system.