In my experience, 5 seems to be the average critical mass for spraying, fighting, and other territorial behaviors. It's probably best to stick to 4 or less, unless you have a lot of square footage as well as vertical space, several good places for litterboxes, and the time to devote to giving everyone their fair share of attention!
Cheers,
Dr. Jean