It seems that many men have the same issues with neutering as you do, I think because they think of things in terms of how a human would react to something like this. Humans seem to have a psychological attachment to their genetalia, but animals don't. Your cat is not going to care that he has balls or not. He is not going to be any different without them except that he won't spray, won't hump your leg, won't be trying to get out to find intact females (and don't think he can't... unnuetered toms have been known to tear out the screens of windows just to get out and find that female in heat) and won't develop reproductive cancers.
I had to deal with this same issue with my bf, since our kitten Cheddar got neutered last week. I swear it was more painful and upsetting to him than it was to the cat. Cheddar came through just fine, has healed up well and is the same cat he was before the surgery.
Your cat will be better off if you have him neutered and so will you, and you don't need to feel guilty or bad about it.