First, keep him indoors. If he got loose outside he could run/hide out of fear and not know how to find his way back. He may never be comfortable going outside. If you want to get him used to it, I'd only offer him an open door to explore outside for a few minutes at a time before you bring him back in. Always leave the door open for him to dodge back in if he gets frightened. As he gets more comfortable, then he can stay out for longer periods of time. Myself, I prefer to keep my cats indoors as we have too many dangers to lose them to, here.
For socializing, keep making a fuss over him and getting him comfortable with you handling him. Then you can ask people (quiet type people) to come over for a bit and try to get him used to having other people around and eventually handling him.
Give him time. All his life he has been very sheltered with only one person. It will take him time to assess all these new experiences he is encountering; new home, new person, other person he's used to is 'gone', new visitors, new sights-sounds-smells... Keep in mind, some cats remain quite shy and may never be fully comfortable. Also, it is okay if he hides when other people come over. If you go out of town, he can hide when a caregive comes to care for him. As long as the food/water and litterbox look like they are being used normally, that's okay.
...gotta run, I think the horse shoer is here!