I know others will have their own opinion but from what I've researched and experienced this is my what I've come to believe.
I think that when we domesticate and socialize a cat we have kept them in a 'juvenile' state of mind. That is they are kept in a kitten like state. Yes they grow and mature, but they keep a kitten point of view. What do kittens long for? A mommy and litter mates. Well, and yummy food
A house kitty wants a mama, and through human socialization we teach them to look to us for that role. That's one reason they long to sleep with us for example. What do kittens do? They want to cuddle with mommy and litter mates as they sleep. This is a natural role for them and creates bonding. Have you ever noticed that a cat tends to love the whole family, but picks one person above anyone else? That is their mommy and everyone else is beloved litter mates. What do kittens do with litter mates? They play and cuddle with them! Domesticated cats long to do this.
True ferals have been thrown into a world where they are not given the luxury of keeping this innocent mind frame. The are young, then they grow up into a survival and realistic state of mind. That is they are brought into an undomesticated and non-juvenile state of mind. They have to live, they must breed and continue the cycle, they must claim territory, etc. Adult cats of most all species (except lions) are solitary creatures when they reach an adult state. They have a mommy and litter mates, then they mature and leave to live a solitary life where they pick their own territory that is theirs alone. They never again long for a mom or litter mates. The only reason a lot of ferals don't live solitary lives is because of circumstance. Generally because there is a steady food source and limited places for them to go they tolerate living with each other. If a feral had a choice though it would be a solitary cat with it's own territory, a cat that only comes together with other cats for the purpose of mating. This is one reason why there is so much fighting in feral colonies. It is not natural for them and causes a lot of stress. Cats, when not in a juvenile state of mind, are not naturally pack animals. From small ones to big ones (once again, the only exception to this is lions). I encourage anyone if they are interested to look into this. It is fascinating! This is one reason why rehabilitation of adult ferals is such a long process, and doesn't always guarentee success. You are trying to rehabilitate them back into a juvenile role. A role that they have grown out of and most definitely never looked to humans to fulfill. This is why building trust with feral adults is so essential. You can not force yourself upon them.
So then you've got strays. Cats that were socialized and domesticated but then thrown into a world where they must fend for themselves. They search for their 'mama' but she isn't there. They are stuck in a confusing role of having one state of mind but being thrust into another. This is one reason why strays seem to have such a hard time and stick out like sore thumbs in a true feral colony. The quicker a stray can be picked up off the streets the better in my opinion. The longer they stay on the streets the more deep seeded their confusion seems to be. I have seen strays that were forced to live in feral situations for years. The psychological confusion is so evident in them. They think both like a feral and a domestic kitty, and they are trying to figure it out. For example it would be like growing up in New York City. Your social structure is one way and the world around you is one way. Then you wake up in a village in Africa. The social structure will not be anything the same, and the world around you will be different. You will learn to adapt and survive in your environment but the mindset you grew up in will always be a part of you and make you different then native born Africans. It will affect how you perceive things. Once again we rehabilitate them back into a juvenile state of mind in order to make them house pets. This would contribute to why some of your strays want to around you yet are nervous. They are battling trying to combine two worlds that don't really blend well. So then you work with them, slowly bringing them back into one mind set. It is harder the longer they've lived on the streets but a very possible rehabilitation. You are doing right by them by continuing to interact and work with them.
This is just my opinion based on my rescue work and research. I am more then okay if someone has a different thought and look forward to hearing them, I just wanted to share mine