Do you know his background? How old is he? Also, how old are your children?
Way more time is needed! The short answer to your question is YES
! This cat would rather be with you than at a shelter and that you won't be teaching your children anything positive in returning him.
Some cats take a long time to come around. It can be an extremely rewarding experience to take a cat that is fearful and watch it transform into a loving pet, all with some time, patience and a building of trust.
Since he's hiding under the bed... you're right, that's not getting anywhere. You need to block the bed off from that cat. He should have a place to hide, a location you don't disturb, but one more like a cubbyhole instead of a whole bed to hide under. I would start him off in a spare room, if you've got one, or your bedroom. The cat needs to feel comfortable in one
room before being given an entire house to explore.
I'd do some reading here on taming cats, which can also in general apply to fearful cats:
Taming Feral Kittens and Cats
When I was about 15 I tamed a kitty (Blacky), it took a year until she'd let me close enough to pet her. At first she was scared of the entire house to the point that she leaped out a broken window to get back outside. Eventually, she's become a cat that sleeps on my bed every night and often on my lap at the computer as well, and is just a really awesome cat. I also recently tamed a cat (Jasper) that was aggressive. I needed leather gloves around him, and he was so food aggressive he would attack viciously when I tried to pick up his food dish. It's been about four months and he's totally transformed into a cat that plays fetch (and knows not to use claws even), isn't aggressive about food, wants to roll around on my lap and chirps/meows all the time. When I was about 8 I had a feral kitten tamed as well (Blaze)... he was more fearful than anything, I remember he didn't move off the couch the first month. Even a year later his personality was still emerging. It was great to watch. It just takes love and trust.
That's the type of lesson you should try imparting on your children.