Hi all, OP here.
Thanks for all the replies
The cat is spending much time outside. But he visits us like 4 times a day now. And he does sleep at the new heated cat house we bought. We bought a clear shower curtain and covered the cat house with it, so rain doesn't get into the cat house.
Cat does seem like the new house a lot. He doesn't wanna come out when we go out to check him. So we are happy with it.
The reason I can't have him inside is because we have two mid-size dogs also.
Now they are all over 12 years old, and all dogs are kinda OK with one another now. But our oldest dog is pretty aggressive, So we can't keep the cat inside.
Maybe when old dogs leave us, we will let him inside. But not just yet. It won't be good for either.