Welcome! Congrats on saving a kitty and giving him a home! For one, definitely do not let kitty out at this time, you may never seem him again. I brought home a stray and gave in to her demands to go outside and had to spend the next 2 days walking around the neighbourhood looking for her until she came crying to our door. She was lost and very hungry. If you are feeling guilt already (you sound like me!) you may want to consider keeping him strictly indoors if you are not in a safe area. I live by a busy road and even thought there are days I really want to let my girls out, I can't live with the thought of them getting hit by a car, but it is totally your choice, just know many many cats are happy being strictly indoors.
Secondly, it is advisable that you keep you kitty in a small area as coming from a cage to a big house can be very stressful. I would say let you kitty out when he starts to show interest in exploring the rest of your house, 4 weeks is just a guideline I am guessing, I am sure you will know when you cat is ready. In my case, we adopted Sully and he is around 5 years old, he spend his first month in the closet and then gradually started exploring. We moved him to the kitchen so he could be part of the family and it's been almost 2 months and he is starting now to want to come out and explore. Sully has fear anxiety issues so I am sure your cat won't need that much time.
Thirdly, it is true cats sleep most of their day away. I stress about mine being left alone for over 12 hours a day but when I get home at night they are sleepy eyed, haven't touched their food and rarely use the litterbox so I know they slept most of the time I was gone.
Hmm..I never had litterbox issues but many have, you will find lots of advice in the behavior section.
Good luck and don't stress!
your kitty will come around when he is ready!