What kind of milk are you giving him? Personally, I would cut out the milk altogether and stick to one good quality brand tinned kitten food, which you could try heating up a little in the microwave or with warm water (make sure you test it first). He needs kitten food to get all the nutrients he needs, and will not get these from human foods.
First, I would get him a full check up at the vets to explore the possibility of a medical problem. He needs to be treated for worms also, with products from your vets.
I would also recommend feeding your kitten seperately from your other cat. The kitten needs around 4 meals per day. Don't let your older cat eat the kittens food, as it will cause weight gain.