Most adult cats are lactose-intolerant and will develop diarrhea (or vomiting) from dairy products. Keep an eye on your cat's litterbox. If his stool turns soft or runny over the next couple of days, or if he starts vomiting, you'll know that giving him milk is a very bad idea.
If he's an older cat, dairy also isn't a good idea. Dairy is high in phosphorus, which can be very hard on aging feline kidneys.