I don't vacuum every day either. Sometimes not even every week (shh!). But I keep a mini dustpan and brush by each litterbox and sweep up after I roll the litter mats and shake them out over the toilet (I have flushable litter). I try to do this whenever I see that someone's been in the box, as the problem is definitely worse if I don't sweep up frequently.
Oh yes, fur. I always had shorthairs and they still left spectacular amounts of fur. I agree with cat owner again about attacking the fur at its source. I brushed my Celia every day, and still, in the summer, if she'd been lying on her side as I was brushing her, once she got up, you could tell exactly where she'd been as there was a fur outline left behind. I used either a small rubber brush or a damp paper towel to get the fur off fabrics as soon as I could, to keep that fur from traveling. That at least generally ensured that my clothes would come out of the dryer without those little clumps of fur.
It helped that my kitties generally slept in their beds, too, so I didn't need to do this everywhere. I only focused on the places where they spent a lot of time.
Who knows, maybe vacuuming would have been faster in the long run. But to me getting the vacuum out required a lot more effort and motivation.