I know exactly how you feel. Every vacation I've taken since 2005 (the first year I ever had a cat) has been diminished a little, or even a lot, by missing Murphy and worrying about him while we're gone. And we have a cat sitter who comes twice a day!
This is just too true. Ever since I got my Fab Four two years ago, I've started dreading going away. I feel that way several weeks in advance. I, too, have a cat sitter who comes twice a day. The reason for my guilt has evolved. At first, when Snowball was terrified of Blizzy (i.e., the first year), I kept her isolated in my bedroom when I went away, so I felt guilty she was bored. I also dreaded the damage Hersh and Blizzy would do chasing each other around the house, though I guess that was more fear than guilt! (Once, I found a five-foot by two-foot oak book shelf, that had been sitting on brackets five feet above the ground, on the floor of my family room when I returned home. Amazingly, it had not smashed anything to pieces en route to the floor.)
More recently, now that there is comparatively more peace in the household, I just feel guilty that they will be bored to tears. I can tell from their behavior when I return that they sleep more than normal when I'm gone for a few days (this is possible!), but they've never seemed worse for wear. My cat sitter writes me nice notes telling me how cute they have been, and how hungry they appear to be at every meal (an old trick).
Someone on this Forum once wrote about installing a remote camera and then looking in on his cats periodically. I thought that very clever but am afraid it would make me feel more guilty, not less.
On the bright side, I'm always really happy to see them when I return, and they seem super-affectionate the first few days I'm back--at least after Hersh seems to ignore me the first few hours.