How to have a relationship with a cancer man?

Seems you weren't really paying attention to what's going on with him, if he didn't want to go why not say okay see you when I get back, if you knew it's been only a few months after a 4 year break up and then turn around and want to go one on one on a holiday well that's moving way too fast.

Reality is, he could still be attempting to make the 4 year relationship work with his ex, you are a rebound relationship for him thus why he can forget about you so easily after he's out of your presence. If I were you I wouldn't put all all of your hopes into this one man, he most likely won't be the guy you hope he'll be. I'm sure after such a fresh wound from breaking up he's still mulling over that failure emotionally.

I suggest you take a deep breath, stop being so damn serious all of the time, lighten up and get busy with your own life interest, if he's only present when it's convenient for him there is NOTHING you can do to change that so stop being whiney and pissy over his behavior, he's still in love with someone else even if they're not together anymore, if you're patient and allow him some space to get over it he'll reward you with more of his time and energy, if you can't be patient and have to play phone tag games and act pissy then you probably should bail out before you get your heart broken.

Good luck!