Clear up my Cancer Confusion

This is my second post, I posted back in Feb about a Cancer guy I was seeing for a few months who refused to have a relationship with me, despite us hanging out weekly, cooking together, doing things with our kids, and having weird romantic evenings that he would initiate to which he would act weird after. I know he was talking to other women while we were hooking up. We had a falling out of sorts in March because I "got weird" meaning I wanted too much and he was seeing this girl who was really young. We hooked up one more night (basically a booty call) while he was seeing this girl and he was very cold to me and I didn't like it. I was a typical Cancer chaser. So I backed off and didn't talk to him for a bit and about a week ago I texted him about a random question to which he answered and then after told me he had a sex dream about me and was being really flirty. So we ended up hanging out at his house this weekend and I was expecting it to be like the last time we hooked up strictly FWB but we ended up talking for five hours and having a great time, sharing a lot of deep stuff, laughing a lot, cuddle etc then we hooked up later (it was how it was when we first began hanging out again). He has been pretty aloof for the past week since. Im sure he's still talking to other women, and they are always horrible choices, trashy, they need help and way too young. (I was in a relationship for 10 years, I am established, own a house, have a good job and I know what I want in my life) I'm just confused about why he chooses to bond with me and have sex with me but can't take it any further. It always feels like we are on the same wavelength, we challenge eachother in a good way, complete eachothers thoughts, ideas etc and he's even said we are like twins before. I am playing it cool, backing
Off and not initiating conversation because I am afraid of scaring him away again but also bc i dont want to be his second choice. I care about this man a lot and I'm afraid to lose him as a friend. I am just very confused as to why he shares such deep things with me, will sleep with me, but then ignores me and keeps me at arms length? Help...

He has options and like any man who is not ready to commit, he's going to keep as many options as possible for as long as possible.

Why stop getting his cake and eating it too?

If you want more make it clear and break off all contact for as long as you continue having feelings for him. It seems like you were open to seeing other people from the beginning. don't worry about losing your friendship with him, worry about your self respect.

If you feel like you're the best option why should you have to sway him? He wouldn't be keeping options if he was looking for something serious.