Cancer Man - Need outside perspectives please

Im going to give a little history first. This cancer man and I have flirted at work for a year. We started texting each other and he came over to my house and we continued texting, but then it turned into me texting and that's when i laid off. Everytime I saw him at work, he would start flirting with me all over again, but then for months on end i would start dating a man and not even think of him, but when it ended i would wonder.

Most recently, about 5 months ago, we had agreed to fwb with the idea that we would see what happens if we are sexually compatible. I think he said this because he was into not your average, run of the mill sort of sex, not bad, just different. He is a very withdrawn and recluse cancer, and i know that he is hurt for whatever reasons and this is why he is so shielded.

But, the way things ended up being is that i would initiate all convos over text, he would only when he wanted sex. But then when I'd see him in person he would say we can just talk, we don't have to do anything, and he even suggested dinner the last time we were physically together. This is typically outside of fwb rules.

However, everything went very badly when i told him over text that i cared about him, he responded that that's not why we are doing this, and I said maybe we should stop, as that kinda hurt, I did think we were friends too. He then proceeded to ignore me for days, had never not responded to my texts. I lashed out on the last day and said thanks for hurting me, he responded finally with a lot of angry and told me to go f# $ % myself.

We do work together - and after several months have passed I decided i would reach out and say I hope he's well. One thing led to another and he wants to go back to FWB. I agreed, but am putting it off.

I don't understand why I am even considering it. We never even discussed the argument.

A part of me that maybe if I do it once, and stay aloof the situation and show him that I am over it, its just sex to me now, that that would make me feel better.

I'm not sure what I should say, but I definitely want to recover my dignity and self respect from how the argument went.