Before you wander off, have you tried this yet?
"Hey, I miss you. Do you miss me? When you go from communicating all the time to pulling back, and then back and forth, it bothers me, because I don't know what to think. I like when I can predict something, because it makes me feel comfortable."
You were so worried about pushing him away, and now you're ready to leave, yourself. What can this hurt?