Hun, I'm sweet too, I'm Libran and dont like peole thinking badly of me BUT....I've also recently learnt that people take advantage of sweet people so I've learnt to stop things before they get to that situation or quickly put a stop to something once you realize it's going down the wrong track. Generally it is much better and has less damage to nip it in the butt quickly!
Honestly he will only be upset if he is in fact wanting something more from you than just friendship because by pulling back you are telling him that you dont want that type of relationship with him. If it is just friends then no harm done!