I wouldn't drop someone, but give them pointers on how to improve losing weight, workout and eat healthy together.
Even if we would eat out a lot I'd still want to go for a run/walk or do some other activity/sport like kayaking with my partner.
Best to not see it as a chore, but another form of quality time.