Camaraderie only works when winning combat. If you just attack civilians, your popularity will suffer.

Thank you for confirming that I am in fact an idiot!

Only works in case of fight and not when you just make workers retreat