Families Disappear? WTH is going on?

Im mid game, nothing bad going on and I suddenly have no one building. Free population is at 0. So I unassgn a worker from somewhere and he disappears. Worker count still at 0. Do this 2 more times until suddenly 0 changes to a 1. I'm down 3 people and I don't know why. Anyone else?