Your Sims also have personal ranks, and lowering the prestige and unity meter may also lower their rank, and another member could challenge your leadership. And then you can just start a new one, so ...