They've been trying to push him out