A crowd attacked the leisure centre hours after a minister revealed that migrants were being temporarily housed there.