Labour MPs are worried removing the payments from millions of pensioners is damaging the party.