People from Pakistan, Nigeria and Sri Lanka may find it more difficult to come to the UK, under new plans.