Lord says 'we're thinking of nothing but the rights of foreigners'