Conservatives using Brexit as 'licence for xenophobia'