Boris Johnson told to 'leave my town'