Boris Johnson refuses to apologise for ‘gross incompetence’ leading to restrictions