Boris Johnson says he was ‘at work’ during No 10 garden gathering