Boris Johnson has insisted the Government's three Brexiteers are a "nest of singing birds" as he faced accusations of changing his mind over EU single market access.