Former Tory minister says Boris Johnson is "the author of his own misfortune"