Former Tory minister says PM’s Savile slur ‘a mistake’