Iain Duncan Smith said it's time for 'expert to handle Brexit'