He told The Andrew Marr Show: "We seem to have the numbers in the House of Commons. A lot of people say 'get this done and move on'"