PM leaves Downing Street for Queen's speech