Queen Elizabeth did not back Brexit says former Buckingham palace communications director Sally Osman.