Conservatives will never win power again after Brexit 'disaster', former Tory government adviser says