'You don't vote for me' comment was a 'little joke'