Princess Diana's brother explains why The Crown 'worries' him