Minister appointed to negotiate Brexit wanted EU 'torn down'