Emma Roberts' dad Eric Roberts cites 'the loss of relationship with my daughter' as his toughest period