Colbert Reveals ‘Heartbreaking’ Part of Hasty Set Exit