A-Lister Reveals Surprising Reason He Turned Down SNL