Lawyer Milloy turned down Patriots ultimatum