Priti Patel said "it's absolutely virus" that Covid measures remain in place