COVID mask mandate calls from Victoria's top public health official have been rejected by the state's Health Minister.