'Harry Potter' star Miriam Margolyes has told adult fans of the franchise to grow up because they "should be over that by now".