AEverytime Hrothgar would stand against Grendel and challenge him to a fight, Grendel would reply, "Not you." It's not until the end of the movie where it is revealed that Hrothgar is Grendel's father. This development is not part of the poem.
AA definitive answer is not given in the movie (nor in the poem). A good guess is that Grendel was able to shape change, e.g., into the mist that can occasionally be seen floating in the background. Another possibility is that Grendel was able to turn himself invisible, taking form only when killing.
ACutting off Grendel's arm and displaying it on the roof has resulted in easing everyone's fears. Hrothgar opens the wine cellar, the villagers abandon their watch around the outpost periphery, and Beowulf prepares to depart, thinking that his work is done. Although everyone thinks that Grendel has lumbered off to die, they are wrong. He has returned to the cave where he lives with his mother (Layla Roberts), who is now eager to avenge her son. She sneaks into the castle and seduces Roland (Götz Otto) while, at the same time, Kyra (Rhona Mitra) attempts to seduce Beowulf into staying with her. Suddenly, Beowulf senses the presence of Evil in the outpost. As Beowulf goes in search of it, Kyra warns Hrothgar, and they head for the dining hall to warn the others, only to find everyone already slain, including Roland. Grendel's mother appears before them, explaining to Hrothgar that she is the succubus who has been sharing his bed and that Grendel is Hrothgar's son. She further explains that the outpost was hers long before Hrothgar and his people moved in and that Grendel has merely been trying to claim what, by rights, belongs to him. When Hrothgar goes for her, Grendel shows up and pulls him off, crushing him to death in his strong grip. As Kyra lays on the floor in a daze, having been batted across the room by Grendel, Beowulf drops from the ceiling and plunges his sword through Grendel's chest. His mother attempts to seduce Beowulf, but he resists, so she transforms into the monster that she really is and tries to kill Beowulf. Just when it looks like she's got Beowulf by the throat, he uses his sword to slam open a gas jet in the wall. The firey gas bursts forth and sets mom on fire. As the whole outpost goes up in flames, Beowulf and Kyra, the only survivors, escape on horseback. In the final scene, Kyra convinces Beowulf to take her with him.