Andrew Bryant, a planner with the Lincoln County Planning & Inspections Department, said no permits had been issued.
On the tape, Arwood said he didn't know what happened.
"They were inside the hole helping to get something and the wall collapsed," he said.
At one point, the dispatcher warned him not to put pressure on the dirt. But Arwood said he had to reach the children.
"If this was you and your children in the dirt, you'd be moving the dirt, too," he said.
Arwood's house was at the end of a gravel-covered road dotted with modular and mobile homes. It's a tight-knit rural community where neighbors sit outside on front porches and look out for each other.
When word spread about the disaster, they ran to Arwood's house and began helping. On Monday, they were somber, saying they were heartbroken for the family. They said Arwood told them it happened without warning and that he tried to grab the children, but they were just beyond his reach.
It was no secret that Arwood was digging a two-story deep hole. Neighbors said it wasn't unusual to see children in the pit when the girl's father was working there.
Neighbor Bradley Jones, who works in construction, said there was no structure to support the pit's tall dirt walls and that there was some concrete on a ledge on top of the hole.
In recent days, the hole was muddy from the rain. He said he warned his daughter, Chelsea, who babysits for the children, not to go in.
"It was dangerous. There was nothing to reinforce those walls," he said.
Chelsea said Arwood told her that he was building the structure to "protect his family" - it was going to be a bunker.
"It's so sad," she said.
Biesecker reported from Raleigh. Associated Press news researcher Monika Mathur in Washington contributed to this report.