Tracy's father Emmet, of Grosse Pointe Farms, Michigan, said that in his last email his son had asked for some airline miles so he could fly to the United States so they could be together for the father's 80th birthday.
The prosecution said he would be brought to a court hearing Thursday to be formally charged under Venezuela's anti-terrorism laws.
The police had been friendly to Tracy during the previous incidents, with some even agreeing to appear in his documentary, the filmmaker's father said. Emmet Tracy said, however, that the family had begun urging his son to leave the country in light of the volatile political situation.
"Frankly it's the kind of scenario that we were concerned about and kept telling him," Emmet Tracy said.
Tensions in the country have been rising since Maduro beat opposition candidate Henrique Capriles in the April 14 election by less than 2 percentage points. The government insists the opposition fomented violence directed at ruling party supporters and official buildings in the days after the election. The opposition is demanding an audit of the vote, which it says was stolen.
Venezuela's government has long accused the United States of trying to undermine it, moving closer to Cuba, Iran and Russia after a failed 2002 coup attempt against Chavez that the George W. Bush administration initially recognized.
Tracy is the first American in recent memory to be detained in Venezuela on politically related charges, however.
"I gave the order that they detain him immediately, hand him to prosecutors with the proof that there is because nobody can be destabilizing this country, whatever they believe, because they're on the side of the bourgeoisie, no," Maduro said.
James said Tracy's Spanish is passable but not great.
He said Tracy "literally has no political agenda. He is very sympathetic to all sides. He's telling stories about people and what their life is like there."
"He has been involved in telling stories that told that international component. But he certainly never worked for the government," said James.
"He's trying to tell a human story," said James. "My fear is that he's gone in deeper than he should have."
Bajak contributed from Lima, Peru. Associated Press writer Luis Alonso Lugo in Washington contributed to this report.
Frank Bajak on Twitter: http://twitter.com/fbajak
Michael Weissenstein on Twitter: http://twitter.com/mweissenstein