Brumberger said his misgivings grew after Hunter demanded to travel with Edwards on private jets, rather than commercial flights like other staff and consultants.
"Ms. Hunter felt she pretty much had an all-access pass to everything," he said. "I disagreed."
Brumberger said he attempted to bar Hunter from the flights, but the candidate overruled him.
Edwards also ordered Brumberger to make sure the PAC paid for Hunter's health insurance, unheard-of for a consultant not on the full-time staff. Concerns were also raised among senior staff that Hunter didn't appear to know much about shooting video. Tapes filmed by Hunter played for the jury showed shaky camera work where those speaking were sometimes not in focus or not in the frame at all.
"It was shoddy and unprofessional," Brumberger said.
Brumberger also described accompanying Edwards on his first trip to the Virginia estate of donor Rachel "Bunny" Mellon in December 2005. The visit unfolded pleasantly with her recounting her visits to the Kennedy White House, where she helped plant the rose garden.
Soon after, the wealthy heiress made the first in a series of substantial donations to Edwards' political committees and his anti-poverty foundation that would eventually total more than $6 million.
Prosecutors said Edwards used money from Mellon, who's now 101, and another wealthy donor to hide the mistress. Edwards' attorneys have said he didn't know it was being used to hide her and that another former aide, Andrew Young, spent much of it on his dream house.
Brumberger described how Edwards and his associates made efforts to stay in touch with Mellon, including calling on her birthday and sending flowers. Brumberger said that it was typical for Edwards to have "call time" with major donors.
Months after the first meeting with Mellon, Brumberger was traveling with Edwards when he called her on her birthday from North Dakota. Brumberger sent Young — a key adviser — an email that it had gone well.
"JRE called. Bunny is still in LOVE," Brumberger wrote in the email, referring to Edwards by his initials.
After the email was displayed in court on Wednesday, prosecutors asked Brumberger what it meant.
"I believe what I meant by that is Ms. Mellon was still supportive of Mr. Edwards's causes," Brumberger testified to laughter in the courtroom.
Prosecutors have said Mellon offered under-the-table cash to cover Edwards' personal expenses after the candidate was embarrassed by media reports that campaign funds were used to pay for $400 haircuts.
Weeks later, Hunter informed the candidate she was pregnant. According to the account in Young's 2010 tell-all book about the affair, Edwards was unable to access his own money to support Hunter without his wife, Elizabeth, finding out. So, Young says, Edwards decided to take Mellon up on her offer.
For most of 2006, Hunter traveled with Edwards for months to meetings across the country, as well as on an overseas trip to Africa.
Concerned about the affair, Brumberger said he twice tried confronting his boss. After Edwards made no effort to send Hunter away, Brumberger said he talked to two senior staff members.
Edwards learned of the meeting and confronted Brumberger in a private lounge at the Chicago airport just before the men were supposed to board a plane to China.
Using expletives, Edwards began yelling and his face turned red.
"He said he couldn't trust me anymore," Brumberger recounted. Edwards informed the aide that he might be fired.
Brumberger said he quit, instead.
"I told Mr. Edwards I was no longer interested in working for him," Brumberger said. "I was kind of in shock."
It was almost exactly a year before news of Edwards' affair began to tumble out in the heat of the Democratic primary campaign.
Follow AP writer Michael Biesecker at twitter.com/mbieseck