"It should be unacceptable," Zajac said of the loss. "We shouldn't have lost that game."
New Jersey has lost four of five.
"Between a big save or the group finding a way to stop the bleeding, we've got to find a way to win that game," coach Peter DeBoer said.
The Blue Jackets trailed 3-1 before scoring twice in 49 seconds in the second period against Brodeur, who came in 7-2-2 with a 2.05 goals-against average and .916 save percentage against Columbus.
Off a faceoff, Jack Johnson fired a big slap shot from the point. Atkinson got a whack at it, with the puck dribbling behind Brodeur where Dubinsky pounced on it before the goalie could recover.
Soon after that goal was announced, Dubinsky played keep-away with the puck, going twice back and forth along the back wall, before feeding Atkinson, who lifted a shot high over Brodeur's shoulder from a hard angle to tie it.
"It's nice to be rewarded," Atkinson said of the victory. "We haven't gotten the bounces in a while."
Calvert broke a 3-all tie when he scored on a rebound after Brodeur blocked Dubinsky's shot 18 seconds into the third period.
Despite a two-goal deficit with a future Hall of Famer in the opposing net, the Blue Jackets still found a way to get two points.
"It's too bad to mess up a game we scored four goals," Brodeur said. "It doesn't happen often that we lose a game like that."
NOTES: New Jersey D Marek Zidlicky returned to the lineup after missing the previous game because of an upper-body injury. ... Columbus was playing its second game of a back-to-back after losing 2-1 at Pittsburgh the night before. The Devils had been off since Saturday. ... It was the second of a three-game road trip, all against Metropolitan Division opponents, for the Devils. ... Johansen extended his career-high point streak to six games (2-6-8).
Follow Rusty Miller on Twitter: http://www.twitter.com/RustyMillerAP