Dziurzynski's goal came from the top of the faceoff circle off the rush on a shot Price should have handled with ease at 6:49 of the second.
The Canadiens finally solved Bishop with their 30th shot when Markov ripped a point shot on the power play with just 4 seconds remaining in the period.
"The result was not on our side and the story of the game was that Bishop was outstanding," Canadiens coach Michel Therrien said. "I liked the way we played. We dictated the play, we played hard, we were aggressive on the puck and we did a lot of good things."
Tied 1-1 after two periods, the score stayed that way through the third and into overtime, which started 3-on-3 because of penalties. Ottawa got one skater back and enjoyed a brief power play before Kyle Turris was sent off for hooking. It was back to 3-on-3 before the Canadiens got their fourth power play of the game, but Montreal couldn't get the puck past Bishop.
The teams played to a scoreless first period, although the Canadiens did have the best opportunity with Max Pacioretty hitting the post behind Bishop. That was the beginning of the Canadiens' assault on the posts. Erik Cole rang a shot of the crossbar in the second before Raphael Diaz hit the post to the right of Bishop.
"It was one of those nights," Therrien said. "The story of the game was their goalie and he deserved a lot of credit for that win."
Notes: The Canadiens were in first place on Feb. 25 for the first time since 1993, the year they last won the Stanley Cup. ... Canadiens D Francis Bouillon played his 500th game with Montreal. He has played 695 total games in the NHL. ... Entering Monday's game Ottawa captain Daniel Alfredsson had 420 career goals. The rest of the Senators lineup a combined 524. ... After scoring four goals in his first four games, Senators F Kyle Turris has now gone 16 without a goal.