Paul Stastny and Cody McLeod also scored for Colorado.
Varlamov's strong play beforehand helped Colorado maintain a 2-1 lead. The Sharks outshot the Avalanche 11-3 in the second, many of those coming early when they were trying to tie it.
Joe Pavelski and Dan Boyle scored for the Sharks, who are 1-2-2 in their past five games but still hold a slim lead for first place in the Pacific over Anaheim.
Goalie Alex Stalock had 19 saves for San Jose before being pulled for an extra attacker in the final minute.
"I thought we played extremely well, thought we generated a lot of chances," Joe Thornton said. "It's just sometimes you don't win those types of games."
Boyle cut the lead to 3-2 when he took a pass from James Sheppard and scored off Varlamov's left shoulder 2:03 after Mitchell. It was his 10th goal of the season.
The Sharks got a chance to tie on a 4-on-3 power play after Ryan O'Reilly took his first penalty of the season, a minor for playing with a broken stick.
Stastny's 26th goal 1:33 into the game gave Colorado a 1-0 lead. Pavelski tied it on the power play less than two minutes later with his 38th.
The Sharks had another power-play chance midway through the first, but McLeod sneaked behind the defense, took a long pass from Nick Holden off Boyle's glove and beat Stalock to give Colorado a 2-1 lead.
"I took my eye off the puck for a fraction of a second to make sure I wasn't going to get drilled," Boyle said. "I just caught a piece of it and missed it. That's an unfortunate bounce."
NOTES: Mitchell returned to the lineup after missing four games with a back injury. ... The Sharks reassigned F Freddie Hamilton to Worcester of the AHL on Friday. ... McGinn played after skipping practice Friday. He took a shot off his foot in Thursday's win over Vancouver.