Companies "are just backing off and not making any moves until things clear up a bit," Bradley Holcomb, chairman of the ISM's survey committee, said.
Consumers also appear nervous about higher taxes. Economists cited the fiscal cliff as a key reason consumer spending fell in October by the most since May.
When consumers cut back on spending, businesses typically reduce their pace of restocking. Both trends are expected to slow economic growth at the end of the year.
The economy grew from July through September at an annual rate of 2.7 percent, largely because of strong growth in inventories. Most economists predict growth is slowing in the current October-December quarter to a rate below 2 percent.
Superstorm Sandy had little impact on factory activity last month, according to the ISM survey. The storm hit the East Coast on Oct. 29 and affected businesses in 24 states.
A gauge of production in the ISM survey rose in November for the third straight month. That's a sign that Sandy didn't force many factory shutdowns.
A slowdown in global growth has weighed on U.S. manufacturers. New export orders slipped in November for the second straight month.
Surveys show consumers remain upbeat about the economy, despite the looming taxes and spending cuts. A measure of consumer confidence reached a five-year high in November.
If lawmakers and President Barack Obama can work out a budget deal that averts the tax increases, most economists predict a good year for the economy.
Follow Chris Rugaber on Twitter at: www.Twitter.com/ChrisRugaber