The evidence is mounting that American consumers and businesses are starting to seriously sweat the fiscal cliff. The question now is: Will it matter?

The National Federation of Independent Business index of small business optimism plummeted 5.6 points to 87.5, the NFIB said Tuesday morning. That brought the index to near its lowest levels in its 26-year history (the only times it was lower were in the 2008-2009 recession and its immediate aftermath). Unlike some other recent economic data, the weakness can’t even be chalked up to Hurricane Sandy; the NFIB said it excluded states affected by the superstorm from its calculations. Rather, this drop was driven by poorer expectations about the future. The proportion of small business owners who expect the economy to improve in the months ahead plummeted by 37 percentage points.

Source: National Federation of Independent Businesses

It would be one thing if the NFIB numbers were an aberration, but in fact it fits with an emerging pattern in survey data. Friday, the University of Michigan consumer confidence index preliminary numbers for December came out, and also plummeted, to 74.5 from 82.7.

None of that means that the economy is sure to take a dive at the close of the year. It is possible, of course, that the holiday shopping season will be a bust and firms will shut down hiring until there is some resolution of the looming austerity crisis. But what people tell pollsters about their degree of confidence about the economy is less tightly correlated with their actual spending behavior than one might think. (Here is one study that does the math).

What these surveys add up to, though, is the most definitive evidence to date that the possibility of huge tax hikes and spending cuts on Jan. 1 is creeping into the psychology of those making business decisions. It is easy to imagine a CEO weighing bringing on a new employee or buying a new piece of machinery—and saying “Nah, I’ll wait until January to see how all this shakes out.” I have heard anecdotal reports from executives of times being tight at some companies as their customers “slow pay” their bills, as they aim to preserve cash in case the fiscal cliff sparks an economic downturn.

Solid evidence that this is happening is more spotty, but there are clear signs of the business sector holding back: Most notably, equipment and software spending by firms declined in the third quarter for the first time in nearly two years.

As for consumers, the risk goes deeper than the possibility that scary headlines about the fiscal cliff will prompt them to spend less on holiday gifts. Consumers were relatively buoyant in September and October, with high confidence levels and growth in spending. But it may have been a levitation act, because those positive numbers weren’t matched by gains in incomes.

Over the four months from July through October, personal consumption spending rose 0.9 percent—yet incomes rose only 0.4 percent. Wage and salary income, the predominant source of earnings for most working families, rose a measly 0.1 percent.

What that means is that consumers’ spending in the crucial holiday months will already be under pressure from the fact that their wages haven’t been rising in any meaningful amount, and they may have gotten ahead of themselves in spending. That will be compounded if the lack of confidence among businesses means fewer jobs, fewer raises, and fewer holiday bonuses.

Those with more optimistic forecasts for consumer spending in the final months of 2012 pointed to high consumer confidence indexes in September and October, which suggested that Americans were bullish enough on the future that they might keep spending despite the stagnation in their incomes.  But that theory is looking weaker with every new survey that comes out suggesting their outlook is souring.

Confidence surveys aren’t the same as the economy, and should be treated with a measure of skepticism. But this may be an occasion when they are telling us something. And it isn’t very good.