He was only kidding when he said his presence at a fundraiser in Albany for Gov. Hugh Carey of New York proved that the vice president is always at the center of power. But Fritz Mondale, as usual, had a point.

The important elections this year, the ones that constitute the true referendum on the Carter administration, are for the governors. For the gubernatorial races, unlike those for the House and Senate, test the decisive unknown of national affairs - which is what the public expects of its political leadership.

The conventional view, to be sure, is that the midterm congressional elections will provide the interesting measure of how the administration is doing. Senators and congressmen do, in fact, have to stand on their records. Those records are heavily mixed up with the national issues - inflation, unemployment, welfare, taxes and foreign policy - that are the warp and woof of presidential politics.

But virtually all congressmen are judged much less for their stands on national issues than for their record of personal service to constituents. Similarly with most senators - at least those from the smaller from the small states.

"What counts in the election of a congressman or a senator," chairman John White of the Democratic National Committee put it in an interview the other day, "is how the candidates have met constituent demands for getting Social Security checks delivered faster, or helping smooth out a dispute between Washington and the local school district."

"I couldn't agree more," Bill Brock, the Republican national committee chairman, said when I repeated White's statement to him.

Those views are shared by most senators and congressmen. They tend to work very hard on personal services. As a result, except in years of national tidal waves - like 1966 when the country went ape over Vietnam, or 1974 when Watergate hit the fan - there tend to be only marginal shifts in the House and Senate.

This year there is no explicit national issue with the punch of Watergate or Vietnam. But if there is no explicit national issue, there is an underground question that runs throughout the country. It is the question of how well government is performing, of whether leaders are living up to expectations.

President Carter is clearly not doing brilliantly in the performance test. While nothing particularly bad has happened to the country during his term in the White House, his performance ratings in every poll have dropped very sharply to ominously low figures. What remains in doubt is whether those low figures mean that he is in serious trouble for the nomination and reelection.

The gubernatorial races provide answers to those questions because several governors have been, or are, in the Carter position. Brendan Byrne of New Jersey, for example, showed so poorly in the polls that he sought an exit as ambassador to Ireland in 1976. Denied that opening, he was obliged to seek reelection. To virtually everybody's surprise, he won - part because of an inept opponent, but more because he could claim to have done at least one thing: pass an income tax.

If that suggests a lowering of expectations, another case shows that the public tolerance is not unlimited. Dolph Briscoe, the governor of Texas, was roundly beaten in the primary three weeks ago despite general prosperity in the state - apparently because he had done absolutely nothing as governor.

Several governors up for reelection this year fit between Byrne and Briscoe. Jerry Brown of California does well in the popularity polls, but a majority of the voters agree he had done nothing important as governor. In New York, Carey had shown very poorly in recent polls, but it looks as though he will be able to claim at least one achievement: helping to save New York City from bankruptcy.

Ella Grasso in Connecticut was shown in early polls to have a very negative image, but that weakness was redeemed by a show of effective action in meeting the emergency caused by the snowstorms last winter. Much the same applies to James Rhodes, the governor of Ohio.

I have no idea how these races will come out, and the pollsters I trust tell me not to bet. What does seem clear is that the results, whatever they are, will provide a clue as to what that unknown god public opinion really thinks about the way political leaders should perform.