In rejecting the League of Women Voters' invitation to a three-way debate, President Carter has taken a gamble that may mark a turning point in the 1980 campaign.

Carter is gambling that he will be better off politically by rejecting the three-way debate now, and accepting the criticism that is sure to follow, than by going to the debate and thereby elevating independent John B. Anderson to heights from which he could siphon crucial votes from Carter in November.

The president's strategists have concluded, based largely on surveys by their pollster, Patrick Caddell, that a strong Anderson candidacy could greatly endanger Carter's reelection prospects. Anderson could attract enough votes from Carter to enable Republican candidate Ronald Reagan to carry several crucial northern industrial states.

The Carter strategists have also concluded most recently, based on surveys by Caddell and others, that the Anderson candidacy has been declining of its own accord. Their goal is to do nothing that will help revive the Anderson candidacy.

Thus, they decided a few weeks ago that the first presidential debate of 1980 should be Carter versus Reagan, a move that would clearly relegate Anderson to the status of an also-ran.

And thus, they reaffirmed that position by deciding quickly yesterday to reject the league's invitation to a three-way debate on Sept. 21 in Baltimore.

"There's been very little dissension on this" within the Carter hierarchy said one of the president's senior advisers.

"It's more important . . . not to give him [Anderson] that kind of legitimacy."

The president's top advisers conceded that their position was sure to draw heavy criticism from a number of quarters.

"No doubt about it," said another senior Carter official, "I certainly think we will hear a lot of criticism." He said it would come from the Reagan camp, from the Anderson camp and from some newspaper editorial pages.

"As far as some of the editorialists are concerned, they seem to have the same interest as Reagan, namely, to promote the candidacy of John Anderson."

"Our assumption," said the Carter official, "is that Reagan's goals have been to avoid a one-on-one debate with the president and to promote Anderson's candidacy. Our goal has been to have a one-on-oned with Reagan."

To accept a three-way debate first would preclude the possibility of ever getting a debate between Carter and Reagan alone, the Carter officials say. "We'd lose the most advantageous forum for us," he said.

Anderson has slipped in the polls in the past few months from percentages in the mid-20s to the mid-teens.

The most recent survey, by Louis Harris, also shows the extent to which Anderson's candidacy jeopardizes Carter's chances of carrying the large northern industrial states he needs if he is to win reelection.

In eight big northern states, according to the Harris survey taken Sept. 3-7, Carter led Reagan 47 percent to 45 percent when the two were matched up head-to-head. But when Anderson was added to the list, Reagan led Carter 37 to 35, with Anderson receiving 23.

It is in those northern states where the Anderson impact is usually most pronounced.

Nationally, the Harris survey showed Reagan leading Carter 41 to 37, with Anderson at 17. When Anderson's name was omitted, Reagan's lead over Carter was reduced to two percentages points, 48 to 46.

The latest California Poll, conducted by Mervin Field, shows that without Anderson Reagan's lead in his home state over Carter is reduced from 10 percentage points to just six. The poll, taken Aug. 30-Sept. 4, showed Reagan with 39, Carter with 29, and Anderson with 18. In July, Reagan led Carter 50 to 21, with Anderson at 20.

Reagan's advisers have made their own calculation and have decided that it is in their candidate's interest to keep Anderson high in the polls -- for more than one reason.

A senior Reagan official, James A. Baker III, told reporters yesterday that there are circumstances in which Reagan could suffer more than the president from Anderson's presence on the November ballot. Baker said that if Anderson were to fall below 6 percent, he would then be taking more votes away from Reagan than Carter -- most of them moderate Republicans.

Reagan's camp knows that Anderson's appearance in a nationally televised debate is likely to prevent him from falling that low.

The Carter strategists, steering clear of the mathematical gyrations of politics, had concluded most of all that Anderson's candidacy had slipped enough so that, as campaign chairman Robert S. Strauss contends, Carter can now carry New York. Strauss also says he believes Carter can carry California but not if Anderson is running strongly. The inclusiion of Anderson in the first debate can only help Anderson and hurt Carter.

Although Anderson had said he "wouldn't roll over and play dead," if the league did not invite him to the debates, his followers have long regarded the debate as the single most important event of the fall campaign. To stand any chance of winning in November, Anderson simply had to be invited.

The question for Anderson was never whether he would "win" or "lose" the debates. It has always been that he would win credibility by being on the same podium as Carter and Reagan.

Yesterday Anderson accused Carter of "running away from the American people."

"I used to say Jimmy Carter was really running in place . . . that the problems we faced in 1976 and he promised to solve are stil there," Anderson said, his voice booming across the Bergen County Courthouse square. "Yet he seemed to be running for office, running harder than ever before, but running in place."

But, Anderson said, he has changed his mind. "Jimmy Carter isn't even running in place.He seems to be running away. He's running away from the Carter record. Running away from the American people. Running away from the great issues that are framed and are ready for debate."

The gamble that the president and his advisers are taking now is that criticisms such as these will not come to dominate the rest of the campaign. "Ours is the more reasonable position, and I think in time this will come home to people," said one Carter official. And another senior Carter adviser added, "Everything in a camgain is a risk."