John Anderson's campaign committee yesterday gave up its unsuccessful attempts to borrow millions of dollars from banks, charging its efforts had been sabotaged by President Carter's reelection team.

Anderson general counsel Mitchell Rogovin alleged a Carter campaign memo questioning the legality of the loans raised fears of "retaliation" from the Carter administration among bankers.

The Carter campaign dismissed the charge as completely unfounded. "Basically, they were seeking very sizable loans with their standing in the polls as collateral, and doing it in the face of new banking law," said Carter campaign counsel Tim Smith.

"It was never in the cards that they would get large bank loans," he added. "I think this is an attempt to find a scapegoat, or to divert attention from their declining standing in the polls."

The memo, prepared by Smith and leaked to news organizations by high White House officials, said the loans could result in "legal fees, adverse publicity, and associated time and effort [by banks] to defend such transactions. . . it is clear the prudent lender should proceed with caution and consult counsel before making any unsecured political loans."

Rogovin said a number of banks had expressed interest in making loans before a report of the memo was broadcast on ABC-TV in mid-September. Suddenly, he added, "there was an enormous drop-off in interest."

Rogovin, however, refused to reveal the names of any banks that were interested in making the loans before the memo became public, or the names of any banks that then lost interest, to substantiate his allegation. He also said he had no personal knowledge of any banks receiving copies of the seven-page memo.

The possibility of loans for the financially strapped Anderson campaign arose Sept. 4 when the Federal Election Commission ruled that the independent presidential candidate would be eligible for retroactive government campaign subsidies if he received 5 percent or more of the vote. Carter and Ronald Reagan, as nominees of the two major $29.4 million in public campaign subsidies.

The size of Anderson's subsidy will be based on his share of the vote. If he receives 5 percent, Anderson will receive $3 million. If he receives 15 percent, he will receive $10.5 million.

At the time of the ruling, Anderson was attracting the support of about 15 percent of the public in national polls, and the Illinois congressman talked confidently of borrowing up to $10 million from a group of New York banks. His standing has been in a slow but steady slide ever since, and was at only 8 percent in the most recent Gallup poll, released Tuesday.

Anderson had hoped to use the loans to finance a national advertising campaign. Since then, his campaign has shifted gears and has begun seeking small loans from Anderson supporters. Campaign manager Mike MacLeod said yesterday almost $1 million has been raised in this manner and he expects to raise another $1.5 million to $2 million for a last-minute ad campaign.

"The banks may have let us down, but the people have not," he said.

Rogovin said five banks initially agreed to lend up to $2.5 million if other banks could be persuaded to join them. But he alleged other banks were unwilling to do so because they feared they might face retaliation from federal regulatory agencies if they did so.

"The bulk of the banks were simply concerned not whether Anderson would get the required popular vote or whether the funding would be attacked by the courts," he said. "There was concern expressed by a number of bankers about retaliation for making such loans. We decided we would be better off in dealing with loans from the American public than continuing with the banks."