President Carter yesterday formally rejected calls that he appoint a special prosecutor to take over the Justice Department's investigation of alleged South Korean influence-buying in Congress, saying the inquiry is making "substantial progress."

In a letter to the Republican congressional leaders, Sen. Howard H. Baker Jr. of Tennessee and Rep. John J. Rhodes, of Arizona, the President said that naming a Watergate-style special prosecutor "would be inappropriate and unwarranted and would probably impede the investigation."

Carter said that the investigation had proceeded to where "potential prosecutions have been identified and in several cases, the evidence gathering process is nearly completed."

Benjamin R. Civiletti, head of the Justice Department's Criminal Division, said in a phone interview late yesterday that his attorneys were near the point of analyzing the strengths of cases against several targets for possible indietment.

He added that the Carter letter had been drafted before last weekend's resignation of the special counsel for a parallel House investigation of similar charges.

The House cloakrooms were busy yesterday as Speaker Thomas P. (Tip) O'Neill Jr. (Mass.) met with other Democratic leaders to decide how to stem a flood of criticism over the sudden departure of attorney Phillip A. Lacovara, who quit as head of the House ethics committee's inquiry after a sharp personal attack on him and his firm by Rep. John J. Flynt Jr. (D.Ga.), the committee chairman.

Flynt had called for an audit of Lacovara's bills after a personal clash over the pace of the investigation.

O'Neill announced last night, after meeting privately with Flynt and other committee members, that he would work with them to find a new, nationally known attorney to replace Lacovara. He also said he would work with the committee to set up guidelines to guarantee the independence of the new special counsel.

Republican members of the troubled committee, however, said yesterday they will push for keeping Lacovara as counsel. Rep. Albert Quie of Minnesota said he will introduce a motion at Wednesday's committee meeting to refuse to accept the Lacovara firm's resignation.

He suggested that Flynt and Lacovara might be able to "hammer out their differences and start over," and said some Democratic members of the committee had promised to support him.

Lacovara, who is vacationing in London, could not be reached for comment yesterday. But a colleague at his law firm, Hughes Hubbard & Reed, which had been retained by the committee, said he couldn't rule out the possibility that the firm might reconsider.

O'Neill's actions yesterday were being watched more closely than usual because of an ABC news report over the weekend that both the Speaker and Majority Leader Jim Wright of Texas might be implicated by the ethics committee investigation.

O'Neill and Wright both vehemently denied the report, which apparently came from an interview with Lacovara. A Lacovara colleague said yesterday that Lacovara did talk with the ABC reporter but that the resulting report was an inacurate reading of Lacovara's answers to hypothetical questions.

The Lacovara resignation led to continued calls yesterday from some members for a special prosecutor, a demand that puzzled Justice Department officials conducting a criminal investigation separate from the internal House inquiry.

President Carter noted in his letter to Baker and Rhodes that there has been no evidence that the Justice Department is unable to adequately pursue the case. He referred to the hundreds of persons questioned and thousands of pages of testimony gathered in the more than year-old department inquiry.

Civiletti, the department's top criminal attorney, said yesterday, "It's unfortunate that some people lump everything in the same boat. We're interested in criminal violations and being able to prove them. We couldn't care less about what is important to the House using a different, though valid, ethical standard."

Among the House's junior Democrats, among whom the Korean investigation was a prime subject of conversation all day, there was a sense of frustration and anger at Flynt and the Democratic leadership.

"We're in a real mess now," said a third-term Democrat, "and I don't see how we'll get out of it. No House investigation now wiil have credibility with the public."

Flynt, in contrast, was scrappy and combative, insisting that the weekend's events were due to Lacovara's impatience and "immaturity." The ethics committee chairman insisted that he saw no need to change the procedures of his ongoing investigation.

Speaker O'Neill displayed a striking weariness, and almost palpable sadness, about the new delay in the investigation.

"I wish this thing had been finished yesterday," O'Neill said. "All I want to do is to complete the investigation as quickly as we possibly can."

A group of first- and second-term Democrats who met with O'Neill yesterday afternoon said he seemed genuniely interested in moving ahead as quickly as possible. They said the Speaker recognized clearly the potential damage to his party - and particularly its junior members - if the investigation were dragged on or came to an inconclusive end.