William F. Baxter, the Reagan administration's controversial antitrust chief who engineered the breakup of the Bell System, announced yesterday that he will leave the Justice Department next Friday to return to his old teaching post at Stanford Law School.

Attorney General William French Smith said that J. Paul McGrath, assistant attorney general in charge of the civil division, will replace Baxter as head of the antitrust division.

Baxter, 53, came to Washington nearly three years ago decrying "the rubbish" that he said had passed for antitrust policy. Since then, he has steered the department away from interfering with large corporate mergers, refused to prosecute vertical price fixing by manufacturers and distributors, and injected so much conservative economic theory into antitrust analysis that many legal experts say he crippled traditional antitrust enforcement.

Yesterday, asserting that his goals had been largely fulfilled, Baxter said he was eager to return to his academic career.

"I find myself burned out, I guess is the traditional expression," he told a crowded press conference. "I'm tired. I've fallen behind in the several books of literature that I have regarded as the core of my intellectual activity."

McGrath, a 43-year-old Harvard Law School graduate and former corporate lawyer, pledged to continue Baxter's policies, but associates describe him as more pragmatic and flexible than Baxter.

"I agree with the policies that Assistant Attorney General Baxter has been following," McGrath said. "Obviously a great deal of additional work needs to be done."

Baxter's resignation provoked an immediate debate about the impact his stormy tenure will have on the future of antitrust law. Business groups heaped lavish praise upon his record, while consumer advocates attacked him as a dogmatic, uncompromising ideologue. Even antitrust experts who disagreed with his policies acknowledged, however, that his legacy will be long-lasting.

Sen. Howard Metzenbaum (D-Ohio) called Baxter's resignation "an early Christmas present" for believers in antitrust enforcement and scored what he called Baxter's "stubborn insistence that large corporations are seemingly incapable of wrongdoing.

"The Reagan administration has made many mistakes in its appointments to high positions," Metzenbaum said. "But none has had the negative impact that Mr. Baxter's stewardship--or lack of it--has had on consumers."

But the policies that Metzenbaum deplored were vigorously applauded by businessmen who believe that, prior to Baxter, antitrust law had lost sight of economic realities. In particular, they argued, the growth of international competition--with the encroachments of the Japanese and Europeans on the U.S. auto market a prime example--had made earlier antitrust hostility to domestic market concentrations largely obsolete.

"His major importance has been a policy of benign neglect," said John Albertine, president of the American Business Conference. "He's allowed activities in the past that would have raised the hackles of the antitrust division. I think he's been the finest assistant attorney general for antitrust that we've ever had."

Baxter's major contribution, Albertine added, was recognizing "that we're really in a new ball game, that we're in a world of global international competition. The old rules in antitrust with respect to market structure really don't apply anymore."

Ironically, Baxter's most notable achievement came on Jan. 8, 1982, when he announced an agreement with American Telephone & Telegraph Co. to break up the communications giant, forcing it to spin off its 22 Bell system operating companies, effective next month. Despite his aversion to the old "bigness is badness" credo of earlier trustbusters and the vigorous opposition of Defense Secretary Caspar Weinberger and Commerce Secretary Malcolm Baldrige, Baxter had pulled off the largest corporate divestiture in history.

Yet close analysis showed that the agreement to break up the Bell System and end an eight-year government antitrust suit was fundamentally consistent with Baxter's aversion to government regulation. The divestiture of AT&T was in part designed to liberate the company from whole areas of federal regulation and unleash a new wave of competition into the fast-growing telecommunications industry.

Many consumer advocates, warning of spiraling local phone rates and potential disruptions in telephone service, have been second-guessing Baxter lately. But he was unyielding yesterday.

"I have no doubt at all it the breakup was the right thing to do," he said. "There isn't the slightest doubt in my mind that the consumer will benefit on the average."

Although the bulk of the department's work on the case is over, the antitrust division will continue to play a major role in implementing the divestiture and thus overseeing the telecommunications industry. McGrath, who had previously represented AT&T in one of its private antitrust cases, said yesterday he will disqualify himself from all matters relating to the Bell System.