J. Paul McGrath, the Reagan administration's top antitrust enforcer, said yesterday he plans to leave office shortly after the president is inaugurated.

McGrath, whose attempts to encourage joint ventures as alternatives to mergers met with mixed results in his year as assistant attorney general in charge of antitrust, said he probably will go back to the private practice of law in New York, the field from which he joined the Justice Department as head of the civil division four years ago.

"I've been out of private life for some time, and I thought it was a good time to go back to the real world," he said in an interview. "I'm not going to teach or take another government position." McGrath has not yet submitted his letter of resignation, but hopes to leave office around February.

There have been no immediate indications of who might succeed McGrath, who was appointed to replace William F. Baxter in December 1983.

McGrath has advocated joint ventures between competitors as a way to bring cooperative efforts to bear against foreign competition without reducing competition in some industries. Earlier this year, when the antitrust division blocked a proposed merger between Republic Steel Corp. and LTV Corp. and indicated its opposition to a proposed merger between U.S. Steel Corp. and National Steel Corp., McGrath suggested joint ventures to the companies as alternative plans, but they demurred. Republic and LTV later modified their plans and merged, while U.S. Steel and National Steel dropped their merger efforts.

But in another case this year, a proposed merger between Atlantic Richfield Co.'s aluminum division and Alcan Aluminum Co., the companies agreed to a Justice Department proposal that they run some operations on a joint venture basis.

McGrath said that while a merger might be anticompetitive, a joint venture arrangement might be the only way to develop new products or new technologies or to fund expensive new industrial projects.

"Where joint ventures are affirmatively helpful, they ought to be availed of," McGrath said. "That's a more affirmative course than has been taken in the past."

This philosophy, McGrath said, is born of an attempt to fit antitrust policy to economic and business realities, such as increased foreign competition. "What we've tried to do is bring antitrust policy up to date, make antitrust policy more consistent with what's good for the economy," he said. "That's something that's not done overnight."

McGrath said he hopes the antitrust division will continue to follow this new course, as well as liberalize the antitrust treatment of patent rights. He said he also believes that the rules for awarding damages in antitrust cases should be modified to eliminate the payment of treble damages in cases where there was apparently no criminal intent -- as in the case of two companies that enter into a joint venture in good faith only to have a court rule the combination illegal several years later.

"I think we've got to take a very close look at treble damages," he said. "Areas that are not economically bad in themselves perhaps should not be subject to such a Draconian penalty as treble damages." Conversely, he said, the penalties could be made even stiffer for such antitrust violations as secret price-fixing. "In areas where the conduct is really bad and criminal, I think we ought to be at least as hard as we've been," he said.