When the Supreme Court decided to tackle the abortion issue last year, Solicitor General Rex E. Lee took a step that would prove to be one of the most controversial since he took office: he entered the case on behalf of the government, although it was not involved, and challenged the core of the court's 1973 decision legalizing abortion.

Now, in the wake of the court's strong reaffirmation of that ruling, Lee says he wonders whether he came on too strong.

"We may very well have firmed up the majority to make its opinion a little stronger" in support of a woman's right to an abortion, he said in an interview. "It was unusual for the government to come in on that side, and that may have induced our adversaries on the court to fire off heavier guns."

Unlike some of the other men who have represented the government before the Supreme Court, Lee took over the office of solicitor general with an agenda, now described on a sheet of paper he displays labeled "Outline of Supreme Court Cases Implementing Administration Policy."

He wants to limit judicial activism, "encroachments" by the courts on legislative and executive authority. He shares President Reagan's views on civil rights, busing and quotas. And, like Reagan, he says he believes that the Supreme Court usurped the powers of state legislatures when it created a fundamental right to an abortion in 1973.

"I thought, what's the point in having the engine at your disposal if you don't take it out and drive it in the direction you really want when the chance comes? . . . That's one of the reasons you elect a president and one of the reasons a president appoints a solicitor general."

But Lee, by his own candid speculation about the impact of his brief in the abortion case, has become acutely aware that balancing an ideological agenda with the unique and tradition-bound role of his office is sometimes a delicate matter.

The solicitor general's office represents the United States, the most frequent litigant at the Supreme Court, and traditionally has maintained what Lee calls a "unique relationship" of extraordinary trust with the justices. It is based on an understanding that while the office is a hard advocate on behalf of the government it is also a creature of the court itself, and is expected not to push it too far, too fast.

That relationship has enabled the office to get its way with the justices more often than has anyone else. This does not mean it wins all its cases. But when the solicitor general asks the court to review a case, for example, the court complies 65 or 75 percent of the time.

That record is intact. But in an administration with a commitment to using litigation as an instrument of social change, some observers say, the relationship is showing some signs of strain.

There are lawyers, both inside and outside the government, who say that the court was, in fact, trying to "teach the government a lesson" in the term it just completed, and not just on the abortion case.

There was the Bob Jones University case, in which the court, in an 8-to-1 decision, rejected the Reagan administration's argument that the Internal Revenue Service lacked the authority to deny tax exemptions to discriminatory schools.

In addition, there was the controversy over the powers of the states to tax multinational corporations, in which Justice William J. Brennan Jr., writing for the majority, went out of his way to say in a footnote that the court was ignoring the government's views.

"I don't think there's been much damage," Lee said of the relationship. He noted that the court handed the government major victories when it struck down the legislative veto and upheld Minnesota's tuition tax deduction for parents of private and public school students. "Nevertheless, I really worry about it," he said.

"There is a constant tension between our role of advocates for special causes and our relationship with the court. If I'm too timid, then I'm not adequately representing a client," he said. "But if I'm too assertive and depart too much from that unique relationship, then I'm not only abusing that relationship, I'm also diminishing my effectiveness."

That "tension" has been intensified under the Reagan administration. Last week, for example, a group of female members of Congress lobbied heavily with the White House to stop Lee and the Justice Department's Civil Rights Division from filing a brief in an important case coming before the court. The brief, which advanced a limited view of the law barring sex discrimination at federally assisted educational institutions, was filed anyway.

That kind of political controversy over Lee's positions has become commonplace. He has found himself pressured from all sides, accused alternately of being too political and not political enough, attacked from the right for not being part of the Reagan team and from the left for being too much a part.

Amidst it all, Lee says he believes he has preserved the traditional independence for the office that has contributed to its credibility at the court. But there are definite limits to that independence, he said.

His immediate boss is the attorney general. And while the solicitor general sometimes contributes to the formulation of government policy through litigation, he says he believes it is the "clients" he represents, the agencies and divisions of government, that should prevail on issues of "substantive policy . . . . We start with the very heavy presumption in favor of the substantive policies adopted by our client," Lee said.

"The question we ask ourselves is not whether the substantive policies are right or wrong. We start with the presumption that they are right. We ask how can we best get them implemented through the positions we take in court."

The abortion brief, he said, "was not generated from the White House. It was a matter of me taking it to the attorney general and telling him that this is what I want to do.... was an opportunity for me to express my views on something I feel very keenly about . . . . Given that, I'd say I'd do the same thing" again.