The nation's federal judges sued President Carter yesterday, saying they are entitled to a 12.9 percent pay increase instead of the 5.5 percent he says they can receive along with other federal employes.

The judges contend that for 12 days last month they were supposed to receive a 12.9 percent pay raise, until Congress passed a resolution limiting that amount to 5.5 percent. Under the Constitution, the salaries of federal judges cannot be cut while they are serving.

The suit was filed by the administrative office of the U.S. courts on behalf of approximately 600 federal judges after several complained that it was unconstitutional to deny them the full amount. The administrative office had earlier sent a memo to the judges that tried to get them to accept the 5.5 percent increase and sign away their right to the larger amount.

Practically every judge refused to sign the document, sources said, so instead the administrative agency brought its dilemma into the federal courts.

The suit arises because judges were included in a federal pay raise of 12.9 percent that became effective on Oct. 1.

On Oct. 12 however, a joint House-Senate resolution was signed by the president, limiting any federal raises, including those approved on Oct. 1, to 5.5 percent.

The judges say they are covered by the U.S. Constitution, which protects judges from having their salaries "diminished during their continuance in office," or, in other words, from having their pay cut. That is, in effect, what President Carter did to them according to the suit.

The suit affects a wide range of feral judges, from Chief Justice of the United States Warren Burger, whose current salary is $75,000 a year, to more than 400 trial court federal judges who make $54,500 a year.

The administrative office told the court yesterday that it is caught in the middle because of the president's actions. If it pays the judges less than 12.9 percent the judges can sue the administrative office for diminishing their pay unconstitutionally; if it pays the judges more than 5.5 percent, it would be subject to federal prosecution under a federal law that prohibits it from using impounded federal funds to pay salaries.

"It's a Catch-22," said Carl Imlay, general counsel of the administrative office. He acknowledged that "every judge in the system is very upset" by the whole situation."

Although Chief Justice Burger is known to follow the activities of the administrative offices closely in many areas, Imlay said Burger had nothing to do with the lawsuit being filed.

Chief Justice Burger's office referered all queries about the pay raise suit to the Supreme Court press officer, Barrett McGurn, who said, "We won't have any comment."

'It was purely our effort over here,' Imlay said of the lawsuit."We had to go to court. The only way for it to be settled was to go to court."

Although this is the first lawsuit known to have been filed by the administrative office on behalf of judicial pay, there have been at least three other suits filed in recent years by groups of judges who claimed they were unconstitutionally underpaid.

In one of them, the judges contended unsuccessfully that the lack of any significant raises over a period of high inflation in the early 1970s had the effect of cutting their pay. In the other still-pending suits, groups of judges are claiming generally in a Chicago court that government pay freezes are unconstitutional when applied to the judiciary.

Judges either directly or indirectly affected by yesterday's lawsuit include all Supreme Court justices, U.S. Circuit Court of Appeals judges, U.S. District Court judges, and a wide range of other federal court judges.

In addition, the ruling could affect the pay of District of Columbia judges, who earn 90 percent of what certain federal judges make, under another statute.

The suit was assigned to U.S. District Court Judge John Lewis Smith Jr., who would be affected by his ruling. He is unlikely to remove himself from the case, since there is a general legal theory that if all judges would be affected by a ruling there is no need for a single judge to refuse to hear the case.