The Supreme Court ruled unanimously yesterday that the First Amendment does not give special protection against libel suits to people who write private letters to the president giving recommendations on potential appointments.

The ruling came in an unusual case calling for an interpretation of the constitutional right "of the people . . . to petition the government for a redress of grievances."

That right, Chief Justice Warren E. Burger said, is not absolute. "The right to petition is guaranteed; the right to commit libel with impunity is not," he wrote.

The case involved a libel suit against Robert McDonald, who wrote two letters to President Reagan opposing the appointment of North Carolina attorney David I. Smith as U.S. attorney in that state. Smith once represented a client in a North Carolina slander suit against MacDonald.

In the letters, McDonald, who called himself a "staunch" Republican, accused Smith of "fraud and conspiracy to commit fraud, extortion or blackmail" and "violations of professional ethics." Copies of both letters were sent in late 1980 and early 1981 to several members of Congress and to White House counselor Edwin Meese III, now attorney general. At least one was sent to FBI Director William H. Webster.

Smith, who did not get the job, sued McDonald, saying the letters "contained false, slanderous, libelous, inflammatory and derogatory statements" and that McDonald knew they were false.

McDonald asked a trial judge to dismiss the suit, arguing that the right to petition, unlike the right to freedom of speech and press, deserves an absolute immunity from libel suits so that petitioners will not be deterred from "redress of grievances."

The judge refused, and so did the 4th U.S. Circuit Court of Appeals, although some state courts have agreed with McDonald. The Supreme Court, reaching back to 1845 to find a precedent, agreed with the 4th Circuit yesterday.

"To accept McDonald's claim of absolute immunity would elevate the petition clause to special First Amendment status," Burger wrote.

Under North Carolina law, Burger said, McDonald will be protected by the same safeguards afforded the press in libel cases involving public figures. Smith will have to prove that the statements were false and that McDonald made them knowing that they were false or without caring whether they were false.

Justice William J. Brennan Jr., joined by Justices Thurgood Marshall and Harry A. Blackmun, issued a concurring opinion.

"Although we have not previously addressed the precise issue before us today," Brennan said, "we have recurrently treated the right to petition similarly to, and frequently as overlapping with, the First Amendment's other guarantees of free expression.

"There is no persuasive reason for according greater or lesser protection to expression on matters of public importance depending on whether the expression consists of speaking to neighbors across the back-yard fence, publishing an editorial in the local newspaper, or sending a letter to the president of the United States," Brennan continued.

Justice Lewis F. Powell Jr. did not vote.