Baltimore County State's Attorney Sandra A. O'Connor appears to be the leading candidate among three Republican lawyers proposed by Sen. Charles McC. Mathias (R-Md.) as successors to Democrat Russell T. Baker, the outgoing U.S. attorney for Maryland.

Mathias has submitted the names of O'Connor J. Frederick Motz of Baltimore and Breckinridge Willcox of Potomac to the Justice Department for a final selection.

Sources indicated that among the three, O'Connor may be the most attractive to the Reagan administration, on the basis of guidelines which place a premium on experience as a prosecutor and a "willingess to support and carry out the policies of the president and to accept the direction of the attorney general."

O'Connor, who was easily reelected to a second term in 1978 as prosecutor of one of Maryland's most populous jurisdictions, also would add to the short supply of woman appointed by the Reagan administration. Of 94 U.S. attorneys nationwide, three currently are women.

But both Motz and Willcox have stronger ties to Mathias, and if the senator were to have the final decision, sources suggest the nod would go to Motz, general counsel to the Maryland Republican Party, and an establishment lawyer with the prestigious Baltimore firm of Veneable, Baetjer and Howard. Motz, 38, also worked two years as an assitant U.S. attorney in Baltimore.

Willcox, who is on leave from him job as chief of the government fraud branch of the Justice Department, worked for Mathias as a legislative assistant about 10 years ago, but has less experience in Maryland circles than either O'Connor or Motz.

Baker was named to the job of chief federal prosecutor in Maryland by President Carter in December 1977. Although U.S. attorneys are appointed for four-year terms, it is customary for them to submit their resignations at the start of a new administration.

Baker said yesterday that Mathias had "made outstanding selections in the people he has picked" as possible successor and said he was flattered to have been among those consulted by Mathias on the nominations.

Mathias also consulted with former U.S. attorneys Jervis Finney, Baker's predecessor, as well as George Beall, a Republican, and Stephen H. Sachs, a Democrat who is now attorney general of Maryland.

Justice Department spokesman John Russell said yesterday that Attorney General William French Smith has received recommendations for 55 of the 94 U.S. attorneys, but that no nominations have yet been made.

Mathias also has a chance to help name a federal judge as the result of the announcement that U.S. District Court Judge Edward S. Northrup will accept senior status upon reaching 70 on June 1. Mathias was named a panel of eight Marylanders, headed by former Wicomico County Circuit Court Judge E. Dale Adkins Jr., to assist him in the search for nominees. c

O'Connor said yesterday that her reading of the guidelines led her to believe that her 15 years of experience as a prosecutor, including seven as an assistant in the Baltimore city office of the state's attorney, "gives me hope," although she noted she lacks one of the recommendations, that the nominee have experience as an assistant in the office to which he or she would head.

"I think that's incestuous," said the outspoken prosecutor, who also admitted that while she is "a fan of the president's," she was a Bush delegate to last summer's Republican national convention.

A native of Louisville, Ky., O'Connor, 38, graduated from high school in Catonsville, where she still lives, with her husband and two children, and from Indiana University and its law school.

Baker will turn over the office to his chief assistant, Herbert Better, on June 1, and after taking four months off, will rejoin his old Baltimore law firm, Piper and Marbury, on Oct. 1.

In his 3 1/2 years as U.S. attorney, Baker carried on the tradition of the Baltimore office being one of the most active in the nation in the pursuit of white collar crime.

His staff was the keystone of a nationwide investigation of corruption within the General Services Administration and convicted two dozen persons who defrauded the Veterans Administration with phony educational benefits.