Democratic incumbents in Montgomery County swept to victory yesterday in three hotly contested courthouse primary races, and the outcome of a fourth race remained in doubt.

In the closest race, incumbent James A. Young held a 784-vote lead over challenger Tony Fisher in the race for county sheriff .

Young, at his home, claimed victory, saying, "I won by a small margin, but I won nonetheless." Fisher said early this morning that he was not conceding and would wait until after the absentee ballots are counted..

With all 202 precincts reporting, State's Attorney Andrew L. Sonner fended off challenger Daniel J. Cassidy for the second time, winning by more than 5,000 votes.

Circuit Court clerk Howard M. Smith handily dispatched opponents Ruth K. Vurek and Frederick M. Russillo, and will face Jerry O. Anderson in the general election. Anderson beat William J. Neptune in the Republican primary.

Four sitting judges in the 6th Circuit Court -- Rosalyn B. Bell, James S. McAuliffe Jr., William C. Miller and Irma S. Raker -- won by close to 2-to-1 margins over challenger Peter J. Messitte. Because the four judges won in both the Democratic and Republican primaries, they are elected to 15-year terms and will not run again in the general election in November.

Cassidy, who came much closer this year than when he was trounced by a 2-to-1 margin running against Sonner in 1978, waited for returns at his Rockville headquarters. Twice he was the victim of false reports that he was ahead of Sonner, only to find he trailed the incumbent.

"I still believe victims have a right to be heard in this county and police officers have a right to be consulted before his case is plea-bargained away," said Cassidy, sounding two of his campaign's principal themes.

Chevy Chase lawyer Messitte, who brought distress to four sitting judges in his primary challenge, conceded defeat before the final tally. "This race was largely on principle," he said, "Winning would have been the icing on the cake, but a large number of people heard my message and responded."

Messitte, 41, passed over two times for an appointment to the Circuit Court bench, was highly critical of the judicial selection procedure that failed to select him. Even though his wife was a member of the commission, Messitte contended throughout his campaign there was widespread "cronyism" surrounding judicial appointments and stated that many unqualified judges are nominated by their friends.

The four judges who formed the "sitting judges" slate and were endorsed by the "Democrats for 82" argued in their campaign that judges who were chosen through the selection process and have performed satisfactorily on the bench should be allowed to continue their judgeships.

In the state's attorney race, Sonner, 48, said he never considered Cassidy "serious competition."

Early last evening, Sonner said "the contest is over" and that the results were what he had expected.

During the entire campaign, the two candidates appeared only twice on the same platform at the same time. Each time, Sonner unleashed sharp criticisms of Cassidy, who Sonner said lacks the legal experience he needs to be an effective prosecutor.

Cassidy, 34, hit voters with a blitz of buttons, posters, T-shirts, bumper stickers, and radio and televison advertisements. He outspent the incumbent five to one and said that the $25,000 donated to his campaign compared to only $5,000 collected by Sonner was proof that county residents wanted a tougher prosecutor.

"The people of Montgomery County want a prosecutor who is going to try some of the major cases, work with the police and develop a stronger victims' rights program," said Cassidy, who was endorsed by the 700-member Fraternal Order of Police and the Coalition of Black Police Officers of Montgomery County.

Sheriff Young, 58, who was appointed in 1976 to complete two years of his predecessor's term, said he probably would retire at the end of a new term.

Fisher, a 12-year veteran of the Montgomery County Police Department and, at age 32, a youngster compared to Young, waged an energetic and well-organized campaign that promoted a theme of "fresh ideas and new blood."