Joseph C. McGrath, a bank vice-president and political newcomer who waged a precedent-setting court battle to get his name on the ballot, claimed victory last night in his race for the Republican party nomination for Montgomery County executive. McGrath defeated Del. Luiz R. Simmons and realtor John P. (Jack) Hewitt, according to unofficial final returns.

McGrath will face incumbent Democrat Charles W. Gilchrist, who piled up a 2-to-1 victory margin over his only primary challenger, businessman Wade Dunn.

Gilchrist, however, lost in his efforts to elect a council slate to his liking. Incumbent Ruth Spector, who was running on the Gilchrist-backed United Democrats slate, lost to her opponent on the Merit Team slate, former Rockville mayor William Hanna Jr. Hanna won 46 percent of the vote to Spector's 41 percent, with citizen activist Elvera Berson a distant third with 13 percent.

Incumbents won in the other six council races, according to unofficial final returns. On Gilchrist's slate, council president Neal Potter, Rose Crenca and Scott Fosler were nominated. On the opposing Merit Team slate, council members David L. Scull, Esther P. Gelman and Michael Gudis won handily.

Spector, Gilchrist's former aide in Annapolis, has been the executive's principal ally on the council. Saving Spector's seat was considered crucial for Gilchrist since her loss, if all the other incumbents won, would shift the balance of the council from a majority of Gilchrist's allies to a majority of his critics, pledged to a more independent council.

But the colorful Hanna, four times elected mayor of Rockville's mayor, won some key endorsements and was a more forceful campaigner than the low-key Spector. Hanna, at a victory celebration with his Merit Team running mates at Gatwick's restaurant last night, said he would work with Gilchrist. "The key issue in this campaign was the independence of the council," he said.

Gilchrist said last night that he had expected his own victory but was somewhat disheartened at Spector's loss. "I was disappointed because I think she did an excellent job on the council and she's a first-rate human being," Gilchrist said. He added that he had worked with Hanna before, when Gilchrist was a state senator from Rockville and Hanna was the city's mayor.

He said he did not expect new tension from the Merit Team council majority.

In the GOP executive race, McGrath won a surprisingly strong 42 percent of the vote, with Simmons in second place with just over 30 percent and Hewitt a close third with 27 percent. McGrath, who successfully challenged the county's strict residency requirement for candidates, stressed throughout the campaign that despite his inexperience in public office, he was more loyal to the Republican party than the more liberal Simmons, whose consumer-oriented voting record in the General Assembly had angered the GOP right wing.

A beaming McGrath claimed his victory before a cheering crowd at GOP headquarters shortly before 11 p.m. "Ninety-seven days ago, I said we had a couple of hurdles to clear," McGrath said. "First was to get my name on the ballot. The second was to win the primary, and that's been taken care of. The third is to beat Charlie Gilchrist."

At the same time McGrath was claiming victory, a dejected-looking Simmons was at his own headquarters, conceding the defeat that will take him out of public life -- at least for now. "I intend to come back and win again," Simmons declared, his wife Claire, crying, at his side.

Simmons, a moderate-to-liberal one-term delegate, blamed his defeat on the conservative faction of his own party, which he had antagonized for some of his positions in the General Assembly. He pointed particularly to his support of a bill increasing the tax burden on country clubs, and said he fared poorly in those affluent county precincts where country club members live.

Simmons himself didn't help matters much when he opened his campaign with a strident attack on President Reagan's policy of laying off federal workers. And he further alienated GOP conservatives when he was quoted in a newspaper interview revealing his general election strategy of distancing himself from Reagan and running as a liberal Republican in the Mathias-Gude-Steers tradition.

McGrath, on the other hand, is a traditional Republican, more conservative than Simmons and a general believer in Reagan's economic program. But he has consistently avoided the label "conservative," casting himself instead as a consensus candidate who has attracted support across the ideological spectrum.

McGrath and Hewitt both had backed their own slates of council candidates, but Republican voters appeared to be splitting their choices. Leading the field for two at-large nominations were Malcolm Lawrence, a Hewitt backer, and Leonard Robinson, the only black Republican council candidate and a McGrath backer.

Also leading in the GOP council race were Alvin J. Arnett, John M. Saveland, Ronald S. Bird, and John A. Dean.

In legislative races, the three incumbent Democratic delegates in the 18th district -- Patricia Sher, Donald B. Robertson and Helen Koss -- were staving off a strong challenge from lawyer John Hurson.

In the 15th legislative district, incumbent Democrat Sen. Laurence Levitan was leading challenger Anthony Puca. In the Republican senate primary in that district, state GOP chairman Allan C. Levey was easily defeating his primary opponent David C. Bastion.

In the 16th legislative district, Democratic incumbents Nancy K. Kopp and Marilyn Goldwater were leading a crowded field, followed by attorney Gilbert Genn for the third seat. In the Democratic primary for state senate, newcomer Brian E. Frosh took an early lead over John J. (Jack) Sexton, a party worker who had the endorsement of the incumbents.

The council campaign was considered one of the most bitter and divisive Democratic party feud in the last two decades. The last such bloody primary fight was in 1962, when three separate Democratic slates did battle and left the party in shambles going into the November general election.

This year, some Democratic party leaders were wondering whether history would repeat itself, allowing Republican candidates to make inroads into a county with a 2-to-1 Democratic voter registration edge.

That is precisely what Republicans are hoping for this November. The Republicans this year viewed their own primary fights as a healthy sign that the GOP, which once could not even find enough candidates to fill out its ticket, now is blessed with tough primary races for executive, council and in some legislative districts.