A challenge to the political establishment in the upper Montgomery County town of Laytonsville failed yesterday as residents of almost a score of municipalities in suburban Maryland voted in local elections.

In some other towns, the voters upheld their tradition of political participation by turning out to cast votes even where entire slates of candidates went unchallenged. In three of eight mayoral elections in Prince George's County and one of two in Montgomery County, incumbent mayors were unchallenged and received courtesy votes.

In Laytonsville the first contested mayor's race in memory launched challenger Larry L. Barker on a spending spree that almost reached $15. But Barker, who got 33 votes, lost to three-term incumbent Charles T. White, who spent no funds at all and declined to campaign, but came away with 72 votes.

In District Heights, former mayor David H. Goldsmith, who lost to incumbent William E. Hay in 1983 by 40 votes, took back the mayor's chair by a wider margin, 463 to 367. Goldsmith had been a vocal critic of Hay's administration, using the city's broken leaf-collection truck as a symbol in the campaign. In the Ward 1 contest, E. Kathleen Shoap defeated Americo Sesso, 205 to 191.

In Fairmount Heights, Mayor Robert Gray won his fifth term by a more than 3-to-1 margin over challenger Arthur H. Jackson Jr., who had attempted to have the election postponed. Jackson, who had portrayed his campaign as a new-blood, new-broom candidacy, received only 60 votes to Gray's 195.

In a five-way battle for three council seats, incumbents Margaret Brooks and George B. Wicks were returned, but challenger Edward Morris Jr. polled more votes than incumbent George Davis and challenger Roxanne Morrison.

The 11th-hour write-in campaign by Brentwood Mayor George D. Denny, who last week reversed his decision not to run for reelection, fell short, and ex-mayor Leonard A. Johnson Sr. was returned to office. Johnson received 205 votes, vice mayor Lydia A. McConnell 97 and Denny 90.

In other Brentwood races, Ward 1 incumbent John Farrell barely retained his seat, gathering 41 votes to challenger John P. Bennett's 40. Ward 2 incumbent Robert J. Powers received 52 votes to Racquel Bywater's 39. In Ward 3, Dick Curtis defeated Rafael Rios, 73 to 17, and in Ward 4, incumbent Jane P. Moden held off challenger Earl Purks, 52 to 33.

A last-minute write-in campaign provided a surprise winner in Edmonston's Ward 2. Patricia Creelman, with 61 votes, overtook Richard W. Marion, the only candidate printed on the ballot, with 52 votes.

In Cheverly, challenger John F. Sullivan unseated Ward 3 council member Dorothy Ryan, 104 to 84.

In Mount Rainier, where all races were contested, only 870 of 3,000 registered voters turned out. Incumbent Mayor Stanley Prusch won reelection with 509 votes to challenger Lavinia (Linda) Nalls' 349.

In Ward 1, incumbents Alice Zickafoose, with 269 votes, and Donald Hibbard, with 229, retained their seats; challengers Edward Turner and Mary Hayes got 180 and 126 votes respectively. In Ward 2, with two seats open, incumbent David Neimann was returned with 312 votes, and newcomer Otis Hayward gathered 202. Robert Creamer received 158 votes, Janice Craiger 116 and Sharon Ambrose 64.

In North Brentwood, incumbent Mayor Raymond A. Hall survived two write-in challenges. Hall received 62 votes, Rudolph Jones 11 write-in ballots and Randolph Richardson one write-in vote.

Incumbents also were returned to office in Cottage City, where Ward 1 council member George P. Louizes gathered 129 votes to challenger Larry A. Lint's 26. At-large incumbent Donna Bartlett received 127 votes to Charles E. Hubbard's 34.

In Somerset, with three council seats open, incumbents Stanley Baleinger and Charles Lerner lost to newcomers Doug Besharov (221 votes), Jack Frankel (216 votes) and Joseph Leiter (202 votes). In Garrett Park, where three council seats were open, incumbents Barbara Frost and George Payne were returned with 173 and 172 votes, respectively; Allen Dittman, with 166 votes, is the third winner.

Riverdale's one contested race, for the Ward 2 council seat, was won by incumbent Edna F. Wade, who received 46 votes to 32 for Charles K. Fogle Jr. Hyattsville's Ward 3 race was won by incumbent Polly Rogers, who received 212 votes to challenger Scott Wilson's 107.

A three-way mayoral scramble in Glenarden among incumbent Stanley D. Brown (281 votes), former council member James C. Fletcher (327 votes) and council chairman Andrew J. Ward (131 votes) will be decided in a runoff election between Brown and Fletcher June 3.

In Ward 1, incumbent Jean W. Eason was reelected with 265 votes, but incumbent Robert M. Blue, who got 186 votes, lost by one vote to Viveca E. Jones. In Ward 2, newcomers Everett M. McMullen and James R. Cousins III were elected with 46 and 98 votes, respectively, defeating incumbent D. Violanthia Davis, who got 36. Louis M. Vaughn defeated Santo A. Chase, 154 votes to 43, for the Ward 3 seat.