Elizabeth Spencer, the soft-spoken candidate who had little money and even less name recognition when she started her campaign for the Republican nomination for Congress in Maryland's 8th District, defeated her controversial rival, Marian Greenblatt, yesterday in the hard-fought, often bitter race.

Spencer, 56, a moderate who was the ideological sparring partner of the conservative Greenblatt on the county school board, pulled nearly 40 percent of the vote in the four-way race that was a test of strength for the conservative wing of the county GOP.

She will face Democrat Michael D. Barnes, a two-term incumbent who was unopposed in the primary, in what is expected to be an uphill battle for the GOP nominee.

Spencer, who had no campaign headquarters, watched the returns come in at a friend's home and shortly before midnight went to county Republican headquarters, where she was greeted by cheers and applause. True to her low-key style she gave no rousing victory speech and instead thanked supporters and told them, "The real work is about to begin."

The harshest words for the incumbent came from the defeated Greenblatt, who in a written statement delivered to party headquarters, called Barnes "an ultraliberal" and said he "must be defeated for the good of the county and the nation."

In the neighboring 5th Congressional District, incumbent Democrat Steny H. Hoyer soundly defeated perennial candidate Michael I. Sprague. In the Republican primary, the Rev. Perry A. Smith III, the handpicked candidate of the Republican National Committee, was beaten by William P. Guthrie, a doctoral candidate at the University of Maryland.

In Maryland's sprawling 1st District, former representative Robert E. Bauman, who formally withdrew last July from the Republican primary for Congress, nevertheless made a surprising showing against the winner, C. A. Porter Hopkins, a former state senator. Bauman's name remained on the ballot because he pulled out of the race after the withdrawal deadline.

Hopkins will face Democrat incumbent Roy Dyson, the youthful state legislator who defeated Bauman in 1980 after Bauman admitted to problems of alcoholism and homosexuality.

In the 8th District, the bitter Republican contest between Spencer and Greenblatt had been viewed as a test for the county GOP's conservative faction, some of whom recruited Greenblatt to run. They hoped to prove that even though most of the party's nominees who have succeeded in Montgomery have been moderates to liberals, in the mold of Sen. Charles McC. Mathias and former representative Gilbert Gude, a candidate to the right-of-center could win in the county. They also hoped to show thast it was not a fluke when Ronald Reagan got a plurality of 50 percent in the county in November 1980.

But Greenblatt was defeated, and the liberal slate formed to oppose the conservative school board majority, of which she was a leader, led the balloting in the nonpartisan school board primary.

In the GOP congressional primary, Phillip N. Buford, a staunch conservative apparently siphoned off some of Greenblatt's support, getting about 16 percent of the vote. But that contest had come to focus on the personality and tactics of Greenblatt more than any other issue.

From the moment Greenblatt announced her candidacy, her outspokenness became an issue. In her first press conference, Greenblatt declared that Barnes is "a supporter of the PLO Palestine Liberation Organization ."

The attack on Barnes, who is seen by many as a staunch supporter of Israel, brought editorial chastisement in the Jewish Weekly and alienation among some key Republicans.

In July, Spencer, resigned from the school board and entered the race.

Her late entrance, her lack of campaign funds and the fact that she is not as well known as Greenblatt handicapped her candidacy. But shortly before the primary, the Baltimore Sun and two county newspapers endorsed her over Greenblatt, citing their contrasting styles and positions on the school board.

The Sun said Greenblatt was "given to irresponsible statements" and described her as "a combative right-winger whose work on the school board has been a source of division and tension." Only the Gaithersburg Gazette endorsed Greenblatt.

Still, Greenblatt, with contributions of more than $40,000, outspent Spencer by as much as five to one. The final days of the campaign were marked by a literature drop to 50,000 households by the Greenblatt campaign and a last-minute blitz of television commercials in which Spencer called on voters to "elect a voice of reason."

In the 5th District, Republicans had handpicked Baptist minister Smith as their candidate after he switched to the GOP in January -- one of a group of black ministers from Prince George's County who defected from the Democratic ranks.

Smith's announcement last May that he would run brought a $5,000 check from the Republican National Committee and praise from GOP national chairman Richard Richards.