Republican Audrey Scott and Democratic Steny Hoyer swept to easy victories yesterday in the special primary elections held for Maryland's vacant 5th Congressional District seat.

The two victors, both of whom came from behind in the crowded and compressed campaign to defeat candidates initially thought to be easy winners, will face each other May 19 in an election that could serve as a referendum on the Reagan administration.

In the Republican primary, Scott, the mayor of Bowie, trounced her only serious opponent, Lawrence Hogan Jr., the 24-year-old son and namesake of the Prince George's County executive, by a lopsided 4-to-1 margin with over 63 percent of the Republican vote cast. The easy Scott victory was apparent as the first votes were counted last night, and a forlorn Hogan called Scott 45 minutes after the polls closed to acknowledge defeat in his first major electoral endeavor.

In the Democratic race, a field overflowing with 19 candidates turned into a two-man race between Hoyer and Reuben Spellman, the husband of former four-team congresswoman Gladys Spellman, whose vacated seat the election was being held to fill. A political neophyte who had never liked the world of politics, Reuben Spellman had hoped to keep the seat in the family by translating his wife's popularity into an electoral victory of his own.

But the strategy's lack of success became clear when the final vote tallies came in, and Spellman had lost to Hover by 2,000 votes. Final results from the Democratic primary showed Hoyer with 14,013 votes or 30.2 percent of the vote, Spellman with 12,364, and all other Democrats trailing by several thousand votes.

In all, 59,974 Democrats and Republicans, or 39 percent of the registered voters in the northern Prince George's district, cast ballots in the special party primaries. The turnout, on a bright spring day, was far larger than the 22 percent of the electorate that county election officials had expected to appear at the polls.

Scott's victory was the more surprising of the two races. She was considered the long shot in a race with Hogan when the 5th District seat was declared vacant and the special election set because of Gladys Spellman's continuing semicomatose state caused by the heart arrest she suffered last October.

Hogan had a name that his father had made famous and popular, first as a congressman for six years and recently as county executive. In addition, the younger Hogan had ready access to his father's political campaign strategists and contributors lists.

But the family ties ultimately hurt him, with Scott and other Republicans charging that Hogan was too young to run for Congress and only in the race as a serious candidate because of his father's desire to expand his own base of power. The strategy apparently worked well and Hogan's campaign staff could only watch in frustration during the last week or so as their initial easy lead slipped quickly away.

Such charges were quickly forgotten last night when Scott claimed victory at her headquarters in Bowie. "The key to the next six weeks [until the general election] is unity and friendship in the Republican Party," she told a cheering crowd, including some state Republican leaders who had used this race to pursue a longtime feud with the older Hogan. "And I want everyone to have that uppermost in mind."

At a muted post-election reception, Hogan told some 75 supporters, his voice quavering with emotion, "Well, as you all know, we've fallen a little short of our goal tonight." As his father stood beside him, the younger Hogan blamed his defeat on "too much to overcome and too little time to do it." Despite the often-nasty tone of the Republican contest, Hogan said he would work for Scott in the general election because "the most important thing tonight is to elect a Republican to support Reagan."

Hoyer, the former state senate president and leader of the Prince George's Democratic Party until he lost a state race in 1978, made a triumphant reentry into politics by defeating Spellman, and easily brushing aside 17 other Democrats, including State Sens. Edward Conroy and Thomas O'Reilly, County Council member Sue Mills and Montgomery County Del. Stewart Bainum.

Hoyer beamed as he stood before supporters and politicians, many of whom had come up through and been participants in the powerful Democratic organization that Hoyer and a few friends constructed in the 1970s. "I will, to the best of my ability in this campaign, commit myself to carrying out the [Gladys Spellman] tradition," Hoyer said. "I hope I can do this job nearly as well as Gladys."

Shortly after Hoyer gave his victory speech, Reuben Spellman appeared at the Ramada Inn celebration, and the winner and the loser embraced as the crowd applauded and Hoyer's wife Judy cried. "We put up the good fight," Spellman told the crowd. "It's up to you, now, Steny. We've just got to get together. In the next five weeks we've got to show the Republicans that no matter how much money they're going to spend, we're going to keep this seat for the Democrats."

Earlier, at his own campaign headquarters, Spellman told a tearful gathering of his supporters, many of whom had worked for and supported his wife for years, "Well, it looks like the estimated figures would make us appear to be behind. I'm sorry to see all the sad faces."

And, as he had done repeatedly throughout this crowded and compressed campaign, he brought up his wife and her works. "One of the things Gladys always said was 'In order to be a good winner, you have to learn to be a good loser.' So we can be a good loser now."

Yesterday's results bring to an end the first leg of a congressional odyssey that 32 candidates embarked upon five weeks ago when House Speaker Thomas P. (Tip) O'Neill, after receiving reports of Gladys Spellman's continuing incapacitation, announced he would be vacating the 5th District seat to which she had won overwhelming reelection last November.

Shortly after O'Neill's announcement and the unprecedented vote to vacate the seat by the full U.S. House, Maryland Gov. Harry Hughes set the dates for the special primaries and general election to fill out the remainder of Spellman's fourth term.

Reuben Spellman immediately jumped into the race, followed in short order by Hoyer, Mills, Conroy, O'Reilly, Bainum and a host of other dreamers, politicians and perennial candidates. On the Republican side, Hogan, Scott and 10 other candidates filed to enter the race.

One of the 19 Democrats, perennial congressional candidate Richard E. Lee of College Park, died early yesterday. Officials at Leland Memorial Hospital reported he died of cardiac arrest.

With just three weeks between the filing deadline and yesterday's election, all the candidates had organized their staffs hurriedly to raise money and put their campaigns into motion.

Among the Democrats, Hoyer had the edge, calling easily upon those longtime party contributors and strategists, such as lawyer Peter O'Malley, who had helped forge the county's Democratic organization in the 1970s.

Spellman turned to his wife's supporters and organizations. Mills called on her constituent files. Bainum, an independently wealthy businessman, dug deep into his own pockets and financed radio and television advertisements to help his almost nonexistent name recognition. Before it was over, Bainum would spend nearly $135,000 for the 5,585 votes and fourth-place spot he managed to get in the final tallies.

Mills campaign workers said that this would not be the end for their candidate -- they expect her to run for county executive in 1982. Bainum told his campaign staff. "They say it's important in life to learn how to lose. I hope you view this as an investment in future political endeavors."

On the Republican side, Hogan had the finances and campaign organization but Scott discovered an effective campaign strategy -- attacking his family connection. Scott also benefited from a continuing feud between the older Hogan and many state Republican leaders, who when given the chance to attack Hogan through his son, willingly came out in support of Scott.

Before the glow of victory had a chance to settle on Scott and Hoyer last night, both had already begun to plot campaign strategies for the May 19 general election. Scott will meet today with members of the Republican National Committee, National Republican Congressional Committee and state Republican leaders to get their fund-raising advice and prepare her campaign.

Having held the seat from 1968 until 1974, when Gladys Spellman succeeded the elder Hogan, the Republicans believe it is possible to win the seat back despite the country's 3-to-1 Democratic registration.

Hoyer's staff had been discussing the structure of a general election campaign for several days, according to campaign manager John McDonough. Today, Hoyer will begin his first active campaigning and may meet with House Speaker O'Neill to emphasize that the Democrats, too, want to win the seat.