Democrat Steny H. Hoyer was elected to Congress yesterday by voters in an overwhelmingly Democratic Maryland suburb who rejected the nation's conservative trend and a Republican strongly backed by the Reagan administration.

With all precincts reporting in Maryland's 5th District, Hoyer, the former Maryland Senate president, won easily over Bowie Mayor Audrey Scott. Hoyer received 55.1 percent of the vote to Scott's 43.5 percent. A third candidate, Libertarian Tom Mathers, received 1.2 percent of the vote.

Hoyer, 41, was considered the leading candidate throughout the campaign because of the Democrats' 3-to-1 advantage in voter registration but the Republicans had made an extraordinary effort to capture the seat, bringing in Vice President Bush and other well-known party members to campaign. On election day, the Republicans sent dozens of workers to Scott campaign headquarters, calling every registered Republican in the northern Prince George's County district three times in an effort to get them to the polls.

In the end, even that was not enough. And when Scott conceded to Hoyer just an hour after the polls closed she attributed her defeat to the district's decidedly Democratic composition. "The Democratic registration was working against us," Scott said last night as she conceded defeat. "I do not think it was a national referendum. Reagan's popularity is very high. I can't predict how each person voted or what influenced that vote but from the very beginning I felt that this was a district race, that I was running as Audrey Scott and that he was running as Steny Hoyer."

But Democrats, who have had few things to cheer about in recent months, were quick to impart national significance to Hoyer's victory. "Steny Hoyer's victory shows that people are beginning to think twice about giving this administration a blank check on issues like Social Security and Kemp-Roth," said Tony Coehlo, chairman of the Democratic Congressional Campaign Committee, which gave Hoyer some campaign assistance.

Hoyer, making a triumphant return to politics after losing a race for lieutenant governor in 1978, attributed his victory to hard work by his campaign and avoided attaching any national significance to it.

"I don't think there's a particular message" to the Reagan administration, he said. "The message is not louder than the people of the 5th District looking at the candidates on the merits. The major element was the determination by the voters as to who could most effectively represent them."

The special election held yesterday to replace former Rep. Gladys Spellman, who has been hospitalized in a semicoma since last October, drew over 45 percent of the district's registered voters despite the soggy and cold spring weather.

The election was the culmination of a hard-fought, six-week race that began April 7 after both Hoyer and Scott emerged from crowded and often bitter primaries.

After the primaries, which featured a total of 31 candidates and some of the best known political names in the district, the two candidates gathered their party loyalists behind them and turned to the national parties for assistance.

Both candidates were given such assistance throughout the race but for Scott, who began the race with lower name recognition than Hoyer, the aid was essential since the local Republican organization is small and Republicans traditionally have difficulty raising money and votes in Democratic Prince George's County.

From the start, officials of the Republican National Committee and the Republican Congressional Committee recognized that Scott had only a slim chance of winning given the county's 3-to-1 Democratic registration. Nonetheless, they got involved in the Scott campaign in part because the race was one of the first since President Reagan was elected and was seen by some as a referendum on the administration.

If the Republicans could win this district, the party officials felt, they would not only reduce the Democratic majority in the House of Representatives but would also demonstrate Reagan's overwhelming popularity.

As a result, the Republican Party poured in money, helped plan strategy, and aided in preparing advertising. It helped Scott contact special interest political action committees that would be willing to make major donations and set up briefings on a variety of national issues.

More important, party officials got the help of the White House: Vice President George Bush was host for an early morning fund-raiser and filmed a television endorsement and President Reagan signed a letter urging votes for Scott that was widely distributed.

By yesterday, this assistance -- which included dozens of election day volunteers -- had made the race competitive but also ensured that a local congressional election, in a county with 3-to-1 Democratic registration, would be seen in part as a referendum on the Reagan administration.

Hoyer, with a large party structure behind him, received less assistance from the national Democratic Party. The Democrats helped raise money from the special interest groups, ran a poll for the campaign and brought in some Democratic notables including House Speaker Thomas P. (Tip) O'Neill and former Vice President Walter F. Mondale.

Like the national GOP, Democratic officials in Washington said the 5th District election was important to the party as a sign that the Reagan momentum had slowed since the 1980 election and that the Democrats are on the rebound.

But, despite the potential national implications of the race, yesterday's election was more than anything a contest between two contrasting personalities: Hoyer, the polished politician who had spent 12 years in the state legislature, and Scott, the mayor of a town of 33,000 whose reputation as a civic activist centered in one small area of the county.

And these contrasts, more than their stands on national issues, were what the two candidates tried to accentuate throughout the campaign. Hoyer's campaign slogan was "pride in leadership," and in speeches and campaign literature he focused on his years of experience in the state Senate and as Senate president.

He tried to portray himself as the only candidate with the qualifications to serve the district's residents in Congress. Scott, with only five years as Bowie mayor and one on the City Council, was not as qualified, Hoyer said.

Scott spent much of the campaign emphasizing her reputation for constituent services and repeatedly referred to herself as a "people person." In doing so, Scott was trying to equate herself and her promise with that of Gladys Spellman, the former congresswoman whose seat both she and Hoyer were hoping to claim.

Like Spellman before her, Scott has spent much of her political career at the grass-roots level, among the civic associations, PTAs and church groups that make up so much of a busy mayor's day. And because Scott did not have many years of legislative experience to call upon, in this race, she took every opportunity to make the comparison.

But Scott also engaged in a more controversial tactic in the race. Coming out of the primary as an underdog according to both her own and Hoyer's polls, Scott was forced to draw support away from Hoyer and, given the Democratic composition of the county, she needed Democratic support.

In order to do so, her media consultant fashioned a batch of hardhitting television commercials attacking Hoyer for his role in the once powerful local Democratic organization that ran the county for much of the 1970s and for trying to appear more moderate than his record showed him to be.

The ads drew criticism from many Democrats and Scott quickly withdrew some of them. They also produced a certain amount of dissension in her campaign ranks that allowed much of her campaign momentum to drop off.

By this week, Scott had dropped all the negative ads and was instead running television commercials that showed Bush and U.S. Sen. Charles McC. Mathias endorsing her. Another commercial displayed patriotic pictures of Reagan, flags, the space shuttle and then Scott and encouraged viewers to vote for Scott.

But while the ads, and other similar squabbles produced headlines and dominated the time of the candidates, issues of national and local concern did emerge. Both candidates said they favored reductions in the size of government and in government expenditures, but Hoyer came out strongly against many cutbacks proposed by the Reagan administration that he said were too harsh.

Scott, on the other hand, strongly backed the Reagan budget and the Kemp-Roth tax cut that Hoyer labeled inflationary. At the same time, Scott said that she would not tolerate cutbacks that had any effect on the 5th District. On many other issues the two candidates held similar views: both oppose constitutional amendments banning abortions; both support gun control measures; both oppose cutbacks in the Metro system, education aid and the cost of living increases granted federal workers.

It was perhaps this similarity on many issues that led many voters back to the candidates themselves when they cast their ballots yesterday. At seven of the district's 112 polling places, many voters spoke of Reagan, their support for or opposition to his proposals and their desire to support their party. But an equal number spoke simply of the two candidates.

"I like Steny Hoyer better for personal reasons," said one woman as she exited Bowie Senior High School in Scott's hometown. Said another, "Audrey Scott's been good for us in Bowie and she's got the same thing Spellman had. That's how I decided how to vote."

The final results from the 112 precincts in Prince George's and the Takoma Park section of Montgomery County were: Hoyer, 42,009; Scott, 33,157; and Mathers, 989.