Maryland Republicans, having delivered their state for President Reagan and toppled an 11-term Democratic congressman, basked in their redemption yesterday and savored the potential for expanding their party's narrow base in a state that remains overwhelmingly Democratic.

"I think it means r-e-s-p-e-c-t," crowed state Sen. Howard A. Denis, a Montgomery County Republican, in an instant postmortem on Tuesday's results. "We have a degree of credibility now that we can build on. We've shown that the Democratic machine in Maryland is vincible."

In the afterglow of their most successful outing since the Nixon landslide of 1972, the euphoria of Denis and his fellow Republicans existed on two levels: in the short term, they have erased the ignominy of four years ago when Maryland was one of only six states to resist the first Reagan onslaught; and they envision a coming day when the GOP becomes competitive enough to win more than the smattering of local and statewide offices it now holds.

The first goal they achieved on Tuesday. "We've rejoined the continent. Now we will get our mail answered from the White House," Denis said. The second goal will probably prove more elusive.

What is clear is that in Maryland, as across the country, Republicans benefited from the enormous popularity of an incumbent who appealed broadly to almost every segment of the population except blacks and Jews, and who tore the heart out of the traditional Democratic coalition.

In Maryland, that meant capturing the Baltimore suburbs and large numbers of voters who not so long ago would have been considered blue collar, but who have gained sufficient economic security to be transported philosophically, if not geographically, into the suburban middle class.

Democratic candidate Walter F. Mondale retained and even improved upon the base that delivered Maryland for Carter in 1980 -- black districts in Baltimore, and Prince George's County, where the Democratic margin was twice what it was in 1980. But he ran well behind the Carter pace in Baltimore County and Anne Arundel County. Large numbers of blue collar and lower-middle-class voters living in the inner suburban rim around Baltimore apparently defected to Reagan.

"Maryland is a classic example of what went wrong with the Mondale message," said Lanny J. Davis, one of Maryland's Democratic National committeemen. "For the first time in memory, a Democrat didn't come out of the Baltimore area with a significant margin."

The numbers tell the tale. Though Reagan won 40,000 fewer votes in Montgomery and Prince George's counties than he did in 1980, he more than made up the difference in the Baltimore area. His margin in Anne Arundel, Harford and Baltimore counties was 91,000 votes higher than four years ago, and he lost Baltimore city by 15,000 fewer votes than in 1980. Statewide, Reagan won by about 79,000 votes.

Within Baltimore city, Mondale outpaced Carter's success in black and Jewish areas, but suffered greater losses in some white ethnic neighborhoods in the eastern, southern and northeastern sections of the city.

Michael Frazier, the Mondale-Ferraro campaign's field director for Maryland, attributes the loss to several factors: a late start in getting the campaign machinery in place following the Democratic convention; a Republican incumbent "who transcended politics," and a competitive congessional race between Rep. Clarence D. Long and Republican Helen Delich Bentley that boosted turnout in Baltimore County.

"We just couldn't keep it Reagan's margin down there," said Frazier of Baltimore County, where Bentley upset the 22-year veteran Democrat. "I don't know how we could have. We just couldn't turn it around."

Whatever the reasons, Republicans and even some Democrats see a lesson in Tuesday's numbers: there is at least the potential for Republicans, given the right candidates, to achieve enough of a realignment to make Maryland a two-party state.

"What this does is create a tremendous source of finances which will help fund the party and enable us to do a lot of things we need to, such as recruit candidates and encourage them to run," said state GOP Chairman Allan C. Levey. "Those who we approach now can realize that we are believable."

"We've lost a basic component which should be voting against the Republican Party," added Democrat Davis. "They [the Republicans] have an absolutely great opportunity, but judging from past history, they're apt to blow it."

Even Levey admits that the state party has a considerable distance to travel. There are still almost three registered Democratic voters for every Republican. In the state Senate, Republicans control just six of 47 seats. In the House of Delegates only 17 out of 141 members belong to the GOP.

Even in this flush Republican year, the state GOP was hobbled somewhat by a lack of money. Montgomery County Republicans didn't even have enough funds on hand to send out their sample ballot this year.

Peter Marudas, a political aide to Democratic U.S. Sen. Paul Sarbanes, insists the majority party in Maryland has history, if not immediate momentum, on its side.

"There was a huge Nixon vote here in 1972 and nothing ever materialized from it," said Marudas. "This doesn't mean anything in the state. It was a vote for Reagan. When you look at the list of candidates for statewide office that the Democrats have to offer and any list, if one exists, that Republicans have to offer, there is no comparison."

Looking ahead to 1986, when the governorship is at stake, Republicans themselves concede they have no one in sight who measures up to the Democratic heavyweights jockeying for their party's nomination, such as Attorney General Stephen H. Sachs, House Speaker Benjamin L. Cardin and possibly Baltimore Mayor William D. Schaefer.

But, Levey said, the perception that the GOP can win in Maryland will bring out better candidates in future. "It makes some of those individuals who are thinking of running statewide believe it is possible. There are a lot of people out there who have thought of running but never dared because we weren't credible."

"We have to be pragmatic," Levey added. "We need candidates to run for county councils, county executive, House of Delegates and state Senate . . . .It [the Reagan victory] doesn't mean we're going to be the majority party in 1986, but it means we can make great inroads in 1986, and who knows what can happen after that."