Sen. Charles McC. Mathias, the liberal Maryland Republican, scored the biggest victory of his 20-year congressional career yesterday by overwhelming State Sen. Edward T. Conroy, his conservative Democratic opponent.

In winning by a landslide while several of his liberal Senate colleagues were losing, Mathias became the first Maryland Republican to be elected to three terms in the Senate, and the first member of his party to carry the city of Baltimore in a statewide race.

Mathias beat Conroy 2-to-1, and carried all of the state's 23 jurisdictions, including Conroy's home base of Prince George's County.

"They say gratitude is the least articulate of emotions," Mathias told his exhuberant followers last night at his downtown Baltimore campaign headquarters, "and I can understand that tonight. I feel very inarticulate." h

He calmed the celebration long enough to talk about such problems as starvation and nuclear proliferation, but roused the crowd again by embracing his sons, Charlie and Robb.

Although Mathias went out of his way during the campaign to avoid supporting the top of his party's ticket, he called Ronald Reagan's victory "a healthy and welcome event. It demonstrates the vitality of the two-party system in America, as their campaign slogan indicated. It is a new beginning and we need a new beginning. I look forward to working with the Reagan-Bush administration."

The Mathias-Conroy outcome was decided quickly. The first of the major news organization declared Mathias the winner less than 20 minutes after the polls closed. But Conroy refused to concede until 10:45 p.m., long after it was apparent even to his staunchest supporter that he had incurred a crushing defeat.

When Conroy finally strode to the microphones in the Constellation Room on the top floor of Baltimore's World Trade Center, President Carter already had conceded defeat, and gloom had settled in among the 100 persons, including Maryland Lt. Gov. Samuel W. Bogley, who had watched the devastating returns on television.

After embracing his wife Mary, Conroy acknowledged that "my message was not accepted by most Marylanders." But Conroy, who has served in the Maryland legislature for 18 years, said that "I'm not out of politics. I'll be around for a long time to come."

After Rep. Gladys Noon Spellman of the Fifth Congressional District was critically stricken last week, Conroy was among those Democrats mentioned as her possible successor, if she is unable to claim the House seat that she won for a fourth time yesterday.

Some of Conroy's supporters blamed his lopsided defeat on his staunch opposition to federal funding of abortion, but Conroy said, "I will always maintain my position on abortion. But I can't point to that for my defeat. They were many other issues."

Mathias, 58, outspent and outcampaigned Conroy, 60, staking out traditionally liberal positions on abortion, the Equal Rights Amendment and defense, leaving Conroy to seek support among largely single-issue conservative voters.

There was no last-minute campaign blitz by Mathias. The graying Maryland aristocrat dropped in at half a dozen polling places in and around Frederick County yesterday, before voting in the Frederick armory at 11 a.m. After lunch in Frederick with his 83-year-old mother, Theresa, Mathias moved to downtown Baltimore, to watch the returns.

There was no blitz at all for Conroy. The Democratic challenger, who saw himself as a Scoop Jackson of the East fighting for a strong national defense, needed wide exposure if he hoped to upset his better-known opponent. But he couldn't muster enough money for a single television commercial.

Conroy attempted to make up in personal vigor what his campaign lacked in slickness. He began yesterday by voting with his wife in Bowie. Then he worked the polls until closing time, concentrating on the blue-collar neighborhoods in the Baltimore area, and headed next for a party room in the World Trade Center to await a political miracle. That bubble burst soon afterward.

Conroy not only failed to encroach on Mathias' suburban and rural strongholds, but he also was unable to prevent segments of Baltimore's highly structured Democratic organization, especially in the black precincts, from slipping into the Mathias camp.

Conroy got little financial support from such dependable sources of Democratic dollars as the labor unions. Mathias, who began raising money more than a year ago, and, unlike Conroy, did it without dipping into his own bank account. Matthias spent about $700,000 from a variety of sources, while Conroy spent about $150,000, nearly one-third of which came from his own, not-so-deep pockets.