One day during the budget debates on Capitol Hill, President Reagan welcomed Maryland's freshman Rep. Royden P. Dyson and several of his Democratic colleagues into the Oval Office to ask their support on a crucial gudget vote.

Dyson waited a few minutes, he recalls, before telling the president that he had only one political interest: "If we vote [with you] will the Republicans campaign against us? That's the only thing I'm interested in."

Reagan chuckled, but it was a serious moment for Dyson, a man preoccupied with getting reelected to the 1st District seat he won last November after Republican incumbent Robert Bauman was charged with sexual solicitation of a teen-age boy.

For the last nine months, Dyson, the one-time Maryland delegate who unsuccessfully pursued the seat for years, has nervously looked over his shoulder at a possible comeback by the deposed but still popular Bauman or a challenge from the many Republicans and Democrats who say he won only because of Bauman's misfortunes.

"Everyone knows he's vulnerable," said Harford County Democratic Sen. Arthur Helton. "Roy represents a district that, all things being equal, he would not have won if the catastrophe hadn't happened to Bauman."

So it was no surprise to Dyson this week when the state Republican Party began a series of radio advertisements that attacked a few budget votes he cast against Republican proposals -- and ignored other equally important budget votes in which Dyson supported Reagan.

Throughout Dyson's conservative district, a huge sweep of farmland and subrubs that bucked the state trend in 1980 and voted overwhelmingly for Ronald Reagan, voters are being told: "Roy Dyson voted against Ronald Reagan . . . . Roy Dyson voted against you."

The ads, said Maryland Republican Party Chairman Allan Levey, are the opening round in an effort to reclaim a district that, politically, is as conservative as the heart of Dixie. Despite a Democratic edge in registration, the Republicans have controlled the 1st District seat for two decades and come 1982, Levey said, "the seat is very winnable . . . whether Bob Bauman, [former Republican State Sen.] Porter Hopkins or Frank Perdue runs."

"They've been waiting to do this for six months," said Dyson of the ads. "In a sense, I've been damned from the day I was elected. They're trying to erode confidence in me."

While the Republican radio campaign is Dyson's current worry, it is not his only one. As he frequently traverses the district -- which stretches from the Pennsylvania border to the Eastern Shore to southern Maryland -- he finds that many people believe his election was a fluke and that several Democratic state legislators are openly hungering after his seat now that Bauman is gone.

Dyson has also heard that some of his potential challengers in the legislature would like to make him even more vulnerable by redistricting his southern Maryland home out of the 1st District when congressional lines are redrawn next year.

While Dyson says this is an unlikely possibility, he says he has nonetheless been scouting around for houses in Salisbury, safely in the heart of the Eastern Shore.

An aggravating as all this is to Dyson, whose energy for campaigning is legendary, he is most jumpy about the man he defeated in 1980, Bob Bauman, the acerbic, conservative pointman who is now an Easton lawyer.

Dyson pursued Bauman's seat for five years as a delegate from St. Mary's Conty until the Republican's popularity and tight hold over his district foundered on his confessions of "twin compulsions of alcoholism and homosexual tendecies." But despite his problems, Bauman's name is still well-known and his reputation as a good congressman persists in many parts of the district.

"Bauman is the only really viable opponent I have," Dyson says.

Bauman, his friends say, would like to return to Congress now that the right wing Republicanism he sepouses is much in vogue. While he is not yet committing himself to a rematch, Bauman recently released the results of a poll that showed him beating Dyson in a hypothetical rematch.

According to the poll by the New York-based Arthur J. Finkelstein and Associates, 41.9 percent of the 309 voters surveyed said they would support Bauman and 33.7 percent would support Dyson if an election were held today. The poll showed the eight-point lead for Bauman even though it also showed that 38 percent of the voters believe he is an alcoholic and 46.9 believe he is homosecual.

While Dyson challenges the accuracy of the poll and says he doubts Bauman will want a rematch that is sure to drag his personal problems into the limelight again, he is nonetheless preoccupied with the possibility.

Recently, Dyson showed up at a Talbot County training school function where Bauman, the school's lawyer, was present. Dyson shook hands all around, avoiding Bauman, and then dashed off. "I made him nervous," Dyson says, "that's why I went."

And while Bauman, equally preoccupied with Dyson, says he successor is "paranoid," he takes a certain pleasure in the fact that his first public speaking engagement since the "troubles" will be on Dyson's home turf, at a meeting of the St. Mary's County Young Republican Club.