The two major candidates for the one open congressional seat in Virginia's 10-member House delegation are ending their race agreeing that the size of President Reagan's margin in the district will be crucial to their race.

Republican D. French Slaughter Jr. of Culpeper and Democrat Lewis M. Costello of Winchester have expressed optimism, but Costello campaign aides say their candidate will have to overcome the GOP national ticket's coattails to claim the 7th District seat, held for 14 years by Republican J. Kenneth Robinson. He did not seek reelection.

"Our polls show it's too close to call," said Tom Vandever, Costello's campaign manager. "Of course we're very concerned about Reagan's popularity, especially with so many undecided."

Most politicans give an edge to Slaughter, 59, a lawyer who has preached the gospel of Reaganism but has been dogged by charges that he refuses to apologize for his support of Virginia's policy of massive resistance to school desegregation in the 1960s.

"We feel pretty encouraged about it -- our polls show us about 7 to 8 percent ahead," said Dennis Peterson, an aide to Slaughter, who represented Culpeper in the House of Delegates from 1957 to 1977. Peterson said that Reagan is even further ahead and that the size of the president's margin will be critical to Slaughter.

A third candidate, independent Robert E. Frazier Sr., 59, of Earlysville, is not considered a serious factor in the race.

The district stretches from West Virginia to the Richmond suburbs and includes parts of Prince William and Stafford counties.

Costello, 51, a tax lawyer, has been on the defensive, having had to withdraw a radio advertisement blasting Slaughter for his support of involuntary sterilization laws. The ad had to be withdrawn after it was shown to contain an inaccuracy. Slaughter has capitalized on the mistake, charging that his opponent is "playing fast and loose with the facts" and waging a negative campaign.

The aggressive tone to Slaughter's rhetoric in recent days contrasts with his campaign style, which has been low-key and bordering on the dour.

Costello originally tried to avoid charges of negativism in the campaign, letting his staff members spotlight Slaughter's record in the General Assembly, which included support for segregated schools and opposition to a variety of voting rights and civil rights measures. He insisted that he would not attack Slaughter directly.

In recent weeks, however, he has run radio and television advertisements attacking Slaughter directly for having supported the poll tax in the early 1960s, sterilization for children with epilepsy, and school closings as a means to block integration. Slaughter has refused to repudiate those votes, declaring, "I can't change the record and I wouldn't."

Costello has sought to turn this to his advantage. "I think Slaughter's polices were wrong then, and they're wrong now," he said in a typical ad. "We can't turn back."

"The Costello campaign has been focused primarily on issues of 20 years ago," said Slaughter campaign manager Peterson. "You may find it hard to believe, but people do not come rushing up to French asking about 1959 issues."

Slaughter ads have returned the fire, charging that Costello thinks taxes are too low and tying the Democrat to presidential candidate Walter F. Mondale and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.).

In fact, the two candidates are not far apart on fiscal isses. Both support a constitutional amendment designed to balance the federal budget.

The race is by far the most expensive ever in the district, according to both camps. Slaughter has raised and spent nearly $400,000, including $100,000 in a GOP nomination fight against Guy O. Farley Jr. of Warrenton, a born-again Christian who ran unsuccessfully for state attorney general and lieutenant governor. Costello, aided by the formidable fund-raising efforts of Del. Alson H. Smith Jr. of Winchester, has raised nearly $250,000.