Republicans made substantial gains in the Democratic-controlled Virginia House of Delegates yesterday, but failed to take control of the Northern Virginia delegation as they had hoped.

Late returns showed the GOP capturing 33 seats in the 100-member House statewide, increasing their share from the current 25. In Northern Virginia they won two newly created seats, but the Democrats managed to preserve a slim 11-to-10 majority, comparable to their current 10-to-9 edge.

Democratic challengers turned out two incumbent Fairfax Republicans who were supported by the New Right, John S. Buckley and Lawrence D. Pratt, who were both seeking their second House terms. But Republican challengers defeated incumbent Democrat Earl Bell in Loudoun County and captured a newly created seat in fast-growing Prince William County.

GOP leaders, who had hoped for more dramatic success in the Washington suburbs, said the strength of the statewide Democratic ticket and a well-financed campaign by supporters of the Equal Rights Amendment hurt several of their candidates.

"People accused me of being conservative . . . People accused me of opposing the Equal Rights Amendment. I boast of that," said Buckley, a target of the pro-ERA forces. Buckley, a political consultant and cousin of conservative columnist William F. Buckley, and Pratt, executive director of Gun Owners of America, both told supporters they will run again next year.

The 21 winners in Northern Virginia, as well as others across the state, will serve only one-year terms, during which they must redraw the boundaries for Virginia's House districts. Last summer a federal court ruled that the plan drawn up by the legislature in response to the 1980 census was unconstitutional --although this fall's elections were permitted to proceed under that plan. The court ordered the General Assembly to submit a new plan by Feb. 1.

Most Northern Virginia incumbents held onto their seats in a campaign marked by apathy and confusion over redistricting. But one-term incumbent David G. Speck, a Republican moderate from Alexandria, was defeated by Democrat Marian A. Van Landingham, a civic activist and political newcomer.

John H. Rust Jr., a 34-year-old moderate Republican, led the field in Fairfax County's 50th District, which includes Reston and Vienna, despite having been targeted by anti-ERA forces for defeat. Rust angered conservatives when he switched his position on the amendment from opposition to support.

Rust was elected along with incumbent Democrat Dorothy S. McDiarmid, the dean of the Northern Virginia delegation, and Democratic challenger Kenneth R. Plum, who took Buckley's seat after having lost to him two years ago.

"Two years ago I didn't realize what we were up against," said Plum, referring to Buckley's direct-mail campaign engineered by conservative political consultant Richard Viguerie. "People thought the Democratic Party was dead, but people who support reasonableness in government, that's what won it for us."

Another moderate Democrat, Vivan Watts, led the field in the 51st District, which includes much of southwestern Fairfax. Watts, who ran unsuccessfully for county Board of Supervisors chairman in 1979, defeated Pratt and was elected along with two Republican incumbents, Robert E. Harris and moderate James H. Dillard II.

The Equal Rights Amendment, which the Virginia General Assembly has refused to ratify, was one of the few issues to emerge in the campaign, and pro-ERA forces spent thousands of dollars to defeat Buckley, who at 28 was the youngest member of the Assembly. But anti-ERA forces won a victory in the 49th District, which includes McLean and Great Falls, with the election of Gwendalyn F. Cody, a former unsuccessful candidate for supervisor. Cody, a Republican, strongly opposes the ERA and defeated the ERA's strongest supporter in the race, Democrat Flora M. Crater, founder of the Virginia Women's Political Caucus.

Also elected in the 49th District were Republican newcomer Robert T. Andrews and seven-term GOP incumbent Vincent F. Callahan Jr.

Democrats in Arlington retained all three seats by returning Mary A. Marshall, Warren G. Stambaugh and James F. Almand by comfortable margins.

The state Republican Party was more active this year than ever before in assisting candidates for delegate, sending money and arranging White House appearances for selected candidates. The party watched with particular interest the races in Loudoun and Prince William counties, where traditionally Democratic majorities have been challenged by rapid suburbanization.

The party's efforts paid off in Loudoun, where Leesburg's Republican Mayor Kenneth Rollins defeated three-term incumbent. Bell. It was Rollins' third and most acrimonious race against Bell, but the first since Loudoun was split into a separate district from Prince William.

"This time it was a head-on race, and I relished every minute of it," said Rollins, whose contest with Bell was among the most hotly contested in Northern Virginia.

In Prince William County, former Manassas Mayor Harry J. Parrish was elected to a newly created seat. Parrish's campaign was dogged by a state police probe into conflict-of-interest charges. But in the last week of the campaign the county prosecutor announced that Parrish had been cleared of those charges.

Parrish will join Prince William incumbents Floyd C. Bagley and David Brickley, both Democrats.