RICHMOND, SEPT. 14 -- Members of the Northern Virginia legislative delegation said today they will challenge the Wilder administration's plans to take back $20 million a year in tax money promised to the region as part of a delicate political deal.

"How many times do we have to win this?" complained Del. Leslie L. Byrne (D-Fairfax). Byrne, a member of the House Finance Committee, noted that Washington area legislators already had agreed to take only one-half of the proposed windfall in the coming year.

"It's like playing a carnival game," said Byrne, who usually is a strong supporter of Gov. L. Douglas Wilder. "You knock down the milk bottle, and then they tell you {that} you have to knock it down five more times before you can take home the stuffed bear."

The stuffed bear in this instance was a creature of the 1989 session of the legislature. It would allow Northern Virginia governments to retain an additional $20 million a year in real estate filing taxes for five years and dedicate the money to transportation and education.

That was to pay back Northern Virginia and some other parts of the state for a similar amount of money that had been taken from a statewide pot and dedicated to the reconstruction and widening of Route 58, which runs the length of the state along its southern border.

Because the formula for distributing the real estate transaction taxes was tied to the value of local real estate transactions, about half of the total was expected to go to high-growth Northern Virginia cities and counties.

The 1990 legislature, reacting to warnings of an economic slowdown by outgoing Gov. Gerald L. Baliles, voted to give local governments only half of the extra money, beginning next January, with the full share to begin in 1992.

Meanwhile, however, money for the road project, a pet of House Speaker A.L. Philpott (D-Bassett), is being paid out, allowing the highway work to begin.

The political deal was struck when the state was enjoying a record budget surplus; since then, the economy has gone into a tailspin, forcing Wilder to propose cuts of nearly $1.4 billion in the current two-year budget.

Wilder's plan to void the real estate tax deal "smells of impoundment," said Sen. Robert L. Calhoun (R-Alexandria). Calhoun recalled that after President Nixon arbitrarily withheld money for appropriated projects, Congress passed legislation outlawing that practice. Calhoun suggested the legislature might follow a similar tack when it convenes in January.

While Byrne insists that "it's a basic question of a promise made, and promises should be kept," Del. Warren G. Stambaugh (D-Arlington) said canceling the arrangement is "not necessarily that big of a deal. There is a lot of pain {because of the revenue shortfall} and everybody's got to share. I don't think the localities are exempt."

House Finance Committee Chairman C. Richard Cranwell (D-Vinton), who was a key player in the 1989 compromise, dismissed a suggestion that lawmakers have a moral obligation to uphold the tax half of the deal in return for providing money for the downstate highway.

"I've only heard about that in the newspapers," Cranwell said.

Del. Marian Van Landingham (D-Alexandria), who is a member of the House Appropriations Committee, where any restoration of the tax would begin, pointed out that the loss of the real estate tax money won't be as great as expected because of the slowdown in the housing market.

For example, the payout to Fairfax County, using a five-year average of real estate transactions, would have been $11.8 million annually, beginning in 1992, but using last year's sales figures alone, the total would have been only $10.8 million. The exact amount at stake won't be known until January, because the payouts were to be based on transactions in the preceding three months.