Virginia's state and local governments could lose as much as $400 million in federal funding next year under the Gramm-Rudman-Hollings deficit reduction law, with the largest cutbacks in money for highways and services to the elderly, a new state study shows.

The figures mark the first public assessment by the state's Department of Finance of the impact of the federal cost-cutting measure, which has dominated budget discussions in the state capital as legislators grapple with finding money for programs already considered threadbare.

"It will trigger devastation to the state," said Del. Bernard S. Cohen (D-Alexandria). "There's no way we can pick up all those programs to be cut . . . . In the short run, this means chaos. In the long run, it's unknown."

State officials and Northern Virginia legislators said it was unclear what impact the cuts might have on the Washington area's Metrorail system.

The projected cutbacks range from $130 million to $400 million, according to the study, depending on the outcome of upcoming budget battles between President Reagan and Congress.

Using the more conservative total, Gramm-Rudman could slash highway aid contributions by more than $13 million in 1987, according to state government budget analysts. Those cuts would come at a time when Gov. Gerald L. Baliles has labeled transportation improvements as one of the state's most pressing needs and has called a special session of the legislature to tackle the issue.

"It will make our job much harder," said Del. Alan A. Diamonstein (D-Newport News.) "It's going to make it transportation a bigger problem."

State officials said cutbacks resulting from the act would total about $20 million in 1986, but would escalate dramatically next year when the full impact of the measure hits.

Federal funds now represent about 18 percent of the state's proposed $18.5 billion budget for the next two years, officials said. The minimum expected federal cut of $130 million is about 6 percent of the 1987 portion of the budget.

Under the $130 million figure, social service programs ranging from day care to services for the elderly would feel the brunt of the budget reductions in Virginia with a $13.4 million loss projected for 1987, the study shows.

Other programs that would sustain losses include: low income energy assistance, $8.6 million; compensatory education for the disadvantaged, $3.5 million; health programs including maternal and child health care, $3.4 million; and urban mass transit, $1.2 million.

Legislators say they are uncertain how to approach the issue because of the uncertainty of future cutbacks.

"It'll probably only cost us about $20 million in 1986," said Sen. Clive L. DuVal II (D-Fairfax). "But from there on, the lid's off."

"We can't just let it all sit until next year," said Diamonstein, a member of the House Appropriations Committee. "We can't legislate by crisis . . . . But what we don't know is how bad it's going to hurt."

The new law sets deadlines for action on the national deficit and requires automatic cuts if there is a stalemate between the executive and the legislative branches. That has left local and state government officials scrambling.

"This can't be a way to solve anything," said Del. Dorothy S. McDiarmid, (D-Fairfax), chairman of the House Appropriations Committee. "Certainly not for us state government ."

But two legislators used the report to plug proposals for a state lottery, which have failed in past years.

"That's why we need the lottery money," said DuVal, the sponsor of one lottery bill. "It would net $150-200 million a year and no one pays who doesn't want to. It's between that and an increase in the sales tax or income tax.

"Opponents of the lottery cite the moral issue, and I agree it's there, but that will have to yield."