A federal judge in Alexandria yesterday refused to stop construction of a controversial high-rise office building in Rosslyn opposed by the Interior Department as a "visual intrusion" on Washington's parks and monuments.

U.S. District Court Judge Oren R. Lewis, criticizing the government's action as "arbitrary, capricious and unreasonable," denied a request by Justice Department lawyers for a temporary restraining order to halt work on the Rosslyn Center building.

Rosslyn Center is scheduled to be the tallest of five high-rise office structures whose height the Interior Department is seeking to limit through a lawsuit against the Arlington County Board and the buildings' developers.

Rosslyn Center, which developers estimate will cost $31 million, will be 22 stories high upon partial completion by the end of next month. The building and penthouse are expected to rise 351 feet when finished. As of Tuesday the building was 297 feet high, already 114 feet above the highest buildings in Rosslyn, the government contended.

"It's arbitrary, capricious and unreasonable on the government's part to stop something that will run into the millions when they delayed and they were negligent," said Lewis, an Arlington resident.

"I'm not going to give you any temporary restraining order. You want me to stop a building that would cost stockholders millions of dollars in losses."

Lewis said he was unwilling to issue such an order "unless the government tells me they're perpared to pay for it. But you're not willing to do that," Lewis said, staring at Justice Department lawyer Andrew F. Walch. "I just don't think you have that right."

Lewis said the government had the right to condemn the property and pay the developers for their losses, but Walch said the government would not agree to that.

It tis 23-page Iawsuit against the County Board and the developers of the five buildings, Interior contends that Arlington violated its own zoning laws by approving the "excessively" tall buildings, that the buildings are a public nuisance and that the county failed to give adequate notice to federal official before approving the buildings.

"Are you saying the government didn't know this was going on" Lewis asked. "This is inconceivable."

Lewis yesterday set Jan. 22 for trial of the suit.

Justice Department lawyers said they sought the restraining order because the "uppermost floor remaining to be constructed will have the greatest visual impact on the nearby parks, memorials and monuments of the United States." Construction has "proceeded far encough to imperil the federal objective of halting further degradation of the Rosslyn skyline...", they said.

Government lawyers yesterday said the Interior suit is meant to serve as a precedent. "We need to establish this reight... around the country" that a building can be a nuisance.

Rosslyn Center's developers, Rosslyn Center Associates, have said in court papers that any delay in construction would cost them $22,750 a day plus costs of renegotiating contracts and leases.