It's now-or-never for fixing up Union Station and the National Vistor Center, congressional and administration boosters of the venerable building said yesterday during a tour of the deteriorating structure.

The toughest talk came from an aide to Interior Secretary Cecil D. Andrus. Andrus was said to be prepared to "board it up, put a padlock on and walk away from it" if Congress again fails to appropriate money to restore the complex, which detractors have ridiculed as a monumental white elephant.

The latest effort to get money to finish a project that began 10 years ago and already has cost the taxpayers $44 million is expected to come to a vote in the House in a month. This year's legislation seeks $36.1 million, which is $3 million less than was rejected by the House last Dec. 19 by a vote of 247 to 139.

The money would be used to restore the train terminal to the concourse area, complete a partially constructed 1,400-spacing parking garage, relocate the vistors center in less space and make critically needed repairs to the 71-year-old classic Roman building. An additional $22.84 million already has been appropriated from Northeast Corridor rail funds to improve the train portion of the complex.

To illustrate the urgency for repairs, hard hats were distributed yesterday as House Public Works Chairman Harold T. (Bizz) Johnson (D-Cal.) led reporters and photographers on an inspection of the crumbling roof and walls.

Robert H. Mendelsohn, Andrus' assistant, said that if Interior carried out its threat to close the Vistor Center, which occupies the main concourse, rail passengers would be forced to walk around the building to the relocated terminal in the rear.

The vistor center, which took over the concourse as part of the bicentennial celebration, has been the target of the sharpest criticism, and it was there that the $3 million was eliminated from last year's legislation. Only $100,000 of the $36.1 million would be spent on reconfiguration of the vistor center.

Johnson said the legislation that passed his committee by voice vote last month will be "the last attempt" to restore "this beautiful building which is a landmark." Johnson said it will be "hard to sell" his collegues in the Congress to support the bill, especially in an election year when the emphasis is on budget-cutting.

"But this [$36.1 million] won't make much difference in the total budget," Johnson said.

The sponsor of the legislation, Rep. Elliot H. Levitas (D-Ga.) conceded that "clearly most of the House regards this as a white elephant." Levitas said it would be even more of a waste of money not to fix up the building, because the government must pay $3.5 million a year in rent to the railroad owners for the next 21 years, whether or not the building is used. "It would be an act of fiscal irresponsibility not to protect the investment we have," Levitas said.

The train faclitites, including ticketing, waiting room, restaurant and newsstand, were relocated behind the building in a low-budget, low-ceilinged space that has been said to look like a motel lobby.

The move occurred before the energy crunch and the subsequent increase in train ridership. The vacated space was to become a permanent display area for tourists. But cost overruns and complaints by rail users resulted in halting work on the elaborate visitor center at the height of the bicentennial.

Rep. John Ashbrook (R-Ohio), in urging defeat, of the legislation last December, said the incomplete project could stand as a symbol to taxpayers, who could point to it as a "water line. . . where we stopped" big federal spending programs.

Even some of the harshest critics, such as Rep. Robert E. Bauman (R-Md.), who dubbed the grand scheme as "the great train station robbery," agreed that "something has to be done on this building on which all this money has been spent."

An Amtrack representative who took part in yesterday's tour estimated that the number of train passengers using the station annually will double, to about 10 million, by 1990.

If the restoration is carried out, Mendelsohn of Interior said the complex would be "one of the best multimodel facilities anywhere," with easy transfer among auto, taxi, bus, subway and train, including the Metro route to National Airport.

The big pitfall to acceptance of the legislation may be the continued existence of the pit just inside the main entrance, which was the site of an expensive multi-screen slide show until it was shut down because of cost.

The legislation does not mention the pit. But Mendelsohn said it is likely that part of the $100,000 for the visitor center will be used to cover it "in the least expensive way."