The House voted yesterday to eliminate one-fifth of the mileage in Amtrak's nationwide rail passenger system in an effort to concentrate a reduced federal subsidy on the busiest trains. The Carter administration had wanted to cut services by 43 percent.

Rep. James J. Florio (D-NJ), chairman of the House Commerce subcommittee on Transportation, said the compromise, to which ththe Department of Transportation agreed, will retain a 22,000-mile network that will be "cost effective and fuel efficient."

The legislation, approved by the House, 397 to 18, goes to the Senate, where another effort will be made to retain the 27,500-mile system in light of soaring ridershhip resulting from the energy crisis. A proposal to preserve the existing system with a one-year freeze was defeated in the House Tuesday night, 214 to 197.

The Senate may try to get a vote on its Amtrak bill before the August 3 recess. Supporters of the full system are rallying behind an amendment by Sen. Frank Church (D-Idaho) that is similar to the amendment that lost in the House.

Florio said the threat of a Carter veto may have made the difference in defeating the freeze proposed by Reps. Albert A. Gore Jr. (D-Tenn.) and Wyche Fowler (D-Ga). One of the administration officials who lobbied most forcefully against the Gore-Fower plan was Acting Department of Transportation Secretary W. Graham Clayton, former president of Southern Railway System.

One source said Clayton, who is scheduled to become deputy secretary of defense when a replacement is found for resigned DOT Secretary Brock Adams, urged a member of the House Armed Services Committee to reject Gore-Fowler "because we can use that money a lot more at the Pentagon."

Clayton, through a spokesman, acknowledge telephoning House members and urging defeat of the amendment. But the spokesman said Clayton "wouldn't have put it that way. He is doing only one job now, and that is as acting secretart of transportation."

Generally, long-distance trains will be eliminated if they average fewer than 150 passengers and lose more than 7 cents per passenger mile. Under the House bills, shorter-haul trains must carry at last 80 passengers and canot losse more than 9 cents per passenger mile. An amendment by Rep. Robert Duncan (D-Ore.) assures that no section of the country will be without passengers service as a resulrt of the new standards.

Florio said trains targeted for cancellations which account for one-third of Amtrak's operating losses, will permit Amtrak to shift cars to train on more profitable routes.

Which trains will be halted will be decied by DOT this fall, based on 1980 ridership projections. Among those likely to be canceled are the Chicago-to-Miami "Floridian" Chicago-Seattle "Hiawatha" Chicago-Houston "Lone Star" one of three New York-to-Florida runs and the "Hilltopper," which runs from Boston to Cattlesburg, Ky., via Washington and West Virginia.

Florio cited the Hilltopper as an example of a "political train." Even at the height of the gas shortage this summer, Florio said, the Hilltopper was operating 60 percent empty. The route of the Hilltopper includes part of the district of Rep. Harley O. Stagers (D-W. Va.), chairman of the House Interstate and Foreign Commerce Committee.

While Stagers did not succeed in getting the Hilltopper included in the House bill, another political train that runs through his state, the Wasington-Chicago "Cardinal," appears to be in better shape. Senate Majority Leader Robert C. Byrd (D-W. Va.), managed to get the Cardinal included in the proposed Senate bill. Neither train would survive under the House proposal, however. And differences must be resolved in a conference proceedings.

The House bill retains these trains that would have been eliminated by the DOT proposal: Washington-New Orleans "Crescent," Washington-Montreal "#montrealer," Wasington-Cincinnati "Shenandoah." Chicago-Laredo Tex. "Interamerican," Salt Lake City-Seattle "Pioneer," Seattle-Portland "Mt. Rainer," Oakland-Bakersfield 'San Joaquin" and Chicago-Los Angeles "Southwest Limited."