The Senate Commerce, Science and Transportation Committee voted 15 to 1 yesterday to approve the nomination of Dr. James C. Fletcher to head the nation's embattled space agency. But panel members also vowed to end congressional "back-scratching" and toughen their scrutiny of the agency and its new chief.

Senate confirmation of Fletcher as administrator of the National Aeronautics and Space Administration, a post he held in the 1970s, is expected within a few days.

Sen. Ernest F. Hollings (D-S.C.), discounting recent criticism of Fletcher's earlier performance in the job, praised his experience and judgment, saying, "I think the fella is really just exactly what the doctor ordered."

Sen. Albert Gore Jr. (D-Tenn.) cast the dissenting vote. Gore said that while he does not question Fletcher's integrity or personal fitness, "I do not have confidence in Dr. Fletcher's judgment."

Gore was referring to what Hollings dismissed as "10-year-old audits" showing billions of dollars in cost overruns at NASA, as reported in The Miami Herald in March and The New York Times last month. The overruns occurred in part during Fletcher's tenure.

Gore noted that as NASA chief, Fletcher told Congress the shuttle would lift payloads into space for $100 a pound. The figure turned out to be in excess of $5,000 a pound or, after adjustment for inflation and other factors, "still a 400 percent increase," Gore said.

He also cited, as another example of Fletcher's tendency toward "wildly optimistic" projections, Fletcher's assertion to Gore on Tuesday that "there's a 50-50 chance" that the Reagan administration's Strategic Defense Initiative would provide a "nearly perfect" missile shield by the year 2000. As head of a study panel, Fletcher helped develop the administration's SDI proposal.

Fletcher, 66, headed NASA from 1971 to 1977. During his tenure, the Apollo moon-landing program and the Skylab space station mission were completed, and two Viking robots landed on Mars.

Fletcher also helped shape the shuttle program, now in crisis following the Jan. 28 loss of the Challenger with its crew of seven.

In a letter to Gore, Fletcher said that one of the published newspaper reports "grossly overstates the increase" in flight costs because of the way they were figured. He expressed a similar objection to the reported cost per pound of payload.

Fletcher added, "There was never an attempt to 'mislead anyone on the cost aspect of the shuttle.' " Early predictions of costs were raised based on experience as the experimental and complex program moved into actual flight, he said.

In an interview with The Washington Post at the time of his nomination in March, Fletcher acknowledged having made some "possibly wrong" decisions and said he had been "over-optimistic" in some of his projections.

Some panel members suggested that Congress bears part of the blame for "failures" in the space program.

Sen. Donald W. Riegle Jr. (D-Mich.) described the House space panel as a "special-interest committee," and said that status was "damaging to Congress and to the space program." Riegle pointed to report that the committee's chairman, Rep. Don Fuqua (D-Fla.), will become president of a space and defense manufacturers trade association after his retirement in January.

Riegle also criticized Sen. Jake Garn (R-Utah) and Rep. Bill Nelson (D-Fla.) for riding the shuttle. "I think it creates a set of tangled relationships that make clear oversight difficult . . . ," Riegle said. "I think we have to put an end to the back-scratching, an end to the less-than-arms-length relationship" between Congress and the space program.

Hollings, despite his support for Fletcher, echoed concerns about NASA's management. "Dr. Fletcher will know we're going to be looking over his shoulder," he said.

Fletcher has given assurances that, despite past attachments to the agency, he is "willing and able to go in and remove people who may need to be removed" and undertake "whatever internal overhaul" may be needed, Riegle said.

Fletcher, a consulting engineer based in Virginia and a professor at the University of Pittsburgh, would replace James M. Beggs, who was forced to resign early this year in connection with an investigation unrelated to NASA. Dr. William Graham is serving as acting administrator.