An 11th-hour attempt by the Carter administration to quadruple the size of a minority business assistance program backfired this week, threatening its chances for survival in the Reagan administration.

The ill-fated effort to expand the pilot program from a single-agency trial run in the Department of the Army to full-scale operations in other major departments -- despite specific congressional orders not to do so -- was led by a White House aide and apparently was supported by Small Business Administration officials and black congressional leaders.

Already faltering under criticism by participants and federal auditors, the program, which is up for renewal later this year, and its sponsors have drawn fire from an incoming Senate oversight committee for an unauthorized attempt to extend it to the departments of Energy and Transportation and the National Aeronautics and Space Administration.

The aborted attempt to expand the program started with a White House memorandum signed by outgoing President Carter on Dec. 19. It was met by an embarrassing legal challenge from one of Carter's hand-picked inspectors general, and officials were forced to draft a last-minute retraction of the presidential memo just one day before Carter left office.

But, in the around-the-clock rush to win freedom for the 52 American hostages in Iran, Carter failed to sign the retraction.

The Carter aide who drafted the memorandum, Pauline Schneider, has refused to testify -- claiming executive privilege -- before the Senate oversight committee that has called a hearing for today to investigate the episode.

Even without the presidential retraction, opposition and criticism from several quarters have caused the program's sponsors to balk and have reduced the program's chances for expansion, or perhaps even renewal, in the Reagan administration to "slim to none," Scheider acknowledged.

The two-year program was set up by Congress in October 1978 as part of the Small Business Administration's ongoing minority business assistance effort. But under the pilot program, the president was instructed to pick a single federal agency to test a new concept. The Army was selected.

SBA officials were given the unprecedented power to pick and choose among Army contracts, granting some to minority contracting firms whose skills appeared to match the required work.

Giving one federal agency the power to grant the contracts of another is a bitter enough pill for any administrator. But outgoing Army Secretary Cliford L. Alexander Jr. said he also vigorously opposed the selection of the Army as the test agency because, during his tenure, the share of Army business going to minority firms jumped from $98 million to $575 million. "Clearly this [pilot program] should have been applied to some agency that has not been as imaginative as the Army in accomplishing more minority business," Alexander said.

A forthcoming General Accounting Office report agrees with Alexander. In addition, the report says that the pilot program has not been properly managed from the beginning. To date, only eight contracts totaling about $30 million have been awarded.

Federal auditors reviewed four of the contracts last year and discovered several questionable practices:

Instead of SBA officials selecting the contracts and matching them to minority firms, the auditors found that in three of four instances, the contracts had been selected by the minority firm. The selection was then rubber-stamped by the federal official.

One of the contracts awarded went to a "joint venture" between a minority firm and a non-minority firm. After the minority firm was awarded the contract, much of the work was subcontracted to non-minority firms.

One of the four contracts went to a firm that had never performed the type of work called for. As a result, the small Business Administration awarded the firm an outright grant of $1.2 million so it could retool for a new line of work.

Sources familiar with the GAO report said SBA officials did not challenge these findings, but claimed some of the problems had been corrected.

Bu last August, when the pilot program was about to expire, congressional hearings were called to consider extending the program for one year. William H. Mauk Jr., deputy administrator for small business, acknowledged during testimony that, "My reading of the laws . . . indicates that the president would designate an agency -- one agency."

Despite his stated understanding of the law, Mauk and other officials in his agency advised Schneider at the White House that an expansion of the program was legal and proper, Schneider said in an interview. "No one ever raised a question of whether the president had the authority to designate more than one agency. I was never aware of [Mauk's] testimony," she said. "It never came to my attention."

Mauk could not be reached for comment.

Congress voted to extend the program to September 1981 and maintained the one-agency limit. Three months later, Schneider drafted the memo extending the program to three new agencies. When Carter signed on Dec. 19, he said, "This program offers opportunities for minority businesses to become more competitive and to participate more fully in high tecnology contracts, which are an increasing portion of federal procurement activity."

The inspector general at the SBA, Paul R. Boucher, read the president's memo Jan. 7. The next day his chief legal counsel issued an opinion saying, ". . . I believe authority does not exist."

Boucher, who has been asked to resign along with all other Carter-appointed inspectors, said in an interview, "To me it was just patently illegal and improper and I was concerned about the discredit it could bring to the office of the president."

Also alerted to the memo was the staff of Sen. Lowell P. Weicker (R-Conn.), the new chairman of the Small Business Committee, which called today's hearings.

Scheider, reacting to the suggestion by congressional investigators that the presidential memo was a last-minute attempt to circumvent congressional intent, said, "There was a hell of a struggle to get a simple one-year extension on the Hill this year and I believe that the probability of extension beyond Sept. 30 [1981] is probably slim to none, so you have to maximize any opportunities when they're presented.

"You only go around once in life."