When Anthony Solomon, under secretary of the treasury-designate for monetary affairs, recently invited a group in for lunch to discuss international financial problems, he asked how the meal was to be paid for.

"Don't worry about it," a Treasury Department administrative aide responded, "we'll charge it to the Exchange Stabilization Fund."

According to a source, an indignant Solomon retorted:

"The hell you will!" Solomon says he can't remember the exact words he used, but acknowledges he clearly expressed his disapproval of tapping the ESF.

Virtually unknown in the city with hundreds of alphabetized agencies, the ESF is a $4 billion, almost totally unaudited fund whose purpose is to aid in the stabilization of the U.S. dollar.

Since it was set up in 1934 with a stake of $2 billion derived from the profits arising from revaluation of the gold price, it has been under the control of the many secretaries of the Treasury.

Carter administration officials believe that it is urgent to bring the administrative expenses of the ESF - which now run some $20 million annually - under regular budget control.

At the moment, like other "off-budget" federal agencies that derive their capital funds outside the normal appropriation process, the ESF escapes the scrutiny of Congress and its expenditures do not count in the congressionally set budget ceiling.

"What they've got there," says a Capitol Hill budget expert, "is a slush fund. The Secretary of the Treasury can spend whatever he wants to out of it, and there's no control."

Rumors of numerous blatant abuses in the expenditure of ESF funds have circulated in Washington over the years. One Treasury official says flatly that "there is no doubt there have been widespread abuses in the past."

But there is no proof of such charges, and Solomon, more cautionsly says only that he thinks "there is a large gray area."

Quite apart from the luncheon incident, he came to "the independent conclusion soon after arriving at Treasury that "some type of ESF reform consistent with President Carter's objective of openness and co-operation with Congress is necessary."

Carter's officials at Treasury stress they do not want to tinker with the actual financial operations of the fund, which in some cases involve secret and high-level negotiations with other governments.

Since its inception, the ESF has played an active role in stabilization efforts, especially during World War II, and in a series of worldwide monetary crises that followed.

In recent years, it has been put in charge of U.S. holdings of Special Drawing Rights (SDRs), a special international credit distributed by the International Monetary Fund. As of last year, the ESF held $2.3 billion worth of SDRs.

The ESF also earns money on investments in U.S. securities - a sort of out-of-one-pocket, into-the-other-pocket bookkeeping. As of the end of last year, the ESF had accumulated earnings of about $550 million.

What has agitated Congress for the past couple of years in the ability of the Treasury Secretary to dip into these monies for almost any expenditure, so long as it can be ven vaguely tied to international finance.

For fiscal 1978, which begins Oct. 1, the outgoing Ford administration proposed a $20 million ESF administrative budget, including a payroll of 553 persons who earn an average of $20,102 a year. This include the entire international financial establishment at Treasury.

All international travel - including the recent exceptionally heavy expenses for former Treasury Secretary William E. Simon - comes out of ESF money.

The use of the fund appears to have been limited only by the imagination of a particular Treasury Secretary. Simon, one of the most creative, responded to the energy crisis by setting up a new post of assistant secretary for trade, energy and financial resources. This office now has a staff of 109, with 85 per cent of its budget paid by ESF.

Although Simon consistently at tacked "offbudget" spending as a deceit covering up the true magnitude of government spending and deficits, the Ford administration made no effort to limit ESF and other off-budget operations - a total of $9.2 billion for fiscal 1978 - until the day before it left office, when it called for more supervision off-budget items.

Until now, efforts to find out how ESF spends its administrative money have successfully been resisted by the argument that the Treasury Secretary must have complete latitude in the delicate field of international money matters.

Economist Edward M. Bernstrein, now a private consultant, who handled many secret ESF operations during World War II, told the House Budget Committee last year that the treasury needs the "flexibility" provided by the existence of the ESF.

For example, in the 1960s it was necesssry during one of the many gold crises to transport a number of gold bars from Fort Knox to the Bank of England to be sold on the London market.

The gold was transferred by Air Force planes and the expense was charged to the ESF. Extraordinary travel expenditures were incurred again in connection with the devaluation of the dollar in 1972.

Nevertheless, Bernstein testified, the ESF "as an agency of the U.S. government . . . should be subject to the same standards, the same scrutiny, as any other funds."

Congress appeared to be on the verge of recommending last year that ESF administrative expenses be made part of the regular budget process, but it failed to agree on the overall off-budget problem.

As an exmaple, of the total leeway to use ESF money, a House Budget Committee report cited a 1971 case when the ESF advanced money needed to set up the Cost of Living Council, established by President Nixon to control domestic wages and prices. The rationale was that the council had something to do with inflation, and inflation affected the value of the dollar. Subsequently, the money was repaid to the ESF.

In response to a request by Rep. Elizabeth Holtzman (D-N.Y.), former Under Secretary Edwin H. Yeo III (Solomon's predecessor) revealed that at various times the ESF had advanced money to the central Intelligence Agency, the State Department, the Army, the Air Force and other agencies.

The payments to the CIA - a total of $241,000 in 1974 and 1975 - were not for financing CIA activities, Yeo said, but covered salaries of CIA employees "on reimbursable detail." The only further explanation offered by Yeo was that the CIA formed part of Simon's staff analyzing "national security-related foreign programs."

There has been one limited audit of ESF administrative expenses, done by the General Accounting Office in November, 1975, at the request of Rep. Sam. M. Gibbons (D-Fla.). It detailed the wide range of operations - wholly comparable to other Treasury functions - funded by the ESF, but found no excesses.

In a telephone interview, the GAO specialist in charge of the report, Melvin Berngartt, pointed out that "we had only limited access to ESF. By law, we aren't given much of an insight."

Thus GAO simply was able to look at job descriptions supplied to it by the Treasury, Berngartt said. It had no way of making a determination, for example, whether payrolls were padded, or expenses excessive.

Meanwhile, it was learned that on its own initiative, the GAO has started a second audit of ESF administrative expenses, and given the current climate of the Carter administration favoring greater disclousure, thinks it may come up with more information.

At the Treasury, the matter is now in the hands of Secretary W. Michael Blumenthal, who is studying internal reports on the ESF commissioned by Solomon.

Solomon intends to protect the role of the ESF in exchange stabilization. "But there's no reason why administrative expenditure can't be handled in a more sophisticated way," he said.