President Carter has ordered government agencies to develop a substantially larger U.S. foreign aid program for the next five years, even as Congress is demanding reductions instead.

Carter's order, which has launched an interagency study to be completed by Sept. 1, followed a promise to poorer nations by Secretary of State Cyrus R. Vance that the administration will seek "a substantial increase" in U.S. bilateral and multilateral aid progams over the next five years.

Both Carter's order and Vance's promise came before the House action last week to cut 5 per cent from the foreign aid appropriations bill. While suprised and dismayed by the strength of the anti-aid forces, administration officials said the vote underscores the need to improve effectiveness in a redesigned aid program to gain the support of Congress and the public.

In a preliminary step toward a "fundamental reorganization and restructuring" of the program, Agency for International Development Administrator John J. Gilligan recently issued a sweeping critique of his agency and ordered a cutback in paperwork and decentralization of decision-making.

Gilligan, in a message to senior AID officials here and all field missions abroad, said "the agency is strangled in paperwork, there is too much emphasis on Washington operations, and too much of the agency's funds are spent on studies and preparing to act and too little in actually delivering goods and services to the poor majority in the developing countries."

Elaborating in an interview, Gilligan criticized AID's top-heavy Washington bureaucracy as "overage, over rank, overpaid and over here." He said that despite dramatic overall reductions in the agency operations and personnel since their height during the Vietnam war, there has been little reduction in the agency's Washington staff. Beginning in 1974, AID had more Americans working full-time in its. Washington headquarters than in the rest of the world combined - and at the end of last year, about twice as many here as abroad.

Gilligan's June 14 message to AID announced a decentralization of decision-making from Washington to overseas missions on project development and operations, and the modification or elimination of several extensive internal reports. Gilligan said these changes will reduce the voluminous paper flow between Washington and the field by about 40 per cent.

One new internal study is "a top to bottom review of the personnel system" in the agency, Gilligan said. Another study seeks to review some 700 AID contracts with outside groups and individuals worth a total of $500 million. About 30 such contracts have so far been targeted for termination, he said.

As chairman of the interagency Development Coordination Committee, Gilligan will head the effort to develop a larger and move effective U.S. aid program under the directive from Carter. Gilligan said Carter and Vance are "deeply concerned" about the declining U.S. role in assistance to developing nations.

"The general assumption is that we are playing a very active if not dominant role in this area and it turns out we're not doing very much at all. We can hardly expect to call the tune without paying the piper," Gilligan said.

AID figures show that American economic assistance abroad, both bilateral and through the World Bank, International Monetary Fund and other multilateral agencies, has declined almost steadily as a percentage of U.S. gross national product. In dollar terms U.S. aid has grown hardly at all despite inflation, U.S. economic growth and the rising needs of poorer nations.

Last year the United States allocated $4.4 billion for economic aid abroad, about one-fourth of 1 per cent of its gross national product. At latest report, the U.S. ranked 12th among the world's aid donors in relation to GNP, behind Sweden, Netherlands, Norway, France, Australia, Belgium, Denmark, Canada, New Zealand, Germany and the United Kingdom.

No figure has been proposed for the "substantial" aid increase with Carter is considering. Among the other things, the study is expected to recommend new goals and justifications for U.S. economic aid, improved methods of coordination within the government and a better-planned "mix" of different types of aid programs.

To help run AID and devise a new program, Gilligan, a former Democratic governor of Ohio, has brought in a mixture of political figures and career officers:

Robert H. Nooter, a 15-year AID veteran and most recently its Near East chief, is AID deputy administrator, Gilligan's top assistant.

Ted Van Dyk, former assistant to Sen. Hubert H. Humphrey (D-Minn.) and campaign aide to several Democratic presidential nominees, is chief of a newly created coordination bureau.

Sander Levin, former Michigan state senator and unsuccessful gubernational candidate, is chief of the population and humanitarian assistance bureau.

Anne Clark Martindell, a former New Jersey state senator and Carter supporter, heads the foreign disaster relief office.

Arvonne Fraser, a figure in the women's movement and wife of Rep. Donald M. Fraser (D-Minn.), heads the women-in-development office.

Donald McDonald, once head of the massive AID program in Vietnam, is chief of program and management services.

The new chiefs of AID's geographic bureaus are: Africa, Goler T. Butcher, former counsel of the African subcommittee of the House International Relations Committee and a foreign policy adviser to Carter's presidential campaign; Asia, John H. Sullivan, formerly a senior staff official of the House International Relations Committee; Latin America, Abelardo L. Valdez, a Washington attorney and the first Latin heritage official to serve in this post; and Near East, Joseph C. Wheeler, a career official who was formerly AID mission director in Pakistan.

Because of civil service restrictions and past use of policymaking jobs as rewards for longtime civil servants, Gilligan said, he has been able to bring in only 24 new officials to give political and administration direction of an agency of 3,600 Americans workink full-time - 2,300 of them in Washington.