Top leaders of American business and labor, in a rare joint appeal, called yesterday on the Carter administration and Congress to take "strong and immediate" action to counteract a "stunning increase" in textile, apparel and fiber imports.

One of their main proposals called for legislation to exempt textile products from tariff-cutting negotiations now under way in Geneva.

Such an exemption had previously been rejected by administration trade negotiators on grounds it could jeopardize the entire round of talks aimed at reducing world trade barriers.

The appeal came in the form of an elaborately staged press conference in the Caucus Room of the Cannon House Office Building, featuring AFL-CIO President George Meany, Du Pont Co. Chairman Irving S. Shapiro, who is also head of the powerful Business Roundtable of top corporate executives, and representatives of both labor and management in the textile and apparel industry.

Only a week ago, the two sides were at each other's throats over the administration's labor law revision bill, which was backed by labor and opposed by business. But the prospective loss of more production, jobs and profits in the textile industry brought them together yesterday in the manner of old friends and natural allies. At one point Meany told a Wall Street joke and several of the businessmen smiled politely. Shapiro appeared to laugh a little.

In their statements, both Meany and Shapiro described the textile imports situation as a deepening crisis requiring forceful action. "The U.S. cannot allow any industry . . . to be destroyed by unfair trade competition," said Meany. "A change in this disastrous import trend must be made and it must be made immediately," said Shapiro.

Meany said the American textile and apparel industry has lost more than 350,000 jobs over the last decade and is currently losing jobs at the rate of 20,000 a year.

Shapiro said textile imports rose last year to nearly $6 billion, with imports for the first four months of 1978 nearly double the level for the same period last year. The textile industry trade deficit of $3.5 billion for 1977 accounted for about 10 percent of the nation's total trade deficit, he said.

Companies like DuPont will be hard-pressed to invest in modernization, expansion or research and development "in the face of continued signals from our government that the domestic textile and apparel industry is being allowed to wither away," Shapiro added.

Both labor and management in the textile industry have been quietly pushing the White House for months to remove textiles from the bargaining table in Geneva, where long-stalled international trade talks were revived last year and are expected to result in some preliminary agreements next month. Their argument has been that shoes, stainless steel, color television sets and other import-threatened domestic industries have been exempted on grounds that lower tariffs would only increase imports, and textiles should get equal treatment.

Having failed at the White House, they're now turned to Congress, where legislation to exempt textiles has been introduced in both houses - with 26 co-sponsors in the Senate and 168 in the House. Hearings are scheduled for next month on the House bill.

Other steps urged by the labor-management group include tighter administration of present trade agreements, negotiations of a quota agreement to curb rising imports from the People's Republic of China, renegotiations of some existing agreements with other Asian countries and development of a new trade program that would freeze import growth rates.

The Amalgamated Clothing and Textile Workers Union, one of the participants, also announced it filed charges yesterday accusing Malaysia, Mexico, Pakistan, Singapore and Thailand of illegally subsidizing exports. The Treasury Department, in a preliminary ruling, recently upheld similar union charges against seven other countries.