U.S. Trade Representative Clayton Yeutter yesterday criticized Third World nations trying to block a new round of global trade talks and said the Reagan administration would go ahead next year with its own meeting outside of the General Agreement on Tariffs and Trade (GATT) if they succeed.

"GATT is in jeopardy," Yeutter told the Senate Finance Committee.

Later, however, Yeutter told reporters it was "most unlikely" that the United States would pull out of the 90-nation Geneva-based agency that regulates world trade.

"We simply can't afford to have a handful of countries, responsible for 5 percent of world trade, dictate the destiny of a large number of countries who deal with 95 percent of that trade," a clearly angry Yeutter told the committee.

He also threatened "a strong and harsh response by the United States and other trading nations" against the lesser-developed countries, headed by Brazil and India, who are trying to keep the annual GATT meeting in two weeks from considering a report supporting the new trade talks.

Yeutter's outburst was precipitated, he said, by a cable he read on the way to Capitol Hill detailing how "several countries blocked an official report" of a meeting of senior officials last month from being placed before GATT's annual meeting in Geneva over the Thanksgiving weekend. This report showed 65 GATT members favored the new round of talks.

Meanwhile, yesterday in Geneva, negotiations that have gone on for 4 1/2 weeks in an effort to reach agreement on an agenda for a new round of trade talks ended in failure when a committee of GATT member nations was unable to agree on a communique' about what they had been talking about.

The failure of the talks of "senior officials" of member nations threw the ball back into the court of the GATT annual meeting. The United States was not displeased with the failure.

"We always thought it was going to be difficult to negotiate in this forum, and it was," said Peter Murphy, U.S. ambassador to GATT. "Now let's get it all out in the open Nov. 25 and see who is prepared to talk seriously."

A new round of talks to expand the authority of GATT has been a cornerstone of Reagan administration trade policy for almost two years. Opposition by France to President Reagan's efforts to set a date for the global negotiations disrupted the unity of May's economic summit in Bonn.

The administration wants the talks to focus on ending restrictions to trade in services such as banking, engineering and investments, as well as farm products and high technology; improving copyright and patent protection to stop the piracy of intellectual property, and strengthening GATT's procedures for settling trade disputes. Services, the fastest-growing sector of the U.S. economy, is the administration's top priority.

Yeutter told both the Finance Committee and an earlier breakfast meeting of the National Association of Manufacturers that his 4 1/2 months as Reagan's point man on trade has underscored the need for the talks to start as soon as possible. "I'd like to begin them tomorrow," Yeutter said.

Third World nations, however, fear that the new round will cost them a chance to get a foothold in the emerging service industries. Sources in New Delhi said this summer that India believes protectionism that blocks Third World sales to the industrialized countries should be ended before GATT takes on new areas such as services.

Yeutter, though, said the Third World nations want U.S. concessions on their exports to America without ending their restrictions on U.S. service industries. "They want to have their cake and eat it too," he said.

He listed the major nations trying to block the new round as Brazil, India, Yugoslavia, Nigeria and Argentina.

He added the Reagan administration was considering taking retaliatory measures against these countries on a bilateral basis, including restricting their ability to ship some manufactured goods to this country without duty under the Generalized System of Preferences (GSP).

"One must wonder," Yeutter told the Senate committee, "how generous the U.S. should be with GSP with nations who are trying to torpedo the international system."

Sen. Daniel Patrick Moynihan (D-N.Y.) said Yeutter's comments indicated that "the GATT arrangement may be breaking down and the United States might have to consider a new regime in international trade all together."

"You heard me right," replied Yeutter. "We are prepared to use alternative ways of bringing about a more free and open trading system."

Heading the list is a meeting in Washington next year of "like-minded nations" that agree on the need for a new round of trade talks. Countries that don't participate, Yeutter said, "would not enjoy any of the benefits of such international agreements."