The General Agreement on Tariffs and Trade (GATT) agreed today to begin talks on a new round of world trade negotiations after the United States threatened to walk out unless the booming service sector, such as banking, was included.

"In a spirit of compromise," Brazil and India, the two main opponents of the inclusion of services in a new round, said they would agree to an agenda that permitted discussion of any subject that any member wanted to raise during the three-day special session.

Deputy U.S. Trade Representative Michael B. Smith, head of the American delegation to the three-day meeting, hailed this as a victory.

"We wanted an agenda which permitted anyone to raise any issue they wanted, and we got one," he said. "That means we can discuss services and we certainly intend to do so."

Other GATT sources, however, pointed out that by accepting an agenda watered-down from their original demands, Third World countries could say that they were not responsible for the final outcome of the meeting.

"They could say that they were prepared to accept discussion of services in the interests of harmony, but at the same time they could say they were not responsible for the outcome of the talks, whatever that will be," a GATT source said.

The special session of the members of GATT, the first in the organization's history, was called at the insistence of the United States after the regular consensus procedure had failed to produce agreement on a U.S. demand that services be included in the new round.

The United States asked for a vote of all the organization's 90 members; 65 countries lined up with the United States to support the special meeting that began today.

The United States is looking for an agreement at the current meeting that a working party be set up to discuss how the new round will be conducted.

"We have 60 days to get this show on the road," Smith told reporters in a reference to the regular November meeting of GATT members, which will have to take a formal decision on a new round next year. "If we don't have an agreement by then, then we're going to look at the whole trade situation more closely."

This was a reference to concerns that if GATT does not agree on an agenda for the new round that includes services, protectionist pressures in Congress and elsewhere in the United States will make it impossible for the country to play a meaningful role in the new round.

Felipe Jaramillo of Colombia, chairman of GATT for 1985, said he was optimistic that the current meeting would produce some kind of formula that would enable the working party on a new round to be formed.

"The question is, what is this working party going to discuss?" he said. "The Americans want services, the Brazilians and Indians don't. We shall see some tough talking before this issue is resolved."