After a decade of war and two years of mutually hostile diplomacy, the United States and Vietnam opened formal talks here today on establishing diplomatic and trade relations.

Details of the 3 1/2 hour opening session were withheld by the two high-ranking delegations, but there were signs that each went into the meeting with hopes of rapid progress on some outstanding issues.

In unpublicized preliminary contacts here, the Carter administration agreed to drop the once-firm American insistence on a complete accounting from Hanoi of all missing American servicemen as a precondition for talks on normalizing diplomatic relations according to diplomatic sources. The Vietnamese were also said to have softened their demands for the economic aid they say President Nixon and Henry Kissinger promised Hanoi in 1973. Previously, Hanoi has made discussion of aid a precondition to holding talks.

Both issues will be raised, and even if they are not fully resolved the delegations are believed to be discussing the establishment of a diplomatic liaison mission as a step toward full ties. Washington and Peking already have exchanged such missions.

A pullback from previous hardline positions on both issues was forecast in the report of a March mission to Hanoi led by Leonard Woodcock, president of the United Auto Workers union. The mission received the remains of 12 U.S. pilots shot down during the war.

The commission, which Carter praised for a "superb" mission, said there is "no evidence to indicate that any American prisoners of war from the Indochina conflict remain alive."

A similar conclusion was reached in December by Rep. G.V. (Sonny) Montgomery (D-Miss.), chairman of the Select Committee on Missing Persons in Southeast Asia. Montgomery was a member of the Woodcock mission.

The report characterized the Vietnamese as being "prepared to deemphasize references to . . . aid as coming from U.S. obligations under the Paris agreement," although the report added: "It remains to be seen how forthcoming the Vietnamese may be in accounting for the MIAs if the U.S. does not take some steps on aid."

Assistant Secretary of State for Asian and Pacific Affairs Richard Holbrooke said after today's talks that the "discussion was frank, friendly and very useful and we will meet again tomorrow morning."

Today's meeting was at the Vietnamese embassy. The talks are to alternate between the two embassies here.

Vietnam's deputy foreign minister, Phan Hien, exchanged handshakes and smiles with Holbrooke at the end of the session.

Comments last week by Prime Minister Phan Van Dong during an official visit to France suggested that the change of administrations in Washington had encouraged the Vietnamese to hope for better relations and eventual economic aid.

"I believe positive results are possible given good will on both sides," he said. He emphasized that Vietnam wants more trade with France and the United States, and said that his country is prepared to grant offshore oil-exploration rights to American companies.

The delegations were clearly composed with an eye toward trying to remove substantive and symbolic legacies of the war in Indochina and the diplomatic stalemate that has followed.

The presence of Frank Sieverts, the State Department's director of humanitarian affairs, on the U.S. delegation indicated that the fate of the 800 American servicemen the Pentagon still lists as missing will be a key item on the agenda.

The Vietnamese are known to feel that they can offer little more than symbolic help on this issue.

The dropping of the U.S. veto of Hanoi's proposed membership in the United Nations and the lifting of the trade embargo are two intermediate steps Vietnam is seeking here.

Hanoi has previously demanded full payment of $3.25 billion in reconstruction aid allegedly promised by the Nixon administration before Communist forces occupied Saigon two years ago and reunified Vietnam under Hanoi's control.