The first shadowy outlines of the secret diplomatic negotiations on Soviet troops in Cuba emerged yesterday in guarded comments of informed congressional and administration officials. They expressed optimism about a mutually acceptable settlement.

Following a third round of talks between Secretary of State Cyrus R. Vance and Soviet Ambassador Anatoliy F. Dobrynin at the State Department, officials still refused to set a target date for a conclusion. Vance said another meeting with Dobrynin is likely Sunday or Monday, and diplomatic sources reported that Soviet Foreign Minister Andrei Gromkyo is expected to arrive early next week in New York. This could bring a change in the tempo and perhaps the level of the negotiations.

"The Vance-Dobrynin meetings are progressing well," said Senate Democratic Leader Robert C. Byrd (W.Va.), one of the handful of congressional leaders who has been briefed by the tightlipped secretary of state about his sensitive negotiations.

"We are getting a quick turnaround (from Moscow) on the information we are seeking. The Soviets are being responsive," he said.

With the future of the strategic arms limitation treaty (SALT II) in the balance, Byrd said he has been cautioning undecided senators against taking positions based on the Soviet troops issue until the current negotiations are completed.

An administration official familiar with details of the negotiations said the United States has not made a formal proposal to the Russians at this stage about the presence in Cuba of what the United States describes as a Soviet combat brigade.

The official, who asked not to be quoted by name, said President Carter and his administration have agreed on "some general notions about what would be a mutually satisfactory outcome." The official implicitly conceded that the final resolution well may be short of the U.S. maximum objective, which is verified withdrawal of the Soviet force, by saying that "the clearer the resolution, the more acceptable the resolution will be and the better it will be for U.S.-Soviet relations."

In addition to holding out a promise of better superpower relations if the impediment of the troops issue is removed, the Carter administration is also brandishing a stick of potential trouble for the Soviets in their border regions if the issue remains unsolved.

Referring to "the principle of reciprocity in international relations," the administration official noted that Soviet combat troops in Cuba impinge. A country as large as the Soviet Union also has special sensitivities, he said, in an apparent reference to historic Soviet concern about its long borders with Europe on one side and China on the other. on an area of special U.S. sensitivity.

"The United States has shown respect for their concerns. We've generally shown ourselves sensitive to them. We would like to continue respect for their concerns," the official said.

He added that failure to resolve the issue of Soviet troops in Cuba could bring "certain negative consequences not of our desire." He declined to specify what he had in mind, or how explicitly the Soviets have been warned of such possibilities.

It was clear that he was speaking of troubles beyond the effect on Senate approval of SALT.

At the White House, the State Department and on Capitol Hill there is common agreement that the effort to find a solution to the troops issue through quiet negotiations with Moscow is a high-stakes operation fraught with difficulty. While nobody is flatly predicting success, the course of the Vance-Dobrynin talks as they develop has generated notably greater optimism than at their start on Monday.

Despite the unusual secrecy imposed at the insistence of Vance, reporters have been reliably told that:

The current negotiations deal with the limited issue of the Soviet brigade in Cuba and will not undertake a solution to the broader problem of Soviet-Cuban military activities in Africa and elsewhere.

Despite intensive and continuing study by intelligence agencies, the United States still does not know precisely what the Soviet brigade's mission is. Intelligence analysts are reported to be "swamped" by information still needing study.

It is considered possible that the Soviet brigade has been engaged in training Cubans from time to time. But in the oficial U.S. view, this is not a crucial question. The issue, as seen by an informed official, is not what functions the brigade may have performed "but its combat capability" as an organized unit with a headquarters, combat arms and combat equipment.