The United States and Cuba are close to agreement on the return to Cuba of up to 2,500 criminals and mental patients who came to the United States in the 1980 Mariel boatlift, officials of the two governments said yesterday.

But both sides said a final resolution will depend on further discussions, which may take place in New York as early as Thursday.

A third round of U.S.-Cuban negotiations on this and related immigration questions was completed in New York last Wednesday. The State Department said the following day that "some progress was made" but that "another round of discussions is foreseen."

A New York Times report yesterday that "an agreement in principle" has been reached brought forth a variety of official and unofficial reactions from representatives of the two governments. By the end of the day State Department officials were saying they are hopeful an accord can be concluded speedily but that the next round of talks will be more than a formality.

"There will have to be more talks, another engagement between the two sides. However, we are making progress," Assistant Secretary of State Langhorne B. Motley said. He said the timing of an agreement is "wide open." A Cuban spokesman, Angel Pino, said "there will be a next round" in the negotiations and that "the intention of both parties is to reach an agreement."

The U.S. objective is to send back to Cuba those judged to be criminals and mental patients among the 125,000 Cubans who came to the United States from the port of Mariel in the 1980 boatlift. Most of these "excludables," as the unwanted group of about 2,500 persons is called, are in U.S. prisons or mental institutions.

In return, the United States has offered to resume the processing of visas for the emigration to the United States of political prisoners who were jailed for their beliefs, and their families. About 3,000 Cubans are expected to come here in this phase of the agreement, State Department sources said, adding that the exact number is among the things under discussion.

According to the Cuban diplomatic outpost here, which operates as a branch of the Czechoslovakian Embassy, Cuban President Fidel Castro said late last week that if an immigration accord can be reached, it will be proof that agreements are possible between the two countries "by peaceful means." Castro was quoted as having said that, "It is not possible to reach agreements through force."

Officials of both governments said any agreement at this point would be limited to immigration questions. State Department officials declined to speculate whether this could lead in time to broader accords of greater political importance.

Wayne Smith of Carnegie Endowment for International Peace, who previously was the senior U.S. diplomat in Havana, said Cuba began negotiations about the Mariel emigres with the Carter administration in 1980 and agreed to continue these talks with a "more flexible" position in February 1981. At this point the Reagan administration was not interested, he said, but two years later began diplomatic feelers about resolving the situation and formally asked the Cubans for talks last May.

Cuba initially considered the administration's interest a pre-presidential election gambit but agreed to resume the discussions this summer after the intercession of Democratic presidential hopeful Jesse L. Jackson, who visited Havana in June, according to Smith.

Jackson said yesterday that success in the immigration talks "should be a first step in improving relations" between the two countries. "These talks show we are able to sit down with the Cubans and work out our problems," he said.

The Reagan administration's immigration negotiations with Cuba so far were held July 12 to 13, July 31 to Aug. 2 and Nov. 28 to Dec. 5, all in New York City.