The United States and Cuba quarreled publicly yesterday over the Cuban buildup in Africa. Fidel Castro in effect told President Carter, who had sent him a message urging the withdrawal of Cuban troops from Africa, that it was none of his business.

"It has nothing to do with Carter, it has nothing to do with the United States," Castro told a group of American reporters in Havana. "Our relations with Africa - that we can't discuss, that we can't negotiate."

Castro also accused President Carter's national security adviser, Zbigniew Brezezinski, of creating an unnecessary problem by releasing U.S. intelligence reports three weeks ago on the Cuban buildup.

"Why did Brzezinski magnify the problem," Castro exclaimed. "Why did he do that? That cannot become an issue.

While obviously reluctant to get into a public debate with Castro, the Carter administration made it clear yesterday that it considered the presence of 27,000 Cuban troops in Africa very much an issue.

Hours after Castro spoke, U.N. Ambassador Andrew Young renewed his weekend attack on Cuba's "new colonialism" in Africa in an unusually sharp address to the U.N. General Assembly's political committee.

"The presence in Africa of nearly a quarter of Cuba's armed forces and the interjection of Cuban military advisers in troubled areas throughout the continent can only lead to more deaths and suffering," Young declared.

The recent tough statements by Young - who earlier in the year said he felt Cuban troops had brought "stability" to Angola - reflect a growing feeling in the Carter administration that Castro has misled the United States about his intentions in Africa.

Far from following through on his promises to reduce his forces, Castro has increased Cuban personnel in Angola by 20 per cent - and has also established a worrisome military presence in Ethiopia.

"The President feels that Castro has been something less than straight on this issue," a top Administration official said yesterday.

Administration sources also said there was every indication that the buildup of Cuban forces in Africa, which Brzezinski reported last month had reached 27,000 military personnel and civilian advisers, was continuing.

"There has been an increase in the past three weeks," an administration official said. Added National Security Council spokesman Jerrold Schecter, "There has been no indication that the pattern has changed."

So last week, when two Congressmen who were about to leave for Havana, Reps. Frederick W. Richmond (D-N.Y.) and Richard Nolan (D-Minn.) met with President Carter, he asked them to tell the Cuban leader to "get out of Africa."

In a 3 1/2-hour meeting with Castro, the two Congressmen also conveyed Carter's warning that the Cuban presence in Africa is an impediment to restoration of normal trade and diplomatic relations between Washington and Havana.

But Castro, whose controlled press has repeatedly stated in recent weeks that Cuba's involvement in Africa was of no concern to the United States, rejected the Administration's representations yesterday.

"I cannot understand these people who want to create a problem artifically," Castro declared. "If it becomes an issue, it's going to be an impediment.

The Carter administration emphasized yesterday that the Cuban troop question is an issue, they regard it as one of legitimate - and serious - concern to the United States.

"Obviously, we feel peace and stability in Africa is a matter of concern to us as well as to other Africa nations, and the presence of Cuban troops does not contribute to that," Schecter said.

While the Cuban presence in Africa is largest in Angola, where 19,000 troops are helping the Marxist government battle insurgents, one State Department source indicated yesterday that the U.S. was also extremely concerned over the introduction of Cuban troops of Ethiopia.

More than 400 Cuban military advisers are now aiding the Ethiopian Army in its two-front war with Eritrean rebels and Somali reguars.

"Are the Cubans going to become as involved in Ethiopia as they are in Angola?" asked the State Department official. "It leads to all sorts of disturbing questions."

Despite the controversy over the Cuban troops, Castro told the visiting American reporters yesterday that he felt the climate between Washington and Havana had improved since President Carter took office.

"I think we've worked for peace," he said. "But there have been complicated problems for 18 years, and cannot be jumped into."

Castro also declared that he would have "no objection" to a face-to-face meeting with President Carter, but said he did not expect one to take place during the President's first term.

"I think Carter should be free to set a date that would be satisfactory," Castro said. "I don't want to put pressure on him by speculation."

Richmond and Nola, during their meeting with Castro, also expressed Carter's concern over the five remaining American political prisoners held in Cuba. The two asked Castro to release the five by Christmas as a gesture toward improved relations.

One of the five, Lawrence Lunt, 54, a former Wyoming rancher, confessed to the Congressmen that he was a CIA contact on the island at the time of his arrest. Another, Frank Emmich, 63, of Toledo, Ohio, denied charges that he was the CIA station chief in Havana, but admitted smuggling out a letter in 1972 or 1973 to then CIA director Richard Helms.

Castro, the target of several assassination attempts linked to the CIA, made no commitment to release the five Americans.

"It's not easy for me," he told reporters. "We promised to analyze that problem. We didn't make a commitment to solve it."