With a new wave of Cuban refugees headed for Key West, Fla., the Carter administration yesterday beefed up the security forces at Fort Chaffee, Ark., and pondered what to do with the refugees who were involved in rioting at the camp Sunday night.

White House press secretary Jody Powell said the number of Army troops on duty at Fort Chaffee was being more than tripled, from 600 to 2,000, and he predicted a substantial increase in the pace of the processing of the Cubans at the refugee center by the end of the week.

Presidential assistant Eugene Eidenberg was sent to Fort Chaffee with instructions from President Carter "to make sure there is no recurrence" of the rioting, during which three Cubans were shot, more than a dozen others injured and a number of buildings set on fire, Powell said.

Meanwhile, Coast Guard officials in Key West said they had confirmed that a flotilla of boats, including a freighter with as many as 850 Cubans aboard, was headed for Key West from Cuba's Mariel harbor.

[The freighter, Red Diamond V, which flies the Panamanian flag, was taken under escort by two Coast Guard cutters late yesterday and the skipper, crew members and any others aboard who might have chartered the vessel to pick up refugees in Cuba would be charged with importing illegal aliens.]

The vanguard of the flotilla, a pleasure craft with 78 people aboard, arrived in Key West late yesterday.

Officials had no firm estimates of the number of new refugees on the boats, but guessed that it could be between 20,000 and 30,000.

The State Department, however, said it had no confirmation of reports that Cuban President Fidel Castro had ordered all U.S. boats out of Mariel, thus effectively ending the boatlift that has brought almost 95,000 Cubans illegally to the United States in the last six weeks.

Powell said the military personnel at Fort Chaffee "have been given instructions to use reasonable, nonlethal force to make sure these legal detainess [the refugees] do not leave the camp."

"These people are being legally detained, and when they leave the camp without authorization they are in violation of the law," he said.

However, neither Powell nor other administration officials could say what action, if any, will be taken against about 50 Cubans described as "ringleaders" of the Sunday night uprising who are now being held by the military separate from the other refugees.

Officials said the unrest at the camp appeared to revolve around about 300 refugees, mostly young men, who either actively encouraged the rioting or were susceptible to agitation. They said there was no evidence that any of these are Cuban agents planted among the refugees to stir up trouble, but that the Justice Department was investigating the causes of the violent outbreak.

A decision on actions to be taken against the refugees involved in the rioting will probably not be made until completion of the investigation, a Justice Department spokesman said.

Part of the problem, the officials said, appears to be the emotional strain placed on the refugees as they wait to be processed and released from Fort Chaffee for resettlement elsewhere in the country.

The officials also stressed that of about 19,000 refugees at Fort Chaffee, the vast majority took no part in the rioting, and that many intervened in an attempt to quell it. Some donned white armbands as a sign to law enforcement personnel that they were trying to restore order.

Powell said the Fort Chaffee facility had been processing about 100 refugees a day, a pace that at least some of the Cubans found unsatisfactory. Be he predicted that the camp will be processing from 300 to 500 a day by the end of this week or early next week as obstacles in setting up the processing system are overcome.

A certain defensive crept into the statements of administration officials yesterday in defending the U.S. response to the refugee tide.

"Given the circumstances, we think it's been handled admirably," said State Department spokesman noted that in the 1970s, the average stay for Indo-chinese refugees at relocation centers was 100 days, while for the Cubans it has been 10 days. He also said that about 45,000 Cubans -- roughly half the refugee total -- have been resettled at least temporarily since April 21.

Powell said a large numer of the Fort Chaffee refugees do not have relatives in the United States, which has slowed the resettlement process for them.

Another official said the administration is caught in a "damned if they do, damned if they don't" situation. In the early days of the boatlift, he said, the processing of refugees at Key West was loose, drawing criticism from private agencies concerned with the resettlement of the Cubans.

Now, the official said, some of the same agencies are complaining that the more tightly controlled processing system set up at Fort Chaffee and other refugee centers is too slow.