A rugby match here between an all-white team from South Africa and a Washington group was abruptly canceled yesterday by District recreation officials, who said they became concerned about possible violence after they received several telephone calls from people who expressed outrage over the scheduled contest.

Protests about the match, scheduled for a playing field near the Washington Monument, came from several people and groups, including Transafrica, a group that promotes African interests in this country; the Central Labor Council and Rep. Ronald Dellums (D-Calif.), chairman of the House District Committee.

The protesters complained that the South African team should not be allowed to play here because of that country's apartheid policy, as well as the symbolism of the team's playing in the shadow of the Washington Monument and Lincoln Memorial.

"This is an overwhelmingly black city," said Dellums' spokesman Max Miller. "Obviously, people are going to be upset."

Although the 3 p.m. game was not announced to the general public, word of the match circulated among area rugby groups yesterday, touching off the protests to city recreation officials, the mayor's office and Congress.

"I suppose from a prudent point of view we have done something kind of stupid," said Fred Byrne, president of the local Potomac Rugby Union, a group of about 35 rugby teams in the Washington-Baltimore area that invited the Durban Collegians to play an all-star team here as part of their United States tour. The Collegians are marking their 75th anniversary this year.

Byrne said the Potomac group had avoided publicizing the match, arranged eight months ago, because of protests earlier this summer when the Springboks, South Africa's national team, played three games in the United States.

However, Byrne said the Durban team, which represents an integrated private club in South Africa, was not associated with the South African government.

Before the game was canceled, the rugby group asked the mayor's office for police protection after it learned of the protests and a planned demonstration at the field.

Several players and a group of about 20 persons, including six protesters, showed up at the field at 15th Street and Independence Avenue SW, but dispersed without incident after U.S. Park Police said the game had been canceled.

Robert J. Fleming, administrative officer of the city's recreation department, said the people who telephoned protests "were in an uproar."

He said the city routinely grants permits to the Potomac teams that play on land owned by the National Park Service but obtain their permits though the city governemnt.

Fleming, who said several international teams have played in Washington, said the Potomac group routinely asked for its permit without identifying its opponent. "I would not have allowed it," Fleming said. "I know there have been problems in the past. They should not be created here."

Byrne, who said he was surprised by the controversy, said another South African team, the all-black Mbabalas, played here last spring without incident and were guests at the South African embassy.

An official of the embassy who asked not to be identified said his government had no connection with the Durban team, but he was obviously miffed at the controversy.

"The D.C. government should also decide to stop the Soviet Union's Bolshoi ballet at the Kennedy Center," he said. "They should apply their principles throughout."

Byrne said the Durban team is a privately financed group of players that has several all-star players on its roster. It is only one of several clubs in South Africa that play in national associations and compete in world tours.

Bryne said the team has been in the country several weeks, playing without incident in Dallas and the Chicago area.

Officials of the Durban team could not be reached for comment. One rugby official said the team has a game scheduled later this week in Philadelphia before ending its tour.