Eleven Western European countries reached agreement today on a common but limited package of punitive measures against South Africa's apartheid regime, but Britain withheld its approval pending an assessment of the impact such action could have on its extensive ties with Pretoria.

Nine of the 10 European Community states, plus incoming members Spain and Portugal, set forth a joint policy calling for a "rigorously controlled" arms embargo along with a ban on all nuclear and military cooperation with South Africa -- measures that many of the countries already had imposed individually.

The limited advance represented by the adoption of a joint accord, and the abstention of Britain, which has wide-ranging economic and other interests in South Africa, reflected persisting divisions among European countries with varying interests that have thwarted the coordination of a more united western policy designed to maximize influence with the South African government and to hasten change of the apartheid system.

A three-man EC delegation, consisting of foreign ministers Jacques Poos of Luxembourg, Hans van den Broeck of the Netherlands and Giulio Andreotti of Italy, visited South Africa earlier this month on a mission to find ways to harmonize European views.

The three-member delegation has also sought to broaden a dialogue with South African black leaders. Poos met today with representatives from the African National Congress, the first high-level meeting that ANC figures have held with EC officials.

The ministers' report formed the basis of discussion today, along with an accompanying list of measures for common approval that were drawn from existing policies of individual EC member states toward South Africa, including various punitive measures.

Even though the delegation recommended no dramatic new actions, the list of proposed measures for communitywide action provoked a heated dispute between Britain and other European countries with looser ties to South Africa.

France and Denmark argued for a strong package that would include suspension of new investments in South Africa, banning the sale of krugerrand gold coins, and visa requirements for all South Africans visiting Europe.

But those steps were rebuffed by Britain and West Germany, which saw them either as damaging to their own national interest or to the welfare of South African blacks.

At one point, Britain sought to adjourn the meeting to allow more time to consider a program of common action. But other states insisted that Europe had to take a stand in the wake of President Reagan's announcement Monday that the United States would undertake several restrictive measures, including a ban on sales of nuclear technology.

Several delegates said there was evident chagrin among left-of-center European governments that felt the community had been left in the lurch by the latest U.S. action.

Secretary of State George P. Shultz sent letters Monday to the allied foreign ministers explaining the U.S. steps and exhorting the allies to act in concert so a unified western approach could be established. Shultz's message reportedly irritated West German Foreign Minister Hans-Dietrich Genscher, who complained about the shifting nature of U.S. policy requisites, according to officials in the delegation.

Shultz sent letters this summer to the Europeans advocating the avoidance of any measure that would surpass Washington, yet now he seemed to be shifting gears and urging quick action, the official said.

All of the participating European states endorsed a political statement urging South Africa to lift the state of emergency and to repeal all forms of apartheid. The statement also called for the release of black leader Nelson Mandela and other political prisoners, as well as the abolition of forced relocation and detention without trial.

As for the arms embargo, the United Nations imposed one eight years ago that nations selling arms have professed to enforce although there have been occasional charges of violation.

The limited economic and diplomatic steps taken today after nine hours of wrangling among the European foreign ministers barely amounted to a basic consensus among the states. Several ministers admitted after the session that "nothing new" was in the package.

The debates, at times bitter, focused on how to reconcile disparate economic interests with the common desire to exert more pressure for the dismantling of apartheid.

France, Denmark and the Netherlands have repeatedly urged other EC members to embrace more serious economic sanctions such as curbs of investments and trade. But Britain and West Germany, with their economic interests, have opposed such moves.

More than 300 West German firms are active in South Africa, a country that provides most of the strategic raw materials for West German industry. A loss of chromium supplies from South Africa for two years could lead to the elimination of 700,000 jobs in the machinery sector, according to a Bonn Economics Ministry estimate.

In addition, Frankfurt banks have lent more than $2 billion to South Africa and are worried about prospects for repayment of $850 million due in the next year.

Even left-wing opposition groups, such as Britain's Labor Party and West Germany's Social Democrats, have expressed caution about the repercussions of sanctions and the risks of losing many jobs in the domestic economy.

As a result, conservative governments in London and Bonn have faced little political pressure at home to yield to the growing international outcry for tougher economic sanctions to press change in South Africa.

Elsewhere in Europe, the South Africa issue has not achieved the emotional intensity felt in the United States. European governments pushing hardest for mandatory sanctions also have the smallest economic stakes in South Africa.

Britain's minister of state for foreign affairs, Malcolm Rifkind, said today that unlike most states with few ties to Pretoria, Britain "could not afford the luxury" of drastically curtailing economic links with South Africa to underscore anti-apartheid principles.

Apart from economic restrictions, Britain objected to the recall of military attaches from South Africa and the "discouraging" of cultural and scientific relations adopted by other European states.

Rifkind said military attaches provide a key channel of information with the South African Army, which represents "a vitally important force" subject to influence through European counterparts.

Despite its opposition to such restrictive measures, Britain agreed to join its European partners in supporting "positive" aid programs that would support blacks.