Heads of the nine nations in the European Economic Community ended two days of tense and occasionally acrimonious negotiations here tonight with a decision to create a European monetary system to start up on Jan. 1.

But the European summit meeting -- termed "a limited success" by its chairman, West German Chancellor Helmut Schmidt, -- failed to persuade the leaders of Britain, Italy and Ireland to allow their currencies to join other EEC moneys in a broader exchange rate agreement which would have involved all nine currencies.

Described by top EEC aide Roy Jenkins as "the central mechanism" of the new European system, the exchange rate accord creates a "zone of monetary stability" as a response to dollar instability and as a platform for renewed economic growth in Western Europe.

The key element of the new exchange rate mechanism will be the restriction of fluctuation margins between major European currencies.

As of next year, the French franc will join the five member currencies of the current European "snake" -- the German mark, Danish crown and the Benelux currencies -- in a float against the dollar which will restrict their maximum movements against each other to 2.25 percent on either side of parity.

These margins will be protected by "compulsory" interventions by the central banks of the participating countries, according to the text of the agreement reached tonight. In addition, a credit worth some $32 billion will be available to central banks to enable them to provide currency support as and when necessary. Central banks will receive this support in European currency units. This new currency unit, the ECU, will be issued in return for the deposit of 20 percent of their gold and dollar reserves.

The ECU will be used "as a means of settlement between monetary authorities of the European Community," says the text issued by Schmidt late tonight.

As monetary experts had predicted, the British pound will not join the exchange rate agreement for the moment.

Prime Minister James Callaghan's stance is seen here as reflecting Britain's fears that linking sterling to a mark-dominated money plan would put excessive restrictions on his government's growth policy.

Italy and Ireland, which with Britain have the EEC's weaker economies, also refused for the time being to accept the economic risks of too close a link with the highly disiplined German economy.

But Callaghan may bring sterling fully into the European plan if he is victorious in next year's elections, according to observers here. They point to left wing opposition to the European monetary system in Britain as an important determinant of the London government's position on currency links with Europe.

However, the British prime minister complimented French President Valery Giscard d'Estaing and Schmidt, the prime movers of the new monetary systems, on their "very bold effort."

The British leader also appeared to feel that the prospects for European monetary stability had improved with President Carter's action on the dollar.