West European leaders took a series of symbolic steps early this morning to reform their cumbersome joint institutions with the ultimate aim of competing more effectively with the United States and Japan.

The reform package, agreed at a two-day summit of 10 heads of government, includes the first revisions of the 1957 Treaty of Rome, which set up the European Community. But political analysts here said it fell short of a charter for the "relaunching of Europe" demanded by some European leaders following a lengthy period of economic stagnation and political squabbling.

The most important change was the encouragement of a system of majority voting in areas where individual states do not consider that their vital national interests are at stake. For the past two decades, decisions have been made by consensus of all EC countries, a system that has resulted in frequent political stalemates.

After the talks broke up shortly after midnight, British Prime Minister Margaret Thatcher described the results as a "modest step forward" for the European Community.

French President Francois Mitterrand said that France would have preferred more radical reforms, but accepted the outcome as a compromise.

The summit was politically important because it was designed to pave the way for the admission of Spain and Portugal as new members of the European Community from Jan. 1, 1986. Practically all European leaders are agreed in principle that institutions and treaties created by the six original states 28 years ago are no longer adequate to the needs of a much larger and more heterogeneous community.

The two days of difficult debates served to underline the obstacles involved in fulfilling the EC's target of creating a unified internal market, equivalent in size to that of the United States, by 1992. While declaring their readiness for greater European cooperation, individual leaders also resisted measures that would have resulted in a significant reduction of national sovereignty.

The combination of pious hopes and modest achievements that characterized the meeting was captured by an adviser to Thatcher who quoted Saint Augustine: "Oh God, make me pure . . . but not yet."

Both Britain and Ireland cited the need to protect their island populations from continental diseases such as rabies as justification for opposing a call for the eventual dismantling of frontier posts and customs barriers. Denmark insisted on its sovereign right to set stringent environmental standards in factories. France wanted special rules designed to keep out cheap charter flights and protect its insurance and banking system.

Under a compromise, EC member states will be allowed to continue to insist on exemptions from legislation enforcing a united market in order to protect their "working environment" or "natural environment." But it will be up to the European Commission, the EC's executive arm, to rule on whether the individual requests are justified.

The demands for national exemptions to a united market are important because, considered cumulatively, they represent a major obstacle to economic growth. European Community economists have pointed to the lack of a genuine common market 28 years after the signing of the Treaty of Rome as one of the main reasons for Western Europe's relative economic decline.

A document released this week by the European Commission estimated that customs formalities alone were costing European firms about $10 billion a year. It estimated that genuine Europe-wide competition for public contracts could save European taxpayers about $35 billion.

The need to face up to the technological challenge of U.S. and Japanese multinational companies was a major theme of a speech by Mitterrand, who warned other EC leaders that "if we do not succeed in unifying our market, others will do it for us."

The EC leaders reached agreement on a separate treaty on political cooperation that codifies an informal system of consultation on important international issues. The treaty envisages the creation of a small secretariat in Brussels to coordinate foreign policies and attempt to reach a common position on votes in the United Nations.