Leaders of the seven nations attending the economic summit conference here vowed today to fight terrorism "relentlessly and without compromise" and singled out Libya for action. But they left the politically sensitive question of economic sanctions to be considered privately by each country.

On the first working day of the 12th annual economic summit meeting, the leaders also approved a statement criticizing Soviet handling of the Chernobyl nuclear power accident. They began to focus on proposals -- including a major U.S.-offered plan to reform the international monetary system -- for closer economic cooperation among the industrialized democracies.

Decrying the "blatant and cynical use" of terrorism "as an instrument of government policy," the leaders approved a statement that went somewhat beyond measures backed last month by Western European foreign ministers. It also was the first time Japan had agreed to a document naming Libya as a source of state-sponsored terrorism.

While repeating familiar themes, the statement also promised more forceful actions against terrorism than were approved by the allies following the 1984 London summit. A senior U.S. official said that while the measures were "not spectacular," the "strength of the wording" and "the evolution which we expect to take place" by individual countries' actions will contribute to Libya's problems.

For its part, the United States apparently has indicated in private discussions here that it will force U.S. oil companies still in Libya to leave.

The measures agreed on today include improved extradition procedures for bringing to trial those accused of terrorist acts and more restrictive rules of entry for persons suspected or accused of terrorist acts. Other measures include restrictions on arms sales to nations sponsoring terrorists, limits on their diplomatic missions and personnel and closer cooperation between law enforcement authorities.

Privately, according to an informed U.S. official, some allies have agreed to take other steps as well, including monitoring of a wide variety of Libyan financial transactions to try to trace the source of money for terrorist operations, monitoring or closing down Islamic Call Societies, which channel Libyan funds to a wide range of projects worldwide, and their bank accounts in European countries, and closing down Libyan front groups.

Secretary of State George P. Shultz predicted today's statement could further isolate Libya at a time when the Reagan administration believes the regime of Libyan leader Muammar Qaddafi is already on edge after the U.S. bombing raid last month. "I think you'll see a gradual rolling in of the isolation of Libya -- diplomatically, politically, economically," Shultz said.

In a final day of jockeying over the declaration, British Prime Minister Margaret Thatcher took the lead in pushing the others to name Libya in the document and also to include a call for improved extradition procedures for bringing accused terrorists to justice, European and U.S. sources said.

At a dinner Sunday night, the sources said, Thatcher, backed by Canadian Prime Minister Brian Mulroney and West German Chancellor Helmut Kohl, had called for a strong statement by the summit partners. President Reagan had circulated a 10-page paper with his views, including that the allies should "hold out prominently the possibility of using military force" to fight terrorism.

France had refused to allow U.S. warplanes to use its airspace in the raid. In the dinner meeting, the sources said, French President Francois Mitterand was reluctant to make as strong a statement as Thatcher was seeking, and particularly unwilling to name Libya.

Aides worked overnight to fashion a statement, but when reporters saw Thatcher at the opening of today's session, she was holding a draft of the statement with a note, apparently by an aide, saying it was "pretty weak."

While Reagan was satisfied with the draft, Thatcher pressed for a tougher statement, and one senior U.S. official said, "Margaret really wrestled old Francois to the mat."

The stronger language in the final statement brought exultations from Shultz, who said the document will send a message to Qaddafi: "You've had it, pal."

At the same time, the leaders of the seven states meeting here -- France, Britain, the United States, Canada, Italy, West Germany and Japan -- approved a separate statement saying the Soviet Union did not provide adequate information about the nuclear accident at Chernobyl and urging the Soviets to do so "urgently."

Reflecting the heavy concentration of nuclear power plants in the industrialized democracies, the leaders said today that "properly managed," nuclear power would "continue to be an increasingly widely used source of energy."

The leaders said they noted with "satisfaction" that the Soviets have now agreed to talk with the International Atomic Energy Agency about the accident. They called for more international cooperation and exchange of information in the event of nuclear accidents.

On terrorism, the leaders avoided mentioning two responses used by the United States against Libya: military retaliation and economic sanctions.

Shultz acknowledged that some nations had a "very good set of reasons" not to make public statements on economic and military action. Some, such as Italy and West Germany, are large trading partners with Libya. But he and other European diplomats predicted the summit nations would privately attempt to isolate Qaddafi further. He said economic sanctions "as they roll in" against Libya "are going to wind up being effective."

However, others were less enthusiastic about economic sanctions. French Foreign Minister Jean-Bernard Raimond said they are not worth it. While France strongly opposes Libyan terrorism, he said, "we do not want a rupture" in relations "because we want to maintain contacts with the Arab world, including even with Libya."

Reagan met today with leaders of some nations that had criticized the U.S. military raid against Libya last month, but an effort was made to demonstrate unity. A senior aide to Mitterand said, "There is absolutely no tension between France and the others over Libya."

Japan, which relies heavily on Middle East oil, has rarely if ever allowed itself to be associated with criticism of Arab states before. But Foreign Minister Shintaro Abe said today that Japan came to a "better understanding" of Libya's role in international terrorism after the United States and European nations presented evidence linking Libya to terrorist acts.

One result of the private discussions at the summit on economic sanctions appears to be a renewed effort by the United States to force American oil companies to withdraw from Libya. Earlier this year, Reagan ordered all Americans out of Libya, but gave the oil companies extra time to transfer their assets so they would not fall in the hands of Qaddafi and give him a "windfall."

Today, Treasury Secretary James A. Baker III said the oil companies will be out "shortly," and he noted that there had been pressure from other summit nations for such a withdrawal as an example of U.S. determination to keep pressure on Qaddafi.

Shultz expressed uncertainty about whether the U.S. firms could transfer or sell the facilities and said they may have to be abandoned. Baker said the United States would not indemnify them against seizure of their facilities but that Libyan assets frozen by the United States earlier this year could be used to offset the losses.

The statement on terrorism went further than one offered April 24 by the foreign ministers from countries of the European Community, according to a senior U.S. official. He noted the call for other states to join with the summit partners in fighting terrorism through such bodies as the International Civil Aviation Organization.

Another new aspect, he said, is a call for "strict limits on the size of the diplomatic and consular missions" as well as "other official bodies abroad" of nations that engage in state-sponsored terrorism. Today's declaration called for restrictions on travel of members of such missions and "where appropriate, radical reductions in or even the closure of" the missions and groups.

France, Britain, West Germany and Italy -- recently have expelled Libyans. The leaders also called for denial of entry to all persons who have been expelled from one of the summit nations on suspicion of involvement in international terrorism or conviction of a terrorist offense.

The leaders agreed to make the 1978 Bonn declaration on hijackings "more effective." The new declaration deals with all forms of terrorism, such as airport attacks and airplane bombings.

The statement's call for improved extradition procedures created a last-minute hang-up because of French objection that the summit leaders had no standing to infringe on extradition issues, since these should be set in a treaty. But the objection was met by making the statement more vague, officials said.

A senior U.S. official, asked in what cases the extradition procedure might work, suggested that of Abu Abbas, a Palestinian accused of masterminding the hijacking of the Achille Lauro cruise ship, who is charged with crimes against U.S. citizens. Similarly, he said, the British feel strongly about extradition of alleged IRA terrorists in the United States who have been accused of crimes against British citizens.

The leaders also called for refusal to export arms to states that sponsor or export terrorism. The senior U.S. official said none of the summit nations now exports arms to Libya, but he could not say whether arms are sold to Syria and Iran, both of which the United States has accused of sponsoring terrorism.

Officials noted that the allies had gone further than their last major statement on terrorism, in London two years ago. It had called for a "review" of arms sales to nations that sponsor terrorism and "scrutiny" by the counties of "gaps" in their laws that might be exploited by terrorists.

Here in Tokyo, meanwhile, police reported that a dozen small explosive devices were planted at sites in the subway system. No damage was reported. A similarly ineffectual rocket attack Sunday was laid to leftist radicals.