Behind the high-security electronic defenses of a plutonium plant, 22 Japanese and American nuclear scientists are probing for the solution to a complex problem. They must succeed where months of political negotiations have failed. The answers they seek lie at the frontiers of nuclear knowledge.

Their task is to forge a compromise that will salvage a new $300 million Japanese plutonium reprocessing plant without undermining President Carter's stand against global proliferation of the nuclear fuel which can be used in the production of nuclear weapons.

The stakes are high for both nations. American approval of the scheduled start-up in late July of the Tokaimura plant 60 miles north of Tokyo would set a precedent for other nations anxious to possess deadly yet precious plutonium. A flat rejection would provoke a grave confrontation with Japan, which is America's closest Pacific ally and second largest trading partner.

The Tokaimura plant has become a burning political and nationalistic issue here. The current joint inspection survey is getting saturation coverage in the press. Government officials, usually oblique and reserved in talking with foreign reporters, are echoing a tough, uncompromising position: Tokaimura must open.

"Japan has no intention of stopping or delaying the reprocessing program," says Masahiro Kawasaki, an atomic energy division director in the Science and Technology Agency. "It is a very vital national issue for Tokaimura to go critical" - to begin operation.

Eleven years in design and construction, the plant is capable of extracting six tons of plutonium a year from spent nuclear fuel. Officials contend that Tokaimura's plutonium will be essential to the country's energy program over the next two decades.

The United States has sought to persuade the Japanese that reprocessing is unnecessary and uneconomical - particularly since the plant can treat only 210 tons of nuclear waste a year. The effort now is to determine whether reprocessing plants like Tokaimura can be safeguarded against plutonium theft and against its diversion into the manufacture of nuclear arms.

The preferred American solution is for conversion of the plant to the plant to the manufacture of a plutonium-uranium mixture that cannot be used in weapons. The Japanese claim that even if the so-called co-processing technology were proven, it would cost $150 million and a delay of two to three years to adapt Tokaimura. Officials say that they are ready to cooperate with the United States in development of coprocessing techniques for use in future plants, however.

The American scientists believe there are 10 to 15 possible modes of operating Tokaimura, and not all would require extensive and time-consuming modification. Japanese officials are prepared to offer a compromise solution. They would not disclose the details, but since the first eight months of Tokaimura's operation are scheduled to be a 'hot' test run that will not produce plutonium, there may be room for negotiation.

The final agreement, according to well-informed observers, could involve a limited start-up that would confine radioactive contamination to a portion of the plant while reprocessing research, and possibly adaptation, would continue in the remainder.

Shelving the plant and its 400 employees, who cannot be fired, would expose the ruling Conservative Party to charges of knuckling under to Washington. The whole dispute - with the accompanying hazard of lost prestige and face - is critically timed for Premier Takeo Fukuda. He is to lead his party into a difficult upper house election July 10.

The United States is Japan's sole supplier of the enriched uranium burned in 12 of its 13 nuclear power plants. Under a 1958 agreement, the United States must be consulted before any of the exhausted radioactive wastes are reprocessed into plutonium. American officials, keeping faith with Carter's stand against the proliferation of nuclear weapons and the manufacture of weapons-grade nuclear materialy, so far have refused approval for Tokaimura.

Summit discussions between Fukuda and Carter in March and two subsequent negotiating sessions failed to break the impasse. Neither side expects the scientists report will submit after their joint nine-day inspection to end the matter. Yet both countries are hoping the scientists can find a forumla for a technical compromise that will lead to a political agreement.

The Japanese are confident. They assume that when Carter and American negotiators say they understand resource-poor Japan's special vulnerability to energy shortages, it signifies a willingness to somehow allow Tokaimura to operate. This week, Sasuke Uno, the Cabinet-rank Director-General of the Science and Technology Agency predicted the plant would open as planned in late July.

Most observers find that increasingly unlikely. It is suggested that the Japanese may be underestimating the Carter administration's determination to draw the line against plutonium manufacture. The U.S. delegation chief, Lawrence Scheinman, a State Department senior adviser, has carefully balanced American confidence in Japan's peaceful nuclear intentions - "beyond question," he says - with concerns about the international impact of Tokaimura go-ahead.

Japan's policies would have a profound influence on the global development of nuclear power. Scheinman told a press conference after arriving in Tokyo.

His deputy Harold D. Bergelsdorf of the U.S. Energy Research and Development Administration, added: "It has to be recognized this is a major decision with broad precedental implications."

The first major test of Carter's non-proliferation policy is as tough as they come. The appelant is Japan, the world's third-richest industrial nation, which has been headed toward development of plutonium-fuelled nuclear energy for the last 30 years. Workmen were completing Japan's French-constructed reprocessing facility at Tokaimura when Carter came out against the entire concept earlier this year.

The 1973 oil embargo exposed the risks of Japan's near-total reliance on imported energy. Contrary to Carter administration predictions. The Japanese believe there will be a world shortage of uranium. Plutonium extends the energy yield of a given amount of uranium by 60 times, thus offering a far greater measure of independence.

Indignant Japanese officials say the U.S. knew of the country's plutonium plans and actively encouraged them as recently as three years ago. "We were good students and did what the teacher told us," one senior bureau crat said, "and you abruptly changed the policy."