President Carter's bid to open a new phase of diplomacy aimed at reducing tension on the Korean peninsula has been rebuffed by North Korea in a message passed to Washington last week through the Indonesian government.

But American officials have not given up hope that sooner or later the Carter plan will make its mark on one of the world's most dangerous trouble spots.

Carter's proposal, announced two weeks ago today during his trip to Seoul, was a joint U.S.-South Korean offer to open three-party talks with communist North Korea on a broad range of questions from person-to-person contacts all the way to eventual north-south unification.

It was Carter's first direct response to an insistent series of secret messages from North Korea since Dec. 20, 1976, a month before his inauguration. His answer, which fell far short of what Pyongyang wanted, required a modest but hard-won shift in position by South Korea.

Seoul's cooperation in the diplomatic bid was obtained, in part, through hints that Carter's program for the withdrawal of American ground troops will not be pushed in the immediate future. American officials expect a new presidential decision on the controversial troop withdrawal issue to be taken and announced before the end of this month.

The political-diplomatic-military maneuvering of the past few weeks has been cause for greater satisfaction in Seoul than in Pyongyang. tnevertheless, the episode may lead to direct United States-North Korean contacts, a longtime objective of the Pyongyang regime.

Seven years ago this month, on July 4, 1972, the two Koreas stunned the world by announcing the start of north-south dialogue after a quarter of a century of bitter confrontation arising from the post-World War II division of the country. When the dialogue foundered, North Korea in early 1974 began making bids for direct negotiations with the United States, over the head of their South Korean rival.

Henry A. Kissinger, as presidential adviser and secretary of state in the Nixon and Ford administrations, sent feelers to North Korea through Pakistani and Egyptian governments and made it knows he would like to to to Pyongyang to try his peacemaking wizardry.

Unlike in Vietnam, an example that is still vivid and alarming in Seoul, Kissinger refused to negotiate with the adversary communist regime in the north without full participation by the ally regine in the anti-communist south.

In September 1975 and July 1976, Kissinger proposed four-power negotiations involving China and the United States as well as the two Koreas. China as well as North Korea refused to take part.

North Korea's bid to Carter for direct bilateral talks began with a personal letter from President Kim Ilsung to the president-elect, delivered to Plains, Ga., through a Pakistani channel a month before Inauguration Day. This was followed by a secret letter from North Korean Foreign Minister Ho Dam to Secretary of State Cyrus R. Vance in February 1977.

Over the following months North Korea sent additional messages to Washington through Gabon, Romania, Yugoslavia and an American professor at a Canadian university. The message of all was essentially the same: Pyongyang and Washington should negotiate a peace agreement to settle the Korean War once and for all, and other countries concerned" (South Korea) might be able to join the talk in some fashion as negotiations progressed.

If Carter was tempted, the controversy over his plan to withdraw U.S. troops from South Korea made a further blow to Washington-Seoul relations impractical then. This was followed by tensions with Seoul over human rights and Tongsun Par-Koreagate scandal.

Only recently, with the tensions subsiding and the withdrawal plan temporarily "in abeyance," are Carter's relations with Seoul stabilized enough to permit consideration of diplomatic iniatives with Pyongyang.

New feelers from North Korea this spring spurred Washington's interest. Hints were dropped to U.N. Secretary General Kurt Waldheim and to American journalists that Pyongyang might be flexible about a role for South Korea in Washington-Pyongyang talks.

With Carters approval, the State Department began sounding out South Korea last month through ambassadors here and in Seoul on a joint proposal for Washington-Seoul-Pyongyang talks. South Korea is said to have been reluctant at first, because of fear of another Vietnam-style negotiation in which the U.S. partner would lose out. And Seoul, by some accounts, considered it much more likely than Washington that North Korea would accept such a joint negotiation.

When South Korea's approval was obtained on the eve of Carter's trip to Seoul, the United States got Indonesia agreement to serve as a communications link with Pyongyang. Indonesia has diplomatic relations with both Koreas, so messages can be passed with relative ease. Indonesia also agreed to be the site of three-way meetings if Pyongyang accepted the U.S.-South Korean offer.

A joint U.S.-South Korean message was sent to North Korea through Indonesia shortly before the announcement of the bid for talks two weeks ago, to tell Pyongyang in advance that this was a serious proposal. The same channel was used for the proposal itself, signed by Vance and South Korean Foreign Minister Park Tongchin and addressed to North Korean Foreign Minister Ho Dam.

Simultaneously, the United States notified Japan, a close U.S. ally with strong interests in the Korean peninsula, as well as China and the Soviet Union. China made clear it would line up squarely behind the North Korean position, as it has since in private meetings with American officials and in a public statement in peking by Foreign Minister Huang Hua. The Soviets are reported to have been immediately unhappy with the U.S.-South Korean offer, for fear that Moscow might be cut out of diplomatic maneuvering that affects its interests.

North Korea's answer was published as a Foreign Ministry statement in Pyongyang last Tuesday, calling the Washington-Seoul proposal for joint talks "utterly infeasible" and sticking close to its previous position against full-scale South Korean participation in negotiations on a peace agreement. About the same time a nearly identical message was dispatched to the Indonesians for transmission to Washington and Seoul.

American # officials considered the North Korean reply very negative in tone and substance, although they took heart from some airy phrases which seemed to leave room for a change of position and from Pyongyang's statement that "we still leave the door open" for a dislogue with South Korean authorities and groups, and with the United States.

Several officials said the Pyongyang statement was an expected first response to the Washington-Seoul initiative. One body of official opinion holds that major North Korean movement toward three-way talks, if it comes, will be depicted as Pyongyang's initiative rather than as a response to the other side's proposals.

With the U.S. estimate of North Korean military strength sharply increased recently, South Korea is continuing its military buildup and opponents of the troop withdrawal plan appear to have the upper hand in Congress and the bureaucracy. Despite continuing tension and danger, American officials expressed hop that the process started in the past two weeks may permit diplomacy to function on the Korean peninsula sometime in the future.