Prime Minister Yasuhiro Nakasone of Japan today began his first overseas trip since taking office with a visit to South Korea, where he is expected to emphasize his determination to bring about a stronger three-way alliance among Seoul, Tokyo and Washington.

Nakasone opened his two-day stay with a promise to South Korean President Chun Doo Hwan that he intends to usher in "a new and vital stage" in relations between the two neighboring countries, which have been beset by bitterness and distrust in the post-World War II era.

Nakasone is the first Japanese leader to travel to South Korea for political talks. Japan annexed the Korean Peninsula in 1910 and ruled it until the end of World War II.

The diplomatic overture reflects a bid by Nakasone to underscore his flair for decisive action and to stress the importance of stronger ties among Japan, South Korea and the United States. He is expected to improve relations between Tokyo and Seoul, Washington's key Pacific allies, by capping an 18-month-old feud with the offer of $4 billion in economic aid for South Korean development programs.

Nakasone's visit here comes less than a week before he is scheduled to go to Washington for talks with President Reagan. According to diplomatic observers here, Nakasone intends to reaffirm ties with the United States, which have been badly strained over the issues of trade and defense preparedness, by demonstrating Japan's willingness to shoulder expanded responsibilities for maintaining peace and stability in northeast Asia.

In a speech at a state dinner hosted by Chun, Nakasone said, "It is our common task, of the very greatest historical significance, to continue to build and maintain a dynamic and mutually beneficial relationship of absolute equality."

Relations between the two countries have remained chilly because of the bitter feelings that many older Koreans harbor toward the Japanese as the result of the 36 years of Japanese colonial rule. Among younger South Koreans, sensitivities dwell on what they view as an economic relationship that vastly favors Japan and on Japanese arrogance in dealing with its neighbor.

Nakasone said that he viewed "with deep regret" that the history of "our relations has its unhappy pages, and this is a fact that I and all Japanese must recognize and take to heart."

Chun called Nakasone's visit "a monumental milestone" and said, "there is a pressing need to further develop and deepen ties of friendship and cooperation between Korea and Japan. . . as members of the free world."

Nakasone, who has been known for his strong anticommunist views and a hawkish stand on defense issues, told U.S. reporters in Tokyo last week that he favored a stronger alliance among Japan, South Korea and the United States to help offset a rapidly growing Soviet military presence in the Pacific. Unlike the military ties Tokyo has with Washington under the Japan-U.S. mutual security treaty, Japan's relationship with South Korea would focus on economic aid "to help make the country more stable," Nakasone said.

Nakasone demonstrated his determination to put a stronger personal stamp on Tokyo's foreign policy than his predecessor Zenko Suzuki by placing a widely publicized telephone call to Chun shortly after he took office Nov. 26. That initiative, observers said, helped set the stage for Nakasone's surprise announcement on Jan. 5 that he would travel to Seoul at Chun's invitation and that there had been a decision on the long-pending $4 billion economic aid package that has churned up strong emotions on both sides.

The settlement of the aid issue is important for Chun, observers here said, because the South Korean economy, to which political fortunes in this country of 38 million are closely tied, has been struggling to break out of a recession.

Chun's government was rocked last year by a series of political mishaps, including a billion-dollar money market scandal here involving close relatives of the president's wife.

While Chun appears to have weathered those difficulties, the fact that Nakasone has come to Seoul, something rare for a Japanese leader, can only help enhance Chun's image at home, western diplomats said. Two of Nakasone's predecessors, Eisaku Sato and Kakuei Tanaka, visited South Korea following normalization of Tokyo-Seoul diplomatic ties in 1965 to attend ceremonial functions but not for political discussions.

In placing Seoul first on his diplomatic agenda, Nakasone is widely thought to be laying the groundwork for his talks in Washington next week with President Reagan. U.S. officials have expressed strong dissatisfaction with Tokyo's recent decision to boost defense spending in 1983 by 6.5 percent instead of the minimum of 7 percent they had expected. By strengthening ties with South Korea, political analysts said, Nakasone apparently hopes to impress on Washington Japan's determination to play a stabilizing role in northeast Asia. "Nakasone is going to Washington by first stopping off in Seoul," a government source said.

Meanwhile, political analysts here have expressed concern that Nakasone's overtures to Seoul, with strong implications of improving ties with the Chun government as a bulwark against communism in the region, may become a major obstacle to reopening long-stalled talks between the South and Communist North Korea on the issue of political reunification. Pyongyang's media attacked Nakasone's visit as part of a plot to set up a military alliance and warned that the trip will further damage prospects for an early resumption of negotiations.