South Korea today agreed to return a hijacked Chinese airliner, its passengers and crew to China, but rejected a Chinese demand to extradite the six hijackers to China.

The hijackers are now likely to stand trial in a South Korean court on air piracy charges.

The decision, announced by South Korean authorities tonight, followed a round of impromptu talks in Seoul during the weekend between officials of the two governments on the fate of the airliner, which landed in South Korea on Thursday with 105 persons on board. It was flying for the Civil Aviation Administration of China.

The hijacking, the first successful one involving a Chinese commercial jet, resulted in the first official contact between the two countries since the Communists came to power in China in 1949. It confronted both governments with a series of sticky diplomatic issues and left the two sides sharply divided over the handling of the six Chinese hijackers.

In talks between South Korean Deputy Foreign Minister Gong Ro Myong and the director general of the Chinese aviation agency, Shen Tu, South Korea agreed to the early return of 96 passengers and crew to China along with the hijacked British-made Trident airliner. Officials, however, refused a Chinese demand for extradition of the hijackers on the grounds that they should be tried under South Korean penal codes.

Knowledgeable observers in Seoul said that 94 passengers and crew members were likely to leave Seoul for Peking sometime Monday on board a Boeing 707 of the Chinese aviation agency. Two crew members who were wounded by pistol shots in a midair scuffle with the hijackers will remain in a Seoul hospital until they are able to return to China. Three Japanese passengers have already returned to Japan.

The surprise arrival of the hijacked plane in South Korea last week amplified the deep political crosscurrents in Northeast Asia where South Korea, a key Pacific ally of the United States, faces a hostile Peking-backed Communist government in North Korea across a heavily armed border.

The diplomatic equation is complicated by the fact that the hijackers have requested political asylum in Taiwan. South Korea has no diplomatic ties with China and is now the only Asian country to recognize the rival Chinese government on Taiwan as the sole legitimate government of China.

In seeking extradition of the hijackers, Shen Tu, who arrived in Seoul yesterday at the head of a 33-member official delegation, pressed the South Koreans to abide by existing international agreements on air piracy and to return the hijackers to China immediately for punishment.

Both countries are signatories of the 1970 Hague Treaty governing airline hijacking. Under that agreement, South Korea can either send the hijackers back to China or try them under South Korean law. There is no extradition agreement between South Korea and China.

According to observers in Seoul, South Korean authorities appeared ready to put the hijackers under arrest pending trial in a South Korean court. Should the hijackers be found guilty, these observers suggested, they might then be expelled from South Korea, a legal twist that would, presumably, allow Taiwan to grant them political asylum.

A decision to allow the hijackers to go to Taiwan, where authorities are reportedly willing to welcome them, would anger Chinese authorities and complicate South Korean efforts to improve relations with Peking. Chinese authorities have become increasingly embarrassed in recent months by a growing number of Chinese citizens seeking political asylum abroad.

On the other hand, turning over the hijackers to China for punishment would inevitably damage relations between South Korea and Taiwan, two powerful noncommunist industrial countries in the region. Echoing Taiwanese concerns, the government in Taipei yesterday expressed "grave concern" over the visit of the Communist Chinese delegation to South Korea.

While there are no diplomatic ties between South Korea and China, trade and unofficial contacts have expanded in recent years. South Korea is believed to be interested in establishing official ties with China, in part as a counterweight to its bitter Communist rival in North Korea.

Observers in Seoul said that the talks on the hijacking incident could lead to a thawing of relations between the two countries, but they pointed out that the Chinese are being careful to limit the discussions to the settlement of the hijacking.

For its part, South Korea, which will host the 1988 Olympic games in Seoul, has promoted an open policy in recent years toward Communist Bloc countries and pressed for the cross-recognition internationally of both South and North Korea as a prelude to the eventual peaceful political reunification of the Korean Penninsula. China, however, has stepped back from any official diplomatic opening with the government in Seoul because of its long-standing support of Communist-ruled Pyongyang.