Large numbers of Chinese road workers are being removed from the Karakorum Highway between Pakistan and China because of fears for their safety, according to a highly informed source. Thousands of the Chinese were held hostage last month by armed tribesmen demanding the ouster of Prime Minister Zulfiqar Ali Bhutto.

Some 4,000 Pakistani troops are now in control of the highway, following a three-day battle last month in which 36 tribesmen and six soldiers were killed, said the source, who has access to Pakistan army intelligence.

No Chinese workers are known to have been harmed.

At the height of the battle, some 10,000 tribal riflemen faced an estimated 12,000 troops. The army had moved heavy weapons and several fighter aircraft to the area, but, according to a Western military observer, casualties were kept at a minimum because the army did not use its heavy equipment."

He noted that this was in marked contrast to an action last year in the nearby area of Dir, when the army killed "between 150 and 400" tribesmen in an uprising.

The tribesmen severely damaged three heavy concrete bridges, the source said, and pontoon and other temporary spans have been put in place and the road is now described as "secure."

Within the last two weeks the military attache of the Chinese Embassy in Islambad, Yi Fei, visited the area where the workers had been held hostage. He was accompanied by his assistant, a man identified only as Mr. Chang, and a senior Pakistan army officer. The workers are reportedly being removed because China fears for their continued security.

Although the Chinese embassy has refused to discuss the matter with Western journalists, there are no indications that China put pressure on Bhutto. China and Pakistan have long had excellent diplomatic relations, so much so that Pakistan played an important role in clearing the way for former Secretary of State Henry A. Kissinger's first trip to Peking in 1971.

One neutral Asian diplomat said that the Peking government had been "deeply concerned" about the welfare of workers at the time they were being held hostage. "There may have been some slight strain between the two governments for a moment," he added, "but if so, it's past because the Chinese know Bhutto was at least as worried as they were. And in the end, no harm came to the workers."

But until Pakistan can guarantee better security for the road builders, it does not seem likely that the full complement will be restored.

Until the withdrawal of workers began, as many as 40,000 Chinese were believed to be at work on the road at any time. Construction, mainly by China, has been going on since 1969.

It is not clear how many workers have been pulled back across the border. But a source in contact with residents of the rugged, mountainous area, said several construction camps had already been closed down.

The removal of large numbers of the workers, who are members of engineering units of the Peoples Liberation Army, has slowed work, now mainly application of an all-weather surface, on the 493-mile road that links Pakistan's Northwest Frontier Province with China's Sinkiang province.

Pakistani authorities have made no recent comment on the situation. Brig. Gen. Tafazul Sidiqui, head of Inter-Services Public Relations, refused to discuss the matter.

But, on June 5, the government's Press Information Department issued a statement that said the highway had been blocked in six places by tribesmen acting under direction of Bhutto's political opponents, the Pakistan National Alliance.

"Supplies to 4,500 Chinese working on the Karakorum Highway remained cut off for 10 days. Some of the supplies had to be carried to the Chinese camps by helicopters during that time," the statement said.

An Alliance source said that the tribesmen, who belong to several small tribes within the Kohistani group, had surrounded the Chinese workers in their camps for two reasons.

"First," the source said," they wanted to do their bit in driving Mr. Bhutto out of office. Second, they wanted the government to compensate them for land taken from them for the highway."

Thirty-four tribesmen are now understood to be undergoing military trials at Ugi Fort, near the town of Manshera, in the Northwest Frontier Province. They are being tried for treason, a source said.

Political observers said they were unable to establish any direct link between the tribal action against the Chinese workers and Bhutto's decision a few days ago to agree to hold fresh national elections this fall.

But an alliance source said that the opposition was convinced that the tribal uprusing "played a role" in forcing Bhutto's decision. "The army was spread that much thinner," the source said. "This put pressure on senior officers who, in turn, put pressure on Mr. Bhutto."

This view concurs with assessments by several foreign diplomats.One said that pressure by the armed forces on Bhutto "was the key" to his agreement to hold fresh elections.

The opposition claims that general elections held last March were rigged in favor of Bhutto's Pakistan Peoples Party. For nearly three months, opposition followers clashed with Bhutto's party and with soldiers and police throughout the country. The government has stated that 240 people were killed, but the opposition says the figure is in excess of 1,000.