North Korea withdrew its invitation to a U.S. envoy who was headed to Pyongyang to request the release of imprisoned, ailing American missionary Kenneth Bae, the State Department said on Friday.

North Korea canceled talks with Robert King, the U.S. special envoy for North Korean human rights issues, who was expected to visit Pyongyang on Friday and Saturday.

“We are surprised and disappointed by North Korea’s decision,” State Department spokeswoman Marie Harf said.

“We remain gravely concerned about Mr. Bae’s health and we continue to urge the DPRK authorities to grant Mr. Bae special amnesty and immediate release on humanitarian grounds,” the spokeswoman said in a written statement, referring to the North by its formal name, the Democratic People’s Republic of Korea.

There was no immediate explanation for the decision from North Korea, which does not have diplomatic relations with Washington.

King’s trip, announced this week as he was visiting U.S. allies Japan and South Korea, was seen as a potential signal of the start of a gradual thaw in relations between Washington and Pyongyang.

The State Department had termed the trip a “humanitarian mission” and played down any connection between Bae’s release and diplomacy over the North’s sanctioned nuclear weapons program.

Bae’s sister in Washington state issued a statement saying their family was disappointed and worried about Kenneth Bae’s health but are “not giving up hope for a peaceful and timely resolution.”

“We hold on to faith that DPRK and U.S. diplomats will resume talks soon, ultimately leading to my brother being released,” said Terri Chung, Bae’s sister.

Rep. Rick Larsen (D-Wash.), whose district includes where Bae’s family lives, urged the North Koreans to free Bae.

“The North Koreans gain nothing from this course reversal. It is time to let Kenneth come home to his family and get the medical attention he needs,” Larsen said in a statement.

King secured the release of another Korean-American missionary, Jung Young Su, in 2011 as part of a trip to assess North Korean pleas for food aid.

— Reuters