Labor Secretary Raymond J. Donovan has decided his department lacks authority to pursue the removal of Teamsters union President Roy L. Williams, who has been accused of having ties to organize crime.

The decision was announced in a letter to the Senate permanent subcommittee on investigation, dated July 9 and released by the panel yesterday.

The letter was a response to an interim report the subcommittee issued in May urging the Labor Department to grill Williams on supposed ties to the Kansas City mob, and to go to court to demand his dismissal if he refused to answer. Last year Williams claimed the Fifth Amendment and refused 23 times before the subcommittee to answer questions about alleged involvement with a reputed organized crime boss.

In refusing to consider seeking Williams' ouster, Donovan wrote that "the Labor-Management Reporting and Disclosure Act of 1959 does not provide any procedure for removal of officers of international unions by civil or administration proceedings" by the Labor Department.

Sen. Sam Nunn (D-Ga.) retorted in a statement yesterday: "The Labor Department policy is clearly to interpret the law in a very narrow manner. In summary, it is a hands-off policy designed more to protect the interests of the union hierarchy than the rank and file."

Nunn said he was drafting legislation that explicitly gives the Labor Department the authority to seek the removal of union leaders who mishandle funds.

Marty Steinberg, chief counsel to the Democrats on the subcommittee, said yesterday that under several statutes as well as common law, the department could go to court and demand Williams' removal if he refused to answer questions. Steinberg said that Williams, as a union officer, is legally obliged to explain relationships that have a bearing on union funds.

Steinberg said the subcommittee would address Donovan's letter in a report expected to be released in two or three weeks.

The 66-year-old Williams was indicted in May and accused of conspiring to bribe Sen. Howard W. Cannon (D-Nev.) in an effort to defeat legislation that would deregulate the trucking industry. Cannon was not charged.

Representatives of the 1.9 million Teamsters elected Williams on June 4 to a five-year term as president of the nation's largest union. At that time, he dismissed the charges against him as "a damned lie" and said he had refused to answer questions posed by the subcommittee because he was not informed in advance of the accusations against him.

Williams was unavailable yesterday and his attorney in Washington, Thomas A. Wadden Jr., declined to comment because he had not seen Donovan's letter of Nunn's response.