Labor Secretary Raymond J. Donovan yesterday said "there is a tremendous defect" in the special prosecutor law under which he is being investigated for allegedly corrupt business practices.

"When unsubstantiated charges can be made by anyone, including convicted criminals, triggering this law, it deserves review," Donovan said at a breakfast meeting with four reporters.

He called the meeting to talk about other matters: his official duties, departmental programs, his views on the growing unemployment rate and his relationship with organized labor. He also wanted to make one thing clear: rumors to the contrary, he said he is not leaving.

"I have paid a large entrance fee to this city and I intend to stay for the double feature," Donovan said, alluding to his often expressed belief that President Reagan will serve two terms.

But the cloud that accompanied Donovan's arrival in Washington last year has not gone away. His Senate confirmation hearings were marked by allegations that, as a New Jersey construction firm official, he violated federal law by paying off union leaders for labor peace.

Now a special prosecutor, appointed under the Ethics in Government Act of 1978, is looking into allegations that Donovan's former company, Schiavone Construction, made payoffs to New York officials of a Laborer's union local.

The 1978 law requires the attorney general to conduct an investigation of federal charges against a president or other high-ranking federal official.

A special prosecutor may be brought in if the attorney general, in consultation with a special court unit, feels circumstances warrant further probing.

Such a prosecutor, New York lawyer Leon Silverman, was named Dec. 29 by the U.S. Court of Appeals here to look into the Donovan case.

Donovan said yesterday that he agrees with the intent of the law, to eliminate the kind of official wrongdoing that turned the former Nixon administration into the "Watergate Era."

But the way in which the law is applied "is not consistent with justice under the American system," he said.

However, Donovan said he is pleased with Silverman's performance. "The special prosecutor indicated that he would not dawdle, and by his actions to date, he is living up to that . . . I applaud that," the secretary said.

He said discretion forbade him from saying anything more about the matter. He said he preferred talking about his record in office, which he described as good.

Donovan credited his administration with bringing strong management to the Labor Department and with streamlining and improving the operation of the much-maligned Occupational Safety and Health Administration.

He also takes credit for saving workers' programs, such as as the black lung benefits fund for coal miners stricken with the debilitating respiratory illness.

Donovan said the Reagan administration's performance, as well as his own, "has eased that pain" inflicted on himself and his family by the charges of wrongdoing.

"I came here to serve, and I am doing that," he said. "But, I would like to have paid less of a price."