A delegation of 20 American physicists sponsored by the National Academy of Sciences yesterday canceled a scheduled trip to the Soviet Union to protest the imprisonment of dissident Soviet physicist Yuri Orlov.

"We have repeatedly informed Soviet authorities that the issue of human rights threatens to erode the willingness of American scientists to cooperate with their Soviet counterparts", National Academy of Sciences President Philip Handler said on the eve of a visit he began yesterday to Poland. "And now our predictions are being borne out."

Orlov's conviction last week for slandering the Soviet state does not bode well for any future scientific exchange between the Unites States and the Soviet Union, which annually involves about 500 scientists from each country, Academy officials said. Handler hinted that the upcoming trials of Alexander Ginsburg and Antoly Scharansky could have a greater impact than Orlov's sentencing.

"Should the trials of Ginsburg and Scharansky turn out similarly," Handler said, "Soviet-American scientific relations will have been profoundly damaged."

In addition to the 20 American physicists who decided yesterday to stay at home, two leading American scientists about to leave for the Soviet Union on separate missions have also cabled their cancellation. One is Nicolaas Bloembergen of Harvard University, the other is Dr. Robert Marshak, head of City College of the City University of New York.

"The closed trial of inhuman punishment imposed on our high-energy physics colleague, Yuri Orlov, make it impossible for me to attend the seminar [in Moscow] next week," Marshak said in a cable to academician A. Lugonov of the Institute of Nuclear Research.

While Handler issued a formal statement in the same of the academy yesterday, neither he nor the academy had a hand in the cancellation yesterday of the trip by the 20 physicists, who voted unanimously to remain at home. They had been scheduled to leave today.

"It was very spontaneous," Dr. William F. Brinkman of Bell Telephone Laboratories, cochairman of the delegation, said yesterday by telephone. "We just asked ourselves, 'Hey, why are we going?"

The 500 American scientists who annually visit the Soviet Union in the formal U.S.-U.S.S.R. exchange program are sponsored on their trips by either the National Academy or some federal agency like the National Science Foundation or the National Institutes of Health.

Next week, a delegation of six nuclear physicists is due to leave for the Soviet Union on a trip sponsored by the Department of Energy. By late yesterday, it was not clear whether any or all of the six would go.

"I'm confused," said one of the delegates, who asked not to be identified. "I don't know what I'm going to do."

No such confusion was expressed by the 20 physicists who voted not to go.

"I have great fondness for the Soviet scientists I've met in this exchange," said Dr. Raymond Orbach, professor of physics at UCLA. "But the atmosphere has been poisoned to such an extent that it's impossible to continue this dialogue. My guess is that the Soviet Union will not be regarded as a healthy place to hold any kind of exchange in the foreseeable future.