The debate over the Reagan administration's controversial plan to expand testing for the AIDS virus revolves around one important question: What makes people change their behavior?

Many public health experts who oppose mandatory testing of prisoners, immigrants and others say that education programs coupled with voluntary testing and counseling are the way to influence behavior that is as deeply ingrained as is sexual practice.

"There is not a shred of evidence that widespread testing will alter people's behavior in the right direction," said Prof. Larry Gostin of Harvard's School of Public Health, who recently completed a study of AIDS policies for the Department of Health and Human Services. "In fact, there is some preliminary evidence that testing can change people's behavior in a reckless way," particularly if unaccompanied by education and counseling.

On the other side of the debate are Education Secretary William J. Bennett and his former deputy, White House domestic policy adviser Gary L. Bauer. At a series of White House meetings in which the testing policy was hammered out, they argued that the acquired immune deficiency syndrome epidemic demands more aggressive public health measures that will produce behavioral change.

"We did syphilis and other venereal disease tests because we have always thought that informing people of the results can result in changing their behavior," said Bauer. "If you have a test and discover you're carrying the {AIDS} virus, then I believe that most of my fellow citizens will change their behavior."

The wrangling over testing is pragmatic as well as ideological. Critics predict that the policy of testing federal prisoners, immigrants, illegal aliens and the recommendation that states institute premarital testing programs will prove to be unworkable and ineffective.

When Attorney General Edwin Meese III announced plans to test inmates in federal prisons, he said he did not know how the information would be used. Some Justice Department officials said privately that the policy may present serious management problems in an overcrowded system where rumors are hard to control and confidentiality difficult to protect.

A majority of states have abandoned testing marriage license applicants for syphilis because of the low incidence of infection, the cost and because most already have had sexual intercourse with each other. Nearly all state health officers say the president's call for premarital testing will drain scarce resources from AIDS prevention efforts aimed at high-risk groups.

Some critics question whether mandatory testing will cripple the new federal amnesty program that will permit many undocumented immigrants to apply for citizenship. Critics say requiring an AIDS test may drive illegal aliens further underground.

"There is one big question about all this testing," said Dr. Mervyn F. Silverman, San Francisco's former health commissioner, who is president of the American Foundation for AIDS Research. "Since there is no cure or vaccine, what is the administration planning to do with the person who's {infected}?"

The testing debate has simmered since last winter, when word leaked that the Centers for Disease Control, the federal agency charged with tracking and controlling the disease, was considering recommending expanding testing to 40 million Americans: hospital patients, marriage license applicants and women seeking prenatal care.

At a two-day conference in February that drew 800 public health officials and gay-rights advocates, a consensus emerged in favor of offering "routine" voluntary testing to those seeking family planning, prenatal or drug treatment counseling.

But when President Reagan made routine testing the cornerstone of the policy he announced May 31, it became "mandatory" for federal prisoners, immigrants and, possibly, patients in Veterans Administration hospitals.

At congressional hearings and other forums recently, prominent public health specialists led by Surgeon General C. Everett Koop argued against mandatory testing, saying it would drive AIDS underground and discourage those most in need from seeking treatment.

At the same time, Bennett and Bauer were pushing for expanded mandatory testing, both publicly and at meetings of the Domestic Policy Council, a group of Cabinet members that was developing an AIDS policy. In an April 30 speech at Georgetown University, Bennett called for testing immigrants, prisoners, hospital patients and marriage license applicants.

In some cases, he said, the right of privacy might not be absolute.

In a May 26 White House memo Bennett also proposed that the president include "at least some groups of hospital admittees."

"One takes a host of medical tests upon admission to a hospital anyway," Bennett wrote. ". . . given the recent report of the infection of three hospital workers, hospital testing is easily defensible . . . . As with immigrants, and unlike marriage licenses, the federal government has a direct role here. We run a host of VA hospitals as well as others."

The wording of the president's first speech on AIDS was the subject of much internal discussion.

At a White House meeting May 27 Koop did not oppose mandatory testing for federal prisoners and immigrants but objected to having the president endorse required testing for other groups. Koop also advocated tougher confidentiality and antidiscrimination laws.

At a meeting the following day, Reagan recalled the 1930s campaign to eradicate syphilis and said he did not understand why public health officials objected to similar testing for AIDS.

Koop told Reagan that syphilis testing was not required until a cure was available. After Bennett and others agreed to change "required testing" to a recommendation that states "offer routine testing," Koop concurred and the decision was unanimous. Koop later said he agreed because individuals had "the right to opt out."

Reagan's speech, delivered on the eve of opening of the Third Interantional Conference on AIDS, did not mention new confidentiality or antidiscrimination laws. That omission and the significant expansion of testing alarmed some researchers and clinicians, 1,317 of whom signed a letter protesting administration policies.

Some officials applauded Reagan's call for expanded testing at drug abuse and sexually transmitted disease clinics, provided it is voluntary and accompanied by informed consent and counseling.

Dr. Donald I. Abrams, assistant director of AIDS activities at San Francisco General Hospital, said that the administration's focus on testing, rather than education, is misguided. In San Francisco, which has sponsored intensive public education campaigns, the incidence of several venereal diseases among homosexual men has fallen by more than 90 percent. "The single most important impetus to behavior change is perceived risk," he said. "Maybe what we need to do to educate people is to use photographs of people at the end stages of this disease. Testing alone is just not going to do it."