The American Medical Association yesterday rejected the Reagan administration's call for significantly expanded routine testing for the AIDS virus in favor of widespread voluntary testing and counseling and new laws that would protect the civil liberties of those tested.

The AMA, the nation's largest and most politically influential medical organization, endorsed testing military personnel and blood and organ donors, which is under way, as well as screening immigrants and prisoners, but refused to support the administration's call for widespread routine testing of hospitalized patients and persons applying for marriage licenses. The AMA also said that patients should "knowingly and willingly give consent" before taking a voluntary test.

"I think this {report} tells the White House that this is what the doctors of this country feel," said Dr. Robert McAfee, a surgeon from Portland, Maine, and a member of the AMA's Board of Trustees, which drafted the recommendations that were voted on yesterday by the AMA's 406-member House of Delegates. "National policy is not just federal policy."

The AMA's new AIDS policies were applauded by some public health officials but greeted with less enthusiasm by White House domestic policy adviser Gary L. Bauer, who has led the fight for expanded AIDS testing. Last month President Reagan called on states to institute routine testing for inmates, persons applying for marriage licenses and those seeking treatment for drug abuse and sexually transmitted diseases. Secretary of Education William J. Bennett has also called for testing hospital patients.

"I'm very pleased that the AMA has endorsed the president's policy on military personnel, immigrants, prisoners and blood donors," said Bauer. "We will review the other recommendations, but I suspect that in this case the American people seem much more willing to have routine testing than does the medical establishment."

Laws protecting confidentiality and outlawing discrimination "take a clear back seat to the protection of those Americans who are not yet infected," Bauer said. "It continues to be puzzling to me why testing for syphilis is appropriate but testing for AIDS is not."

Dr. June L. Osborn, dean of the School of Public Health at the University of Michigan and a leading opponent of mandatory testing, said she was "delighted that the AMA is getting into the act with a very constructive set of recommendations."

"Until now the medical profession has not had help in discerning an appropriate stance," said Osborn. "To have a group as conservative in the good sense as the AMA say, 'We are not going to follow the obvious,' is wonderful."

Recommendations by the 271,000-member AMA are advisory but influential with state medical groups and lawmakers. The report was issued during the governing body's annual meeting in Chicago.

Although most of the recommendations were adopted without debate, the proposal on testing of inmates was controversial. Opponents argued that segregating inmates was prohibitively expensive and unworkable and identifying infected prisoners in the general population could endanger their lives.

The AMA also endorsed a bill introduced by Rep. Henry A. Waxman (D-Calif.) to increase funding to voluntary testing and counseling. The group also said the $1 billion Congress is expected to allocate to fight AIDS in 1988 is insufficient.

The group also supported recommendations by Surgeon General C. Everett Koop that elementary school students receive explicit AIDS education and that condoms be advertised on television.

One recommendation likely to be controversial if adopted by states would let doctors warn "unsuspecting" sex partners of infected persons that they may be at risk.

Some states, including California, prohibit doctors from notifying sexual partners of those infected with the AIDS virus, even though other laws say that physicians have a "duty to warn" if they have reason to believe a person is at risk.

"The confidentiality of the doctor-patient relationship is vitally important but not absolute," the report said.