The American tradition of handing out ambassadorships as political rewards to amateurs and second-raters is too risky in the modern world and is demoralizing the nation's Foreign Service professionals, according to a series of witnesses at a Senate hearing yesterday.

Yet, the Reagan administration has reversed a recent trend toward greater professionalism by appointing political friends at the highest rate of "any administration since that of Herbert Hoover," Charles S. Whitehouse, president of the American Foreign Service Association, told the Foreign Relations Committee.

"The Reagan appointees are by and large persons less distinguished and capable than many selected by earlier presidents," he charged.

Of Reagan's top diplomatic appointments so far, 48 percent have been political, compared to 37 percent overall under President Carter, with the Foreign Service professionals suffering a net loss of 11 ambassadorships so far, Whitehouse said.

Richard T. Kennedy, undersecretary of state for management, defended the administration's record as being more moderate than that, arguing that the rate of political appointments tends to be greater in the first year. He also contended that if ambassadors left in place are counted along with new appointees, the mix is more favorable to the careerists.

The president has "made it clear that he views the ambassador as his personal representative to the host government," Kennedy said. "We do not accept the sometimes stated convention that the ambassador is just a messenger for the foreign policy decision makers in Washington."

The administration opposes a bill introduced by the hearing's chairman, Sen. Charles McC. Mathias Jr. (R-Md.), on the grounds that the bill is unconstitutional and unnecessary, Kennedy said. The proposal would set a 15 percent limit on political appointments to top diplomatic posts.

Mathias expressed dismay at a recent comment by White House personnel director E. Pendleton James, who was quoted as saying, "The question is not whether we have too many political appointees. We don't have enough."

The Senate's own record of complicity in allowing the appointment of unqualified diplomats, and the silence of top Foreign Service officials on such matters, also came in for criticism.

The Mathias bill should not be necessary, former ambassador to Moscow Malcolm Toon told Mathias.

"It would be much better to have a more responsible attitude on the part of the administration as well as, frankly, the Senate," he said.

The Senate really has failed the country" in this respect, said Theodore L. Eliot Jr., a former ambassador, now dean of the Fletcher School of Law and Diplomacy.

Toon created a stir earlier this month when he told the Foreign Service Journal that the U.S. diplomatic ranks abroad are so laden with unqualified "Mr. Klunks" that the conduct of the nation's foreign policy is jeopardized.

Toon slammed several Reagan appointees by name. Referring to John Gavin, a Spanish-speaking actor who is a friend of the president, he said, "We have a Hollywood actor and a not very good one at that" in Mexico.

"We have a man in London," he went on, "who owes his place in life to the fact that his parents founded a furniture polish dynasty . . . ." The ambassador referred to is John J. Louis Jr.

Agreeing that some of our best ambassadors have been political appointees, Toon and other career diplomats urged the Senate to consider the creation of an independent review panel to help the Senate evaluate the competence of noncareer candidates, possibly patterned loosely after the American Bar Association's review of judicial appointments.