In a voice trembling with emotion, Sen. Edward M. Kennedy (D-Mass.) yesterday paid tribute to his brother, Robert, on the occasion of the first Robert F. Kennedy Memorial Human Rights Award -- a $30,000 prize to a Salvadoran organization -- which has already caused a political stir.

The day would have been Robert Kennedy's 59th birthday.

"He had that special grace, a full heart, the fundamental wisdom of caring for others," the senator told the 700 guests at Georgetown University's Gaston Hall.

His hair grayer and his face fuller and redder than usual, Kennedy spoke of his slain brother in words he has often used over the past 16 years. He spoke slowly and deliberately until the very end.

"Tomorrow and tomorrow, we will renew the faith of Robert Kennedy that someday the walls of indifference and inhumanity shall be swept down," he said, his voice beginning to vibrate. "For in a year when the most destructive missiles can span the globe in a matter of minutes, his shy, insistent voice is still heard around the world, across the years, calling to us: Some men see things as they are and ask why. Other men dream things that never were and ask why not.

"And so he gave himself," Kennedy said, "and he was there even when a cause was unseasonal or unpopular."

This could be one of those causes. Organized by Robert Kennedy's eldest child, Kathleen Kennedy Townsend, the award was given against a backdrop of controversy surrounding the first recipients.

The Reagan administration had denied entry visas to four of the five members of, the Co-Madres, who are the mothers and relatives of Salvadoran political prisoners of the civil-war-ridden nation.

The Reagan administration supports El Salvador's government, which is battling rebel forces, and critics maintain the administration wants to minimize reports of the violence occurring regularly in the country.

One woman, Emelina (Alicia) Panameno de Garcia, was granted a visa to accept the prize, which she did with a chilling address about the cruelty and savagery inflicted on her and her family. Four large red leather chairs sat empty on the stage to dramatize the absence of the other award winners.

"You would think our country would embrace these people," said Townsend, "for what they have been through . . . I think they did not want anyone here that might be critical of the U.S. government."

"This is not an isolated incident," said Steven Shapiro of the American Civil Liberties Union office in New York. "It is the latest example of a concerted strategy by the Reagan administration to keep foreigners who are critical of its Central American policies out of this country."

The State Department released the following statement in response:

"One member of the Salvadoran mothers' committee has been found eligible for a visa. Four members of the committee were found ineligible to receive visas under section 212 (a)(28)(F) of the Immigration and Nationality Act, which prohibits the issuance of visas to people who advocate the use of violence. The decision was based on the applicant's personal advocacy of acts of violence, and actual participation in terrorist activities."

Those denied visas were: Maria Teresa Zula de Canales, Elivia Cosme Hernandez Garcia, Consuelo Amanda Lemus Salinas and Dilia Haydee Melgara Alvarado.

"We cannot discuss the particular information except to say that these activities took place over periods of time," said Kathleen Lang, a State Department spokeswoman. "The visas were denied for the reasons given -- personal advocacy and participation, on the basis of their individual actions."

Lang added: "These visa decisions have not prevented the presentation in the United States of a full range of political positions espoused by organizations of which the various applicants are members."

Emelina Panameno de Garcia, speaking through an interpreter, said that Co-Madres was started in 1977, after murders, kidnapings and other acts of violence by Salvadoran death squads. She told the audience that she had been raped by eight men and sexually brutalized with the butt of a rifle.

Said Townsend in her closing remarks: "My father would have been proud of this international human rights award . . . He told us that government by fear is unacceptable."