More than a third of the graduating seniors in American University's College of Arts and Sciences staged a silent protest yesterday at their commencement angered by the choice of George Gilder, favored economist of the Reagan administration supply-siders, to deliver the main address.

The students, joined by a dozen faculty in the audience and two of the 17 campus officials on the platform, wore white armbands on their robes to protest their displeasure with Gilder, whom they labeled a racist and sexist for some of his writings.

But yesterday the protesters -- including 50 who stood and turned their backs during his 15-minute speech -- didn't hear any of Gilder's views on race or sex. They heard only a discourse on the virtues of small business and the decline of large corporations as the centers of innovation and creativity in the economy.

It is that economic philospophy extolling capitalism and small business that has endeared Gilder to many in the Reagan administration. OMB Director David Stockman has said Gilder's new bestseller, "Wealth and Poverty," is "Promethean in its intellectual power and insight."

It was that conservative philosophy and Gilder's newfound notoriety that led university Dean Frank Turaj to invite him -- over the objections of many of the 500 students who attended the graduation ceremonies. About 675 students earned degrees from the College of Arts and Sciences.

Many students, even those not wearing armbands, said they objected to what they called the conservative dean's use of their graduation ceremony to invite a speaker to his liking, without consulting faculty and students.

During his introductory remarks, Turaj issued a combative denial of the charges against himself and Gilder. Turaj said the charges against Gilder are "unsubstantiated."

He also said he had discussed the invitation with other faculty and said the speaker he selected last year was former Carter administration spokesman Hodding Carter. He reminded the audience they were in Constitution Hall, "the home of freedom of speech."

Parents of some students involved in the protest said they had no objections to their children's actions. "That's her business," said Ann Schenck of Charlotte, N.C., whose daughter, Ann Shirley Hunter, turned her back to Gilder during the speech.

Gilder, who got $2,500 for the appearance, said afterwards that the protesters "didn't bother me at all. They were very polite and allowed me to speak -- what more could I want?"

Protest leaders passed out leaflets criticizing Gilder for saying things such as: "There is no such thing as a reasonably intelligent feminist" and "there is little evidence that black women suffer any discrimination" and "differences between sexes fully explain all gaps in earnings."

Gilder complained afterwards that the students were protesting his earlier writings on women and blacks and "took quotes out of context" because they "just want to misrepresent me. I am not a racist." He said he had been told there would be a protest, but that he did not alter the speech in any way.

After the ceremony, Turaj said that "after this flap," future speakers may be chosen by committee, although he said he was "absolutely glad" he had selected Gilder.

"It proved a lot of things," Turaj said. "It proved that intellectual integrity has something to fear from the self-satisfied, smug, old-fashioned, knee-jerk liberal atmosphere on most campuses today."

Jan McGovern, one of the protest organizers, said he was "proud" of the large number of students who joined the protest at the normally placid university. "It takes a lot of guts to stand up on graduation day in front of parents and friends and do this."

About 700 students from other departments in the university received degrees during ceremonies at different locations and times. Speakers at the other commencement activities were: Virginia Lt. Gov. Charles S. Robb, who spoke to graduates of the College of Public and International Affairs; U.S. Court of Appeals Judge Ruth Bader Ginsburg, who spoke to the law school graduates; Herbert E. Striner, dean of the Kogod College of Business Administration, who addressed the business school graduates; Doris Mosley, former dean of the Lucy Webb Hayes School of Nursing, who addressed the graduates of that school; and WRC-TV anchorman Jim Vance, who spoke to graduates of the Community Studies program.