Howard University president James E. Cheek, facing an angry crowd of about 500 students protesting the expulsion of the campus newspaper's editor, declared yesterday that he would not give in to student leaders' demands that he resign.

The confrontation took place at an outdoor rally around a flagpole, with Cheek and Howard University Student Association president Howard Newell speaking to each other through an electric bullhorn that they passed back and forth. The crowd occasionally jeered Cheek and some raised their fists.

Meanwhile, the ousted editor, Janice McKnight, a senior from Northeast Washington, filed suit in D.C. Superior Court seeking reinstatement as a student and as the newspaper's editor, as well as $100,000 in damages. In court papers, McKnight said she was expelled because she printed articles detailing sex discrimination charges filed by a male staff attorney against the university's general counsel.

The university has said McKnight was dismissed because she gave "untruthful" information on her admissions application in 1979.

At yesterday's rally Newell demanded that McKnight be reinstated and accused Cheek of curbing free expression and "intimidating" faculty members. He also blamed Cheek for what he called "the mediocrity that has crept up on this university" and denounced him for developing "a close nexus" with the Reagan administration. The federal government supplies about three-quarters of Howard's academic funds.

"You have done well at fund raising," Newell said to Cheek, who was standing just a few feet away. "But we are tired of the cover-ups and of pushing things under the rug. We believe your effectiveness as president is over. Will you resign as president of Howard University?"

Then he handed the bullhorn to Cheek who said: "You know, Howard Newell, the answer to that question is 'no.' "

When someone in the crowd asked if McKnight would be allowed to return, Cheek said he could not answer because of her lawsuit.

"If there had been any way to prevent what has happened I would have done it," he said. "I am awfully sorry."

As boos came from the crowd, Cheek added, "The division of student affairs handled McKnight's case the way it has handled all other cases. That's what was advised by our outside legal advisers, and we followed that advice."

Several hours before Cheek appeared at the mid-afternoon rally, Newell and other student leaders denounced the university president at a meeting attended by about 750 students in Cramton Auditorium.

Several students recalled that Cheek, now 50, was named Howard's president in 1969 following the resignation of James Nabrit in the wake of widespread student protests. One of the leaders of those protests was Michael Harris, then a student and later the attorney for Howard who filed the sex discrimination complaint that was the subject of the articles McKnight published in The Hilltop newspaper.

Harris, who filed his complaint with the Equal Employment Opportunity Commission Nov. 1, was fired by Howard last week for "activities . . . at odds with the university's objectives."

When Cheek first became president, he established good relations with student leaders, including Harris. But yesterday Lori Johns, a senior, told the meeting in Cramton Cheek "has been here for 13 years, and now he thinks only he can run Howard University."

Newell said Cheek and his aides were invited to speak in the auditorium at noon. But instead of appearing there, he said, the president asked about a dozen student leaders to meet with him in his office at 2 p.m.

Newell said the leaders would not meet with Cheek by themselves and he urged other students to come along. About 250 did, overflowing a room next to Cheek's office. The president sent word that he would talk to the whole group in a small auditorium in the School of Social Work, and went there with some aides. But Newell and most of the students said they wouldn't come where Cheek wanted them to be, and instead walked outside to the flagpole.

Cheek then walked to Cramton Auditorium, and said he would talk to students there. But as the flagpole rally grew, Newell and the crowd demanded that Cheek come to them.

A few minutes later Cheek arrived, accompanied by a security guard, an aide, and his driver. He explained that he had not appeared at the noontime meeting in Cramton because the invitation sent by the student government was not signed. "We cannot have a reasonable discussion in this kind of atmosphere," he said, but then Newell started his denunciation, and the exchange between the president and the student leader ensued.