A D.C. Superior Court judge urged Georgetown University officials yesterday to consider setting aside their plans to close the university's 87-year-old dental school, but his request was firmly rebuffed by a university lawyer.

"I urge, I beg you to look deeply into this problem," Judge Eugene N. Hamilton said during a court hearing. "It wouldn't hurt anything to sit down and talk to each other . . . . You see, the courts should not be doing things like this if we can avoid it."

But J. Alan Galbraith, the university's attorney, replied, "There is no possibility that the university would or could change its mind about reopening the dental school. The decision has been made . . . ."

Earlier, the university's president, the Rev. Timothy J. Healy, told the judge that if he issued a preliminary injunction to block the closing, the university would appeal the order.

The university's board of directors, acting on Healy's recommendation, decided in March to phase out the school by mid-1990, citing a looming "financial disaster." No first-year class was enrolled this fall, and deposits were returned to the 120 students who had been admitted. Second, third, and fourth-year classes are continuing.

Students and faculty members at the dental school sought a court order to require the university to enroll a first-year class, starting in January.

The university has attributed the planned closing to a nationwide decline in the number and quality of students going into dentistry.

In a related development, John Greenbaum, an associate vice president for the university's medical center, testified that a study of Georgetown's Nursing School by the Price Waterhouse accounting firm had "indicated concerns about enrollment . . . and future financial projections . . . very similar to the dental school report." But he added that there has been "no final decision by the board" on the Nursing School's future.

Medical center spokeswoman Cynthia Byers later said closing the nursing program is "not under consideration at this time."

The suit by dental students and faculty members charges Healy and the directors with breach of contract and breach of their fiduciary duty. It contends that the decision to close the school was made with "limited discussion" and in violation of university rules, which mandate consultation with the Faculty Senate.

The school had 570 students last spring and operates a large dental clinic.

Executive Vice President John F. Griffith testified that a Price Waterhouse study last fall projected an annual deficit of $3.6 million by 1992. In December, the board ordered the faculty to prepare a plan to cut costs and raise revenue, but that plan was rejected by Griffith and Healy as unworkable.

Yesterday, Healy said that faculty members had not been told about deliberations over closing the school to prevent news from spreading. "The university's internal business is not for press release," Healy said. "I wanted the board to have as free a terrain as they could."

At the hearing, several witnesses argued that Georgetown might have scaled back its dental program, as many other schools are doing, without closing it.

A Faculty Senate resolution, introduced at the hearing, complained that university officials had violated rules by closing the school without consulting the faculty. Healy argued that "when no more than one school is involved, it is not grist for the Faculty Senate mill."