The problem would bedevil any academician: Given a finite number of dollars, classroom seats and professors, how does a university balance a dramatic jump in full-time enrollment with the demand for continuing education classes for 2,600 part-time adult students?

At George Mason University outside Fairfax City, the answer turns out to be both simple and painful: Slash by more than a third the number of part-timers in the Continuing Education division's extended studies program so that more full-time students can be admitted.

That decision last month by university officials has divided faculty members and administrators, has angered hundreds of part-time students in the program and has raised questions about the future of Northern Virginia's leading state university.

At the root of it all is what amounts to an embarrassment of riches for George Mason.

At a time when most colleges and universities in the country are starved for applicants, GMU will open its spring semester on Jan. 24 with more than 800 new full-time degree candidates, including transfers from other universities and new freshmen.

To make room for them, top administration officials decided to trim the number of available slots in the extended studies program from an expected 2,600 enrollees to roughly 1,500. A school spokeswoman said the cut will not affect others who are admitted specifically as part-time students in other George Mason University programs.

Last week, as word of the cut spread, more than 200 angry and bewildered students telephoned George Mason's counseling office to find out whether they would be permitted to take classes this semester. About 100 of them were placed on a waiting list with little hope of finding room in spring classes, officials said.

"People are going to be turned away, and that's going to hurt," said Tony Hickey, an assistant dean in George Mason's graduate studies program that includes dozens of extended studies students. "They don't see the bind we're in. They see it as a closing of opportunity."

For Hickey and others, there is a painful irony to President George W. Johnson's decision to make the cuts. Unlike hundreds of other colleges and universities across the country, George Mason has never treated its continuing education program like a poor relation.

"Continuing ed has always been everybody's stepsister, but we never did take that position," Hickey said. "That's how we got our law school: It was the nontraditional students and the graduate people who had some political clout in the community and who went to the General Assembly and fought for it."

George Mason's faculty and staff always have been proud of the close ties the school has forged with the region. Indeed, it was by attracting the sons and daughters of Fairfax and Prince William counties that George Mason grew from an obscure arm of the University of Virginia to the school for the region and became a rival to the University of Virginia.

In just eight years, the student body has grown from 6,134 students to nearly 15,000. Enrollment in the extended studies program, long popular with the Washington area's huge pool of professionals, also has mushroomed, climbing from 819 students in 1974 to 2,670 last fall.

George Mason now gets respect, but, as the current glut of students is showing for the first time, the respect has a price.

"We're glad to see George Mason grow and its reputation enhanced, but it sure is putting a crimp on what we can do here," said Sally Reithlingshoefer, associate director of the continuing education division. "It's almost too ironic."

Last week, several university officials privately said opposition to the cuts in extended studies was fierce--and fragmented. Acknowledged one top administrator: "We didn't want this to happen, but our hands were tied. Things are rather bleak all the way around. It all came down to money."

That view is disputed somewhat by Johnson, who said he based his decision not only on George Mason's finances, strapped like any other college's by the current recession, but also on the university's own priorities.

"Our first priority is the full-time student," Johnson said. Extended studies participants "eat up sections" because they attend classes along with full-time students. "We overdid things this fall in trying to make room for too many full- and part-time students together," he said.

The enrollment cut was "a matter of suddenly being alerted to the fact that the tub was filling up awfully fast and starting to spill over," Johnson said.

"It's easy to speculate on why we had this sudden increase in students: In hard economic times, this area has a fine university on its doorstep, plus the fact that we are now a more highly visible institution. And we benefit from that."

Johnson insisted that continuing education remains "a very important mission of George Mason's in this community. At the graduate level it is extremely crucial to the high-tech industry, which calls continuing ed its 'first priority.' But there was no other place to cut back."

Lisa Meher, 31, of Falls Church, is one of dozens of prospective part-time students awaiting a decision on an application to the continuing education division. Encouraged by instructors in Fairfax County's adult education program to continue her poetry studies at George Mason, Meher applied for a class at the university--only to be notified last week that she may not be able to take the class.

"Apparently, there is still room in this class I want to take, but it looks like I won't be admitted" because of the ratio of full- to part-time students, Meher said. "I don't see any point to closing a classroom to somebody who's willing to pay for that class.

"I squeezed what I had in my checkbook for that class. It's a ridiculous situation."

George Mason administrators said the heavy enrollments should ease in time for summer courses, but they and other education experts are uncertain whether cuts in extended studies will be needed in future semesters.

"George Mason is really an anomaly," said Kenneth G. Young, executive director of the National University Continuing Education Association in the District.

"The growth in full-time students is contrary to what's happening every place else. I don't know whether they're experiencing an unusual demographic bulge or what, but I bet that influx of full-time people doesn't continue."