Joel Berliner, a student at the College of William and Mary here, has a simple philosophy about college football: it's not whether you win or lose, but how big the stadium is.

He thinks it's big enough.

But the board of visitors at the prestigious college, the nation's second oldest and the alma mater of Thomas Jefferson and four other U.S. presidents, disagrees. Last December, while students were home for Christmas break, the board voted to double the size of modest, red brick Cary Field to 30,000 seats and 70 feet in height.

Since then, the chorus of boos and jeers against the $4-million decision has been rising.

"The old stuff hit the fan," said Glen Shivel of the Association for the Preservation of Williamsburg, which is fighting the expansion. "It was a big shock because the stadium is right downtown. Any expansion is going to be a monstrosity."

"Downtown" here means the college's own historic campus and, side by side with it, Colonial Williamsburg, the 18th-century restoration funded by the Rockfellers and a busy seasonal tourist center. Cary Field is within walking distance of both.

"Some people say we've had more spirit in this than we did during the Vietnam war moratoriums," said college spokesman James Rees. In fact, nobody can recall so widespread a flap at William & Mary since 1951, when then-college president John E. Pompret resigned during a scandal involving special admissions and grading policies for athletes.

To the stadium's student foes, the issues now are much as they were then. "The board of visitors is trying to change the thrust of this college, which has been known for academic standards," said Berliner, a senior (from Fairfax.)

He heads a group called the Amos Alonzo Stagg Society named after the legendary college coach who invented the huddle and the onside kick but opposed vestiges of professionalism in college athlletics.

"We feel it's going to destroy the whole character of the school, because it's going to put athletics before academics," said Larry Fineran, a student member of the Young Republicans.

The supporters of stadium expansion disagree. "A lot of people have gotten mixed up on what this is all about," said Dr. George J. Oliver, a local resident and football fan. "There seems to be some feeling that excellence in athletics and excellence in academics are two things apart. They're not."

And men's athletic director Ben Carnevale argues that the football team, which now plays seven of its 11 games on the road, can't get more home games because the 15,000-seat stadium doesn't generate enough gate receipts to draw good teams, which get a share of the money.

The controversy began, opponents say, because of the way the board made its decision. The December meeting occurred while most students were away. "They left every impression they were trying to sneak one by," Berliner said.

"They approached this thing without saying a word to anybody in town," said Richard B. Sherman, a faculty opponent who lives near Cary Field. "It antagonized a number of faculty members who like football."

Much of the community opposition centers on the character of Williamsburg, a town of subdued charm where the red brick buildings and cobblestone streets are imbued with history. The impact of William and Mary on the town is already enormous: If the college vanished, the population would be halved to 5,000.

James McCord, a history professor at the college who lives two blocks from the stadium, says the expansion poses "a danger of crushing" the city. Others point to the traffic problems they say the stadium would cause.

Furthermore, college officials concede that they rarely fill the current stadium seating capacity-the average attendance for the four home games is less then 14,000-although Carnevale says that a larger stadium would draw better opponents and thus larger crowds and more money. Opponents challenge his financial projections, but Carnevale says that unless the school builds a larger stadium, the football team faces the danger of losing its Division 1A status in the National Collegiate Athletic Association.

The opponents say that doesn't matter.

"Faculty members are already leaving because they're not getting paid enough," said Fineran, citing national figures that put the state-supported college in the bottom quarter of colleges and universities as far as faculty compensation goes.

Expansion advocates say the funds for the stadium probably would come from donors who are not interested in giving to other college programs. But the opponents say that a larger stadium would mean pressure for better teams, which in turn would force William and Mary to recruit players who would not meet the school's present standards.

"I do go to football games, and I like football," said Berliner. "However, I would rather see William and Mary play James Madison and have a good time than to play Tulane and get smashed."