Rejecting the recommendations of three other federal agencies, the Army Corps of Engineers announced its approval yesterday of a controversial $600 million oil refinery to be built in southeastern Virginia.

Opponents of the refinery, which would be the largest on the East Coast and the first major refinery to be built there in 20 years, immediately responded with stinging criticisms.

"I've had refineries up to my ears and the Corps of Engineers up to my ears," said Cranston Morgan, former president of the Virginia Seafood Council, one of the groups bitterly fighting the refinery. It would be located on the Portsmouth waterfront, near downtown Norfolk and near the James River's rich oyster beds.

"I think the oyster industry and the shell fish industry have just had their death sentence pronounced," Morgan said.

The corps' decision, made by Lt. Gen. John W. Morris, chief of engineers, will give the project its last needed environmental permit. The decision is likely to be appealed by the Interior Department to Army Secretary Clifford Alexander, who can overturn the permit. Such a step would be unusual, an Interior spokesman said last night.

The Fish and Wildlife Service, a division of Interior, had opposed the re finery because of potential harmful effects on James River oyster seed beds on wintering beds for Chesapeake Bay blue crabs.

The refinery is also opposed by the Environmental Protection Agency and the U.S. Marine Fisheries Service on environmental grounds.

The huge refinery, to be built on the banks of the Elizabeth River, a tributary to the James, would process 175,000 barrels of Persian Gulf oil a day brought in by tankers. It would be built by the Hampton Roads Energy Co., a Washington-based company backed by the Cox Enterprises an Atlanta newspaper concern.

The corps' decision, announced at a hastily called Norfolk press conference, appeared to catch Interior Department officials and Virginia environmentalists by surprise. "I didn't think the administrative decision process had been completed yet," said W. Thomas Olds Jr. of Fish and Wildlife.

In a prepared statement, Morris said: "I have carefully considered and weighed the entire case record and found that the benefits of the proposal outweight the adverse impacts."

He said the need for increased oil production on the East Coast "outweighs the anticipated adverse environmental effects."

The controversy over the refinery has been going on since the early 1970s when Hampton Roads Energy president John K. Evans, a refinery specialist who has developed another refinery in Hawaii, announced the Virginia proposal.

The refinery originally was opposed by the Norfolk Corps of Engineers chief, Col. A. Newman Howard, now retired. But it has been supported strongly by Gov. John N. Dalton, his predecessor, Mills E. Godwin, and Norfolk labor unions.

Tim Finchem, a Virginia Beach lawyer and aide in the apparently unsuccessful Senate campaign of Democrat Andrew P. Miller, arranged a White House meeting in October for refinery supporters to press their case with environmental officials. Army officers and presidential aides. The opponents, who were not notified later protested and were granted a meeting of their own.

Because financial backing for the refinery comes from the Cox family of Atlanta, who have been strong backers of Carter, opponents have charged that their political backers of Carter, opponents have charged that their political influence would favor the refinery. "We feel it has been a political decision," said Mrs. Kirk Clarkson of Norfolk, a leader of the Citizens Against Refinery's Effects, an antirefinery group.

A study by several federal agencies determined that the Portsmouth site would environmentally be the next-to-last choice among a score of possible locations on the East Coast, the opponents said.

"I'm disappointed in the decision," said Robert Hicks of Richmond, whose Conservation Council of Virginia represents 37 groups statewide. "There's no doubt in my mind where the political support was, and it certainly wasn't for the environment."

"We're going to have to review the rationale they used in overruling us," said William Dickerson of the Environmental Protection Agency.

"The Army Corps of Engineers has just made one of the worst decision I've seen in all my years in conservation work," compalined Jack Lorenz, executive director of the Izaak Walton League of America. "The short-term gain of jobs and tax revenue for Portsmouth pales in comparison to the long-term promise of deteriorating air, water, and a vial local (seafood) industry," he said.

A spokesman for the Department of Energy, which supported the refinery, said the department had not been officially notified of the decision yet and could not comment.

Portsmouth Mayor Richard Davis, who elads the refinery's supporters, had argued that the city economy needs the 500 job in refinery would provide.