Maine citizens voted overwhelmingly yesterday to keep open the Maine Yankee nuclear power plant, rejecting a proposal that would have banned nuclear power in the state.

The vote with 660 of 665 precincts reporting was 230,708 votes to save Maine Yankee and 159,761 to shut it down, a ratio of 3 to 2.

It was the first time the fate of an operating plant had hung in the balance of a popular vote, and Mainers took seriously the national attention focused on the outcome.Turnout was unexpectedly heavy in fair weather, surpassing the 1976 presidential election tally in many areas.

Supporters of the eight-year-old plant were jubilant. "Maine and its people are the winners in this referendum, and we feel reassured by the vote of confidence," said Edwin W. Thurlow, president of Central Maine Power Co., which owns 38 percent of the reactor. "We're grateful that voters have confirmed our own belief that Maine must use all of its viable energy resources in order to assure a healthy economy and a quality environment."

The pro-nuclear forces outspent the plant's critics five to one during the intense media campaign and widespread public debating sessions over the last few months. Raymond Shadis of Edgecomb, who organized and led the referendum drive, said his side had made a good showing despite the defeat.

"What we're seeing is the people of Maine giving the nuclear industry and the energy industry in this country one hell of a kick in the pants, regardless of the outcome," he said. His wife, Pat, added, "We would have won easily with equal resources. Many people made their choices on lack of information."

The Committee to Save Maine Yankee disclosed spending of more than $800,000 as of Sept. 12, compared to the Maine Yankee Nuclear Referendum Committee's $179,000. A large proportion of the pronuclear funding came from out-of-state sources, primarily other public utilities.

The 840-megawatt power plant got solid support in the cities and larger towns, carrying its own town of Wiscasset 966 to 481. Neighboring Edgecomb, where the Shadis family lives, voted to shut down the plant, 239 to 227.

Supporters of Maine Yankee argued that closing it would cut off a third of the power supply to the low-income state, forcing residents to cough up about $140 million to pay for replacement power. Television and newspaper advertisements stressed a trippling of utility bills that would follow, telling the already hard-pressed citizens that a plant shutdown would discourage new industry from coming to the state. Although the state Democratic platform supported the drive to close Maine Yankee, not a single major state politician endorsed it.

Shadis and the antinuclear activists relied on volunteer labor and the familiar Maine small forum debates night after night to make their points. They stressed their worries about the safety of nuclear power in general, charging that a major accident at the costal site would wipe out Maine's economy.

The state, they argued, had a unique opportunity to point the way for the nation in replacing the lost nuclear energy with increased hydropower generation, conservation and solar units.

They challenged the pro-nuclear figures, saying replacement power would cost much less than $140 million. However, as Paul Liscord of the Portland antinuclear organization said "We were bucking the tide of decades of nuclear promotion."

The referendum process is a familiar one in Maine. This one began in March 1979 when Maine Yankee was one of five power plants shut temporarily nationwide while their earthquake resistance was rechecked. Shadis said his concern grew after the accident at Three Mile Island in Pennsylvania, and a march to the state capitol in July launched a petition drive to get the question on the ballot.

More than 55,000 persons signed positions to win the single-issue vote. There was just one thing for voters to decide yesterday: "Shall an act to prohibit the generation of electric power by means of nuclear fission become law?" They voted no.