The Northern Indiana Public Service Co. yesterday abandoned plants to complete the Bailly nuclear power plant near Gary in the nation's first cancellation of an atomic project under construction.

Environmentalists were jubilant, claiming their repeated lawsuits over the much-delayed plant's location and structural plans had convinced the company to give up.

"It's very exciting. It's a wonderful victory for citizen efforts to assure the safety of the nuclear power industry," said Diane Cohn, attorney for five challengers to the plant, including the city of Gary.

Edmund A. Schroer, the utility's chief executive officer, agreed in a formal statement that delays had killed the plant. "Ultimately it became a victim of one lengthy delay after another," he said.

When first proposed in 1967, the 644-megawatt plant was expected to cost $187 million and be finished in 1976.

"Because of repeated intervention after the construction permit was granted, the latest cost estimate, based on a 1979 service data, was increased tenfold to $1.815 billion," Shroer said.

Delays that would have extended completion until 1992 would have raised the cost to $2.3 billion, "due wholly to the effect of inflation and the time-cost of money," Shroer said. "Cost increases of this magnitude are simply not bearable," he said.

The plant site, seven miles east of Gary at the edge of the Indiana Dunes National Lakeshore and adjacent to the Burns Harbor Bethlehem Steel mill, was challenged almost as soon as it was chosen. The major complaint was its proximity to a major population center, Chicago, about 30 miles northwest of the site.

After lengthy hearings, which produced an 11,000-page transcript, the Atomic Energy Commission approved the site in 1974. Citizen groups challenged that, and the U.S. Supreme Court upheld it in 1976.

Then came more challenges to the construction methods and structures to be used at the plant.

A coalition of the city, the local steelworkers' union and two environmental groups called Save the Dunes Council and the Bailly Alliance charged that the sandy lakeshore soil required much deeper pilings than the utility proposed. The "short pilings" issue delayed construction further.

The utility filed a petition to extend its construction permit but, after the nuclear power plant accident at Three Mile Island in March, 1979, the Nuclear Regulatory Commission issued criteria for siting new plants.

Bailly appeared to violate the criteria, and the same coalition of citizens' groups, joined by Ralph Nader's Critical Mass Energy Project, filed objections to the renewal of the construction permit.

The alliance argued that evacuation plans for Gary were inadequate and that a wider area around Gary could not be notified of any accident or evacuated safely within a reasonable period of time.

James Keppler, NRC's Midwest regional director, wrote to the NRC last January expressing concern about the technical questions and the age of the construction plans. He said consideration should be given to dropping the plant.

The NRC resolved the short pilings issue earlier this year, ruling that the utility could continue construction. But it failed to call a public hearing on the issue and, when the citizen groups objected, a circuit court ordered construction halted until a hearing could be held.

Last week, the NRC decided not to act on coalition petitions to halt extension of the construction permit, in effect passing the buck to NRC staff director Harold R. Denton. Denton said yesterday that he had not decided how he would have ruled.

A utility spokesman said yesterday that $205 million has been spent on construction, litigation and materials and that the site is less than 1 percent complete.

"Basically, it's a hole in the ground with a few pilings," he said. Coal and oil-fired electrical generators are at the site, but new equipment bought for the nuclear plant will be sold, the spokesman said.

Jack Weinberg of the Bailly Alliance called the decision "A great victory for the people in northwest Indiana." He said it meant "that we know we can make a difference and keep our community safe."