New York state officials yesterday scrapped construction of a huge nuclear power plant in Cementon, N.Y., citing billions of dollars in anticipated cost overruns and problems with Pennsylvania's crippled Three Mile Island facility.

It was believed to be the first cancellation of a United States nuclear project since accidents and errors at Three Mile Island, near Harrisburg, Pa., posed the threat of extensive radio-active leaks.

The proposed 1,200-megawatt Greene County nuclar power project would have been the state's largest capacity nuclear plant.

The Power Authority of the State of New York said it would seek federal approval to build a coal-burning facility to replace the Cementon platn, located in ahamlet of about 500 persons near the Hudson River, 120 miles north of New York City.

Some excavation had been completed but there has been no plant construction. Reactor equipment had been built for the Cementon site by Babcock & Wilcox Co. - the same firm that built the Three Mile Island reactor.

"Events have recently occurred that will make licensing of nuclear plants more difficult," power authority Chairman Frederick Clark said in reommending that the board sell the assets of the upstate plant. "I believe we should not continue this project."

Clark said that his recommendation to sell the assets fo Cementon preceeded by several weeks the events at Three Mile Island.

The chairman also said the action to junk the plant was necessary because the projected construction bill had jumped from an estimated $1.8 billion to about $3.1 billion since 1977 - with further rises expected by the time the plant was to have gone into service in 1989.

A spokesman siad the authority has not found a potential buyer for the site.

"There's been no proof of any real danger of catastrophe, and I don't think we should react in an overemontional way," said power authority board member George Ingalls, the only member of the five-man board of trustees to vote to go ahead with the Cementon plant. One board member was absent.

Trustee Richard Flynn said the board's decision "involved an agonizing balancing of the various interests." Flynn said he was "extremely concerned" with the effect that canceling the plant may have on future energy costs for southeastern New York, particularly New York City.

There were other actions yesterday related to Three Mile Island and the issue of nuclear power:

In Mineola, N.Y., Kathleen Boylan said she would boycott her husband's place of work and demand a halt in construction of a long Island Lighting Co. nuclear generating station at Shoreham. Mrs. Boylan's husband, Hugh, was appointed Long Island Lighting's vice president for purchasing last week. He said he does not agree with his wife's antinuclear views.

In Springfield, III., Gov. James R. Thompson declared his confidence in atomic power but said he wants radiation monitors placed outside all three nuclear plants in his state. "If we are out front in on nuclear power," he said of Illinois where 32 percent of its electricity comes from atomic plants, "we ought to be out front on precaution."

In Fayetteville, Ark., pediatrician Benjamin Spock joined 150 demonstrators who called for a moratorium on building and operating of nuclear plants. The rally was organized by antinuclear people's action for Safe Energy and its student auxilliary.