IN 1976, WHEN large numbers of shellfish and plants began to die off the coast of New Jersey, no one seemed to be in charged of finding out what had happened.

Marine biologists ran tests and learned that there was virtually no oxygen in the water near the ocean bottom from just south of Long Island nearly all the way to Delaware. When asked, various authorities blamed offshore dumping of sewage sludge near New York in an area known as the New York Bight.

They also blamed chemical slurries, or the bilge from some passing chemical cargo vessel, or underground seepage or a red tide. The fact was that not only did no one know then or knows now what caused the situation, or whether it would recur, but no one was responsible for whether it would recur, but no one was responsible for knowing.

Murk in the nation's coastal waters is often surpassed only by the murk around jurisdiction for them. With 80 per cent of the population expected to live within 50 miles of the coasts by the year 2000, state governments have burgeoning problems in that area and little research capability even to understand them. Federal activity, meanwhile, seems to continue with little or no regard for the changing situation.

Thousands of small pleasure boats zoom in and out of major shipping lanes, colliding by the hundreds every year in a national tragedy that is barely even documented. The seagoing traffic is particularly heavy in some areas, like the Santa Barbara channel in California, where Washington is preparing to offer leases for offshore rigs involved in the still-growing national thirst for oil.

In Santa Barbara, the facilities would unload Alaskan crude from giant tankers while other vessels maneuver in and out of the harbor. "It'll be almost like running a slalom course through the docks," said Dail Brown, assistant director for critical area planning at the National Oceanographic and Atmospheric Administration (NOAA).

The question of states' rights versus federal laws has not previously meant arguments over sewage sludge, offshore oilwells, king crabs or supertanker routes, but it does now. It also means genteel struggle over damage claims, royalties, revenue sharing and manganese nodules, which are just the latest of the riches in and under the restless sea.

"More and more players keep coming into the ball game and the rules are constantly changing," observed Phillip Clark, coastal zone management coordinator for the American Petroleum Institute, the oil industry trade association. "There's no real way to resolve conflict . . . it's a real limbo-land of regulations." Conflicts in the Courts

THE SENATE this week considers final passage of a measure of license U.S. private exploration and mining of those manganese nodules, multimillion-dollar lumps of ore just waiting to be scooped up off the sea floor as much as 5 miles down. Since the ore occurs under the open sea, 200 miles or more from U.S. coasts, one would think there wouldn't be much interest at the state level. Not so.

"There could be environmental disruption with currents we don't know about yet. The ore has to be processed somewhere on shore; it has to be transported somehow. Does that mean pipelines or what? There will be new job markets and new taxes to pay; who gets them?And what about dividing up the profits?"

The speaker is Dean Rusk, former secretary of state and now a self-described lone voice warning from his post at the University of Georgia law school that federal-state relations are going to be a major problem in agreeing on the laws of the sea.

"There's a point at which a lack of uniformity among the states could impose a very heavy burden on anyone trying to work out there," he said.

At the moment, most disputes are settled in lengthy court battles that hinge on defining federal powers granted by the Constitution. Washington, for example, has just banned the dumping of sludge from treated sewage anywhere offshore after 1981.

Coping with the change could cost the states millions of dollars in new sewage treatment plants."That's a tremendous problem for them," said Robert Kencht, assistant administrator for NOAA's Office of Coastal Zone Management. "What are they going to do?" Several states are contemplating lawsuits.

Recognizing that 3-mile limit to coastal waters no longer means much, NOAA this week merges its office of Coastal Zone Management and its Oceans Management office to bring them both under Knecht. He and others say that the problems in this area are still being defined and little has been done to solve them.

"The states are being drawn into a rising number of coastal ocean issues [and] the forcing element is outside the states," Knecht said. "Often the argument isn't even over the policy itself but over jurisdiction in deciding."

An example of that occurred in Washington State, where federal action almost by accident has all but settled years of painful state legislation and legal wrangling that went all the way to the U.S. Supreme Court. The high court last year overturned former Gov. Daniel Evans' ban on oversized tankers in Puget Sound, but only on grounds that a state cannot regulate interstate commerce. Sen. Warren G. Magnuson (D-Wash.) has hopes of virtually reinstating the ban through new Coast Guard regulations, which may control such commerce.

Magnuson is against any new oil transshipment port in the Sound, but the Department of Energy is considering a proposal from the Northern Tier Pipeline Co. to build one near Port Angeles and run a pipeline for Alaskan and foreign oil from there east to Clear Brook, Minn.

"One could imagine a situation where the feds say we need the pipeline and the state says, Oh no we don't, and there's nothing the state can do," said a Washington resident watching the situation.

In another case, the state did lose. New Jersey's 1976 challenge to federal offshore oil drilling licenses was at first upheld in U.S. District Court on grounds that state interests in the inevitable pipelines and other possible disruptions had been ignored. But the 2nd Circuit Court of Appeals reversed the verdict in August 1977, saying a possible state veto later could not stop the drilling now. "There comes a point where the chain of 'ifs' gets too long and too tenuous to be of any practical use," said Judge Walter Mansfield. The "Consistency" Doctrine

WHAT LEVERAGE the states have in the future will probably depend on court interpretation of the concept known as "consistency" that was established in the Coastal Zone Management Act of 1972. The measure provides funding for states to come up with a coherent plan, following federal guidelines, for their coastal lands and waters out to 3 miles, and then decrees that any federal action afterward must show "consistency" with the state plan "to the maximum extend practicable."

Since anything done anywhere in the ocean is likely to wash ashore sooner or later - in the form of pipelines through areas the state wants as beaches, for example, or in the form of jobs or refineries or even oil spills - the states are newly able to speak up.In one such case, Alaska's Supreme Court upheld the primacy of the state's restrictions on the taking of king crabs over Washington's looser standards for areas outside the 3-mile territorial sea.

"The 'consistency' doctrine gives the states a brand new grant of power on the outer continental shelf, and we are the ox that gets gored," said the oil industry Clark. Only 13 states and territories of the 34 with coastlines have so far come up with approved coastal zone management plans; and the oilmen are unhappy with most of them.

The petroleum institute took Massachusetts, Wisconsin and California to court on grounds their plans paid inadequate attention to "the national interest" as required in federal guidelines. The oilmen, of course, meant that the states were excessively restrictive on future oil development.

The industry lost in all three cases, only the California case being tried on the merits. However, the judge complained that the ambiguous law had obviously "befuddled" bureaucrats and recommended that the oilmen seek remedial legislation. "That's no remedy at all," Clark said.

The recent Outer Continental Shelf Act amendments require more state consideration and information as federal agencies draw up their plans, and also create multimillion-dollar damage and impact compensation funds.

States are also involved in regional councils set up to coordinate fishing in the newly declared 200-mile economic zone, but all of these measures are too new to have had any impact yet.

"What we really have is only the blunt instrument of yes or no so far," said William S. Gaither, dean of the College of Marine Sciences at the University of Delaware. He wants to set up regional coastal councils that would take over federal decision-making functions, a controversial notion as best.

As the states grope for a new relationship, with Washington, the seaward traffic continues to increase and the problems get more complex.

"Clearly we're moving toward a time when there's going to be some kind of mandatory shared use of marine resources," said Knecht, "zoned like Cape Hatteras is for beaches, water skiing, boats and fishing . . . but that's a long way off."