The Alaska lands bill turned into a textbook cliffhanger yesterday as senators negotiated on its future behind closed doors.

Unable to get beyond a shouting match on the floor Wednesday over disposition of 2.5 million acres, a minor detail in the massive bill, senators agreed to battle privately toward some type of truce.

Talks at the staff level ran most of Wednesday night and the senators themselves took over yesterday, meeting in the office of Senate Majority Leader Robert C. Byrd Jr. (D-W.Va.). Lobbyists and aides hovered outside, exchanging rumors.

At stake in what President Carter has called the environmental vote of the century, one that would assign varying levels of environmental protection to a third Alaska, between 102 million and 125 million acres.

Sen. Henry Jackson (D-Wash.), Mark Hatfield (R-Ore.) and Ted Stevens (R-Alaska) worked to defend Jackson's Energy Committee package against what appeared to be superior voting strength for a tougher environmental version on the side of Sens. Paul Tsongas (D-Mass.), John Chafee (R-R.I.), Alan Cranston (D-Calif.) and Gary Hart D-Colo.).

Democratic Sen. Mike Gravel, Alaska's other senator, moved in and out of the talks, arguing against any bill and hinting that opponents might do better to wait for a possible Republican administration.

"If we can stop it this year," he told reporters, "we can start with a better climate next year." The prospect of President Ronald Reagan, he said, was "one of the factors" that ought to be considered. He emphasized he was neither endorsing Reagan nor criticizing President Carter, who is a strong advocate of the Tsongas view.

Stevens has said he wants a bill this year but cannot stomach any of Tsongas' amendments, which he calls an effort to lock up the state against further energy development. His forces were reportedly stunned Tuesday when two test votes went heavily against them.

The environmentalists argued that 95 percent of available energy lands will remain available under the Tsongas amendments and that this is the last chance to protect a virgin land in advance, rather than trying to repair the damage later. They were exultant over Tuesday's votes.

"Stevens is licked and the question now is giving him something to save face," said Elvis Stahr, past president and general counsel of the Audubon society, one of 52 groups making up the Alaska Coalition of self-proclaimed environmentalists. "We got all the waverers and a few more we didn't expect," he said.

But Stevens changed gears and threatened to join Gravel in filibustering the measure to death unless he got some concessions. When he got into a 10-minute war of words with Hart over 2.5 million acres allegedly promised to the state, the bill was set aside so yesterday's talks could be held.

All sides agonized in the Senate lobbies over what chips might be getting bargained away inside the meeting.

"We wanted a bill but it's got to be one that addresses some of our concerns, and we just aren't sure that's possible now," said John Katz, chief lobbyist for the state of Alaska. Gov. Jay S. Hammond was reportedly leaning toward asking Stevens to join a filibuster rather than agree to even a modified Alaska Coalition version.

Evironmentalists, on the other hand, were resisting concessions to Stevens on the theory that they have most of the lands under the kind of protection they want right now, even without a bill, and don't need to give an inch. President Carter and Interior Secretary Cecil D. Andrus together put 91 million acres into national monuments and wildlife refuges earlier this year in order to force Senate action, and there the lands will stay until Congress acts.

"We want a bill but we're not desperate," said coalition head Charles Clusen before the debate began.

By late evening, some progress had been reported toward reaching agreement on four of the five Tsongas amendments, although all sides said they would wait to see the details before passing judgment.

The possibility remained that environmentalists might move to kill what they regarded as an unacceptable substitute.

Any successful approach would reportedly consist of a single package to replace all five Tsongas amendments, which would be brought to the floor and voted on without delay. Stevens told reporters there most likely would be no more floor action until Monday, and appeared to rule out the possibility of joining a filibuster, at least for now.

"It's a tortuous pass but we can still get a bill," he said. "I think we could go on a week or two this way and still be effective."

Emerging from the meeting later, Jackson said there had been some progress. "We're determined to get a bill. Byrd is adamant," he said.