The State Department said yesterday it is taking "seriously" a threat by key Senate Democrats to derail early approval of the new treaty to eliminate intermediate-range nuclear forces (INF), but expressed confidence that the underlying dispute over the Senate's constitutional role can be resolved.

Department spokeswoman Phyllis E. Oakley said Secretary of State George P. Shultz is "studying . . . carefully" an angry letter on the constitutional dispute sent late Friday night by Senate Majority Leader Robert C. Byrd (D-W.Va.) and Senate Armed Services Committee Chairman Sam Nunn (D-Ga.).

Apparently trying to dampen concerns that the rift over procedures for the Senate's review of the treaty could jeopardize its ratification, Oakley said "we have been in touch with the Senate on these issues . . . {and} we are confident the concerns of the senators can be satisfied."

However, there was no indication yesterday from congressional or Reagan administration officials of how the bitter dispute might be settled.

A White House official, who asked not to be identified, raised the possibility yesterday that the Democratic leadership of the Senate is trying to erect "a roadblock to the treaty ratification," and, if they do, "it is an issue we will take seriously."

The White House official described the legislators' threat to delay Senate ratification as "peculiar" and said "we do not see it as something the American people will support." The official added, however, that "we are endeavoring to cooperate in every way with the congressional committees" reviewing the INF Treaty.

At stake in the dispute, which erupted publicly after Shultz broke off negotiations with the Senate Democrats late last week, is an assertion by the Reagan administration that congressional testimony given by officials of the executive branch during treaty ratification hearings may not reflect a treaty's true meaning.

Nunn and Byrd argued in their letter to Shultz that this theory threatens to undermine the significance of treaty approval by raising questions about what the Senate is actually approving. It also allows a subsequent administration to claim the treaty meant something different from what senior Reagan administration officials now claim.

They said that as a result, the Senate must "exhaustively" scrutinize the INF negotiating record and write its own understanding of the treaty's meaning -- a process that will be "unwieldy and time-consuming" and "could undermine the . . . ratification process."

Both sides agree that the genesis of this debate over legal principles is the Reagan administration's controversial "broad" or permissive reading of the 1972 Antiballistic Missile (ABM) Treaty, a reading that Nunn and Byrd vigorously oppose. The administration sought the "broad" interpretation to justify highly realistic U.S. space tests of "Star Wars" missile-defense technology, which would otherwise be prohibited by the ABM Treaty.

State Department legal adviser Abraham D. Sofaer first advanced the idea that executive branch treaty testimony may not be authoritative in an effort to explain Senate testimony by key officials in 1972 that clearly supported a more traditional, "narrow" reading of the ABM Treaty.

In a statement that outraged Byrd and Nunn, Sofaer told the Foreign Relations Committee last March that when the Senate "gives its advice and consent to a treaty, it is to the treaty that was made {in agreement with another party} irrespective of the explanation" provided to the Senate at the time by executive branch officials.

The committee responded in a report last fall that Sofaer's "assertion that the president is not bound by the Senate's understanding {is} . . . thoroughly unpersuasive constitutionally {and} . . . potentially devastating to the ability of the United States to make new international treaties."

Nunn added that it "would inflict serious long-term damage on the institutional relationship between the president and the Congress," and he and Byrd began meeting with Shultz in an effort to draft a letter on the INF Treaty that would have effectively repudiated Sofaer's view.

The most recent such meeting, on Monday, included Sen. Claiborne Pell (D-R.I.), chairman of the Senate Foreign Relations Committee, and Max M. Kampelman, chief U.S. arms negotiator. Several participants said Shultz fundamentally agreed on the text of a letter that would have met the Democratic legislators' concerns by stating that executive branch testimony on the INF Treaty was "authoritative" and legally binding.

Shultz planned to deny that the language amounted to a repudiation of Sofaer's view, the sources said, and Nunn intended to announce that it did repudiate Sofaer.

The deal fell apart when a group of conservative Republican senators got wind of it and demanded that Shultz hear their objections, aimed largely at preserving the legal basis of the "broad" ABM Treaty interpretation. The group, which met with Shultz Wednesday night in the office of Senate Minority Leader Robert J. Dole (Kan.), included Sens. Richard G. Lugar (Ind.), Pete Wilson (Calif.), Malcolm Wallop (Wyo.), Dan Quayle (Ind.) and Arlen Specter (Pa.).

Sen. Alan K. Simpson (Wyo.), who informally chaired the Republican meeting, said "there was concern expressed about the possibility of striking some kind of arrangement with {Byrd and Nunn} that might not be in the best interests of the Republicans."

Simpson said he asked Shultz "what are we doing getting deeply involved with the issue of 'broad' versus 'narrow' and a negotiating record versus a Senate ratification record when we're dealing with a fresh piece of goods . . . {namely} the INF Treaty."

Simpson said others spoke of Shultz setting "an unfortunate precedent" and "reinterpreting" the administration's 1985 decision that the "broad" reading of the ABM Treaty was legally correct.

Shultz responded by letting the Republican senators review a new version of his letter to Congress the following day, which omitted a pledge that Senate testimony by administration officials on the INF Treaty was legally binding. The senators approved it and Shultz sent the new letter to Nunn and Byrd on Friday afternoon.

The Democratic legislators responded by complaining in a letter late Friday that "your letter is silent in regard to the three key principles we have been discussing over the last two months" on the procedures for Senate treaty approval.

Nunn and Byrd said these principles were:

"1. Testimony of all executive branch witnesses and any submissions for the hearing record by the executive branch on a treaty can be regarded by the Senate as authoritative. "2. The meaning of the treaty as presented to the Senate can be regarded by the Senate as authoritative without the necessity of the Senate's incorporating that testimony and {other supporting} material in its resolution of ratification through understandings, reservations, amendments, or other conditions.

"3. A meaning for the treaty different from that presented to the Senate by the Reagan administration cannot be adopted without the approval of the Senate."

Nunn and Byrd said they would work to delay Senate ratification of the treaty until they had attached detailed "understandings, reservations, and amendments" reflecting these concerns, and they ruled out further testimony by administration officials on the INF Treaty before the Armed Services committee.

Simpson responded that he "could not imagine anything more disruptive or counterproductive or even unbelievable" than such a threat.

A Nunn aide, noting that Shultz has "already indicated he is willing to satisfy our concerns," expressed optimism that the administration will eventually concede.

Nunn and Byrd noted in their letter that Shultz had already repudiated Sofaer's position, also advanced in support of the "broad" reading of the ABM Treaty, that internal U.S. government divisions during treaty negotiations can alter a treaty's meaning. Shultz wrote Friday that "internal executive branch deliberative material . . . cannot be used as a basis for {treaty} interpretation."

Sofaer had argued internal government disputes undercut official U.S. statements supporting the "narrow" ABM Treaty interpretation.