Senate Majority Leader Robert C. Byrd (D-W.Va.) yesterday predicted a compromise between Congress and the Carter administration over extending the U.S.-Soviet nuclear arms limitation accord beyond is Oct. 3 expiration date.

"Congressional action should be had," Byrd said, but "I think we can work it out with the President. I think the matter can be resolved."

The administration has been determined to avoid "a disruptive debate" over extending the expiring accord while negotiations go on for a new strategic arms limitation talks (SALT) pact.

As a consequence, the United States agreed with the Soviet Union that each would issue a "unilateral" declaration to abide by current arms ceilings after Oct. 3, without any "agreement" requiring congressional action. The U.S. statement was issued Friday. The Soviet declaration is expected in the next few days.

This procedure is being challenged by Sen. Henry M. Jackson (D-Wash.), chairman of the Senate Armed Services Arms Control Subcommittee, as a circumvention of Congress' constitutional power to approve treaties.

What Jackson wants, aides say, is not a stormy debate but acknowledgment that congressional action is required.

Although Secretary of State Cyrus R. Vance has said that "I don't believe congressional approval is necessary," State Department spokesman Hodding Carter III yesterday added a newly flexible modification. Said the spokesman: "We are not sitting here saying that Congress should or should not pass a resolution."

The issue will come up Monday before the friendlier Senate Foreign Relations Committee in testimony behind closed doors from Paul C. Warnke, director of the Arms Control and Disarmament Agency. Several members of this committee, and of the House International Relations Committee, have said they support the administration's interpretation.

A communique issued yesterday on two days of talks by President Carter and Vance with Soviet Foreign Minister Andrei AS. Gromyko expressed determination to seek improved U.S. -Soviet relations. It said "both sides consider it necessary to intensify their efforts to find mutually acceptable solutions to existing problems."

The Carter administration and the Soviet Union are both anxious to convey an impression of progress toward replaing the expiring 1972 SALT accord after three years of stalemate on negotiating the replacement pact, which was projected in 1974.

"Progress was achieved in bringing closer together the positions of the two sides," they said in the communique.

When asked for specifies, spokesman Hodding Carter said the main difference between the two earlier Vance-Gromyko meeitings, last March in Moscow and in Geneva last May, was that in the current negotiations there is "hard bargaining . . . without Abrasivenexx," in contrast to earlier formalized declaration of postion.

Vance and Gromyko will resume negotiations Frifsy in New York, here both will be attempting the U.N. General Assembyly. In addition, Carter said, Warnke will return to Geneva for what were termed "intensive negotiations" between the regular American and Soviet SALT delegations.

"While there has been no great breakthrough," Carter said, "both sides are obviously encoraged by the exchanges."

The basic obstacles blocking agreement, Carter repeated, continue to be differences over limitations on American long-range cruise missiles, limits on the Soviet nions' largest intercontinental ballistic missiles and Soviet Backfife bombers and methods of verifying arms ceiling.

In a separate statement, the two nations agreed that the anti-ballistic missile treaty they signed in 1972 "has operated effectively." This accord, severely limiting defensive missile systems, is subject to a five-year review as of Oct. 3, but unless there is objection it will continue in force.

The agreement expiring Oct. 3 limits offensive land-launched and submarine-launched intercontinental missiles and is labeled an "interim" accord. It would be replaced by a treaty running to 1985 and putting ceilings on a broader category of weapons.

Discussions with Gromyko on the Middle East were described as "generally constructive."

Hodding Carter said that in Vance's judgment, the Soviet Union demonstrated "a real interest in working with us to reconvene" Arab-Israeli talks in Geneva and in "helping the parties come to a peaceful resolution in the Middle East."

There are "admittedly differences of views between us on the shape of a settlement," Hodding Carter said, "but there are at least some signs that we are drawing together." The United States and the Soviet Union would be co-chairmen of a new Arab-Israeli conference in Geneva. Both renewed pledges to attempt to reconvene that conference this year.