Senate Majority Leader Robert C. Byrd (D-W. Va.) yesterday became the most prominent party leader to come out in favor of an "open" Democratic National Convention next week in New York City.

Byrd said he expects President Carter to win the nomination no matter which convention rule is adopted. But he said he opposes the proposed rule that would require delegates to vote for the candidate they were elected to support in their state primaries.

His surprise announcement was greeted with cries of joy by supporters of the open convention drive, and with derision at the White House, where one high-ranking official wondered whether Byrd had dreams of glory that the convention might turn to him as an alternative.

Though Byrd's announcement appears to add impetus to the open-convention movement, it is unclear what effect it will have on the position of other elected officials or the delegates.

Recent surveys indicate that most Carter delegates have been resolute in supporting the president. And he told a group of them at the White House Friday night that it would be a travesty of the nominating process to release delegates from their pledges.

Byrd made his views clear in his regular Saturday meeting with reporters. When asked his position on the proposed rule, he said, "The delegates to the convention should not be bound to vote for any particular candidate on the basis of the primary results if there have been changes in economic, political or international conditions that dictate in their own good judgment that they should vote for someone else."

He said he felt his opinion was a logical extension of his position calling for free and open debate among the candidates.

"Any president would have a stronger mandate, a deeper vote of confidence, if the nomination were secured on the personal preference of the delegates at the time of the convention," he said. "I think the incumbent would feel and would be perceived to have a stronger mandate when delegates are not bound by a procedural rule."

Byrd's comments yesterday came after a week of private consultations with Democratic senators. "I just focused on the issue," he said. "I studied it." He even cited a 1975 Supreme Court decision that said a party's convention rules take precedence over state laws governing the conduct of delegates.

Edward Bennett Williams, the prominent Washington attorney who is chief spokesman for the open-convention push, said in a telephone interview yesterday that Byrd's announcement was "a significant break-through."

"I think it reflects the attitude of the Senate generally, and I'm hopeful this will be influential in having the president reevaluate his own thinking on the matter," he said.

Hal Wolff, the coordinator for the Committee to Continue an Open Convention office in Washington, was more blunt. "We're delighted," he said. "It's obvious Byrd is carrying the flag for a number of endangered Democratic senators who can't come out for anything that might alienate some of their workers.

"It's also no secret that Byrd has contempt for Carter," he added."I don't know which contributed the most of his decision. But it's exactly our line."

Robert S. Strauss, nation chairman of the Carter campaign, issued a statement yesterday in response to Byrd's announcement that repeated his previous complaints about the open-convention movement.

"A few elected officials have spoken out for what has been erroneously labeled an 'open convention,'" he said. Strauss said those officials fail to understand that the delegates already are pledged to vote for the candidate "preferred by the grass-roots Democrats who elected them."

Strauss said he was confident that the delegates will vote not to "disenfranchise 20 million Americans and not to return the selection of the Democratic Party nominee to the hands of a few, stopping cold 12 years of party reform."

Williams countered that Strauss "keeps putting blinders on" by ignoring part of the party charter that says no delegate shall be required to vote against his or her preference.

At the White House, a key Carter aide was sharply critical of Byrd's action. He noted that Byrd was taking again a position endorsed by the president's challenger, Sen. Edward M. Kennedy (D-Mass.)

Commenting on the string of reports from Byrd's news conferences, the official added: "They are rife with comments not helpful to the president. He's [Byrd] been a matter of concern to us for some time.

"There are a few at the White House that wonder if he doesn't have dreams of glory for himself at the convention."

This sign of strained relations between Byrd and the White House echoes a recent report about Byrd's dissatisfaction with his role at the upcoming convention. He was invited to speak just a few weeks ago.

The comments yesterday from both Byrd and the White House raise questions about the effectiveness of a working relationship that is a key to getting administration programs through the Senate.

Again yesterday Byrd refused a chance to endorse Carter for renomination. "There's no reason why I should rush to judgment," he said.

Byrd said he would announce his preference if he makes a decision before the convention. "I can support either of them," he said of Carter and Kennedy.

In a related development, a new Newsweek magazine poll shows that Secretary of State Edmund S. Muskie is the only alternative candidate mentioned who runs even marginally better than Carter in a three-way race against Ronald Reagan and John B. Anderson. Reagan placed far ahead of all the potential candidates, according to the poll.

The national survey of 1,036 registered voters on the last two days of July found Reagan ahead of Carter 46 to 28 percent, with Anderson trailing with 18 percent of the vote.

Muskie trailed Reagan by a slightly closer margin, 45 to 31, with Anderson falling off to 16 percent.

Kennedy and Vice President Mondale fared only slightly worse than Carter in a three-way race. Reagan led Kennedy and Anderson 48 to 27 to 18, according to the poll. He led Mondale and Anderson 47 to 27 to 18.

Sen. Henry M. Jackson (D-Wash.), an unsuccessful contender for the nomination in 1976, did less well against Reagan, 46 to 21, with Anderson at 20 percent.