The Senate began the new Congress yesterday by taking up a proposed revision of its filibuster rules, while Senate Republicans installed a generally moderate group in their leadership positions.

The rules changes were proposed by the majority leader, Robert C. Byrd (D-W.Va.), who said it was not "in the national interest... or fair play" to allow a handful of senators to tie up the body even after it has voted to cut off debate.

Stepping gingerly around Senate traditions and sensibilities, Byrd went out of his way not to offend members of the Republican minority who can sometimes make effectority use of the filibuster reles to block controversial legislation. Byrd said he hoped the two parties could negotiate an agreement on rules changes, and he recessed the Senate until Thursday in hopes of producing such a settlement.

But Byrd also threatened to exploit precedents set by three recent vice presidents who have reled that a new Senate can alter its rules by simple majority vote. If literally applied, this could mean imposing rules changes and cutting off debate onthem with the support of just 51 senators, a drastic departure from Senate traditions.

The standing rules of the Senate require a two-thirds vote to change the rules, and 60 votes to cut off a debate. Byrd argued yesterday that the standing rules of a previous Senate cannot bind this new one -- though he admitted he had taken the opposite position in years past.

Informed Senate sources said it wasn't yet clear whether Byrd could find 51 votes to support his rules changes in a showdown vote. Any attempt to push the canges through with a narrow majority could deeply divide the Senate and miaght provoke stalling tactics from a substantian minority that could probably tie up the Senate for a long time.

Aides said Howard H. Baker Jr. (R-Dtenn.), the monority leader, was willing to accept some rules changes, but not all that Byrd had proposed. Baker named a committee of five Republican senators to study the issue and perhaps meet with Democrats to discuss a compromise.

Yesterday morning the 41 Drepublican senators met to elect officers, and with one exception picked moderates over conservatives for contested positions.

In the most closely fought reac, Sen. H. John Jeinz Ill (R-Pa.) defeated Sen. Orrin G. Hatch (R-Utah) by a 21-to-20 vote for the post of chairman of the Republican Campaign Committee. Last week the Hatch camp claimed 24 votes for its man, who was nominated yesterday by the Senate's most popular conservative, Paul Laxalt (R-Nev.), Ronald Reagan's presidential campaign manager in 1976.

Apparently the 10 newly elected Republicans gave Heinz a clear majority of their votes, making possible his victory.

In another close race, Sen. Bob Packwood (R-Ore.) defeated conservative Sen. James A. McClure (R-Idaho) 22 to 19 to become chairman of the Republican Conference. The only conservative to win a close contest was Jake Garn (R-Utah), who beat moderate John H. Chafee (R-R.I.) 21 to 19 for the post of conference secretary -- a closer result than expected.

The Republicans reelected their top leaders by acclamation -- Baker, Assistant Minority Leader Ted Stevens (R-Alaska), and Policy Committee Chairman John G. Tower (R-Tex.)

The opening Democratic caucus was uneventful. Sen. Byrd appointed five mew members to the Steering Committee, the body that decides which committees senators will sit on. The Steering Committee is scheduled to meet today to divide up committee assignments

There were no contests for Democratic leadership positions.

The rules changes Byrd proposed yesterday would eliminate many of the dilatory tactics used in recent years to prolong debate outside the traditional format of the filibuster. Byrd emphasized that the Senate "has a right to filibuster a matter," but only once.

Under present rules, a motion to take up a bill can be the subject of a filibuster. Then the bill itself can be filibustered. If the Senate apporves a cloture petition to cut off a filibuster, opponents can still delay a vote virtually indefinitely by calling up previously filed amendments.

The first and third of these options are the targets of Byrd's proposed rules changes. In addition, the majority leader wants to eliminate some of the procedural tricks now open to senators to delay proceedings.

Baker, aides said, would be willing to accept changes in the "post-filibuster filibuster" rules, but would not be interested in a rule precluding the "pre-filibuster filibuster." Whether more conservative Republicans will accept a compromise along those lines remains to be seen.

Though Byrd recessed the Senate until Thursday, several sources predicted it would take longer than that for various parties to explore possible compromises before beginning a floor discussion on the rules.