After a fifth day of inconclusive discussion of the nuclear freeze resolution, House Democrats said yesterday that they would return to the Rules Committee next week to seek a limit on debate.

Attempts by Republicans and conservative Democrats to attach more than 40 amendments to the resolution are "nothing more than a filibuster by amendment," charged Rep. Norman D. Dicks (D-Wash.). Rep. Edward J. Markey (D-Mass.) complained that "The proliferation of amendments makes it unlikely we would have a vote before July 4...if we allow this unprecedented abuse of House procedures."

Several Democrats said the proposed rule would probably allow eight or nine amendments during two more days of debate. Republicans are likely to challenge the rule, but Markey predicted it would pass easily. "The mood of the House is that it's time to have a vote up or down," he said.

The resolution calls on the United States and the Soviet Union to negotiate "an immediate, mutual and verifiable freeze" on nuclear weapons development and production. It is strongly opposed by President Reagan, who has said it would undercut his arms control negotiators in Geneva and lock in Soviet strategic superiority in certain areas.

Last year the freeze was defeated 204 to 202. This year, with 26 more Democrats in the House, proponents thought it would pass easily. However, a coalition of conservative Democrats and Republicans has stalled it.

Yesterday, the impatience of the House was evident as, for the third time since mid-March, members voted on variations of an amendment to allow for negotiations to reduce arms and concurrently to freeze them, thus endorsing the president's current arms control policy.

The amendment, sponsored by Rep. Mark D. Siljander (R-Mich.), lost by only six votes in March. Yesterday it was set aside by 215 to 194, as the House approved a substitute sponsored by Rep. Jim Leach (R-Iowa) to restore language providing for a weapons freeze first, followed by arms reductions.

"The sequencing is very important," Leach said. "Without it we'd be giving a full vote of confidence to the current negotiating approach: building up in order to cut back . . . . We're saying we have to stop the arms race now."