House Democrats yesterday agreed to grant amnesty to colleagues who broke party ranks to support President Reagan's economic program, but warned them that future defections on key votes could cost them good committee and leadership assignments.

The mild disciplinary measure was outlined by Majority Leader James C. Wright Jr. (D-Tex.) at a closed-door party caucus, during which members vented their complaints against the 50 or so "boll weevil" Democrats whose votes gave Reagan his close victories on budget and tax cuts. There was no formal vote, but members said there was broad support for Wright's formula.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) told reporters it was "not exactly a love feast, but we reached an understanding" that he said would guarantee "smooth sailing" for the Democrats from now on.

O'Neill, pointing cheerfully to Republican divisions over the expected next round of budget cuts, also announced that three Democratic-dominated House committees will hold hearings around the country this fall on the effects of the Reagan reductions in health and education programs and the impact of high interest rates.

Yesterday's caucus was scheduled in July, when many House Democrats were steaming at the defections by Democratic members, largely from conservative southern districts, that enabled Reagan to push through his budget and tax plans over alternatives shaped by Democratic-dominated House committees. They resented the picture of "Democratic disarray," and some of them called for expulsion or other measures to discipline the defectors.

In the interim, however, many of the Democrats have come to believe that they were either smart or lucky to let Reagan have his way, because they think his program is backfiring and they will profit in 1982 from public disillusionment with it.

As O'Neill said, "Our plan is working perfectly . . . . The monkey is off the back of the Democrats. These are Reagan's deficits and Reagan's interest rates."

In this changed atmosphere, there were no demands that the traitors be hanged, and there was broad acceptance of Wright's modest four-point program. As the majority leader outlined it to reporters:

Forgiveness for all past votes and an open-door invitation to the renegades to return to the fold in good standing.

An explicit statement that committee chairmanships and assignments to such prized committees as Rules, Appropriations, Ways and Means and Budget are rewards from the caucus, and "we expect some responsibility to the party and their colleagues" from Democrats who receive such positions.

Designation by the party's Steering and Policy Committee of a few "litmus-test" votes on important policy issues each session. O'Neill is to notify all Democrats of such a designation and, Wright said, "those who hope to be rewarded by the caucus can expect to have their votes scrutinized."

A distinction is to be drawn, even on these key votes, between "occasional aberrations on the basis of conscience, conviction or constituency and a pattern of consistent conniving with the opposition," Wright said.

There were, according to members, a number of critical comments aimed at Reps. Phil Gramm and Kent R. Hance, the two Texas Democrats who bolted from their Democratic colleagues on the Budget and Ways and Means committees and became co-sponsors of the "bipartisan" alternatives embodying the Reagan cuts.

Rep. Toby Moffett (D-Conn.), one of the most vocal of the "disciplinarians," complained about "people on our side who rub our noses in it by standing behind the president every week, getting their pictures on TV, and grinning with him when we've just gotten licked."

But even Moffett said he realized "we can't have binding votes," and called Wright's proposal "a step in the right direction."

Gramm, who sat through the caucus but did not speak, dismissed the importance of Wright's statement or the caucus comments. "Words don't really change anything," he told reporters afterward. "I'm not seeking any confrontation with anybody, but I will continue to speak out."

Gramm said he had "always known my actions might mean the loss of my committee position," but he volunteered the information that he was consulting extensively this week with Office of Management and Budget Director David A. Stockman on the new round of spending reductions.

O'Neill told the caucus that he had some sympathy with the defectors, because he had been regarded as a renegade 15 years ago when he publicly ended his support of President Johnson's Vietnam war policies.

O'Neill said the proper time to exert discipline would come when the caucus elects committee chairmen and members of the four "leadership committees" late in 1982 for the new Congress beginning in January, 1983.

In what some members took as a pointed hint, O'Neill added that Rep. G. V. (Sonny) Montgomery (D-Miss.), one of the leading Boll Weevils, had "only six votes cast against him" for chairman of the Veterans' Affairs Committee last December.

But Wright said the leadership was not eager to make any martyrs. "Nobody's going to be Joan of Arc out there," he said, "unless he lights his own torch to his own funeral pyre."