A chagrined Senate, battered by publicity over the fees that many members get for speaking to special-interest groups, voted 51 to 41 yesterday to limit senators' earnings from speeches and articles to 30 percent of their salaries, or roughly $18,200 a year.

The Senate acted after an angry debate and a painful series of votes forced by Sen. Henry M. Jackson (D-Wash.), who warned his colleagues that their previous refusal to limit outside earned income had "undermined the integrity of the Senate" and created a "scandal waiting to happen."

An angry Sen. Jake Garn (R-Utah) protested that Jackson was casting unfair aspersions on the motives of his speech-making colleagues by suggesting they "go out and hustle" for fees. Moreover, said Garn, "I resent my colleagues trying to impose on my children what my future income will be."

The Senate's action not only would limit senators' speech-making earnings but would also leave Senate salaries at a lower level than House members'.

However, some senators served notice last night that they plan a move to raise Senate pay to the House level or let senators choose between getting higher pay or having limitless outside earned income.

In a deal last year between the two chambers, House members raised their pay by 15 percent to $69,800 a year, while keeping a 30 percent-of-salary limit on outside earnings. In exchange, senators kept their pay at $60,662.50 a year but scrapped a proposed limit on their outside earnings, including honoraria for speeches and articles, that was scheduled to take effect this year.

But senators were embarrassed by recent publicity over their 1982 outside earnings, including handsome fees for many and ranging as high as $135,750 for Finance Committee Chairman Robert J. Dole (R-Kan.). In all, more than half the 100 members were paid more than they would have been with a 30 percent limit on outside earnings.

In a poke at the Senate, the House last month approved a 30 percent limit for senators as a rider to a supplemental appropriations bill for this fiscal year. The rider was promptly taken out by the Senate Appropriations Committee.

When the bill reached the Senate floor yesterday, Jackson proposed four amendments, three of which included the possibility of higher pay and went down to decisive defeat. The fourth--the limit on honoraria with no pay increase--then passed.

"We took 'em by surprise," a jubilant Jackson said. While he would have preferred a pay raise to offset the honoraria limit, he said senators "have got a phobia about no pay increase."

Jackson added: "The garbage collector in charge in Los Angeles gets more than a United States senator. Maybe he does more useful pickups."

There was poignant irony for the Senate in its action yesterday. Senators care more than House members about earnings from speech making because, as better-known figures, they tend to be paid more. But because the honoraria limits are computed on the basis of salary, and House members' salaries would be higher than senators', the Senate limit would be lower than the House's.

During the debate, Dole, last year's record honoraria-collector, suggested that the Senate should consider campaign contributions as well if it were going to limit personal earnings of senators. Jackson offered to accept an amendment to restrict contributions by political action committees. He got no takers.

In defense of limitless honoraria, Garn, who was paid $60,779 in fees last year, argued that senators can reject honoraria if they choose and noted that fees are reported publicly for constituents to assess. Moreover, he said, there are no limitations on unearned income from investments, a situation that favors rich senators. To limit outside earnings would restrict Congress to the "ultra-rich" and the "ultra-ignorant" who can't earn a living on the outside, he asserted.

Among Washington-area senators, only Charles McC. Mathias Jr. (R-Md.) voted against the honoraria cap. Mathias previously voted for the restriction when combined with a pay increase.

The supplemental money bill to which the honorarium limit is attached is not expected to win final Senate approval until next week.

The $15.6 billion measure is under threat of a presidential veto because it exceeds administration proposals for domestic spending.

In other action on the overall appropriations bill, the Senate fended off a move to deny multi-year contracting for the B1B bomber and rejected a proposal to cut foreign military assistance by $376 million.

Sen. J. Bennett Johnston (D-La.) led the fight against multi-year contracting, which was sought by the Pentagon on grounds it would make the project more difficult and costly to cancel or modify if Congress decided later to do so.

Senate Appropriations Committee Chairman Mark O. Hatfield (R-Ore.) said he would resist any spending add-ons to the bill and attempt to trim it in conference with the House, which has approved a smaller version.

Hatfield, enraged initially at the rationale used by presidential advisers in urging a veto, implied that the two sides had reached a better understanding of their differences. "The president has a right to veto anything he pleases," Hatfield noted, but then he added that he would "vigorously oppose" a veto of the legislation.

Meanwhile, House-Senate conferees continued negotiations on a budget resolution for fiscal 1984, mainly making exploratory efforts to stake out positions for real bargaining expected in future days.

And a Senate Appropriations subcommittee approved a bill for the Department of Housing and Urban Development that includes $3.9 billion for the Environmental Protection Agency.