In a little-noticed proceeding as Congress was leaving town in early August, the Senate Budget Committee quietly voted 9 to 4 to do something every small boy knows not to do:

Pick on bigger boys.

The budget panel voted to ask the Senate to instruct seven older and firmly established committees to cut about $4 billion from federal spending by the end of this year.

The purpose is to keep Congress within the spending targets it set for itself in its first fiscal 1980 budget resolution last spring.

When this "reconciliation" resolution reaches the Senate floor the week of Sept. 10, it will set up the most serious test so far of the five year old congressional budget process.

The howls have begun already.

Chairman Russell Long (D-La.), whose powerful Finance Committee would be called upon to vote $1.7 billion of the cuts, has complained he can't do it because most of his committee's bills originate in the House.

Chairman Abraham Ribicoff (D-Conn.) and Sen. Charles Percy (R-Ill.), whose Governmental Affairs committee would be instructed to cut $100 million from government retirement programs, have said they won't do it (though they have promised an "effort" to achieve savings elsewhere).

Chairman Warren Magnuson (D-Wash.), whose Appropriations Committee would have to cut the most of all -- $2.3 billion -- has made no comment. But Magnuson, a member of the Budget committee, bellowed a loud "no" vote on the resolution to call for reconciliation.

"Magnuson is opposed, and one doesn't take Maggie on lightly," acknowledges Chairman Edmund Muskie (D-Maine) of the Budget Committee. "All the committees resent the budget process and they might gang up.

"But if committees adopt the attitude that this is trespassing on their areas, then they ought to repeal the budget process."

The reconcillation process has never been invoked on such a scale before. Budget committees in the past have always set their spending targets high enough to avoid such embarrassment.

This time is different. The next budget deficit, if Congress takes no remedial action, is likely to be $9 billion higher than the $23 billion set as a target in May.

About $4.8 billion of that is a result of deterioration of the economy. Higher inflation and unemployment estimates have led in turn to increased spending estimates for such programs as Social Security, food stamps and unemployment compensation.

The Senate Budget Committee is accepting these increases as more or less inevitable.

Moreover, in May Congress blithely promised in the budget resolution report to find some $5.6 billion in savings through legislative cuts in programs.

As of now, less than 20 percent of those cuts are in progress in the legislative mill.

So the Budget Committee, in marking up its second and final budget resolution just before the August recess, voted to throw down the gauntlet

Reconciliation is provided for in the Budget Act, but it has been used only once before -- and unsuccessfully. It must be approved by the Senate, which could amend or scale down orders to the seven committees to cut spending. It also must survive a House-Senate conference before committees would be ordered to make the cuts.

Muskie argues that the alternative to making the cuts is to increase the budget deficit of $32 billion and reverse a four-year congressional trend of steadily diminishing deficits.

Muskie is counting on this fact to win support for reconciliation. He is gambling that senators up for reelection won't want to face the charge that they talk like fiscal conservatives but vote for higher spending.

"This will affect the public's view of the disposition of this Congress to be fiscally prudent and responsible," Muskie said. "for those going before the voters next year, this is the clearest evidence of their willingness to hold spending down. For those who want to drive home to an increasingly conservative constituency that they are fiscal conservatives, this is handmade."

Muskie declared that this isn't "the Super Bowl" for the Budget Committee -- it could be a life or death test of the reconciliation process.

Until this year, the Budget Committee has simply added up what the authorizing the appropriating committees want to spend and set that as a ceiling. Shortfalls in actual spending have given the committee some latitude, so that if spending exceeded the amount set in the first resolution, it could be absorbed without increasing the deficit because of shortfalls by the time of the final resolution in September.

But this year President Carter, in an effort to bring out a tight budget cut closer to the bone. The customary shortfalls are not there.

What the house is going to do about reconciliation is not clear. The House Budget committee won't begin markup until after the budget is debated in the Senate, so it is taking a wait-and-see approach. But the House committee has had a hard time passing budget resolutions in any form. Prospects for confrontations with other committees are worse there than in the Senate.

How good are prospects in the Senate?

"I have no way of assuring the Senate will accept reconciliation," Muskie said. "It's an obligation which needs to be fulfilled if we are to conform to the fiscal plan adopted in spring." Remarking on other committees' resentment and the prospect that "they might gang up," Muskie said, "Our only weapon is public opinion."

Congress has been steadily holding down the deficit to curb inflation, Muskie said, "To shift to a stimulative policy now is going to look like the same old Congress going for the first excuse to run up the deficit."