The travail of Sen. Edmund S. Muskie (D-Maine) will be understood by anyone who has tried to ward off creditors demanding payment and children wanting toys -- with nothing but an astronomically unbalanced checkbook at hand.

Muskie has been a lonely sort of figure this week on the Senate floor, playing the role of an adamant patriarch trying to bring restraint and horse-sense to a squabbling, grasping family.

The fight this week is over federal spending -- who gets the money, who gets cut and who gets boosted, where the priorities will rest in divvying up the tax dollar.

As chairman of the Senate Budget Committee, which sets the guidelines and rides legislative shotgun on these matters, Muskie is the man with the unhappy task of holding off the fiscal wolves.

Muskie lost his effort to thwart the hawkishness that led to Senate approval Tuesday of a big increase in defense spending. But efforts to hold down domestic spending were successful.

On paper, nothing could be quite as sleep-inducing as the prospect of a Senate debate over what is aridly described as the second concurrent resolution on the budget, fiscal 1980.

But the reality on the floor has been something far different. Contrary to the Senate's often-illusary clashes, this has been a genuine collision between philosophy, politics, personality and senatorial turf.

The central player is Muskie and, in many ways, it has been vintage stuff -- voices lifted to a shout, fingers being pointed, emotion thick enough to slice and no little amount of overstatement.

Muskie's message this week has been simple: The United States faces no greater threat to its security than rampant inflation and that spending restraint -- not increases for defense or anything else -- is the only way to deal with it.

Everyone in the Senate favors restraint, of course, unless it hurts a pet program.

"You cannot have it two ways," Muskie said at one point. "All aspects of the budget, defense and nondefense, have to be controlled by this body if we are to control inflation."

The Maine senator's emergence as an evangel of restrained spending (he always has been viewed as one of the Senate's premire liberals) is due in part to the way congressional process has changed dramatically in a few years.

For a century or more until the mid-1970s, the process called for each committee to pass its programs, each chamber to pass out appropriations and then, if spending happened to coincide with the reality of federal income, it was a nice bonus.

Then, after former President Nixon began impounding funds already appropriated by Congress, the genie of reform was loosed on Capitol Hill. Muskie, the late Sen. Lee Metcalf (D-Mont.) and former Sen. Sam J. Ervin (D-N.C.) in 1973 proposed creation of congressional budget committees.

Those committees' task would involve more than just reconciling federal income with federal spending. For the first time, Congress would have a counterbalance to the budget proposals of the executive branch.

The budget act was passed, the committees set up and Muskie was chosen as chairman of the Senate panel. That was 1974 and every year since, Muskie has been insinuating the committee deeper and deeper into the process.

That has meant stepping on the toes of the more influential committee chairmen, sometimes telling them no -- a thousand times no -- they cannot exceed the spending limits put on their committees each spring.

A watershed, as Muskie staffers remember, occurred on the fiscal 1976 budget. In one week, defense and food stamp spending bills were rejected because they came in over limits.

And during that year, with each appropriation and authorization bill that came to the floor, Muskie would stand up and remind the Senate how each of those measures would have an impact on overall spending.

That underscored the seriousness of the senator and his message was understood.

The process has worked to a reasonable degree in large part, most Senate observers agree, because Muskie and Sen. Henry Bellmon (R-Okla.), the ranking Republican on the committee, have stood shoulder-to-shoulder in chasing away budget predators.

Generally, Muskie, now 65 and in his fifth term, is better known nationally for his work as an architect of the landmark environmental protection laws of the 1960s and early 1970s.

Muskie continues as chairman of an environment subcommittee, but the emphasis of his work these days is on the budget process -- watching it, advancing it, pushing away the empire builders.

"He has said he would rather be budget chairman than majority leader," said one of his assistants. "He takes the budget thing very seriously and he sees it as his lasting legacy here, to solidify the process and make it permanent."

Which means, of course, stepping on toes and saying no -- even to Maine. He caught some hell this week from constituents when he objected to increases in veterans benefits and school lunch spending.