The nation's governors were a study in frustration yesterday as they grappled with pressures from home to seek a balanced federal budget and pressures from Congress to cut federal aid to states.

The governors, attending a threeday meeting of the National Governors' Association that started Sunday, warned Washington that people are, as one governor said, "mad as hell" about federal spending.

In turn, various Washington officials warned the governors that engraving the budget-making process in a constitutional amendment could be dangerous.

The governors themselves were sharply divided over the issue of an amendment; so, in typical Washington fashion, a committee of the association approved a resolution calling for a balanced federal budget by 1981 -- a position the association took twice last year -- but not mentioning an amendment.

The committee on fiscal affairs, headed by Vermont Gov. Richard Snelling, also decided, by a 5 to 4 vote, to study state efforts to balance their budgets or limit spending and see if these held any lessons for the federal government.

The action, or lack of it, represented a rebuke to California Gov. Edmund G. (Jerry) Brown Jr., who has urged a budget amendment drafted by a constitutional convention if Congress refuses to pass one.

Asked if he were disappointed by the committee move, Brown said no and added, "We're here to focus on the issue." He said Congress and the governors now see a balanced budget as a priority and he is talking privately to governors about state legislative calls for the constitutional convention. Twenty-eight states have already issued such requests.

Realizing he did not have support for a motion endorsing a budget amendment, Brown had pressed for creation of a committee "to evaluate the implications of amending the Constitution." Snelling headed him off by proposing to reaffirm the association's existing policy.

Earlier, in a plenary session of the conference, Govs. James R. Thompson Jr. of Illinois, Bruce E. Babbitt of Arizona and Richard W. Riley of South Carolina called the idea of a constitutional convention "simplistic." Gov. Pierre S. du Pont IV of Delaware said that, while he favors an amendment to limit federal spending, he opposes the convention proposal. But he warned Washington officials:

"The people are mad as hell and they're not going to take it any more... Listen to us and our legislatures and don't treat us lightly... Don't underestimate the depth and strength of feeling we have in the states."

But Sen. Henry L. Bellmon (R-Okla.) advised the governors, "Don't go overboard and adopt a mechanistic process that will make it difficult to deal with changing circumstances."

James McIntyre, director of the Office of Management and Budget, cautioned against a constitutional amendment: "enshrining a particular economic policy designed to meet the needs of a particular time would be unwise."

Rep Peter W. Rodino (D-N.J.), chairman of the House Judiciary Committee, deplored "this climate of simplification, urgency and tension" and said the "danger" of it is that "we might be tempted to rush pell-mell into a decision without being aware of the profound consequences."

Sounding a somber note, Minnesota Gov. Albert H. Quie said, "The people are feeling as strongly about the federal government now as they did about the government of England at the time of the Revolution."