As the new year dawned, year No. 1 in the era of the Gramm-Rudman-Hollings law, an idea occurred to Rep. Charles E. Schumer (D-N.Y.) on how to deal with the still unknown mysteries of the balanced-budget legislation that Congress enacted in December.

Schumer, like many other House liberals, wants to protect the domestic programs dear to his Brooklyn constituents. Many of these programs are in danger of severe cuts under any budget that meets the new law's mandatory deficit target of $144 billion for fiscal 1987.

But, Schumer reasoned, if Congress failed to adopt a budget meeting the deficit target, the Gramm-Rudman-Hollings law would trigger across-the-board cuts in most government programs. The programs cherished by liberals would take their cuts, of course, but others, including the Defense Department budget, would likely suffer even more.

An interesting idea, or so Schumer thought.

But when Schumer, a member of the House Budget Committee started going over the numbers in more detail, the notion did not look so attractive. It might work this year, but after about three years of mandatory across-the-board cuts to meet increasingly stringent deficit targets, he concluded, not much would be left of the federal government: "No FBI, no Coast Guard," Schumer said.

Schumer is not alone in making this kind of initial, tentative gaze across the unchartered fiscal waters of Gramm-Rudman-Hollings, named after its chief Senate sponsors, Phil Gramm (R-Tex.), Warren B. Rudman (R-N.H.) and Ernest F. Hollings (D-S.C.).

Scattered across the country and in several foreign countries for the congressional recess, lawmakers have begun to think through the implications of the new law and devise tentative strategies to deal with it. When Congress reconvenes on Jan. 21, it will enter a new world its members created but few of them pretend to understand fully.

Even before Congress returns to Washington, there appears to be a growing consensus that the first dose of mandatory across-the-board cuts, scheduled to fall on the current fiscal 1986 budget on March 1, probably cannot be avoided if for no other reason than lack of time to produce an alternative budget. But under the law, these initial cuts are limited to $11.7 billion, a level of austerity generally considered tolerable. The real test will come in the fiscal 1987 budget, beginning Oct. 1, when mandatory cuts easily could exceed $50 billion.

In the months between now and next fall, the strategies and coalitions that form around the budget issues are expected to be in constant flux and the outcome of the maneuvering uncertain. But based on interviews with several House members, it is possible to chart some of the opening positions in the coming Gramm-Rudman-Hollings debate.

*House liberals are likely to push for a tax increase to meet the $144 billion deficit target without making additional cuts in domestic programs. If President Reagan's opposition to new taxes prevails in the debate, some House liberals may be tempted to fight a budget compromise and hold out for across-the-board cuts. This could be particularly true of members of the Black Caucus and others who represent the most economically depressed districts, because programs targeted to the poor are exempt from automatic cuts.

*At the opposite end of the political spectrum, conservative Republicans and advocates of supply-side economics are likely to fight a tax increase to reduce the deficit. But in making final choices, they face a much more difficult problem than the liberals because if no compromise is fashioned and the automatic cuts are triggered, 50 percent of the savings will come from the Pentagon.

"Look at the defense numbers," said a House Republican. "An awful lot of supply-siders are strong defense advocates, too. That is where Jack Kemp [the New York Republican who is likely to seek the 1988 GOP presidential nomination] has a real problem." As a result, lawmakers said, some opponents of tax increases may eventually relent hike.

*The House leadership probably will try to steer the debate toward what Rep. Richard A. Gephardt (D-Mo.), chairman of the House Democratic Caucus, has called "a grand compromise" involving some higher taxes, some cuts in defense spending, some retrenchment in domestic programs. That would not be entirely satisfactory to almost any faction but might prove preferable to the indiscriminate automatic cuts of the Gramm-Rudman-Hollings law, lawmakers said.

Finding the right formula for a "grand compromise" will be tricky, and not just because Reagan is expected to oppose a tax increase at least until the deadlines near for imposition of the automatic cuts. House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), too, has taken a firm stand in the past against any effort to raise taxes unless first proposed by the White House.

At the same time, there will be 535 separate calculations in Congress on what would be worse at home -- an overall tax increase and budget-cutting package such as that envisioned by Gephardt, or the Gramm-Rudman-Hollings ax.

"Members of Congress are really sizing this thing up, trying to determine where the benefit will be," said Rep. Patrick Williams (D-Mont.).

Schumer, voicing the kind of thinking that first led him to see a seductive appeal in the ax approach, said, "If the Budget Committee put together a budget that most Democrats would support, it's not going to cut defense as much as Gramm-Rudman, it's not going to cut water projects and agriculture as much as Gramm-Rudman. So what is it going to cut? You're going to cut programs that liberals traditionally defend."

But in making this or other initial assessments, lawmakers are operating in a large political vacuum, just as they did last year when they enacted the Gramm-Rudman-Hollings law and promised to begin cutting the budget for real this year. The cuts -- whether imposed automatically or fashioned out of a grand budget-tax compromise -- are still part of the hazy, distant future.

Recently, Rep. Joseph J. DioGuardi (R-N.Y.) polled his wealthy, suburban Westchester County district and found overwhelming support for the Gramm-Rudman-Hollings path to a balanced budget by fiscal 1991. Yet his constituents' top political priority is the preservation of federal income tax deduction for state and local tax payments. That deduction could be imperiled if Congress decides it needs more revenue to shrink the deficit.

"If the question was do you want to balance the budget by eliminating the state and local tax deduction, the poll would turn around," said Kieran Mahoney, a DioGuardi aide.

It is just this kind of political uncertainty that makes the new era of Gramm-Rudman-Hollings a subject of intense interest to those who will be directly involved in the process. Later this month, Congressional Quarterly is holding a seminar on how the new law is supposed to work. The response -- from government agencies, interest groups, state and local governments and even some foreign embassies -- was so "overwhelming" that a second session had to be scheduled, said Charlotte Hodak, Congressional Quarterly's seminar coordinator.

At the end of the month, House Republicans will meet in Baltimore and House Democrats at the Greenbrier resort in West Virginia. The new budget law is expected to be the dominant topic at both meetings. A few days later, Reagan will submit his new budget to Congress and the first Gramm-Rudman-Hollings season will be in full swing.

"The games have not yet begun to be played," said a House Democratic aide. "Once they are, it's going to be a full-time job just keeping up."