Congress is laying aside its rubber stamp and sharpening its pencils for what could be more and more rewriting of President Reagan's script.

But even House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) concedes that Reagan remains personally popular, and such Democrats as Rep. Leon E. Panetta (Calif.), who led the party's damage-control efforts on the budget earlier, caution against underestimating the president's ability to rally his troops.

The new assertiveness on Capitol Hill, which arose after what Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) calls the "shock of August," when interest rates soared despite Congress' approval of Reagan's deep budget and tax cuts. It is causing many lawmakers to adopt more of a show-me attitude.

Baker contends that last week's ideology-defying vote to let the national debt exceed $1 trillion for the first time indicated that the August panic had subsided. But the smooth passage for the debt bill and a stop-gap financing bill for the government may have said as much about congressional leadership skills as it did about the agenda for the future.

Members "are expecting more out of this administration than a blank check that says, 'Sign here'," said a Republican leadership aide. "They're beginning to ask 'Why? . . . What for?"

Some of the restiveness is attributed to what another Republican aide describes as "system overload" stemming from hard decisions made earlier and even harder ones that lie ahead, without -- in the case of Reagan's proposals for new budget cuts -- a handy vehicle for ramming the program through Congress.

Another factor cited by some Republicans is the administration's failure to keep up the high level of consultation that helped achieve the big early victories, causing problems especially with defense and foreign policy initiatives.

On plans for deployment of the MX missile, which were announced Friday, an angry Senate Armed Services Committee Chairman John G. Tower (R-Tex.) complained that he was last consulted in August. Rep. Barber B. Conable Jr. (R-N.Y.), ranking Republican on the House Ways and Means Committee, noted unhappily that he was not consulted on the new economic program until three hours before Reagan disclosed it on national television.

Still another cause for restiveness is what Rep. Richard B. Cheney (R-Wyo.), who was former president Ford's chief of staff, calls part of the "normal cycle in any presidency" when decisions get more difficult, controversy builds, the opposition hardens and "the last election gets a little farther behind us while the next one gets a little closer."

The "next one" is now only about a year away, and there is mounting fear in some GOP quarters that persistently high interest rates or other bad economic news, coupled with the impact of budget cuts, could torpedo Republican hopes of capturing control of the House and even jeopardize the party's hard-won control of the Senate.

In contrast to the first six months of his administration, when Reagan had only to cut a few backstage deals to get what he wanted, Congress is balking on issues ranging from heavier cuts in social welfare programs to the proposed sale of AWACS surveillance aircraft to Saudi Arabia. Republicans also showed anything but a united front in their initial response to Reagan's new strategic defense plans, including the MX missile and B1 bomber.

Congressional resistance has already forced the administration to back off plans to clamp down on cost-of-living increases for Social Security and other popular but expensive entitlement programs, even though Budget Director David A. Stockman and some lawmakers, like Senate Budget Committee Chairman Pete V. Domenici (R-N.M.), view major surgery on entitlements as critical to long-term savings.

Congressional assertiveness is also apparent in widespread demands for deeper cuts in defense, perhaps twice the $2 billion in Pentagon savings that Reagan recommended. It is the price the administration will have to pay for support of many Republican moderates, whose opening demand is $9 billion in defense cutbacks.

The GOP moderates "hold the cards," said a source close to the House GOP leadership. "We're going to have to negotiate with them, period."

Moreover, the momentum, enthusiasm and sense of urgency that Reagan could tap for the first round of cuts, adding up to $35 billion, is not there for the new budget proposals, entailing $13 billion in spending cuts and $3 billion in new revenues.

House Minority Leader Robert H. Michel (R-Ill.) talks in terms of $10 billion to $12 billion in new savings, and Senate Banking Committee Chairman Jake Garn (R-Utah) has said the proposal for 12 percent across-the-board cuts in domestic appropriations, which account for more than half the proposed new savings, is "politically unrealistic."

"People were confident before the first budget reductions were made that there was real fat to cut," says a House Republican leadership source. "Now the question is where is the line between the fat and the meat. No Republican wants to go too far -- down to the bone."

By setting unrealistically high budget-cutting targets and forcing a showdown, the White House is engaging in "shortsighted politics, something you wouldn't expect from an administration that has been as shrewd as this one," Conable complains, contending that Congress "won't come up with anything like $16 billion" in new cuts.

From Rep. Jim Leach (R-Iowa), new chairman of the moderate Republicans' Ripon Society, comes yet another concern.

"By going much further than we currently have, we may be causing social and class and perhaps racial divisions that could be profound," Leach said last week. Leach, calling for a congressional "mea culpa" for tax cuts that were approved last summer, proposes to shield social programs from heavy new reductions by repealing tax breaks for the oil industry and financially shaky corporations -- a lonely idea so far.

Another problem is that the new budget cuts amount to a reneging on accommodations that were made with moderate Republicans and conservative Democrats to win their support for the earlier budget cuts. Victims have held their fire, preferring more negotiations, but Cheney says: "A member of Congress travels on his credibility, his word. The administration will have a long-term problem, it seems to me, if it does this very often. In other words, I would hope that, come next year, they get it right the first time."

What's surprising is that it's the Republicans, who provided such dazzling unity on behalf of Reagan's tax and budget cuts last summer, who are kicking up their heels -- probably not to the point of shaking the ground under Reagan's base on Capitol Hill but surely enough to make the terrain a little rockier.

The independent courses are not being just charted by the so-called "Gypsy Moths," the Northeast-Midwest Republican moderates who have so much to lose when Reagan squeezes the domestic budget to prop up the Pentagon. Reagan got an earful from Tower on the administration's MX missile plans, and the 18 Republicans who are publicly opposing the AWACS sale include Rocky Mountain conservatives as well as Eastern moderates. Moreover, Garn's protests that programs like housing have born enough of the belt-tightening burden can hardly be taken as the lament of a bleeding-heart liberal.

The only strategy the administration has put forward for handling the new budget cuts is to veto excessive appropriations bills. The Democratic-controlled House has shown it would sustain some vetoes, and the Democratic leadership, faced with pressure from its own Budget Committee leaders, agreed Friday to look for more savings in the budget-busting money bill for labor, health and human services.

But, while both Michel and Baker favor slugging it out on appropriations bills for the time being at least, there is no guarantee that this kind of trench warfare will achieve the big savings the administration wants.

So there is increasing talk of another omnibus budget-cutting effort when Congress passes its second budget resolution later this fall or renews its "continuing resolution" to cover appropriations bills not passed by Nov. 20. Reagan won his first round of cuts last summer with the help of such an all-in-one strategy, avoiding votes on individual spending items. A veto strategy, especially if it leads to an impasse on individual spending bills, could add to pressure for a new big package of cuts.

Where are the Democrats in all this?

Enjoying the spectacle, for one thing. Tired of talk about their own "disarray" earlier in the year, they are promoting the notion of Republican "disarray." Why, asked an aide to O'Neill as the new budget troubles surfaced, "do anything to spoil the view?"