As the Reagan presidency enters a new phase, the bold direction and brimming self-confidence that marked the first two years of Republican rule are being blurred by signs of indecision and conflict in the White House and on Capitol Hill after the midterm congressional elections earlier this month.

Senior presidential aides, shaky survivors of the 1980 GOP freshman class in the House and nervous Republican senators looking ahead to the 1984 election know they have fewer than two years left to balance complex, conflicting economic and political demands before a revived Democratic coalition challenges their grip on government.

There is no sign of mutiny in Republican ranks against Ronald Reagan's leadership, but there is a good deal more second-guessing of White House strategy. And, increasingly, there is a willingness among Republicans on Capitol Hill to go their own way -- with or without the president's approval.

Reagan himself must choose between pursuing his original agenda of defense boosts, domestic program cutbacks and tax reductions, with a clear risk of confrontation with Congress, or accepting a policy of compromise that seeks to accommodate those independent Senate Republicans and antagonist House Democrats.

The president has proclaimed his readiness to "stay the course" he outlined in his 1980 campaign. But many congressional Republicans, interviewed last week, said that policy could stumble over the combination of recession, rising deficits and the approach of the 1984 election -- which would inflict serious wounds on the president and his party.

Republicans at both ends of Pennsylvania Avenue now acknowledge that all the goals Reagan set forth two years ago cannot be accomplished at once.

Presidential aides concede that the administration's budgetary policies, which Reagan believes will produce long-term, non-inflationary prosperity and a secure national defense, may entail politically dangerous risks of high deficits, high unemployment and continuing alienation of important constituencies. The president seems ready to run those risks.

But, after two years of extraordinary, almost parliamentary discipline and cohesion, it is clear that the Republican coalition on Capitol Hill is beginning to fragment.

More and more Republicans are plotting their own course with a wary eye on voting blocs that have -- at least temporarily -- turned their backs on some of Reagan's policies.

There was evidence of that in last week's decision by Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) to cross the Capitol and strike an agreement with House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) on a highway, bridge and transit repair program that would create thousands of construction jobs, to be financed by a nickel-a-gallon increase in the gasoline tax.

Baker did not wait for Reagan's approval before assuring O'Neill of his cooperation in passing the needed legislation in the lame-duck session of Congress which begins Monday.

Reagan nimbly jumped aboard that bandwagon a day later, but the message was not lost on anyone that he was no longer calling the tune.

The lame-duck session may pose other challenges to his leadership, including threats of defeat or delay on his new MX missile proposal and his overall defense spending plan.

But the real change will become obvious in January, when the 98th Congress convenes with 26 fewer Republicans in the House -- assuming that the Democrats, as expected, win the two Georgia House elections on Tuesday -- and 19 Republican senators looking nervously toward their 1984 reelection campaigns.

All across the Republican spectrum in the House, from moderate "Gypsy Moth" Rep. Jim Leach of Iowa to conservative "Yellow Jacket" Rep. William E. Dannemeyer of California, there is a recognition that the political realities have changed. "The coalition that dominated the House in 1981-82 is dead," said Leach, referring to the alliance of nearly all 192 Republicans and 40 or so conservative southern Democrats that gave Reagan his tax and budget victories.

Dannemeyer agreed "we'd be kidding ourselves" to pretend otherwise.

"Our coalition was so slim in the 97th Congress , I question if we can resurrect it," he said. "Something's got to give."

Many congressional Republicans agreed with a freshman survivor, Rep. E. Clay Shaw Jr. (R-Fla.), that "we're facing a situation where the great communicator has to become the great compromiser."

But that view is far from universal in GOP ranks. Such influential House Republicans as Rep. Dick Cheney of Wyoming, the chairman of the Policy Committee, and Rep. Barber B. Conable of New York, the ranking minority member on the Ways and Means Committee, urged the president to be "tough" and "consistent" and to "keep his powder dry."

Sen. Paul Laxalt (R-Nev.), probably Reagan's closest friend on Capitol Hill, said his advice is to "pursue essentially the same course as we've been following," in hopes economic recovery will arrive well before the 1984 election.

Senior White House aides fully expect Reagan to stake out a strong bargaining position by submitting a budget in January that reflects his original 1980 program of accelerating the defense buildup, cutting tax rates for three straight years and shrinking the domestic side of government.

But with Republicans as conservative as Rep. Jack Kemp of New York agreeing with most Democrats that "we've reached about the limit of the domestic programs you can cut in a recession," and with almost all of them wary of being cast as the villains in the cutback of Social Security and other entitlements, the stage could be set for a bruising confrontation.

Senate Appropriations Committee Chairman Mark O. Hatfield (R-Ore.) said last week that if Reagan submits a budget that makes no concession to the "fundamental change" in the makeup and outlook of Congress, "it will have a shorter life than the one they sent up last year," which was unanimously rejected by the Senate Budget Committee last May.

But that prospect does not intimidate Reagan.

"Quite candidly," said one of the president's top aides, when told of the Hatfield prediction, "that is the likely scenario. But the president is a good negotiator and he'd rather start from his own program, not theirs."

The idea that a president would submit a budget in the expectation that it will be junked by his own party in Congress is unusual enough. But some strong Reaganites on Capitol Hill, like Rep. Tom Corcoran of Illinois, see a substantive as well as a political risk for Reagan in doing that.

Corcoran and others noted that Senate Budget Committee Chairman Pete V. Domenici (R-N.M.), who is up for reelection in 1984, has announced that his main goal is economic stimulus to reduce unemployment and future deficits.

And, most observers predicted that House Budget Committee Chairman James R. Jones (D-Okla.) will be tugged to the left by the big new class of mostly liberal Democrats elected this month.

With the odds against Republicans controlling the House budget vote, as they did in 1981 and 1982, the possibility is that the non-veto-able budget resolution could confront Reagan with a basic economic and defense blueprint that is not much to his liking.

Still, Corcoran, like others, has urged the president "to stick to his guns," but be ready to enter the bargaining process much earlier than he did this year.

And that is what the White House said he will do. "We see no risk that he will be dealt out of the game," said one senior aide, "because ultimately, nothing is going to go through the Congress that he does not support."

That proposition is likely to be most severely tested on national defense. Reagan's proposed $1.6 trillion defense buildup over the next five years probably is causing Republicans as much heartburn as anything else.

Even some of the most hawkish GOP legislators, and the most loyal of Reagan's lieutenants in Congress, agree that the rate of growth in military spending that Reagan wants must somehow be scaled back, although a consensus has yet to form over the nature and extent of possible cutbacks.

Nearly all worry about the impact of the defense buildup on the deficit. Some also fret about what they call equity: a growing sense among voters that the Pentagon isn't bearing its fair share of the cost-cutting. A few, pointing to the recent success of nuclear freeze initiatives, said they fear a dangerous escalation in the arms race.

But even those relatively few Republicans who voiced skepticism from the start about the wisdom of the speed and size of the buildup conceded that there is little room for big savings in the Pentagon's budget, at least for the immediate future. Quick deficit reductions could be found in maintenance and operations and military pay, but these are the precisely the areas that almost everyone, from hawk to dove, wants to strengthen.

Elimination, modification or delay of weapons systems would save billions in future years as the bills for the new weaponry come due, but immediate savings from the initial stages of development would be modest.

Moreover, few if any GOP lawmakers are advocating a total reversal of Reagan's course on defense.

With the White House apparently holding firm on its defense buildup plans, even to the extent of refusing to be locked into future-year spending limits proposed by Congress earlier this year, some Republicans as well as Democrats have said that Congress will do the Pentagon surgery if Reagan doesn't.

Reagan had "better be willing to accept less because he's going to have little alternative," said Conable, a defense booster who said he finds this turnabout regrettable. "The admirals and generals are out to get all they can, and the impression the public gets is that we're just throwing money at defense problems."

Even winning congressional approval for this year's defense appropriations bill from the lame-duck 97th Congress "will take more cuts than he's willing to accept," said Conable.

"Defense is where he Reagan is going to have to demonstrate a willingness to compromise, or the Senate and House will just set a lower level of spending ," said Rep. James A. Courter (R-N.J.), a conservative Armed Services Committee member.

But Rep. Jack Edwards (Ala.), ranking Republican on the House Appropriations subcommittee on defense, pointed out that "where the battle has to be fought is between the White House and the Pentagon because Congress is really limited in what we can do to cut the defense budget. We can argue that they can save something here or there, but we can't make it happen. It has to be cut in the Pentagon and in the White House."

Congress can signal its intentions, however, by taking a whack at some big Pentagon spending targets, and that is what some Republicans expect during the lame-duck session from critics of the recently announced deployment system for the MX missile.

Many Republicans as well as Democrats are skeptical of the deployment system, and some fear a protracted, enervating struggle.

On the domestic spending side, Republicans talk in at least general terms of big savings from major benefit entitlement programs like Medicare and Medicaid, but are increasingly reluctant to try to squeeze much more out of annual appropriations for programs ranging from education for handicapped children to feeding for the elderly.

Hatfield bluntly rejected further appropriations cuts and is pushing for some increases. And some, like Durenberger, one of a half-dozen Senate Republican moderates who got a reelection scare this fall, have become increasingly cool even to Medicare and Medicaid cuts, except perhaps for some reimbursement savings.

On the other hand, conservatives like Sen. William A. Armstrong (R-Colo.), also a member of the Finance Committee, are poised to fight for more cuts. Some, like Sen. Warren Rudman (R-N.H.), see a strong tie between defense and domestic benefit cuts.

"If the president doesn't cooperate on defense cutting , he won't get the cooperation of the Democrats on the House side for the leverage he needs to get entitlement growth under control," said Rudman.

Social Security presents a unique problem.

Republicans were bloodied on the issue in the campaign and are anxious to have Reagan and O'Neill lead the way, arm in arm, when commission-drafted proposals for solving the system's financing problems come before Congress early next year. Many Republicans, tired of being accused to trying to balance the budget on the backs of the elderly, also are cool to the idea of trying to wring any big deficit reductions out of Social Security.

"Obviously we have to be careful about Social Security," said Conable, adding that "Republicans are quite capable of booting it; we have in the past."

Conable worried that Reagan "just doesn't like the subject" and that there still are people in the White House "who talk about making it voluntary."

In general, the economy has caught Republicans, especially those from hard-hit states, in a fiscal dilemma.

They still want to shrink deficits by reducing spending, but they also want to accelerate economic recovery, which has given rise to considerable GOP support for a little old-fashioned pump-priming, starting with 320,000 highway and bridge repair jobs.

"Monetarism is not the be-all and end-all; recovery is," claimed Sen. Richard G. Lugar (R-Ind.).

Domenici has said anti-inflation efforts have to take a back seat to economic recovery, and Baker even is discussing pulling the reins on the Federal Reserve to force lower interest rates.

Almost lost in the shuffle is the New Right's agenda of abortion, school prayer, busing and other so-called "social issues," which was all the rage just two years ago.

"If there's any message I perceive, it's that Republicans have spent too much time on social issues and not enough on basics of budget and spending issues," said Alabama's Edwards. "The repudiation of Sen. Jesse Helms of North Carolina and his agenda has been nationwide."

Behind all these legislative strategy debates is the growing Republican concern over the implications of the 1982 election for their continuation in power past 1984.

Durenberger said that "the sense of pessimism out there (among his colleagues) is greater than I expected. There's a great deal of fear about 1984 as a Republican."

Not all those running in 1984 feel that way, of course.

Armstrong, a conservative first-termer, said he "can't imagine the class of '78 turning liberal," but added that "even before the bloom was off the rose, I was independent."

And Lugar, who is seeking the chairmanship of the Republican Senatorial Campaign Committee, said, "We're at a very dangerous juncture" for both the economy and for the Republican Party.

The full extent of that danger was spelled out for Reagan in post-election analyses from White House pollster Richard B. Wirthlin, detailing the losses Republicans suffered in 1982 among the elderly, young people, women, farmers, blue-collar workers, union families and ethnics--all groups among whom Reagan ran far above the Republican norm in 1980.

But White House aides see little likelihood that Reagan will shape his budget to please any or all of those groups.

"He's made it very clear in the budget discussions," said one aide, "that he is willing to pay a price and let some of these groups go unsatisfied in the short term, if it will help create the climate for economic growth in the long term."

That contradicts the expectation voiced by Rep. Willis D. Gradison Jr. (R-Ohio), among others, that the approach of another presidential race "will force the president and the White House to focus more on short-term than long-term economic solutions." But it also suggests the potential for more conflict between the White House and Congress.

Charles Black, a conservative campaign consultant who worked with Reagan in the early stages of the 1980 race, said, "The Republican leadership and membership (in Congress) are unwilling to go into 1984 with deficits and unemployment the size we had in 1982. If that means they have to break with Reagan, they will."