In the early days of Ronald Reagan's presidency, when no one knew whether his far-reaching economic program would be a smashing success or dismal failure, Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) described it as a ''riverboat gamble.''

Today, the ''gamble'' has paid hanbdsome dividends to Reagan but also cost him dearly.

On the threshold of his second term, Reagan can look back on a record far more successful than critics thought possible in those early days. But he begins his second term still saddles with some of the biggest failures of his first.

The record shows that Reagan kept his campaign promises of a tax cut and defense buildup but also that he failed to balance the budget and that the superpowers were plunged into new tension.

Today, almost everything on Reagan's agenda for the next four years is overshadowed by these immense difficulties: the huge and growing deficit and the legacy of four years of damaged relations with the Soviet Union.

Reagan, who promised to balance the budget and came into office warning about the dangers of an $80 billion federal deficit, now faces a deficit more than twice that size and has fewer palatable options to control it.

Reagan, who came into office convinced that the Soviets could more easily be persuaded to reduce arms in the face of a U.S. military buildup, now confronts more difficult choices about what positions to take in nuclear arms negotiations with Moscow.

Both challenges, if met successfully in a second term, hold the promise for Reagan of a final triumph after four years of struggle and disappointment. Both will test his talents as a communicator and instincts as a negotiator. Both will test his limits as a politician and an international leader.

Since his landslide reelection, Reagan has almost withdrawn from public view. He has deliberately made no speeches to the American people. In the quiet interlude, he has waded through black loose-leaf notebooks filed with multiple-choice options for budget cuts or worked out the delicately crafted language of an arms-control statement to be read to the Soviets.

But if Reagan's career in American politics is any guide, this has been only the prelude to a symphony of speeches and appearances, starting with today's tone-setting Inaugural Address and next month's State of the Union speech and continuing for four years.

He will try again soon to persuade a doubting Congress to resume aid to rebels fighting the Sandinista government of Nicaragua. He will seek to pry loose more money for the endangered MX nuclear missile. He will be cajoling Congress to give him still deeper spending cuts in politically popular programs that date to the 1930s, such as farm price supports.

Reagan will try defending from congressional critics a defense budget that has soared in five years. He will continue preparing for resumed negotiations with the Soviets on limiting nuclear weapons, with his administration still divided internally on vital arms-control issues.

Reagan also faces one of the most daunting selling jobs of his political career. It was relatively easy to persuade Congress to cut taxes in 1981, but his hope for a simplified tax code, with lower rates for individuals and higher corporate taxes, may prove far more difficult to achieve.

''The people voted against tax increases, and they were right,'' Reagan said a few weeks after he defeated Walter F. Mondale last November. ''They voted against wasteful government spending, and they were right. They voted for the expansion of opportunity for all, and that's what we mean to achieve, with policies that will control spending, simplify the tax system, improe America's productivity and competitiveness, so that we can keep this great nation of ours moving forward.''

Reagan had opened his first term with the same note of urgency. ''We must act today to preserve tomorrow,'' he declared in his first Inaugural Address. ''And let there be no misunderstanding -- we are going to begin to act beginning today.''

He launched a grand experiment in curbing the size and scope of the federal government. It was an experiment as ambitious in many ways as the New Deal of President Franklin D. Roosevelt, whom the young Reagan admired, an experiment seeking to reverse government expansion set in motion by Roosevelt and given momentum by President Lyndon B. Johnson's Great Society programs of the 1960s.

On the eve of the second phase of Reagan's experiment, his performance and prospects, measured against the economic, regulatory and social goals he laid out at the beginning, have these dimensions:

* Reagan said in his campaign five years ago that he could cut taxes, increase defense spending and balance the budget by 1984. He cut taxes and achieved more military spending than promised, but he presided over the largest peacetime deficits in U.S. history. They are estimated at more than $200 billion annually if no further action is taken.

Reagan declared in the last campaign that deficits would melt with economic growth. The trend seems to be the opposite: instead of shrinking in times of economic recovery, the deficit is growing.

Reversing this remains the top domestic-policy item on the Reagan agenda but promises to be difficult. Reagan has largely left to Congress in recent weeks the initiative on making difficult budget choices on defense spending, Social Security and taxes. When Senate Republicans write their own budget, Reagan may agree to accept it as a compromise, then press a confrontation with House Democrats over it.

* Reagan pledged to shrink domestic spending and appreciably reduced its growth rate, from about 5 percent annually in the 1970s to less than 1 percent in 1980-84.

Reagan has often discussed reducing government's share of the economy but failed to meet his first campaign foal of reducing federal spending to 19.4 percent of the gross national product (GNP). Many officials now talk longingly about reaching 21 percent.

One reason that Reagan missed the mark is that the 1981-82 recession slowed GNP growth. But today, in the third year of the economic upswing, spending remains at more than 24 percent of GNP. After two months of agonizing, Reagan is expected to propose a budget next month that would shittle it to 23 percent in fiscal 1986.

One recent White House plan, approved by Reagan's advisers, would have lowered spending to 21.5 percent by fiscal 1988, but officials now say Reagan's next budget will fall short of that goal.

For all of Reagan's promises about shrinking government's size, he actually reordered its priorites. He shifted the mix of federal spending from domestic programs toward defense, while toal spending did not shrink but grew. Many of the budget cuts have come in federal grants to state and local governments, despite Reagan's stated desire to strengthen them.

* As a further step in paring the government, Reagan vowed to eliminate the departments of education and energy. Faced with stiff congressional opposition, Reagan failed to do either and gave up trying during his first term. He recently appointed new secretaries to head each and plans to include both in the next budget, signaling that it is unlikely that the departments will be dismantled soon.

In fact, the only programs wiped out during Reagan's first term were relatively minor ones, such as trade adjustment assistance, or those that Congress was prepared to overhaul, such as the Comprehensive Employment and Training Act jobs program.

Reagan is targeting other programs for elimination at the outset of his second term, including the Small Business Administration, mass-transit subsidies and some urban-aid programs, but is likely to have only limited success with Congress, where each has entrenched defenders.

* The president fulfilled his promise of a three-year, across-the-board cut in individual income-tax rates. Congress also cut corporate taxes and approved tax ''indexing,'' which keeps taxpayers from being pushed into higher brackets because of inflation.

Since enactment of the first tax bill, however, Reagan has accepted four partially offsetting tax increases: one in Social Security, a 5-cent-a-gallon gasoline increase, hefty and largely corporate tax increases in 1982 and another smaller such inrease last year.

Overall, Reagan took back about one-third of his original tax cut in later increases, but indexing and individual tax-rate cuts have survived.