Congress faces a politician's nightmare as it returns to town today: an election-year recession, huge projected deficits that have led to calls for tax increases as well as spending cuts, and a persisting list of other sharp-edged issues ranging from voting rights to school prayer, to abortion, to decontrol of energy prices.

The congressional agenda will be complicated even further if, as expected, President Reagan proposes in his State of the Union message tomorrow a complex swap of programs and revenue sources among the federal government and the states.

Although the budget is expected to be dominant on Capitol Hill again this year, the session will open with consideration of the New Right's controversial agenda of "social issues," which were squeezed out by budget matters last year. Proposals to permit school prayer and ban school busing for racial purposes will come up almost immediately in the Senate, to be followed shortly by anti-abortion measures. Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) predicted yesterday that these issues would provide a "tumultuous" kickoff for the session.

Baker also promised to give "early priority" to legislation denying tax exempt status to private schools that practice racial discrimination, which is also expected to generate major controversy.

If these issues facing the second session of the 97th Congress will make life difficult for its members, they also pose more problems and risks for Reagan, whose political spell held the lawmakers in thrall for most of the 97th's first year of landmark budget and tax cuts.

"This year's going to be a lot tougher for everyone . . . a real test," said Senate Finance Committee Chairman Robert J. Dole (R-Kan.), forecasting a more partisan atmosphere stemming from the November elections as well as strains resulting from the country's economic problems, including deficit projections of $100 billion or more over the next few years.

As for the budget, which Reagan is scheduled to submit to Congress early next month, "to cut again across the board will be much more difficult," said House Minority Leader Robert H. Michel (R-Ill.). Michel, echoing the fears of others, also worries that soaring deficit projections could bring down the whole congressional budget control process, in that Congress will not be able to avoid the deficits but members will be unwilling to vote for the budget resolutions embodying them.

Reagan will propose more domestic cuts, but further reductions in the so-called discretionary domestic programs are expected to encounter stiff resistance on grounds that there is little fat left to trim from them after last year's retrenchment. The other domestic programs, the so-called entitlement or large basic benefit programs, were less severely cut last year and present a more lucrative target, but some lawmakers say the likely resistance to entitlement cuts has been underestimated. Even some Republicans are thus pushing in the only other possible direction, to scale back the president's proposed defense build-up, but they say they anticipate strong resistance from the administration.

Baker and others, however, caution against underestimating the resiliency of Reagan's popularity in the country and his clout in Congress. Democrats remain fearful, too, that congressional obstructionism could prompt Reagan to "run against Congress" in bidding for more Republican seats in the November elections. With Social Security solvency already set aside until after the elections, it is possible that other controversial matters may also be shunted off to a post-election "lame duck" session that is already being predicted as virtually certain.

A problem mentioned over and over again by Republicans as well as Democrats is frustration over the economy's failure to respond as quickly and as positively as they had hoped to the economic iniatives that Congress embarked upon last year under prodding from Reagan.

Even if the recession bottoms out by election time, Congress will be called on to make the grim Hobson's choice of raising taxes or putting its signature on record deficits, with some members fearing that it may wind up doing both.

Congressional action on taxes is possible even if Reagan doesn't recommend it. The specter of huge deficits may prompt Democratic efforts to roll back or limit the individual tax cut scheduled for 1983. While Republicans such as Baker will oppose a rollback, they may counter with proposals of their own to close tax "loopholes" and raise the so-called minimum tax for corporations. Many Republicans are edgy about charges that last year's tax and budget cuts gouged the poor for the benefit of the rich.

Both Senate and House Republican leaders are fearful that Congress--for the first time since passage of the congressional budget control act nearly a decade ago--may be unable to pass a budget resolution. "Reconciliation," the highly successful legislative shortcut under which Congress bunched all its budget decisions into one bill last year, ordering its committees to achieve stipulated savings, may be out for this year because of the difficulty of putting together an acceptable "reconciliation" package, Baker has said.

One reason for Reagan's phenomenal success with Congress last year was that, in contrast with the Carter administration's scatter-shot legislative agenda, Reagan's program was narrowly focused on tax and budget cuts. This conveniently relegated most other controversial issues--most conspicuously including the New Right's favorites of abortion, busing and school prayer--to the congressional closet.

But their planned reemergence early this year poses threats to last year's remarkable unity within Republican ranks and competes with fiscal and budget issues for White House attention, muscle and political capital.

Baker, who, in concert with the White House, kept the so-called "social issues" on the shelf last year, served notice last week that he plans to bring up prayer and busing in connection with pending Justice Department bills as early as this week. Abortion legislation will come later when anti-abortion forces, split over differing approaches, are ready to go with it. Likely to be considered first is a proposed constitutional amendment giving Congress and the states joint power to restrict or ban abortions.

If movement on these social issues is likely to please arch-conservatives, the administration's double turnabout on tax exemptions for private schools that practice racial discrimination is expected to have the opposite effect. Dissension within Republican ranks is expected when Baker, leading the charge for the administration, pushes for legislation to deny tax subsidies for the schools, thereby seeking congressional sanction for the antidiscrimination policy that the administration revoked earlier this month.

Other noneconomic issues expected to provoke controversy include:

* Extension of the Voting Rights Act. The House has passed a somewhat toughened law, and 61 senators, enough to break a filibuster, are cosponsoring the House version, although some may be shaky. The administration, joined by Baker, favors extension of the current law.

* Natural gas prices. The administration wants accelerated decontrol and is weighing whether to propose it to Congress. The last fight over gas prices lasted 18 months. Moreover, some lawmakers, including Baker, are saying that a speed-up of decontrol can't pass without a windfall profits tax on the proceeds, which Reagan has opposed.

* Departmental dismantling. The administration, making good on campaign promises, is expected to try to break up the Department of Energy and may seek abolition of the Department of Education. Both proposals could have tough sledding in Congress.

* Federal regulation. Continued efforts are expected to deregulate intercity bus transportation and the maritime industry, expand deregulation of savings institutions, curb federal rule-making authority and resolve antitrust issues in the telecommunications industry, possibly affecting the recent settlement between the Justice Department and the American Telephone and Telegraph Co.

* Ethics. The Senate is scheduled Feb. 2 to consider the expulsion of Sen. Harrison A. Williams (D-N.J.), one of seven members of Congress convicted in connection with the Abscam probe. The Senate Ethics Committee has recommended expulsion.

* Crime. Wide-ranging legislation to revise federal criminal laws is pending in both houses, with Democrats competing with Republicans to emerge as the election-year champions of a crackdown on crime. Legislation to restore the death penalty for federal crimes is also being pushed by conservatives.

* Jobs. Democrats may push programs to alleviate the effects of unemployment, and proposals are pending for revamped job-training efforts. The administration intends to introduce its plan for urban "enterprise zones" to encourage job expansion in decaying inner-city areas.

* Environment. Revamping of clean-air legislation is a holdover from last year. Other controversies will involve the proposed banning of oil and gas exploration from wilderness areas, extension of the Endangered Species Act, several clean water programs and nuclear waste disposal.

* Welfare. In addition to an expected administration proposal to trade off responsibility for several major welfare responsibilities with the states, the food-stamp program must be reauthorized and is again a target for cuts.

* Miscellaneous. Also pending are tightening of the Freedom of Information Act, legislation making it a crime to reveal the names of intelligence agents, user fees for port development, aviation and other services, efforts to curtail the Federal Election Commission and a standby oil allocation plan. Foreign aid, including aid to Central American countries and payments to the International Development Association of the World Bank, will also be an issue, even though Congress finally approved a foreign aid money bill last year.