The 100th Congress, frustrated in its first seven months by strained relations with President Reagan and its own partisan bickering, returns from a month-long recess today to confront a massive pileup of unfinished business complicated by deadlines it cannot hope to meet.

Congress' problems with the White House -- and within its own ranks -- raise the prospect of a highly contentious session lasting until Thanksgiving or perhaps Christmas.

Reagan has a thin legislative agenda, much of it already rejected by Congress. A senior Republican staffer called it "nearing the point of irrelevancy" on Capitol Hill.

But the president still is a force to be reckoned with. With the help of filibustering Senate Republicans and veto threats, he can block most Democratic initiatives, dimming the Democrats' hopes for a solid congressional record for next year's elections.

"Congress and the president have to find a way of cohabitating, and that's the responsibility of the president as much as Congress," Senate Majority Leader Robert C. Byrd (D-W.Va.) said yesterday in an interview.

When Congress left Aug. 7, there were signs of improved relations with the administration, including an accord between Reagan and House Speaker Jim Wright (D-Tex.) on a peace plan for Central America and compromises over several relatively minor domestic matters.

But the issue of aid to the Nicaraguan contras is expected to erupt again, and tensions over risks to U.S. military forces in the Persian Gulf remain high. In addition, new flash points have emerged, including a possible sale of weapons to Saudi Arabia and potentially serious complications for the president's arms-control strategy.

The battle over Supreme Court nominee Robert H. Bork, with the Senate Judiciary Committee's confirmation hearings to open next week, is expected to be long and acrimonious, adding to partisan hostilities building as the 1988 presidential and congressional campaigns get under way.

Byrd yesterday repeated his warning that GOP delaying tactics on other matters could hold up action on Bork. "If they want to drag things out," Byrd said, "Bork's nomination is going to have to wait, too."

Moreover, all the old budget-related stumbling blocks remain, looming larger as time grows shorter and threatening the government with "crises" as the White House and Congress use the leverage of fiscal deadlines to angle for advantage on pet causes.

In theory, Congress must resolve the impasse over budget controls and raise the federal debt ceiling by Sept. 23, draft deficit-reducing tax increases and spending cuts by Sept. 29 and pass appropriations for the entire government by Oct. 1, the start of fiscal 1988.

In practice, it is expected to extend most if not all of these deadlines, dashing hopes for an October adjournment and putting off most major policy decisions until at least mid-November.

The first hurdle is agreement on new Gramm-Rudman-Hollings targets for deficit reduction and a mechanism to enforce them. Lack of an agreement is holding up passage of the debt-limit extension. Only when the new deficit targets are set can Congress seriously consider the tax increases and spending cuts necessary to achieve them. But compromise on those issues will remain difficult because of Reagan's opposition to tax increases and congressional reluctance to consider further spending cuts.

House Majority Leader Thomas S. Foley (D-Wash.) said last week he was "relatively confident" of reaching a Gramm-Rudman-Hollings compromise without further delay, although other lawmakers are not so sure. Failure could force another short-term extension of the debt ceiling (there have been two this year) and more delay in deficit reductions.

Appropriations present a grimmer picture. The House has passed nine of 13 regular spending bills for fiscal 1988, but none has even been reported out of committee in the Senate and few are expected to pass the Senate by Oct. 1. Senior appropriations aides say the best Congress as a whole can do by then is pass a stopgap "continuing resolution" to keep money flowing at current rates until compromises can be worked out, probably just before Thanksgiving.

This means that showdowns over arms control and other major issues, expected to be decided as riders to appropriations bills, will probably also be delayed. A delay could be significant on issues such as arms constraints, where timing has been critical in the past. Reagan has successfully used the approach of a U.S.-Soviet summit as an argument against congressional action to undermine the president's position. Another summit appears to be on the horizon.

But Democrats, still nursing wounds from their forced retreat on arms-control decisions in the face of last year's Iceland summit, insist they will not be dissuaded this time and have some bargaining leverage, courtesy of Senate Armed Services Committee Chairman Sam Nunn (D-Ga.).

To counter administration arguments that the negotiating record for the 1972 Antiballistic Missile (ABM) Treaty should be decisive in opening the way for expanded testing of the president's proposed space-based antimissile defenses, Nunn is demanding congressional review of all documents dealing with current negotiations on a treaty to ban small- and intermediate-range nuclear forces (INF). This could seriously complicate ratification of the INF pact that Reagan wants and puts pressure on the administration to abandon its insistence on a broad ABM interpretation.

Other arms-control issues, expected to be decided in connection with a final spending bill for fiscal 1988, include House proposals for a nuclear test ban and a resumption of adherence to weapons limits in the unratified SALT II Treaty.

The House, which has made far more progress on legislation than the Senate, is hoping to wrap up its work no later than Thanksgiving. But the more closely divided Senate, stymied by Republican filibusters and faced with a protracted battle over Bork, could be in session until Christmas.

As a result, House leaders want to send members home when their work is done, recalling them when necessary to consider such matters as House-Senate conference agreements. "We'll leave them {senators} our phone numbers and let them call us when they're ready to do business," quipped one House aide.

Furthest along among the major bills is trade. A conference committee has begun working out major differences between House and Senate versions, but reaching agreement is expected to be difficult and the White House has made veto threats. Both chambers are expected to pass textile-quota bills similar to legislation vetoed by Reagan last year.

Legislation to protect the elderly from costs of catastrophic illness has passed the House but is stalled in the Senate, partly because of controversy over extending Medicare coverage to outpatient prescription drugs. The administration objects to the program's cost.

Also pending with good-to-fair prospects for action in one or both houses are farm-credit relief, expansion of Medicaid coverage, increased funding for AIDS research, reauthorization of major education programs, new liability limits for nuclear plants, restrictions on polygraph tests, pension-law revisions, clean-air rules and air-travel protections. U.S. District Court Judge William S. Sessions is expected to be confirmed as director of the Federal Bureau of Investigation, along with C. William Verity as commerce secretary.

But overhaul of the welfare system is likely to slide until next year, along with several civil rights and labor-backed proposals, including an increase in the minimum wage. A Republican filibuster is blocking action on public financing of Senate elections and campaign-financing curbs. Another Senate GOP filibuster is holding up a defense authorization bill for the next fiscal year because of objections to arms-control provisions.

On foreign policy, a Congress that was reluctant to extend military aid to the Nicaraguan contras after the Iran-contra affair is expected to be cooler to the idea now that Central American countries are pursuing a regional peace plan. "Looking at it today, contra aid would have a hard time," Byrd said.

Efforts will continue to force the administration to invoke the War Powers Resolution in connection with Persian Gulf hostilities, but Congress is traditionally reluctant to cut off funding when U.S. forces are in the line of fire. Expected requests for new funding are likely to reopen debate, however.

Strong opposition is expected if the administration, as planned, proposes another Saudi arms sale. A presidential report on South Africa is due Sept. 30 and could provoke an effort to strengthen sanctions approved last year.

Foreign aid is headed for another thrashing, and Pakistan faces particular trouble because of reports that it is trying to develop nuclear weapons.