Congress returns Monday to a thicket of unfinished business dominated on the financial front by two major international issues--one affecting the United States' relations with Third World debtor nations, the other Reagan administration efforts to curb the supply of high technology to the East Bloc.

They are both likely to come up early in the session following the Labor Day recess and are expected to be highly controversial, with angry House Democrats threatening to recant their support for Reagan administration efforts to win final passage of an $8.4 billion authorization bill for the International Monetary Fund.

The administration has called this bill vital to the stability of the world banking system, and would like to have it passed and on President Reagan's desk for signature by the time the IMF's annual meeting opens here Sept. 27. Many other nations are awaiting American action on the IMF quota before making their pledges. The IMF, meanwhile, is going through severe cash shortages and is negotiating to borrow money itself.

The other key international financial matter before Congress is a replacement for the Export Administration Act, which expires Sept. 30. Before the recess, the administration-sponsored version of the bill, designed to restrict exports of high technology to the East bloc and Soviet Union, appeared to stand little chance of passage with both House and Senate committees reporting out bills far less restrictive on high-tech exports than the White House requested.

The fierce battle between administration hawks and doves to forge a compromise bill was repeated during the spring and summer on Capitol Hill, with both sides losing ground in committee sessions to a well-organized business lobby whose arguement was that trade restrictions hurt the United States more than leakage of high technology helps the Soviet war machine.

It is unclear, however, whether the Soviet downing of a Korean passenger jetliner will strengthen the hand of hawks in the administration and Congress and undercut efforts by business to ease trade restrictions.

Before the recess, it appeared that supporters of stiff curbs on high-tech transfers wanted to stall passage of any bill, hoping Congress would give up and vote a simple extension of the present law, considered stronger than either the current House or Senate version. Both congressmen and senators vowed to oppose that strategy, but the current intensification of East-West strains may change their opinion.

Neither house appears likely to take up the Export Administration Act before the second week of the session, and its consideration is expected to follow the final vote on the IMF authorization measure.

Finding itself in the unusual position of having two international finance measures high on its agenda, Congress is forced to take up the bills one after the other because many of the same staff members and legislators play key roles in both.

No matter how a conference reconciles differences between the two versions, the IMF authorization is in trouble in the House where, despite strong White House lobbying, it eeked out a bare six-vote, 217-211, victory last month.

That support remains shaky as the bill was attacked as a bailout for big banks, who lured Third World countries into a record $600 billion in loans they couldn't afford. It also drew opposition from steel state congressmen, who argued that IMF funds helped subsize steel mills in newly industrialized nations so they could ship cheap steel to the United States, costing Americans jobs.

Those built-in antagonisms were compounded by a National Republican Congressional Committee press release that labeled Democratic supporters of the Reagan administration's position on the IMF bill pro-Communist. The release was aimed at the home district of 20 Democrats considered vulnerable by the NRCC.

House Democrats, whose votes helped give President Reagan his IMF victory, reportedly are incensed over the press release. It quotes ultraconservative Rep. Phil Gramm (R-Tex.) as asking, "What gives them the right to support communism?" because Democrats voted against an amendment the White House opposed. The amendment, introduced by Gramm, would have prohibited the United States from supporting IMF loans to countries run by "communist dictators."

House Banking Committee Chairman Fernand J. St Germain (D-R.I.) linked the IMF authorization to administration help for Senate passage of a housing bill he considers a top priority.

The Sept. 30 deadline for extending the Export Administration Act notwithstanding, the timing for action on the bill remains in doubt. This is especially true in the Senate, where the Finance Committee has raised key questions on some provisions of the bill that was cleared by the Banking Committee. That jurisdictional tiff is likely to be resolved, leaving the House and Senate to decide the larger issue of how stiff controls on exports need to be in order to protect America's national security.

The Reagan administration was deeply split over that issue, and Senate aides said representatives from the Commerce and Defense departments took different positions lobbying on Capitol Hill.

The hawks were led by Assistant Secretary of Defense Richard N. Perle, who consistently has held a staunch anti-Soviet position; Stephen D. Bryen, a deputy assistant secretary of Defense who was a close associate of Perle when they both worked for the late Sen. Henry M. Jackson (D-Wash.), and Commerce Assistant Secretary Lawrence J. Brady, a key Reagan campaign aide in New Hampshire.

Brady's boss, Commerce Undersecretary Lionel H. Olmer, led the fight within the administration for controls that reached a middle ground between protecting national security while allowing American high-tech sales overseas.

Even so, a House Foreign Affairs subcommittee led by Rep. Don Bonker (D-Wash.) reported out legislation that loosens restrictions far more than even administration doves wanted. The congressmen appeared heavily influenced by arguments that America's high-tech industry needs exports to remain on the cutting edge of research and development that makes it the world leader and that many of the products the Pentagon wants to ban are readily available from other countries.

Although the Senate bill sponsored by Sens. E.J. (Jake) Garn (R-Utah) and John Heinz (R-Pa.) takes a stronger national security tack than the House version, it still is considered too soft by the Pentagon hawks.

Also on the international front, Commerce Secretary Malcolm Baldrige will continue the administration push for passage of trade reorganization, replacing the Commerce Department with a slimmed-down Department of International Trade and Industry (DITI) modeled after Japan's Ministry of International Trade and Industry (MITI.) The bill is given a good chance of passage in the Senate, where its prime sponsor, William V. Roth (R-Del.), heads the Governmental Affairs Committee which is expected to mark up the bill later this month.

There is, however, strong opposition from Senate trade experts, such as John Danforth (R-Mo.), and some farm groups that could derail Senate approval.

Passage in the House, however, is more in doubt, because the leadership of the Ways and Means Committee, which handles most trade legislation, has publicly opposed the reorganization. The plan also has divided the business community.

Domestically, there is a real chance the nation could be left without a housing bill again this year despite St Germain's attempt to link its Senate passage to House approval of IMF authorization. An appropriation bill has passed, but the authorization measure is stalled in the Senate where William L. Armstrong (R-Colo.) is threatening it with more than 60 amendments.

The bill would extend the Community Development Block Grant and the Urban Development Action Grant programs, and major federal housing assistance for farmers, Indians and the poor. It also threatens the Federal Housing Administration loan program as the FHA authority is due to lapse. But Congress is likely to renew FHA's authority separately because its mortgage loans affect many middle class constituents as well as homebuilders.

In the field of telecommunications, there are strong moves in the House and Senate to repeal a Federal Communications Commission ruling that requires subscribers to pay a monthly fee for access to long-distance service. This could be politically popular because it keeps rates from going up Jan. 1, but defeats FCC efforts to make users pay the true cost of phone service.

Despite the extremely rapid pace of change in the finance industry, which has radically altered the way Americans do their banking, new legislation is unlikely to pass Congress this session.

Still stalled because of wide differences between the Senate and the House is a new bankruptcy law to replace one the Supreme Court ruled unconstitutional more than a year ago. There is little indication those differences will be resolved during this session of Congress.