To hear the contestants tell it, the race to represent Manhattan's East Side in the House of Representatives is being fought block by block, brownstone by brownstone.

Checkbook to checkbook might be more accurate.

In a district that includes Bloomingdale's and the Bowery, the two contestants, mild-mannered liberal Republican Rep. Bill Green, 54, and brash liberal Democrat Andrew Stein, 39, the Manhattan Borough president, may set the spending record this year for House races.

With just days left in the campaign, these scions of extremely wealthy families have spent a total of nearly $1.3 million to win this silk-stocking congressional district.

When the race is over, both agree, more than $2 million, including a healthy chunk from their own pockets, will have flowed into New York's political economy: for television advertisements, pollsters, campaign consultants and sign and leaflet printers.

Green and Stein are not the only ones spending tremendous amounts on congressional campaigns.

In Orange County, Calif., Democratic Rep. Jerry M. Patterson and his aggressive challenger, former representative Robert K. Dornan, have raised a total of more than $1 million.

House Budget Committee Chairman James R. Jones (D-Okla.) and his GOP challenger, former prosecutor Frank Keating, also have topped the $1 million mark and are expected to go much higher.

"Procter & Gamble spends millions each year advertising toothpaste," said Keating spokesman Paul Lee. "Well, this is a little more important than toothpaste."

Since 1978, when the $1 million threshold was crossed in a House race -- as it happens, in Bill Green's district -- the number of megabuck contests has been increasing.

In 1980, according to Federal Election Commission (FEC) reports, two House members spent more than $1 million on their campaigns. In 1982, the number jumped to five.

Democratic and Republican campaign officials say that nearly a dozen races could reach the mark this time.

"We have a system that's out of hand," said Fred Wertheimer, head of Common Cause, a citizens' lobby that has criticized the way political campaigns are funded.

"It reflects an arms-race mentality . . . when you get up to the kinds of amounts being spent now, it just doesn't make any sense," said Wertheimer.

The increase in megabuck races mirrors a trend toward high spending in all House campaigns.

Democratic and Republican officials say there has been a steady increase in the amount of money the average candidate must spend to be taken seriously.

In 1978, the figure was around $175,000, they said. It has risen to about $225,000 today. According to the FEC, successful House candidates spent a total of $55.6 milllion in 1978 and nearly $115 million in 1982.

Why would someone spend $1 million, often dipping into personal funds, for a two-year, $72,000-a-year House post with no job security?

"You get caught up in the heat of the battle," said Adam Levin, a New Jersey Democrat, who holds the spending record for his unsuccessful $2.3 million race ($1.4 million of which was his own money) against Rep. Matthew J. Rinaldo (R-N.J.) in 1982. "Would I do it again to the extent to which I did? I don't think so," Levin said.

House races that cross the $1 million threshold have a few common denominators:

More often than not the race occurs in an urban area, such as New York or Los Angeles, where television costs are high. In hotly contested races, television can be invaluable, quickly reaching hundreds of thousands of people.

A challenger appears to have access to money, either his or her own or through promises of contributions by influence-seeking political action committees (PACs). The incumbent responds by trying to build up a huge war chest well before the race, which in turn forces a challenger to try to raise even more.

In many cases, the incumbent has had difficulty in the previous election and appears vulnerable to a well-financed attack. The opposing national political party targets the seat.

For instance, Jones, a moderate Democrat whose conservative Tulsa district is expected to vote wholeheartedly for President Reagan, was reelected in 1982 with only 54 percent of the vote after outspending his GOP challenger by more than 3 to 1.

Keating, Jones' opponent, is expected to spend at least $650,000, much of it on television time, trying to depict Jones as too liberal and too much in the mold of House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and Democratic presidential nominee Walter F. Mondale.

In response, Jones began raising money early. According to FEC reports, he now has raised more money than any other House candidate. He is also at the top of the list for PAC contributions, with 40 percent of his funds coming from them.

He has waged a television, radio and mass-mailing campaign stressing his record as an "independent" voice for Oklahoma and attacking Keating's record as a prosecutor.

Patterson, the moderate California Democrat, was considered similarly vulnerable after his last race. He, too, attracted a tough Republican competitor with a track record of finely honed fund-raising ability.

The flamboyant Dornan set a spending record for House races in 1980, the year he raisd $1.9 million to squeak by Cary Peck, son of movie star Gregory Peck.

Dornan later was redistricted out of his west Los Angeles seat. In 1982 he unsuccessfully ran for the Senate. Recently he moved into Orange County outside Los Angeles and announced his race for Patterson's seat.

Dornan has raised more than $711,000 and, like Keating, is using his money to send a message that Patterson is too liberal for the district.

Patterson has raised about half that amount, with a final target of $600,000, a figure well above what Patterson and his opponent together spent in 1982 and nearly triple what was spent in the district in 1980.

Patterson is stressing a five-term record of constituent service and is attacking Dornan as a carpetbagger from Beverly Hills.

The price tag is the same, but ideologically the contest in New York's silk-stocking district, so named because of its large number of wealthy residents, is the flip side of the Jones and Patterson races.

Green is a liberal Republican, a "gypsy moth" who has opposed many of the Reagan administration's cuts in social spending.

But his district is decidedly Democratic and liberal. Green has been on somewhat shaky ground there since his first election in 1978. He may be the only incumbent in the country this year who has to worry about Mondale's coattails.

Stein is attempting to paint Green as a Reaganite in liberal clothing. Green has been portraying Stein as a politician who works hard to get media coverage but not to help his constituents. Each has raised questions about the other's integrity.

Stein has loaned his campaign more than $226,000. He is spending most of his funds, some $500,000, on television advertisements.

Green intends to contribute about $50,000 of his own money for the final push, according to aides. Green said that in past races he has given a total of $250,000 that he has not recouped.

Green said last week that having gone through two extremely expensive races he firmly supports some sort of federal limitation on congressional campaign spending. But, he added, if "someone goes on a spending spree . . . , you have to stay in it."

Stein agreed that "it's a crazy system." But, he said, "it's the system. If you're going to get elected, you've got to do it. To me the effort and the money is worth it."