On Nov. 4, President Carter lost an election. On that same day, Patrick H. Caddell and Gerald Rafshoon lost as well -- not just an election in which they were deeply involved, but also their entree to the lucrative spoils of victory.

Caddell is the head of Cambridge Survey Research, the polling firm the president used in his two campaigns and throughout his term in the White House. Rafshoon, who heads Rafshoon Communications, has handled Carter's campaign advertising during the same period.

Records at the Federal Election Commission show that these have been good years for the two firms, yielding millions of dollars in payments from the Carter campaign committee and the Democratic National Committee, particularly during the 1980 presidential campaign.

But in the aftermath of defeat, a squabble has broken out between Caddell and the DNC over a $400,000 bill from this year's campaign. Those involved say they expect the matter to be settled with the eventual payment of the money to Caddell, who blames it all on "personalities" and the political maneuvering to succeed outgoing DNC Chairman John C. White.

At issue is a final bill from Caddell's firm for polling services at the end of the Carter campaign. The president's reelection committee has already spent virtually all of the $29.4 million in federal funds it was allowed during the general election. Thus the bill was sent to the DNC, to be paid out of the $4.6 million the national committee legally could spend for Carter's reelection effort.

But when the bill came due on Nov. 7, the DNC balked. Debt-ridden for more than a decade, the committee now faces a major rebuilding job in the wake of the party's 1980 loss of the White House and the Senate, and a possible cash-flow squeeze at least through the end of the year. The Caddell bill upset some party officials and brought to the surface some long-held resentments toward the Carter inner circle.

Caddell, though his professional expertise as a pollster is not questioned at the DNC, is at the center of many of those resentments. For the last four years he has been, the Democratic Party's official pollster, enjoying the blessing and full backing of the White House.

FEC records show almost $400,000 in payments from the DNC to Caddell's firm for polling services between 1977 and last summer. During the same period, the records show an additional $750,000 in payments and obligations of the DNC to Caddell, including polling bills left over from Carter's victorious 1976 campaign, assumed by the DNC without any sign of a squabble.

Much of this more than $1 million in billings and other obligations incurred by the DNC was for surveys ordered by the White House in mapping political strategy for the president. In many cases, according to DNC sources, the polling results were never seen at party headquarters.

Since last summer and the beginning of the general election campaign, according to the records, the DNC has paid Caddell's firm more than another $1 million. This includes $711,000 for surveys in connection with the Carter campaign and $340,000 for polling for Democratic Senate candidates.

Meanwhile, Caddell has also collected more than $1.4 million for his polling services directly for the Carter/Mondale Reelection Committee, according to the FEC records, which cover expenditures through Oct. 15. Additional expenditures during the last two weeks of the campaign are to be included in reports the DNC and the Caddell committee are to file with the FEC next month.

Like Caddell, Rafshoon has also enjoyed a position of privilege during the last four years. He estimates that his advertising firm handled about $20 million in billings from the Carter reelection committee during the general election campaign and $4 million during the primaries.

In addition, according to the FEC records, the DNC paid the Rafshoon firm $400,000 in October as part of its share of the Carter advertising effort. The national committee also assumed Rafshoon's remaining bills from the 1976 campaign, paying off more than $400,000 during the last four years.

Neither Caddell nor Rafshoon can expect the next few years to be as lucrative as the last four. As for the DNC, it can expect to remain in debt. According to Tim Finchem, staff director of the Carter campaign committee, who is trying to sort out the money matters with DNC officials, Caddell's unpaid $400,000 bill is only part of about $1 million in campaign expenses the DNC will have to pick up from the almost defunct Carter/Mondale committee.

Finchem said that from the beginning of the campaign the DNC agreed to provide the full $4.6 million it was allowed by law to spend as part of the Carter reelection campaign, regardless of whether this involved polling services, telephone bills or other items. He said he is working with DNC officials to ease the committee's cash-flow problems and that he can understand the reluctance to pay off the Caddell bill immediately.

"It's a lot of money to put out the door three days after you lose an election," he said.