The Democratic Party confronts a brutally simple challenge as the preelection year ends. Its two best-known presidential contenders, Jesse Jackson and the self-resurrected Gary Hart, are distrusted by so many American voters they are probably unelectable. Yet both have enough appeal to important sectors of the Democratic electorate that they can probably postpone and possibly prevent the emergence of an electable alternative.

Unless they find and unite behind that alternative, the Democrats will lose the White House for the fifth time in 20 years.

The Democrats are paying the price for saddling the country with a nominating system in which ambition overrides all other criteria. Gary Hart is serious about his policy ideas, and many of those ideas deserve to be taken seriously. But the motivation for his flawed candidacy, expressed at least twice in his interview with ABC-TV's Ted Koppel, is visceral: ''It was something I felt I had to do,'' he said. ''Just something I had to do to finish off what I began.''

In the new edition of his book ''The Presidential Campaign,'' Stephen Hess of the Brookings Institution argues that what distinguishes presidential contenders ''from other high officeholders of their generation {is} not their intelligence, accomplishments, style, or the reality of their prospects.

''What distinguishes them is presidential ambition,'' Hess says. Those with the worst case of the White House bug ''are proven risk-takers,'' a description custom-made for both the public and the private life of Gary Hart.

Jackson is another example of the supreme risk-taker as presidential candidate. He thrives on defying the odds, whether in a self-assigned international rescue mission, some high-profile private diplomacy or the leadership of a ''lost cause'' protest rally. Jackson has arrogated the high moral ground of one who ''puts himself on the line,'' and Hart aspires to a similar distinction.

In a basic sense, both Jackson and Hart are also ''protest candidates.'' They challenged Walter Mondale, the choice of the Democratic establishment in 1984. And both have, at various times, portrayed themselves as victims of rules rigged by the power brokers who control politics and the mass media.

Bashing the press and the political establishment is always good stuff on the stump. Generations of Democratic politicians, from William Jennings Bryan to George Wallace, have shown that you can reap millions of votes from the fertile fields of public indignation by urging the alienated to ''send them a message.'' But such candidates ultimately do not win the White House.

Between them, Jackson and Hart now control 50 percent of the Democratic primary vote, according to the latest ABC News poll, with no other candidate higher than 11 percent. That is scary news for the Democrats, because Hart and Jackson also carry the highest disapproval ratings from the electorate.

The challenge for the Democratic Party is finding an alternative candidate who merits the White House as much as he craves it.

The need to find an alternative may drive some wavering leaders to Massachusetts Gov. Michael S. Dukakis, others to Sen. Paul Simon of Illinois or Sen. Albert Gore Jr. of Tennessee. They represent geographically separate bases of support in the Northeast, the Midwest and the South. But none of them, nor the lower-ranking rivals, Rep. Richard Gephardt of Missouri and former governor Bruce Babbitt of Arizona, has yet demonstrated the ability to project his appeal much beyond his own sector of the party.

It could still happen. If Dukakis were to win the opening Iowa caucuses, or if Simon or Gore were to make a strong showing in New Hampshire, you might see the coalescing of sentiment that Democratic National Committee Chairman Paul Kirk prays for and predicts. But if the active contenders fail to head off Hart and Jackson, you can be sure someone will be enlisted to do the job. I have been skeptical of the scenarios for a late-starting candidacy, a draft or a brokered convention. But with Hart's reentry, they become a real possibility.

There are four men the leaders of the Democratic Party generally believe are of presidential stature who have declined so far to enter the race. They are as ''risk-averse'' as the active candidates, especially Hart and Jackson, are ''risk-addicted.''

Two are Easterners: New York Gov. Mario Cuomo and Sen. Bill Bradley of New Jersey. One is a southerner: Sen. Sam Nunn of Georgia. The last and least publicized is from the region the Democrats forget, the West: House Majority Leader Thomas S. Foley of Washington.

Foley is on everyone's list of the five best legislators in Congress and the three best television performers in American politics. He has brains, wit and a demonstrated instinct and talent for moving the big issues, like budget-deficit reduction, toward resolution. He is trusted by the politicians of both parties. His personal familiarity with the current and emerging leaders of Britain, Europe and Japan is unrivaled by any State Department careerist. He is the one man in Washington who was invited to breakfast, lunch and dinner with Mikhail Gorbachev last week.

Even the Democrats, dim-witted as they are, may find themselves desperate enough to notice him.