It all started at one of Pamela Harriman's famous dinner parties, a Georgetown gathering of the rich and powerful, Democratic warhorses all, brought together amid the French antiques, Chinese porcelain and Impressionist paintings to plot how to save the Democratic Party, the "party of the people."

It was the kind of group few but the Harrimans could assemble. Among the guests were former defense secretary Clark Clifford, Robert Strauss, House Speaker Thomas (Tip) O'Neill, House Majority Leader Jim Wright, Senate Democratic Whip Alan Cranston, Democratic senators John Glenn of Ohio, Dale Bumpers of Arkansas, Gary Hart of Colorado, Ernest F. Hollings of South Carolina.

All were vert introspective. Some argued the Republicans had simply outorganized the Democrats, that they had better comuters, better fund-raisers, a better national chairman. Others, just as forcefully, maintained the party had lost touch with the middle class and run out of ideas. m

Mrs. Harriman and her husband, Averell, the former ambassador and New York governor, were deeply impressed. They have long been big contributors to the party and its candidates.

But Pamela Harriman decided the times called for more drastic action, that they demanded she do just as dentists, steelworkers, General Electric, trial lawyers and the New Right had done before her. She would form a political action committee, called "Democrats for the '80s."

"I firmly believe 1982 is very important not only to the Democratic Party, but to the country," Mrs. Harriman said in a telephone interview from Barbados last week. "I believe if we [the Democrats] lose in 1982, we will no longer have a two-party system."

As one might expect, her PAC won't be quite like any other. First, it has an unusually ambitious mandate and is aimed at helping party moderates. In addition to raising money for candidates, "Democrats for the '80s" want to develop new party policies, and, like several conservative PACs, do political organizing. But more interesting than what the PAC will do is who will do it.

Stuart Eizenstat, Carter's former domestic policy chief, has agreed to head a policy unit. Former Arkansas governor Bill Clinton has agreed to take a similar role in political organizating. Jesse Calhoon, president of the Marine Engineers Beneficial Association, and the John Bowles, New York investment banker, are on its board of directors. Howard Samuels, the former New York off-track betting chief and a onetime candidate for governor, is committee treasurer.

How did this all come about?

A few words of background are helpful here. Pamela Harriman is born to money and position, has spent her life among some of the legendary figures of our time. The daughter of an English baron, Harriman was first married to Randolph Churchill, the late son of Winston Churchill. Her second husband was the late Leland Hayward, the Broadway producer.

As the wife of the nation's most prominent elder statesman, Mrs. Harriman, 60, is best known in Washington as one of the city's leading hostesses -- a title she detest.

"I hate that word 'hostess,'" she said in an interview. "I've always tried to do what I believe in. I just like to communicate."

In putting together her PAC, Mrs. Harriman has demonstrated her ability to communicate. She has touched bases with every important segment of the party, from AFL-CIO President Lane Kirkland and Sen. Edward Kennedy to a host of little-known young southern congressmen. She has held two well-attended dinners and a reception in her home that attracted 85 labor leaders.

The Harriman home is a logical place to talk about what has gone wrong with the Democratic Party, says House Majority Whip Thomas Foley. "An invitation to the Harrimans' is a little bit like an invitation to the White House. People just don't say no . . . She is a dynamic and intelligent woman, and the governor is one of the great figures of the Democratic Party. They provide the catalyst for a lot of things."

One Democrat, who attended a dinner held Jan. 21, the night after Ronald Reagan's inauguration, says Mrs. Harriman deliberately played a background role and made no formal presentation about the committee. "She didn't want to get up and say, 'I have a marvelous new idea,'" he said. "She just let the idea unfold."

The new PAC's initial fund-raising goal is a rather modest $1 million. Unlike several other new Democratic political action committees, this one is interested in helping moderate and conservative candidates as well as liberals, said Mrs. Harriman. "We will be very interested in races where we can win." s

The committee hasn't decided how to raise money. One idea that intrigues some is selling tickets to a round of dinners at the Harrimans'. A dinner invitation could go for $1,000 to $5,000.

When this was suggested to Mrs. Harriman in an interview, she seemed surprised. She said she had no plans for such an undertaking. There will be more dinners, however, she said. "We mean to continue a dialogue with our friends on the Hill."