Rank-and-file Democrats show signs of a potentially sharp division in 1980, according to a new Washington Post poll.

Sen. Edward M. Kennedy (D-Mass.) holds a sizable but declining lead over President Carter in the race for the party's nomination, and both candidates are ahead of the leading Republican challengers, according to the poll.

But among those interviewed who identified themselves as Democrats, one-third of the Kennedy supporters said they would not vote for Carter against a Republican and one-third of those who support Carter said they would not vote for Kennedy against a Republican.

Overall, Kennedy outpaced Carter by 49 to 27 among people who said they were Democrats, with California Gov. Edmund G. (Jerry) Brown Jr. trailing far behind with 6 percent. When only people who said they were registered Democrats are included, Kennedy's margin dropped to 46 to 30 over Carter, with Brown still tallying 6 percent.

Kennedy runs strongest among younger people, blacks and Catholics, beating Carter by more than 2 to 1 among them, and leads Carter in all regions. Only in the South does Carter draw within 10 points of him.

Nevertheless, judging from the poll, Kennedy is not nearly as strong against Carter as he was earlier in the year or even as recently as the beginning of November.

The Post poll, in which 2,505 persons were interviewed, including almost 1,000 who said they were Democrats, was conducted from Nov. 1 to Nov. 12. In a CBS-New York Times poll conducted Oct. 29 to Nov. 3, Kennedy was seen as holding a 4 to 20 lead over Carter among Democrats. Setting aside possible errors in either poll, Kennedy's lead dropped from 34 points to 22 points in little more than a week.

Many policical observers had predicted that Kennedy's huge edge overr Carter would begin to erode once the Massachusetts senator got into the race. In June, for examle, political pollster Robert Teeter told The Post the "right now, Kennedy has the best of all worlds. He can pick his issues. He can do it without any of the negatives. In a campaign it would be a different situation."

Despite Kennedy's lead, The Post poll shows both candidates defeating leading Republican contenders in trial heats. Among people who said they were registered voters, Kennedy leads Ronald Reagan by 51 to 37 percent; he is ahead of John B. Connally by 55 to 28, and ahead of Sen. Howard H. Baker Jr. (R.Tenn.) by 50 to 33.

Carter, who trailed Reagan earlier, is now ahead of him by 47 to 42. The president leads Connally by 52 to 30 in The Post poll, and is ahead of Baker by 46 to 36.

Hidden beneath that good news for the Democratic Party, however, is what could become a sharp rift among rank-and-file Democrats. In all, 55 percent of the self-identified Democrats interviewed said it makes no difference to them whether Carter or Kennedy gets the nomination -- that they would vote for either candidate against Reagan, the front-running Republican.

But 11 percent, in a lonstanding tradition of people who consider themselves Democrats but vote Republican, said they would vote for Reagan against Carter and Kennedy. That leaves one-third of all Democrats who say they would support either Carter or Kennedy -- but not both -- against a Republican.

At this early stage, in other words, a significant segment of the Democratic rank and file appears to have divided into either a Carter camp or a Kennedy camp, much as Republicans in 1976 were divided between Reagan and Gerald R. Ford. In that year, as the campaign developed, the bitterness between the two camps grew and some political observers maintain that their failure to reconcile after Ford won the nomination was what cost the Republicans the White House.

Earlier this year, some Democrats warned that the party could suffer the same fate in 1980 if Kennedy became a candidate. Some Carter supporters continue to warn of such a split. The new Post poll suggests that the ingredients for such a development exist now, if only in a budding stage.

The poll canot explain much of the rift between Kennedy and Carter supporters, but some trends are clear. Many Kennedy backers seem convinced that Carter simply is incapable of handling the presidency. They have given up on him, at least for now.

One poll question asked respondents to state which of these two statements they agree with more: "Jimmy Carter just can't seem to cut it as president," or, "Jimmy Carter is a better president than he is getting credit for."

Slightly more than half the self-identified registered Democrats interviewed said Carter deserves more credit, and they chose him over Kennedy as the nominee by 47 to 33 percent. But among the three in 10 who said Carter "can't cut it" and those who said their views on Carter fell in between the two statements, Kennedy led Carter by 63 to 9 percent.

Those figures represent the views of all registered Democrats in The Post poll; Kennedy supporters, naturally, appear even more convinced that Carter is just not up to the job of being president.

On the other hand, a significant number of Democrats fail to be drawn to Kennedy. At the front edge of that group are many who reject much of the social change the country has undergone in the past 15 or 20 years, and who seem to associate Kennedy with that change.

Much of The Post's poll dealt with views on changing social mores in the United States, including attitudes toward men and women living together without being married, or liberalization of divorce laws, or the existence of coed dormitories in college. Among people who approve such change, Kennedy beats Carter by 2 1/2 to 1. Among those who hold consistently traditional views and object to such change, Kennedy draws no more than 30 percent support from Democrats, and trails Carter.

As the campaign moves on, of course, a key question for the candidates is how much further, if at all, Kennedy's lead will erode. Soundings by The Post in this poll and in others suggest that extreme volatility exists among Democrats.

At this stage in the four-year electoral process, many prospective voters simply have not fixed attention on the campaign. They are either undecided or willing to tell a pollster they prefer Kennedy one day and Carter another. oIn a two-part Post poll conducted in July and August, for example, 40 percent of those who identified themselves as Democrats wavered back and forth -- switching from Kennedy to Carter or vice-versa in a period of two weeks.

Republicans watching on the sidelines, however, have a different key question. It is how big the rift will grow between the two Democratic camps and what effect it will have on the election next November.