An organization of liberal Democratic congressmen said today that its members' polls show that President Carter is trailing Republican nominee Ronald Reagan in four-fifths of the marginal congressional districts it surveyed.

The compilation of polls in 60 districts was released by the liberal Democratic Study Group, whose chairman, Rep. David R. Obey (D-Wis.), said it showed that Democrats "could have a hell of a lot more trouble than people realize" if Carter is the nominee.

The polls, conducted in the last three months, covered 50 Democratic and 10 Republican districts in 26 states in all regions of the country.

According to the DSG statement, Reagan led Carter in 48 districts, with independent John B. Anderson ahead in eight, and Carter leading in four.

The districts were identified only by state. According to Richard Conlon, executive director of the DSG, Carter carried 22 of the 60 districts when he ran in 1976, compared with the four where he is leading now. Conlon said the study "confirms the worst fears members have" about the Nov. 4 election prospects.

The study said "All but eight or 10 of the districts are marginal or potentially marginal in this election," meaning that they are closely balanced and subject to political turnover.

The study said Carter's support declined sharply by the polls. In 20 April-May polls, his support averaged 30 percent, in 17 June polls, it was 24 percent, and 23 July polls, it was 21 percent. Reagan's strength climbed, while Anderson's held steady over the period, the study said.

In 1976, the vast majority of House Democrats ran ahead of Carter in their districts, but politicians figure that the weaker the presidential candidate, the harder for a House member to hold a seat.

Obey said anyone who doubts that Carter's indicated weakness could cost the Democrats House seats is "smoking something that is not legal."

Obey and many others active in the DSG are supporting the effort to defeat the proposed convention rule binding delegates to support the candidates to whom they are pledged.

But Obey, unlike some of the others, is equally critical of Sen. Edward M. Kennedy as a potential nominee. While the polling data released by the DSG gave no indication of Kennedy's comparative showing in the 60 districts, Obey said he believed "we'd be in as bad shape or worse shape" with Kennedy at the head of the ticket.

Obey said he would prefer to see vice president Mondale or Secretary of State Edmund S. Muskie nominated next week.