Representatives of 50,000 small coffee producers met with Honduran congressional leaders Tuesday to complain that the presence of U.S.-backed anti-Nicaraguan rebels along the border between the two countries has forced them from their homes and ruined their livelihood. Wilfredo Castellanos, vice president of the Association of Coffee Producers, said in an interview that the organization planned to demand compensation from the U.S. government for damages. He said the small growers had lost an estimated $2 million in income this year because fighting between the rebels, called counterrevolutionaries or contras, and Nicaraguan troops along the border had prevented a full coffee harvest.

"We want the Honduran congress to debate the problem of security" in the eastern province of El Paraiso where most Honduran coffee grows," Castellanos said.

"The government should expel all foreign troops and affirm Honduran sovereignty over its own territory," he said. "The crisis will get worse soon if this does not happen because of the new U.S. aid program" for the contras.

The protest was unusual. Although Honduran military and government officials say in private that they are concerned about the growing U.S. presence here, there has been virtually no public evidence of any concern.

Neither is there much visible enthusiasm for the contras, who are easily contacted here and move about without apparent restrictions.

In a break with the prevailing apathy, Castellanos maintained that cross-border shooting and contra troop movements had damaged his members' homes and farms and had killed several people, but he provided no figures. A spokesman for the U.S. Embassy here said troops of the leftist Sandinista government of Nicaragua had sparked the border incidents and that the United States had provided assistance to coffee growers in the region.

One U.S. official denied that there had been significant coffee losses.

"There has been little, if any, damage to this year's crop," he said. "The bulk of the coffee was successfully harvested as far as we can determine."

Compensation demands would have to be addressed by the Honduran government because there are no U.S. forces in that area, the spokesman said.

The presence of thousands of Nicaraguan refugees in towns near the border has created what the conservative Honduran press calls "the new Nicaragua," noting that the many children born to the refugees are Honduran citizens. But, although six politicians are already making speeches to position themselves for presidential bids 3 1/2 years hence, none has said anything either critical or complimentary about the contras.

Asked about a solitary anti-American graffito in the town of Comayagua, near the Honduran-U.S. military base at Palmerola, one resident American who would like to see some protest about the contras said glumly that it meant nothing.

"There's probably about three people in that group," the American said.

President Jose Azcona has said he welcomes President Reagan's $100 million contra aid package in hopes it will enable the contras to leave Honduran soil and operate exclusively inside Nicaragua. Like other public figures here, he has stopped short of otherwise endorsing U.S. policy or condemning the Sandinistas.

"He is in a delicate position," said Ramon Zuniga, a University of Honduras professor specializing in national political affairs.

"He has so little real power that he must try not to antagonize any part of the armed forces." The military, which wields independent power here, is divided on the contras but united in its opposition to any military clash with better equipped Nicaragua, according to Army officials.

They said, however, that increased U.S. aid could increase the contras' visibility in Honduras and spark more protests such as the coffee producers' complaint.