When Speaker Thomas P. O'Neill scolded his floor leaders last week for insufficient support of administration programs, they responded with an unexpectedly harsh retort that spells deep trouble for President Carter in the House of Representatives.

As filtered to uncomprehending ears at the White House, the message from Capitol Hill amounted to the same old complaints about not getting telephone calls returned from presidential aide Hamilton Jordan. In fact, that is barely the tip of the iceberg. What really bothers Democratic House members is the president's insensitivity in sending them legislation that runs counter to the mood of Congress and the people.

While O'Neill complained about Democratic nonsupport for Carter's bill creating a consumer protection agency, the Democratic congressmen complained over having to vote on an unpopular bill. Their mood is reflected by these words to O'Neill by one senior House member, long noted for liberalism and party loyalty: "I'm tired of getting my brains kicked out on these issues. What has Jimmy Carter ever done for us? Pretty soon the members are going to start running against Carter, and that's trouble for him."

Such emotions were detonated after defeat of the consumer protection bill by 38 votes in the House Feb. 8 when 101 Democrats (of 283 voting) opposed the president. Revolt in the ranks reached into the noncommissioned officers. Deputy, at-large and regional whips voted only 19 to 16 for he bill.

Tip O'Neill, his Boston sense of party regularity violated, was upset when the party whips held their weekly meeting Feb. 9. Calling Carter "a helluva president," the speaker complained about lack of party discipline, in not supporting Carter. The congressmen, said O'Neill, should not think so much of themselves but seek a higher purpose.

That set off the explosion. Rep. Jim Lloyd, who in the election of 1974 won a conservative southern California seat previously held by Republicans (including Richard Nixon), protested shabby treatment when he approached administration officials with constituent problems. Health, Education and Welfare Secretary Joseph Califano ignored his calls. Secretary of the Navy W. Graham Claytor dodged a meeting with Lloyd until the congressman threatened to confront him in open hearing.

Then Lloyd got to the substance. "Some of these tired old Democratic programs have seen their day," he said. "We'd better get off that big-government kick." Pointing out that candidate Carter had pledged less government, Lloyd criticized him for scrapping the B-1 bomber and proposing a new government agency. "And you say to me," the two-term congressman told O'Neill, "that I have to come down the line for the president's programs. I say politics are not the same in West Covina as in Boston."

Lloyd was supported by one House member after another: Bill Alexander of Arkansas (one of three deputy whips), Sam Stratton of New York, Tom Bevill of Alabama. Complaints varied from irritation that the consumer bill was brought up in the face of certain defeat ("putting us through the agony for nothing") to kicking Hamilton Jordan around ("He couldn't run somebody's district office").

Then Rep. Robert Giaimo of Connecticut, a powerful figure as chairman of the House Budget Committee, fired the heavy artillery: "These things are not necessary. Why bring them up? It's just another layer of more government. The members are starting to find that people back home say 'no!'"

Rep. John Brademas of Indiana, the chief whip, broke in to claim the White House had gained "much better rapport, much better communications" with Congress. "That's for you," responded a regional whip, pointing to O'Neill and Brademas, "not for us."

He betrayed widespread feeling among House Democrats that the speaker, following early thorny relations with the White House, now rubber-stamps the president's programs. The whips grumble O'Neill disregarded their advice a week earlier against bringing up the consumer protection bill since it faced certain defeat.

Nevertheless, Brademas was essentially correct in claiming "much better communications." No president has been more accessible to Capitol Hill than Carter, who has devoted many hours lately to private meetings with congressmen. His chief lobbyist, Frank Moore, has developed into a competent, universally liked agent in Congress. The president's troubles, therefore, are nothing trivial; they concern substance, not style.

Ironically, it was first expected that a liberal, free-spending Congress would object to Carter's conservative approach to government. Instead, the president has bound himself to the agenda of organized labor and other liberal pressure groups, while the House functions as what one senior Democrat there calls "the country's best public-opinion poll."