House Speaker Thomas P. O'Neill Jr. (D-Mass.) has told President Carter that before the year is out Congress will have to pass legislation dealing with the balanced budget issue.

The message, delivered last week at a White House breakfast, is the clearest sign yet that congressional leaders now realize they can't avoid confronting the national drive for a constitutional convention on the budget.

House Democratic leaders made clear last week they still oppose a constitutional amendment to balance the budget. But Rep. John Brademas (D-Ind.), the majority whip, said the leaders discussed with Carter various ways of responding short of a "strait-jacket" on the economy.

"I think the handwriting is on the wall that it will take something like this to prevent a balanced budget amendment appearing on every piece of legislation it would possibly be attached to," said William H. Cable of the White House congressional liaison staff. "it's going to come up again and again until something is done." Cable said there has been "a change in the tone of the discussions that I think is healthy," indicating the talk has gone from thinking of ways to prevent such amendments to "finding a solution."

What brouught the leaders to the point of agreeing to any balanced budget language at all was the surprising strength balanced budget supporters have shown in attempts to attach such language to the debt limit bill in both chambers of Congress.

In the House, an attempt to prevent future increases in the debt limit unless there was a balanced budget for fiscal 1981, or two-thirds of the Congress agreed to a deficit, failed by a narrow 201-to-199 vote, and then only because the leadership promised the balanced budget supporters a vote on the issue later in the year.

The Senate, in approving the debt limit bill Tuesday, passed an amendment requiring the two Budget committees to submit to Congress a balanced budget beginning in fiscal 1981. But Congress, by a majority vote, could then authorize deficit spending.

The strength of the balanced budget supporters has made the Democratic leaders decide to play a part in formulating some balanced budget language before the supporters coalesce around a proposal to leaders don't like. Porposals range from limiting spending to a percentage of the gross national product to the kinds of amendments requiring a super majority to unbalance the budget.

Democratic leaders also hope that by placing balanced budget language in law, they can hold off the demand for a constitutional amendment, or what is worse in their eyes, a constitutional convention. The House Republican Policy Conference has endorsed a call for a constitutional amendment and hearings on such amendments will begin before the House Judiciary Committee Tuesday.

Brademas said, "The question for the Republicans is do they want a balanced budget or do they want an issue."

There is one other factor that is forcing congressional Democratic leaders to act. A balanced budget by fiscal 1981 is already law. Last year, Sen. Harry Byrd (Ind.-Va.) succeeded in getting written into an international monetary fund bill a simple amendment that says, "Beginning with the fiscal year 1981, the total budget outlays of the federal government shall not exceed its receipts."

If nothing else happens, that law would stand, and would either have to be amended or repealed before next year.