Secretary of Housing and Urban Development Jack Kemp broke ranks with the Bush Administration yesterday, insisting that the federal budget deficit "is not ballooning" and that any attempt to reduce it by raising taxes will hurt, not help, the economy.

Kemp, a conservative who in his unsuccessful 1988 campaign for the Republican presidential nomination passionately advocated tax cuts as a spur to economic growth, said he supports the current budget summit as a way to "reduce spending."

But he lamented that the negotiations between the White House and congressional leaders have turned into a "tax summit." "If no one else {in the administration} will make the public case, I will. The transcendent issue is economic growth. Cutting taxes, not increasing taxes, is the route to economic growth," Kemp said.

Kemp's comments came as the budget group held its fourth meeting on Capitol Hill, where bargainers spent a lengthy meeting comparing projections of the deficit, according to participants. They moved closer to agreeing on the size of the deficit and the amount of debt reduction actions needed. Democrats believe a fiscal 1991 deficit reduction package should be between $55 billion and $60 billion, while Republicans set it between $45 billion and $50 billion.

The session began as President Bush praised the group's progress, but suggested it is Congress, not he, that caused the deficit problem. "I will point out that people understand that Congress bears a greater responsibility for this," Bush said, adding, "But I"m not trying to assign blame. . . . "

Those comments irked some Democrats who do not want to make any proposals in the bargaining until Bush does. "That is just a statement that runs from the truth," said Rep. Charles E. Schumer (D-N.Y.). "We've put down a budget, passed a budget. The president hasn't done any of that. The only thing he's done is call for a summit, which means Democrats should come and help."

Kemp, in a telephone interview from Louisville, where he was giving a speech, seemed torn between venting his frustration over the talk of taxes and deficits and the requirement that a Cabinet member be loyal to the president. "I'm saying this as a private person," he said, adding with a laugh, "which I probably will be after I say this."

Kemp took issue with the administration's basic premise for the summit: that the deficit was of serious enough concern that extraordinary measures are necessary. "The sky is not falling," he said, "the deficit is not ballooning. The economy is stronger than it is getting credit for."

Kemp as a congressman voted against the Gramm-Rudman-Hollings law that mandates yearly, steadily shrinking deficit levels until the budget is balanced. Kemp called for a change in the law that would replace specific dollar targets with ones related to the size of the economy. "A $100 billion deficit in a five trillion dollar economy is quite different than a $100 billion deficit in a three trillion dollar economy," he said.

Kemp has told friends and associates of his increasing frustration at not being involved in the economic debate over the deficit -- a debate occurring in the administration's inner sanctums to which the former congressman has not gained entry.

Kemp's concern that the budget summit has come to be seen as a tax summit is widespread among GOP political operatives, who have made the case strongly to the White House in recent weeks that it should have emphasized the summit was about "economic growth." Instead the administration allowed the discussion of taxes as one put it, to get "way out of whack" in the days after the White House and congressional leaders agreed to meet.

Bush yesterday continued to be silent on how the deficit should be reduced. "It is way too early to start talking about remedies here. I want to let the process go forward and then, when I get agreement, I will go out and say to the American people, 'Here is what we recommend.' And I am not going to prejudge it."

Although Bush expressed satisfaction with the process, House Budget Committee Chairman Leon E. Panetta (D-Calif.), one of the negotiators, warned about the slow pace of the talks. He said many members of Congress facing reelection in November are "locking themselves into" public positions opposing tax hikes or certain spending cuts. Panetta said if agreements are not reached in "30 days or a few weeks," it will become impossible to achieve them before Election Day. A "big stall" in negotiations could force a post-election session of Congress or push the hard decisions over into another year, he said, risking a summer political crisis when the national debt limit must be raised.