The House, rebuffing a last-minute compromise by President Carter, gave overwhelming approval yesterday to a $16.3 billion middle-income tax-cut bill that would turn capital gains taxes sharply and provide new tax breaks for home sellers.

Passage came on a 362-to-49 vote after the lawmakers beat down a White House-supported amendment to shift more of the tax cuts to the lower end of the income scale. The vote against the administration plan was 225 to 193.

In a significant statement, House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) told the House that, "If the administration had proposed this [compromise] five months ago it would have been adopted as the Ways and Means Committee bill, and would have sailed through the House."

The House also defeated, by a sizable margin, a rival GOP tax cut plan proposed by Rep. Jack Kemp (R-N.Y.) which would have slashed income taxes by 30 percent over the next three years. The defeat of the Kemp measure came on a 240-to-177 vote.

It refused to delete from the tax bill - despite vehement opposition from the Carter administration - a proposal that would provide for the first time an "inflation adjustment" for capital gain taxes that some experts regard as a first step toward "indexing" the entire tax code. The vote to retain the provision was 249 to 167.

The bill now goes to the Senate Finance Committee, which is expected to enlarge the reduction in capital gains taxes and add new special interest amendments. Sen. Russell B. Long (D.-La.) the committee's chairman, said his panel will begin work on the bill Aug . 21.

The results yesterday marked a political setback for the Carter administration, which had mounted a major campaign to try to reshape the bill drafted by the House Ways and Means Committee. Treasury Secretary W. Michael Blumenthal had spent much of the past week on Capitol Hill lobbying for the administration amendment.

Carter vowed once that he would veto any measure that included any sizable cut in capital gains taxes. However, he since has backed away from that position, saying he will wait to decide until after the bill has cleared Congress.

The bill approved yesterday marked a significant departure from previous Democratic tax bills in that the cuts would go primarily to persons in the $15,000 to $100,000 income brackets, rather than to those earning $15,000 or less.

The measure also would provide about the same size cut in tax rates, proportionally, for most income groups. In all other tax bills enacted since the early 1960s, the cuts have been more "progressive," with larger reductions for lower income brackets.

Rep. Al Ullman (D-Ore.), chairman of the Ways and Means Committee, who served as floor manager of the bill, said the measure was designed to meet complaints of middle-and upper-middle-income taxpayers, who he said often feel that they have been forgot.

The tax cuts for individuals would take effect next Jan. 1. Besides widening the present tax brackets, the measure would raise to $1,000 the current $750 personal exemption now allowed each taxpayer and his dependents.

The bill also would reshuffle existing tax rates, scrap the present $35-a-person general tax credit and repeal the deduction for state and local gasoline taxes. The average tax reduction would be abot $163 per taxpayer.

The major tax break for homeowners would be allow persons who sell their homes a once-a-lifetime chance to escape capital gains taxes on up to $100,000 of the profit they make from selling a principal residence - even if they do not buy a more expensive one. Home sale profits also would be exempt from the minimum tax. The provision would effect home sales completed after July 26, 1978.

The bill also would reduce the maximum tax rate on capital gains to 35 percent, down from the 49.1 percent now paid by a handful of high-income taxpayers. A capital gain is the profit from the sale of stocks or other property.

The reduction in capital gains taxes marks a reversal of a trend toward tightening that has prevailed since the late 1960s. Taking its cue from "tax reform" advocates, Congress raised taxes on capital gains in 1969, and then toughened them further in 1976.

In passing yesterday's measure, Congress formally scrapped - both in size and substance - the massive tax-reduction and "tax reform" plan President Carter proposed in January.

Carter first had sought a $24.5 billion tax reduction, but then trimmed it back to $19.4 billion as an anti-inflation gesture. He also proposed a spate of "tax reforms" - including repealing the deduction for the three-martini lunch.

Yesterday's bill, however, would provide a tax cut $3 billion smaller than the latest tax cut Carter requested. And it contains only a handful of his proposed reforms - primarily the provision to repeal the deduction for state and local gasoline taxes.

Carter had pledged a massive "tax reform" effort during the 1976 campaign in a move to overhaul the tax system, which he branded "a disgrace." But Congress proved later to be unreceptive to his plan.

The defeat of the administration's floor amendment yesterday appeared as much to reflect Congress' growing rift with the White House as actual differences over the substance of the Carter compromise.

In all, 91 Democrats joined 134 Republicans in rejecting Carter's last-minute measure. And in the vote to retain the "inflation adjustment" for capital gains taxes, 107 Democrats supported the position.

In an unusual move, Speaker O'Neill, who nominally supported the Carter compromise effort, criticized the administration in floor debate for having been tardy in trying to reach an accomodation with congressional Democrats.

Yesterday's floor action aslo included a defeat for liberals, who had pushed for some tax relief to offset next January's scheduled Social Security tax increases.

The liberals were barred by the Rules Committee from proposing any amendments, but sought on the floor to overturn the panel's procedural timetable. Their effort failed, 284 to 130.

The primary reduction in capital gains taxeswould mainly benefit taxpayers in the $100,000 and up categories. The "inflation adjustment" provision would provide tax breaks to those in the $20,000 and up brackets.