The House Rules Committee yesterday cleared the way for House consideration this morning of a $16.3 billion tax cut and woted to allow a separate floor vote on the Carter administration's last-minute plan to revamp the measure.

The decision, engineered by the House leadership, marked an initial victory for the administration, which would have been embarrassed politically if the Rules Committee had vote.

However, in a move that could anger some liberal Democrats, the committee also voted to bar a liberal-backed Social Security amendment that would have provided workers some relief from next year's scheduled payroll tax rise.

The panel also rejected a Republican request for a separate floor vote on the controversial Kemp-Roth tat-cut plan. Instead, the Kemp-Roth measure will be consigned to a token motion to send the whole bill back to GOP motion to sent the whole bill back to committee.

The Rules Committee's action predictably reflected the wishes of House Democratic leaders. The procedural plan prohibiting all but a few amendments was outlined at a breakfast by House Speaker Thomas P. O'Neill Jr. (D-Mass.).

However, key Democrats conceded the floor schedule could anger enough factions in the House to force a close vote when the procedural issue comes up for a floor vote today - passibly throwing the parliamentary situation into chaos.

Rep. Richard Bolling (D-Mo.), who offered the O'Neill motion in the Rules Committee, admitted that barring the Social Security amendment "will lose the support" of some liberals, while the Kemp-Roth decision could anger Republicans.

"You add them all up, and I don't know where you come out," Bolling said. Bolling had proposed allowing a vote on the Social Security plan, but was defeated on an 8-to-7 vote - mainly because O'Neill did not actively push the issue.

The Rules Committee action effectively shattered an agreement between conservative Democrats and Republicans on the House Ways and Means Committee in which GOP members supported the pending bill in return for a floor vote on Kemp-Roth.

Rep. Al Ullman (D-Ore.), the Ways and Means chairman, had urged the Rules Committee to go along with the coalition's agreements, which would have barred a separate vote on the administration's plan. However, the committee demurred.

In addition to the administration's plan, the committee's procedural schedule would allow separate floor votes on only two other amendments.

A Republican-backed plan to provide for the first time an "inflation adjustment" for capital gains taxes, which eventually would slash capital gains taxes in half. A capital gain is the profit from the sale of stocks or property.

A proposal by Reps. Charles A. Vanik (D-Ohio) and Jake Pickle (D-Tex.) that would scrap the entire tax bill and simply extend the tax reductions enacted in 1977. The Vanik Pickle measure is not expected to pass.

As a gesture to Republicans, the Rules Committee allowed extra time to permit a floor debate on the Kemp-Roth proposal. Republicans had wanted a separate floor amendment to heighten chances that their measure might pass.

The Ways and Means Committee bill would provide $16.3 billion in tax cuts, including $10.4 billion in reductions for individuals and $1.9 billion in cuts in capital gains taxes. The cuts would go mainly to the $20,000-and-up brackets.

The administration's proposal would reshuffle those cuts to provide more for the lower end of the income scale and less for the upper end, and would trim back the capital gains relief for a few high-income investors with tax shelters.

However, despite yesterday's victory in the Rules Committee, it's not certain whether the Carter plan will succeed. Although Treasury officials contended mementum for the proposal is building, the measure faces an uphill fight.

The developments came as, separately, Sen. Russell B. Long (D-La.), chairman of the Senate Finance Committee, said yesterday his panel may make the capital gains tax cuts effective immediately, rather than next January as in the House bill.

In testimony before the Senate Budget Committee, Ling noted the 1979 effective date could prompt some investors to postpone any transactions in the remainder this year. He argued that the delay would cost the Treasury revenue.

Long already has hinted the Senate will approve in even larger reduction in capital gains taxes than is contained in the House bill. The bulk of the provisions in the House measure would take effect next January.

The administration's victory in obtaining a separate vote on its floor amendment came after frenetic last-minute lobbying by Treasury Secretary W. Michael Blumenthal. Blumenthal spent most of this week in Capitol Hill meetings.

In the Rules Committee vote on the Social Security amendment, three Democrats joined five Republicans to prohibit a separate floor vote. The measure would have allowed taxpayers a tax credit on income taxes of 5 percent of payroll taxes.