The House Rules Committee voted yesterday to send the controversial college tuition tax credit bill to the House floor ahead of President Carter's rival scholarship aid plan, dealing a setback to White House efforts to block the tax measure.

By voice vote, the committee agreed to schedule floor action on legislation drafted by the House Ways and Means Committee that would allow parents of college students to claim up to $100 in tax credits for the school term beginning next August and $250 a year by 1980.

Although the Ways and Means Committee measure would apply only to college expenses, the Rules panel agreed to allow a floor amendment that would extend the tax credit to elementary and secondary school costs as well. It also will permit a vote on whether to double the size of the credit.

The action was a disappointment to the administration, which had hoped to push its scholarship plan through first to gain a tactical edge. The Carter bill had been ready to go to the floor in March, but was held up by the leadership after members demanded a vote on the tuition credit instead.

Rep. Al Ullman (D-Ore.), chairman of he Ways and Means Committee, told the Rules panel yesterday he hoped the House Education and Labor Committee would revamp the Carter legislation to "fit around" the tuition credit, but House sources said this was unlikely.

Instead, Democratic leaders appear to be prepared to push through the scholarship aid plan even if the tuition credit passes, and then leave it to Carter to veto the tax credit legislation. The president said in a letter to House leaders in April he would veto the tax credit bill.

The moves came at the same time backers of the tuition credit legislation made public new Library of Congress calculations showing that the increase in college costs between 1967 and 1976 outpaced the rise in median income of families with college-age children - after taxes are taken out.

The figures, compiled at the request of Rep. Albert H. Quie (R-Minn.), a key supporter of the tuition credit plan, showed that while college costs rose by between 74.2 to 76.7 per cent during the period, the after-tax income of these families rose 66.8 percent.

The Congressional Budget Office published a study last week showing that the income of these families before taxes had risen 78.8 per cent, while that of families with youngsters actually attending college during those years had soared by 87.3 per cent.

Budget office officials appeared to differ with Library of Congress economists on the method of calculating the impact of federal, state and local taxes. However, CBO's calculations showed that even after taxes are considered, college costs did not outpace income by much.

Meanwhile, the Ways and Means Commmittee prepared to start writing this morning a bill to roll back Social Security taxes, although there is no consensus on how to do it. Ullman has proposed a plan that would cut payroll taxes $5 billion.

Panel members seem agreed generally they want to roll payroll taxes back to 1977 levels at least for the 1979-80 period, and "make up" the losses with revenues from general income taxes. But they appear divided over specifies of the plan.

New support for a rollback in Social Security taxes came from a Harris survey made public yesterday showing that Americans favor by 48 to 30 percent trimming the recent Social Security tax increases by at least one third, and using income-tax monies to make up for it.

The House Democratic Caucus voted formally last month to ask the Ways and Means Committee to draft such legislation. Both the Carter administration and the Democratic congressional leadership strong oppose any such change.

Along with the votes on whether to include elementary and secondary schools and increase the size of the tax credit, the Rules Committee mandate yesterday also would allow a vote on a third plan - a proposal by Rep. Abner J. Mikva (D-Ill.) allowing taxpayers to defer taxes equal to college costs.