The Senate Finance Committee yesterday rejected the GOP-sponsored Roth-Kemp plan to cut income taxes by one third over three years and turned to a new proposal by Chairman Russell B. Long (D-La.) to expand the House tax bill with more tax reductions for lower-income groups.

Defeat of the Roth-Kemp measure came on an essential party-line vote of 10 to 8.The panel also rejected two compromise versions.

At the same time, the committee hinted it may undo last week's tentative agreement to enlarge the earned income tax credit for the working poor by paring the size of the increase or possible scrapping it altogether.

The action came after Sen. Harry F. Byrd (Ind-Va), who missed last week's action on the earned income credit, denounced the measure as "an additional welfare benefit." Byrd's comments spurred other conservatives to challenge the decision.

Long's new plan to expand the tax cuts for individuals would provide new tax reductions for those earning $15,000 a year or less, who were all but excluded from the House tax bill. The House cuts were aimed primarily at the $20,000-to-$50,000 brackets.

The new Long proposal would increase the size of the overall tax bill to $20.1 billion, compared to $16.3 billion voted by the House in August. Besides providing additional cuts for lower-income groups, it also would enlarge the House-passed curs for all taxpayers through $100,000 a year.

The Finance Committee chairman's proposal was designed to meet President Carter's major objection to the bill passed by the House - that its benefits were skewed too much toward the rich, at the expense of lower-income groups.

It was not immediately clear, however, whether the panel would approve Long's recommendation intact. A vote could come as early as this morning.

The committee's rejection of the Roth-Kemp proposal marked the latest in a series of congressional defeats for the GOP measure. Party leaders plan to promote the Roth-Kemp bill as a Republican campaign issue in a 'round-the-nation airplane hop later this week.

However, while yesterday's outcome was expected, the vote was surprisingly close. A compromise to cut taxes 16 percent over two years lost on a 9-to-9 tie.

The votes all were along party lines, except for a defection by Sen. Mike Gravel (D-Alaska), who supported the GOP plan throughout. Byrd voted for the two-year compromise plan. Long and other Democrats voted solidly against the plan.

The scaled-back version of the Roth-Kemp bill would have provided a tax cut for individuals of $20.2 billion in 1979, compared to $14.2 billion for Long's Proposal and $10.4 billion in the House bill. It also would have cut tax rates equally for all income-groups.

The Carter administration had opposed the Roth-Kemp proposal vigorously. Donald C. Lubick, assistant secretary of the treasury for tax policy, told the panel Carter almost certainly would have vetoed the GOP plan had it gotten through both houses.

The pressure by conservatives to reconsider last week's decision on the earned income credit could mark a setback for Long, who had hoped to use the provision to help offset Carter's objections to the House bill's cuts for higher-income taxpayers.

The plan would add another $1.8 billion in relief at the lower end of the income scale. Last week's proposal also included a new plan for distributing the benefits directly to employes' paychecks by using the withholding tax system.

Long's plan to expand the size of the tax cuts for individuals would substantially increase the reductions for lower-income taxpayers. For example, a family of four earning $8,000 a year would receive a $36 cut under the House bill, but a $442 reduction under Long's plan.

At the same time, however, the proposal would trim back slightly the size of the tax reductions for single persons in the $40,000-and-up brackets. Committee sources said the disparity stemmed from flukes in the the structure of the income-tax brackets.