President Carter's new proposal to expand existing federal college scholarship programs cleared its first big, hurdle in the Senate yesterday - placing it neck-and-neck with the rival tuition tax credit bill in a race for the Senate floor.

By unanimous vote, the Senate Human Resources Committee approved a slightly altered version of the scholarship legislation that would provide larger tuition grants than the Carter proposal for most middle-income families.

The bill now goes to the floor, where it faces a procedural contest with the tuition credit legislation. Conservatives want both measures considered together, but liberals are pushing to take up the scholarship bill first. Both must wait for a vote on the Panama Canal treaties.

When fully effective, the legislation approved yesterday would cost $1.4 billion - compared to $1.2 billion extra for Carter's proposal and $4.5 billion for the tuition tax credit bill. The first new grants would become available in the fall of 1979.

Catholic school spokesmen are backing the tax credit proposal, which ultimately would extend tax breaks to parents of private and parochial school pupils. By contrast, the scholarship aid bill would benefit only college students. Carter has warned he will "not accept" both.

The committee's version of the scholarship measure differs from Carter's in two ways : First, it would provide larger grants for middle-income families, and, second, it would provide federal loan guarantees for students whose families earn more than $40,000 a year.

Under Carter's proposal, direct tuition grants would be extended to families earning between $15,000 and $25,000 a year, but the maximum payments to students in those income brackets would be $250 a person - the same as the size of the tax break in the tuition tax credit plan.

Under the committee's bill, however, the maximum grant would start at $1,020 for a family earning $15,000 a year and would be reduced gradually as the applicant's family income approached $25,000. A family receive $540.

The Human Resources Committee also added $100 million to increase the size of the existing "supplemental eductional opportunity" grants program, which is designed to provide aid to especially needy students. The money would come from excess work-study grant funds.

Approval in the committee yesterday came on a vote of 14 to 0. However, in one telling move, two senators on the panel, Richard S. Schweiker (R-Pa.) and Orrin G. Hatch (R-Utah), said they plan to vote for the tuition tax credit if they have a choice.

The committee's legislation would increase the cost of the direct grant portion of the program by $1.2 billion, compared to $1 billion for Carter's plan. But the cost figures for the other elements of the program - loans and work-study grants - would be less than in Carter's proposal.

Meanwhile, backers ot the tuition tax credit made public the text of a memorandum to the president from Secretary of Health, Education and Welfare Joseph A. Califano Jr. in which Califano argues the scholarship plan is needed to keep college aid in the education committees.

The memo urges that "it is worth paying a price to avoid giving" the Senate Finance Committee "another piece of executive branch policy." It said the congressional education committees "are so fearful of losing jurisdiction . . . that they will go without us - and at a high price."

Califano has warned if the tuition credit is adopted, it would skew federal education benefits disproportionately toward private school students, raising total federal aids to $575 a pupil for private schools and $128 for public schools.

However, Rep. Barber B. Conable (R-N.Y.) has countered that the disparity would be reduced by addint to the public school figure the deductions allowed on federal income taxes for the portion of a taxpayer's state and local taxes that go to finance public schools.