When the prime rate hit 20 percent for the first time last year, the big New York banks actually were charging at least four percentage points less on most new business loans, according to figures disclosed yesterday by the House Banking Committee.

The bank's own reports to the Federal Reserve Board show they gave as many as two-thirds of their new loans at rates below the quoted prime, which is widely regarded as the lowest rate available to commercial borrowers.

The descrepancy between the publicly announced prime rate and what the banks actually were getting has created "a widening credibility gap," House banking Chairman Fernand St Germain (D-R.I.) complained in a letter to 10 of the biggest banks.

St Germain called on the banks to explain what their prime rate really means, why certain customers can borrow at below-prime rates, and how many loans they've made recently for lower rates.

"Perhaps your more sophisticated borrowers are well aware that the prime rate is not the prime rate, but the small businessman and the consumer are none the wiser," St Germain said in the letter released yesterday.

The letter opened a new debate over one of the most widely quoted economic indicators, a price-tag on loans that has become a symbol of inflation and high prices.

Most consumers, St Germain said, accept the dictionary definition of the prime rate: "an interest rate at which preferred customers can borrow from banks and which is the lowest commerical interest rate available at a particular time." The prime is also often said to be the rate banks charge their biggest and most credit-worthy customers.

Several of the banks that were sent St Germain's letter and an accompanying questionaire on their primary prime lending practices said they have not yet received the message and were not prepared to comment on it.

Bankers have been saying for some time that the prime rate is not as important as it is made out to be, and some have gone so far as to write their own definition, disavowing any claim that the prime rate is the cheapest available.

New York's Chase Manhattan Bank has said its prime rate is "a rate we charge on certain types of credit to commericial borrowers which we publicly announce from time to time."

Most big city banks charge the prime rate only on certain types of commercial and industrial loans, usually short-term (two to six months) borrowings to finance inventories, supplies and the like, said a spokesman for New York's Citibank.

Some consumer borrowing, including auto and home improvement loans, traditionally has been handled at rates that are not related to the prime. Some banks have a small business rate and a farm lending rate that is less than is charged major corporations.

A discount from the prime is also standard in the "broker loan rate" that is charged on overnight loans to stockborkers to pay for the day's purchases.

Some loans on which the rate is supposed to rise and fall with changes in the prime have a "floor and a ceiling" -- a maximinum and minimum rate. When the prime climbed above 20 percent for the first time ever last year, many of those loans hit the ceiling, and borrowers got a relative bargain.

Banking committee staff members have already obtained Federal Reserve Board reports documenting sub-prime loans. Four times a year, the Fed surveys 340 banks and collects data on 20,000 loans.

Starting last year, the Fed data showed a sharp divergence between the prime rate and the rate on major business loans and a dramatic increase in the share of loans made below the prime.

In February of 1980, during the first peak in interest rates, the large New York banks reported 67 percent of their new loans were for less than prime. The May survey of the same banks found 60 percent of the loans were below prime and the portion was about the same in August. By November, when rates were coming down, the share of below-prime loans dropped to 21 percent.

The average discount from the prime was 4.26 percent, according to the Fed data.

If the prime rate does not reflect the best loan terms available, some other indicator may be needed so businesses and consumers can comparison shop for loans, the banking panel chief suggested.