Problem loans at the nation's major banks have grown 63 percent since the start of the year and will continue to increase in the months to come, according to a study by Salomon Brothers Inc., the investment banking firm.

Salomon Brothers analyst Thomas Hanley said the 35 banks he surveys every quarter had "substandard" assets (mainly loans) of $18 billion as of Sept. 30, up from $14.8 billion on June 30 and $11 billion at the start of 1982.

Substandard assets include loans that are not being repaid on schedule, real estate loans on which a bank has foreclosed and taken possession of the property and loans the bank thinks are shaky.

Bankers say most nonperforming loans eventually are paid off in full.

Although the severe recession is the main reason banks are seeing a big increase in problem loans, several banks are holding large problem loans they purchased from the failed Penn Square National Bank in Oklahoma City.

At the head of the list of major banks with large amounts of problem loans was Continental Illinois National Bank of Chicago with nearly $2.1 billion, or 4.7 percent, of its assets classified as "nonperforming" at the end of September. The average for all major, money center banks like Continental was 1.7 percent of assets.

At the beginning of the year, $660 million, or 1.4 percent, of Continental's assets were classified as non-performing. That was about the same as other big banks like Chase Manhattan and Citibank. On Jan. 1, money center banks reported that 1.2 percent of their assets were in the problem category.

Penn Square made more than $2 billion worth of energy loans that it then sold to other major banks such as Continental. Many of the loans turned out to be bad--both because of the sudden decline in the energy industry in Oklahoma and because of fraudulent activity on the part of some Penn Square officials, federal regulators have said.

Continental alone bought about $1 billion of loans originated by Penn Square. It has been forced to write off many of them as uncollectible.

Seattle First, another big regional bank, was also a major purchaser of Penn Square loans. Mainly because of its Penn Square difficulties, Seafirst's problem loans grew from $188 million, (1.7 percent of its assets) on Jan. 1 to $601 million, or 5.9 percent of its assets, on Sept. 30.

Bank of America, the nation's second biggest bank, has problem assets of nearly $2 billion -- about 1.6 percent of its total assets -- in large part because of the sharp decline in the real estate industry in California.

Hanley of Salomon Brothers said nonperforming loans at major banks are going to grow in the months ahead because of continuing high interest rates. Despite the decline in rates--the prime rate has fallen from 16 1/2 percent to 12 percent in the last three months -- interest charges are still well above the 5 to 6 percent rate of inflation.

This means that companies, many of which have hung on so far in the recession, will find it difficult to stay current on their loans.

In many industries, demand remains so weak that companies are not able to raise their prices, but still are cutting them.

Hanley said that most banks have not classified as problem loans their lending to Mexico. He said banks must face the possibility of adding at least a portion of the billions of dollars in loans to Mexico's private sector to their problem lists.

Nevertheless, the list of problem credits at the nation's banks is less imposing than during the severe recession of 1974 and 1975. Bank regulators have stressed that, even though problem loans and problem banks are growing in numbers, the banking system remains fundamentally sound.