The U.S. banking industry is getting progressively weaker because banks cannot earn a sufficient return making loans and are prohibited from getting into more profitable lines of business, the chief regulator of nationally chartered banks said yesterday.

Declining profitability is a "nagging concern" today that threatens to become "the major problem of the banking system by the end of the decade," Comptroller of the Currency Robert L. Clarke said.

Bank regulators such as Clarke and Federal Deposit Insurance Commission Chairman L. William Seidman are preparing a concerted effort to persuade Congress that banks need to get into new businesses such as insurance and some types of securities to strengthen the industry. A seriously weakened banking system will be unable to furnish the credit that most of the economy needs to function, they argue.

Clarke said in a speech to the Boston Economic Club that banks -- both big and small -- have lost their best customers during the last decade and are forced to lend to riskier companies. As a result, the quality of their loan portfolios is declining, and loan losses are further eroding profits.

The average return on assets for the 4,200 federally chartered banks with assets of $300 million or less declined for the sixth consecutive year in 1985, he said. In 1980, these banks earned $1.13 for each $100 in assets -- such as loans, Treasury bills and bonds. Last year, their return declined to 53 cents for each $100 in assets.

About 40 percent of small banks -- those with $25 million or less in assets -- lost money last year.

The biggest banks, on the whole, "have experienced flat earnings for the past six years," he said. Those bigger banks that reported sizable profit increases in the first quarter of 1986 made their money on increases in the value of government bonds they owned, trading profits, foreign exchange dealing, sales of assets and service fees, Clarke said.

"In other words, this income came from sideshows to traditional banking," according to the text of Clarke's address, which was released here.

Big banks used to make their money by lending to solid, blue-chip corporations. In 1966, banks supplied 85.4 percent of all the short-term funds companies needed. Today, banks supply 68 percent.

Most of the corporations that borrow from banks no longer are the big, sound companies, which now find it cheaper to raise money by directly selling commercial paper -- essentially corporate IOUs -- to investors.