The Office of the Comptroller of the Currency has turned down preliminary "nonbank bank" applications by Washington's American Security Corp. and San Francisco's giant BankAmerica Corp.

A spokesman for the comptroller's office declined to give reasons for the rejections, but federal regulatory sources laid the comptroller's action to their large increases in problem loans and declines in earnings.

It was the first time the comptroller's office has turned down preliminary applications for nonbank banks. Such institutions either do not offer checking accounts or make consumer loans, and thus slip through loopholes in federal laws that prohibit interstate banking.

Many banking organizations are using nonbank banks -- often called limited-purpose banks -- to extend their deposit-taking operations into other states.

American Security, which owns the District's second-biggest bank, already had received preliminary approval to set up a nonbank bank in Maryland and Florida and was seeking authorization to establish one in Virginia. The comptroller's office, which regulates all nationally chartered banks, not only turned down American Security's Virginia application but also rescinded its approval for the Maryland and Florida banks.

The comptroller rejected BankAmerica appplications to set up 13 nonbank banks in 11 states and the District.

The comptroller's office, however, continues to churn out preliminary approvals for limited-purpose banks. It approved five more yesterday, bringing to 276 the number of applications to which it has given a tentative green light.

But the Federal Reserve Board, which also must approve most nonbank bank applications, has been more reluctant than the comptroller to approve the applications, and a federal court in Florida recently issued a temporary injunction prohibiting the comptroller from giving final approval to limited-purpose banks.

Only 14 limited-purpose banks are in operation.

A spokesman for the comptroller's office said there are a variety of reasons that the agency could turn down nonbank bank applications, including the capital, management and profitability of the institution, as well as information obtained from examinations of the institution. If an institution is a subject of "more than unusual supervisory concern," an application also can be rejected.

Because of a special $37 million provision to cover potential loan losses, American Security reported a loss of $11.9 million in the final three months of 1984 and a profit of only $4.7 million for the year as a whole.

BankAmerica, which owns the nation's second-biggest bank, reported a decline in profits last year to $346 million from $390 million in 1983 and has entered into an agreement with the comptroller's office to boost its capital to 6 percent of assets by late 1986 from the 4.8 percent level it reported last fall.