The Supreme Court yesterday agreed to step into a major battle in the financial industry by deciding the scope of the Federal Reserve Board's power to regulate limited-service, or "nonbank," banks.

The justices agreed to the Fed's request that the high court review an appeals court's ruling last September that limited its ability to control the rapidly spreading financial institutions.

Nonbank banks, which may make conventional commercial loans or offer traditional checking accounts but may not do both, have proliferated with major brokerage firms, department stores such as Sears, Roebuck & Co. and J. C. Penney Co. Inc. and other businesses entering the field.

The movement has led to interstate banking networks that have put traditional banks, still regulated by the Bank Holding Company Act of 1956, at a "severe competitive disadvantage," the board said in its petition to the high court.

The 10th U.S. Circuit Court of Appeals last year said the Fed improperly tried to rein in the nonbank-bank movement by expanding the definition of what constituted a bank for purposes of regulation under the 1956 law.

In 1966, Congress exempted from the act's coverage companies that do not accept deposits from which the depositor has a legal right to withdraw funds on demand and, in 1970, it excluded companies that do not make commercial loans.

The board redefined its regulations in 1983 to cover nonbank banks, something the appeals court called a "device to freeze the changes" until the board could "persuade Congress to act."

The Fed said in its petition to the high court that the issue must be reviewed. "The rapid proliferation of such institutions will settle the issue by default because even if the board's position were to prevail . . . dismantling the vast numbers of new nonbank banks would present severe practical problems," the board said.

The court's action yesterday means it will hear arguments in the fall over the propriety of the board's efforts to regulate the industry. But moves in Congress, where numerous bills are pending to deal with the issues, may make the legal arguments before the court largely academic, especially if Congress takes broader action than the court takes.

According to an attorney familiar with the legislation, most of the pending bills would provide broad regulation of nonbank banks, but legislation has been deadlocked, in part over disagreement about the appropriate cutoff date for determining which organizations to allow and which to disallow.

The case is Board of Governors of the Federal Reserve System v. Dimension Financial Corp.

In other action yesterday, the court refused to hear appeals by some of the nation's largest investment banks and brokerage firms seeking compensation in the largest municipal default in history -- the $2.25 billion bond default of two Washington Public Power Supply System plants.

Lawyers for WPPSS, Chemical Bank, the American Association of Retired Persons, the American Bankers Association and others asked the court to overturn a 1983 Washington Supreme Court decision that absolved taxpayers and utilities in that state of liability in the default.

WPPSS sold bonds from 1977 to 1981 to about 45,000 bondholders, more than two-thirds of whom were individuals who bought them for retirement income, according to briefs filed asking the court to take the case.

WPPSS was set up by towns in the area that pledged to back up the bonds if the nuclear plant projects failed. But the state supreme court said the towns did not have the power to make those pledges and the pledges were therefore invalid.

Chemical Bank and WPPSS argued in their petitions that the state court decision was unconstitutional in that it permitted a "taking" of property without due process and adequate compensation.

The case is Chemical Bank and Washington Public Power Supply System v. Public Utility District No. 1 of Benton County, Washington.

The justices yesterday also declined to review a federal appeals court ruling last fall that gave the Environmental Protection Agency broad authority to order automobile recalls in emissions-defects cases.

The U.S. Court of Appeals for the District of Columbia overturned a three-judge panel's ruling in December 1983 by ruling last September that the EPA may recall cars that have more than five years or 50,000 miles of use.

The 8-to-3 ruling backed the EPA's move to expand its recall authority to include all vehicles, regardless of their age or mileage. That ruling was issued in a case involving 200,000 1975 General Motors Corp. Cadillacs.

The three-judge panel had said the agency went beyond its authority in redefining the "useful life" provision of the Clean Air Act. The act says useful life is "five years or 50,000 miles, whichever occurs first."

The case is General Motors Corp. v. Ruckelshaus.

The court, with two justices dissenting, declined to review two other local appeals court rulings that limited the Interstate Commerce Commission's efforts to deregulate railroad rates for coal exporters and the use of boxcars.

In one case, Interstate Commerce Commission v. Coal Exporters Association of the United States Inc., the court refused to hear appeals by the ICC and the railroad industry that its deregulation of coal exporters was appropriate under the Staggers Rail Act of 1980.