Health maintenance organizations, prepaid health plans hailed only a few years ago as a solution to escalating medical expenses, are facing an unpleasant new phenomenon as they seek to penetrate the Washington marketplace: red ink.

Most area HMOs took a severe financial beating last year, with several prominent plans, including Group Health Association and the George Washington University Health Plan, reporting losses totaling millions of dollars, according to financial reports filed recently with area insurance regulators. The few plans that made money generally made much less than in the previous year.

While in several cases the losses can be attributed to the costs of starting up a new business, many HMOs are reeling from a resurgence of medical-cost inflation and stiff competition for patients that has forced many to lower their prices below what is needed to cover their expenses.

Doctors and HMO executives alike predict that as HMOs come under greater financial pressure, conflict between the health plans and the medical community will increase. Signs of that rising conflict could be seen in 1986, when the physicians union at Group Health Association went on strike and doctors at MD-IPA, a big Rockville HMO, fought administrators over their tactics to hold down health costs.

Doctors are also chafing over the increasing practice of HMOs to pay them a flat fee for each patient, rather than on the normal fee-for-service basis -- a practice they say provides an incentive to limit care while putting them at increased financial risk.

"As the HMOs don't do well, they are going to try to squeeze more out of the budget," said Dr. Robert Berenson, a Washington internist active in several local plans. "The only thing they have direct financial control over is the payment to the doctor."

Rising competition "is going to increase the amount of strife between clinicians and HMOs," added Dr. John Ott, medical director of the George Washington Plan, which lost nearly $8 million in 1986. "You've got a conflict, because you as a health plan want to pay me less and I as a doctor want to be paid more."

The financial downturn of HMOs here and around the country represents an ironic twist for an industry that enjoyed spectacular growth in the 1980s thanks to the perception that they provide more comprehensive and less expensive medical care than normal insurance plans.

Unlike those fee-for-service plans, HMOs provide comprehensive doctor and hospital services for a preset price, either through a medical staff that works exclusively for the plan or by contracting with individual doctors in the community. HMOs seek to reduce costs largely by carefully reviewing the use of medical services by plan members in order to weed out ostensibly unnecessary care.

As in the rest of the country, HMO membership growth in Washington has been remarkable. Roughly 530,000 metropolitan-area residents belong to HMOs, or about 16 percent of the regional population, according to reliable estimates. That represents more than twice the 214,000 residents enrolled in 1982 and more than a 25 percent increase over the 408,000 of just a year ago.

But the membership growth has been accompanied by an almost equally fast increase in the number of HMO plans in the area. Whereas until a few years ago there were only a handful of such plans, today at least a dozen are up and running across the region. Furthermore, insurance companies and hospital groups are in the process of establishing at least a dozen more. Insurers are also setting up so-called "preferred provider plans" that are similar to HMOs, although less restrictive in the choice of doctors and hospitals available to patients.

The result has been a tremendously volatile health insurance market, with pressure on plans to keep premiums down and raise marketing expenses to attract additional members. Younger plans are also burdened by substantial capital needs, such as buying computer equipment, adding staffing and establishing reserves required by insurance regulators to obtain an HMO license.

"There's clearly pressure on HMOs to keep the premiums down, and costs are rising. Your margins are being squeezed," said Alan Silverstone, the administrator of Kaiser Permanente, the largest plan in the region with more than 160,000 members. Kaiser, a not-for-profit plan, was among the few to report a surplus -- in this case, $2.6 million -- but that still represented a sharp turndown from the year before, an unusually successful 12 months in which the plan made nearly $15 million.

"The notion that the HMO market can accept more and more entries, each of which can be profitable, makes no sense whatsoever," said Steven Sieverts, vice president for health care finance at the local Blue Cross and Blue Shield, which operates its own HMO called Capital Care. "This is not a challenge to the HMO concept, but to the business judgment of people pushing new entries."

The competition has been so tough that many executives predict that smaller health plans will merge with rich insurance companies, or face going out of business, as gaining market share becomes more expensive.

Some say the process of consolidation has already begun; earlier this month Network Health Plan, a Northern Virginia HMO that lost $1.5 million in 1986, announced plans to be sold to Partners National Health Plans, a joint venture between Aetna Life Insurance Co. and a big chain of not-for-profit hospitals.

"It's going to get vicious, bloody," said David Metz, president of Capital Care. "You can not enter this market or compete without deep pockets," he said. "You've got to be prepared to take losses."

Although plans of all stripes are suffering, the HMOs suffering the most in the current market are two of the area's oldest and best-known, both of them nonprofit: Group Health, with 141,000 members, and the George Washington plan, with 26,000 enrollees. Group Health reported a stunning loss of $13.7 million -- compared to a surplus of $1.6 million the year before -- on revenue of $129.6 million. George Washington lost $7.9 million on revenue of $21.5 million, after turning a small surplus the year before.

David Tinsley, Group Health's director of administration, said many of the member-owned cooperative's woes can be traced to a 25-day doctors' strike last year, which forced many patients to seek more expensive care outside the health plan's own system of doctors and hospitals.

The strike also dampened Group Health's efforts to recruit new members, on whom the plan had been banking to raise revenue. Even apart from the strike, Group Health suffered from high labor costs, continuing medical inflation, rising malpractice premiums, and, according to some outside critics, an older-than-normal population of enrollees that makes for higher-than-normal medical expenses.

In George Washington's case, the plan launched ambitious efforts to expand its operations into suburban Virginia and Maryland in 1986 under the direction of American Medical International, the huge California health care conglomerate that purchased 80 percent of the plan in 1985. AMI has since sold the plan back to the university, but the cost of building new facilities and hiring new doctors remained a major factor behind the huge losses, according to medical director Ott.

Ott said the plan was also hit by an unusual incidence of high-cost illnesses among its members. It reported 37 cases of patients being hospitalized for more than 30 days, up from 20 in 1985, including a newborn baby with heart problems who stayed in the hospital for more than six months and cost the plan more than $1 million.

"We've never had it that bad before," said Ott.

Like other plans, Group Health and George Washington have moved to regain profitability with a series of sometimes painful cost-cutting steps. Group Health, for instance, laid off a fifth of its administrative staff last fall to reduce overhead expenses, and has been negotiating with its physicians' union over ways to improve productivity, an often controversial area. Many outsiders believe Group Health will have to take additional tough actions to regain financial health.

At MD-IPA, a big Rockville HMO that reported a narrow profit last year, administrators drew flak from doctors when they moved to limit obstetric costs and payments to anesthesiologists and other hospital-based physicians. The plan also has taken the highly unusual step of refusing to renew contracts with several employer groups whose health costs far exceeded the plan's average.

"It is a tough position to tell people we don't want your business," said Joe Guarriello, MD-IPA's vice president for corporate affairs. But he said MD-IPA was constrained by federal rules that require HMOs to offer the same "community rate" to all employer groups. That rule essentially forces the plan to subsidize high-cost groups, leaving MD-IPA with the choice of raising rates for all groups or ending business with about a dozen, officials said