Maryland's Chesapeake Bay critical area program, designed to save the shores of the nation's largest estuary from overdevelopment and excessive pollution, has dramatically pushed up the prices of existing waterfront houses, according to real estate brokers and a new state study.

At the same time the program has reduced the value of some tracts of undeveloped waterfront property, brokers said, because it limits development to no more than one house per 20 acres within 1,000 feet of the bay or its tidal tributaries. Prior to the legislation, which was enacted in 1984 and affected land that had not been subdivided before December 1985, properties could be divided into parcels of various smaller sizes, depending on the particular county.

First proposed by Gov. Harry Hughes in 1983, the critical area law established the 1,000-foot buffer against intensive development and created a state commission to oversee the program.

While environmentalists hailed the legislation, developers reacted strongly against it. Yesterday, at a meeting in Norfolk of officials from Maryland, Virginia, Pennsylvania and the District who convened to discuss an agreement to reduce pollution of the Chesapeake Bay, Maryland Gov. William Donald Schaefer said he hoped to relax the critical area restrictions. Schaefer said the one dwelling per 20 acres is too stringent.

The law also requires 16 counties and 44 municipalities on the bay or its tributaries to submit local critical area plans this month that must match state restrictions and be in place by September 1988.

But even as the local regulations are being drafted, the program already has had a major impact on the market for Chesapeake Bay real estate and development, in effect accomplishing the goal of limiting development.

Richard Fischer, a Calvert County real estate broker pointed to a 100-acre farm at Prison Point on the Patuxent River. Under prior zoning, it could have yielded 20 buildable lots, he noted. Now, it would yield only five, thus reducing its appeal to developers and pushing down its price.

By contrast, Fischer said, the price of land subdivided before the law took effect, which can be developed more, has gone up 20 to 30 percent in the last nine months.

In the Anne Arundel County bayshore community of Deale, broker George Heine said waterfront houses are appreciating 10 to 20 percent a year, compared with 5 to 7 percent before the critical area legislation.

"Supply and demand applies to waterfront property as well as to anything else," said Solomon Liss, a retired Baltimore city court judge who is chairman of the state commission overseeing the program. "What we've heard is those who own {waterfront houses} think they have an asset of greater value."

A state-financed study, conducted by Rutgers University for the Chesapeake Bay Critical Area Commission, supports such conclusions.

Using state tax records, the Rutgers consultants examined the 605 sales of waterfront houses between 1981 and 1986 in Calvert County in Southern Maryland, and in Talbot and Dorchester counties on the Eastern Shore. From the records, they established a "composite" 10-year-old waterfront house of 2,000 square feet on two acres.

Ascribing price rises partly to inflation and partly to the new law, they found that buyers paid a "critical area premium" last year of $10,662 for such a house in Calvert, $8,677 for a similar property in Talbot. In Dorchester, the county among the three that is farthest from the Washington and Baltimore metropolitan areas, the premium was $684.

In Calvert, a house that sold for $204,000 in 1985 sold for $231,000 last year. Of the difference, $16,000 was attributed to inflation and $11,000 to the property's location in a critical area, the consultants said.

"We've seen a progression," said E. Patrick Beaton, a Rutgers professor directing the study.

Beaton said the impact is even being felt indirectly on the Virginia Eastern Shore, which is not subject to Maryland's critical area law. In Accomack County, he noted, as many properties changed hands last year as in the five previous years combined -- in part a reflection of the demand for land that can be developed at greater density.

The consultants are now looking at undeveloped waterfront property, including farmland, inside the 1,000-foot buffer. "What I hope will come out of this is a much clearer picture of who suffers and who benefits," he said.

Among those most affected, Beaton said, are farmers, who often sell off parts of their land to pay for their retirement. If their land is downzoned, its market value declines.

Beaton and others said they expect the Maryland law to be tested in court, but no challenge has yet been filed. This spring, the U.S. Supreme Court seemed to pave the way for such a lawsuit with two decisions restricting government's rights to downzone property without compensating the owners.

To provide some relief, Calvert County planners are proposing to allow a property owner to transfer development rights of one house per 20 acres from the restricted waterfront to an interior parcel -- in effect, to allow the property owner to develop land at a greater density somewhere outside the critical area.

In the Solomons area of southern Calvert, one developer has been able to proceed with the construction of 23 town houses on six waterfront acres because his plans were submitted before the Dec. 1, 1985, deadline. "Now, he could not do what he's doing here today," said real estate broker Fischer.

Across the bay a waterfront parcel on Kent Island that was subdivided before the legislation sold for $42,000 about 18 months ago, Stevensville real estate broker Reginald W. Jones said. The owner recently rejected a $100,000 offer.

"The property will definitely cost a lot more," said Jones.

"It has eliminated a fair number of people from the market," said broker Heine. "It's a phenomenal change. It used to be a family with an income of $65,000 to $70,000 could buy a decent home. Now, they're not even in the running. There are hardly any waterfront homes under $200,000."

One house his firm sold in 1982 for $100,000 is now on the market for $200,000. "It's just a small, three-bedroom rambler, 25 years old, bulkheaded with a pier," he said, but the water depth is too shallow for most boats.

At times, however, undeveloped tracts are being sold for higher prices because there are fewer bayshore plots to go around.

Heine recently sold an undeveloped 1.75-acre lot for $100,000 cash. "It probably wouldn't have commanded that price, except the buyer is aware they're not making any more subdivisions," Heine said. "It's like a scare when {the price of} gold is running up. All of a sudden, people want to buy waterfront while it's still available."

Developers seeking to comply with the new rules say the law limiting the number and placement of units also means the few they are allowed to build will carry a higher price for them to turn a profit.