The construction of Washington's multibillion-dollar subway system has failed to reduce the number of commuters traveling by car to the downtown area, according to a new study by the Metropolitan Washington Council of Governments.

Despite increased use of public transportation, the average number of rush-hour commuters heading into the downtown area by car on weekday mornings rose 5.7 percent to about 238,000 persons between 1977 and last year, the report said. The Metro subway system opened in 1976 and by last year it had expanded to include 39 miles.

The new findings represent a reversal of earlier trends, which had emerged after the 1979 gasoline shortages. A drop occurred in rush-hour auto travel to downtown between 1978 and 1980, as motorists switched to mass transit.

But the decrease now appears to have been a temporary aberration, more than offset by increases in car trips after the gas shortages eased, the study says.

Rush-hour commuters using public transportation to reach the downtown area climbed sharply to 123,500 passengers last year, a 31 percent increase over 1977.

According to the study, subway ridership rose steeply, but bus patronage dropped almost 40 percent.

Many former bus riders switched to trains either because rail service was faster or because the bus routes that they had used were discontinued, the study notes.

Despite the resurgence of auto travel, the opening of the subway system appears to have had at least a marginal impact on traffic congestion, the study indicates.

In 1977, bus and subway rides accounted for 29 percent of all morning rush-hour trips to the downtown area, with 71 percent of the commuters traveling by car. In 1982, use of mass transit reached 34 percent, while car travel amounted to 66 percent.

The impact of the Metro system on highway travel long has been regarded as a key issue by transportation officials.

"One of the primary goals of the Metrorail system was to reduce traffic congestion by providing an alternative to commuting to and from the central employment area by private automobile," the study says.

Nevertheless, transportation analysts at the Council of Governments said yesterday that the Metro system's failure to reduce commuting by car appeared in line with trends elsewhere in the United States. "If there's capacity on the highway, it will be filled. You can be pretty much guaranteed of that," said Kenneth I. Flick, a transportation engineer who wrote the report.

The study found that the subway system had opened the way for increased commuting both by public transportation and by car.

The rail system's expansion provided more seats for mass transit riders, the study says. As subway lines were extended, many buses were removed from congested downtown streets, leaving room for more cars, it adds.

Robert A. Pickett, assistant planning director for the Metro system, said the study's findings did not conflict with the transit authority's goals. Metro's long-range aim, he said, was to accommodate enough commuters to avoid the need for new highways.

Data included in the study showed that the Metro system has succeeded in attracting patrons who otherwise would have commuted by car, he added.

"As long as highways are there, people are going to use them," Pickett said.

Total morning rush-hour travel to the downtown area--by cars, buses and trains--rose 13.2 percent to nearly 362,000 trips between 1977 and last year, the study says.

Officials said it was unclear why commuting to downtown had increased during a period of economic recession, although they suggested several possible explanations.

They said that employment in the rapidly developing downtown area may have remained sizable despite the recession. A larger proportion of downtown jobs may be held by suburban commuters than in the past, they said. Some commuters may have traveled through the downtown area to jobs elsewhere in the region, they added.

The new study appears significant, in part, because it focuses on travel to the downtown area along corridors the subway system was designed to serve.

Previous reports, by the Census Bureau and other agencies, have pointed to increases in commuting by car, but have attributed the trends to growth in suburban employment centers, which the rail system does not reach.