The District of Columbia Court of Appeals invalidated yesterday the city's tax on professionals who work in the city. The decision eventually could force the city government to refund as much as $45 million in revenue collected since the tax was enacted four years ago.

In a 17-page decision, the three judge appellate panel decided that the City Council was barred under the Home Rule Act from imposing the tax because its burden fell mainly on persons who live outside the city.

For those nonresidents-who paid the bulk of the $45 million-the council action amounted to a commuter tax on personal income, which is precisely what Congress intended to prevent when it granted the city home rule, the court said.

Enacted by the council in the first year of limited home rule, the tax went into effect Nov. 30, 1975. Suburban professionals immediately complained that the tax was designed to capture new revenue from them while leaving the tax burdens of city residents relatively unchanged.

The tax chiefly applies to self-employed professionals and personal service workers, including lawyers, doctors, architects, artists, writers photographers and accountants.

City officials estimated yesterday that about 5,000 persons pay the tax annually. Because the court decision yesterday is subject to further appeals, those officials were reluctant to discuss the prospects for a refund, or the cost or mechanics of it.

If a refund is ordered, suburban professionals-who cannot claim a state income tax deduction for payments to the city government-presumably would get back all of their city taxes.

But city residents, who receive what substantially amounts to a credit toward their city personal income taxes to offset the amount of professional tax, would owe most-if not all-of their refund to the city in higher personal income taxes.

Colin F. S. Walters, assistant city administrator for financial management, said yesterday that he was "shocked" at the prospect of having to refund millions of dollars in taxes.

"It surely would make a significant dent in the mayor's financial program," Walters said.

Mayor Marion Barry was informed of the court's decision early yesterday by Acting Corporation Counsel Judith W. Rogers. Rogers described Barry as concerned about the ruling.

Principal Deputy Corporation Counsel Louis P. Robbins, who represented the city government in the case, said yesterday that he expects the city will ask the nine-member court of appeals to review the panel's decision.

If the full court refuses to rehear the case, or upholds the panel's decision, the city could ask the U.S. Supreme Court for a final review.

Yesterday's court decision was based on a 1977 suit challenging the tax brought by two suburban lawyers-Richard A. Bishop, who lives in Alexandria, and Axel-Felix Kleiboemer, of Kensington. Both men practice law in the city.

"The court upheld the position we have advocated since 1975, before any of the taxes were collected," said Philip L. Kellogg, who brought the case to the appeals court along with several other lawyers on behalf of the two attorneys.

"We believe the court's findings are correct and that the court has made a very courageous decision," Kellogg said in a telephone interview.

The appeals court decision was written by Judge Catherine B. Kelly, who was joined by Associate Judge George R. Gallagher and retired Judge Hubert B. Pair.

In her opinion, Kelly focused on a section of the Home Rule Act that specifically prohibits the City Council from imposing any tax on personal income of persons who are not residents of the city.

The central question in the case, Kelly wrote, was whether the tax was on personal income or "something other than income."

The suburban lawyers argued it was a tax on personal income, and thus clearly prohibited by Congress. The city government argued it was a tax on "the right to earn income in the District," Kelly said.

The issue, however, Kelly said, was "the nature and effect of a tax, not its label."

"The District of Columbia Code calls this an unincorporated business franchise tax," the judge said. Franchise taxes, however, can be considered income taxes, she said. The court found that the city's professional tax is levied against personal income.

"To the extent here that we deal with individuals who are professionals and are not protected by te corporate veil, we must find that the tax burdens the taxpayer personally," Kelly wrote in her opinion.

The court decision yesterday reversed an earlier ruling by D.C. Superior Court Judge John Garrett Penn in which he found that the tax was a "franchise" tax even though it was based on income.

L. W. Garton, associate director of the D.C. Department of Finance and Revenue, said there is no way to determine from tax records how much of the $45 million in professional tax revenue came from suburban residents.

The department's director, Kenneth Back, said any refund to city taxpayers would be counted as taxable income.

Back said, however, that if a refund is ultimately ordered, recipients living in the city would have to file amended personal income tax returns. Taxpayers living in the suburbs would have to file amended federal tax forms to account for the refund.