Colin F. S. Walters, the District of Columbia's top money manager, was forced out yesterday as top officials sought ways out of a financial crisis aggravated by a court decision that will cost the city millions of dollars in tax revenues.

Walters abruptly quit his $50,112-a-year job after being summoned by City Administrator Elijah B. Rogers and told he had to whip the city's new, trouble-plagued financial management computer system speedily into shape.

One of the team of assistant city administrators who joined Mayor Marion Barry's administration on his inauguration day in 1979, Walters was the first high-ranking Barry appointee forced to depart from City Hall. His title is assistant city administrator for financial management.

"There was pressure on me to improve my performance, and in the face of that I had to say, 'I'm doing the best I can.'" Walters told reporters after his resignation was announced. "I did not get . . . high grades for knocking heads."

Rogers said he did not discourage Walters' decision to quit. When a reporter suggested this implied Walters would have beeen fired if he had not resigned, Rogers replied: "I don't think that would be inaccurate."

The resignation came in the wake of a D.C. Court of Appeals ruling Tuesday that the city must refund up to $58 million it has collected over the last five years from such professionals as lawyers, accountants, engineers and architects who conduct business in the city. About two-thirds of the total would be paid to suburban residents.

With the city facing a $28 million budget shortage for the balance of this fiscal year, the decision threatened to throw the city's already-shaky finances into even more disarray. The inability to collect the professional tax for the balance of this year could add as much as $12 million to the cash shortage, even without making the refunds promptly.

Walters' letter of resignation and the Mayor's reply accepting it "with deep regret" both were dated yesterday. Walters will leave the job March 21.

Talking to reporters, Barry suggested that Walters was in a job that was over his head, but he refused to blame the departing official for getting city into its increasingly severe financial crunch.

While Walters is not responsible either for city budgeting or the collection of taxes, he is charged with keeping track of the money once it is in and oversees meeting the payroll and paying municipal bills, which often are paid late.

In one well-publicized goof in Walters' department, the computer system failed to pay $11 million to the Internal Revenue Service in December that had been withheld from the pay checks of D.C. employes. The money was paid a month late.

In another instance, Walters appeared at a D.C. City Council hearing and was confronted with a computer printout indicating city agencies may be overspending their budgets at the rate of $300 million a year -- a report that made the evening news on television.

That was wrong, Walters told the Council members, and he got the computers reprogrammed within days with the damage already done. He explained to the Council members that the computer system at its present stage was able to produce data that was only 90 percent accurate.

Such incidents caused embarrassment to the Barry administration and grumbling among council members, and led in part to a stern letter from Sen. Patrick J. Leahy (D-Vt.), chairman of the D.C. Appropriations Subcommittee, telling the mayor his handling of the city's financial crisis is "a test of your administration."

Walters, 41, is a soft-spoken native Englishman who is personally popular at city hall. His wife, Mildred Bantista, is director of the D.C. Commission on Arts and Humanities.

In discussing his impending departure yesterday, Walters said he had been "under substantial pressure since the turn of the year from my superiors here . . . to speed up the [full] implementation of the computerized management system. His only superiors are the mayor and city administrator Rogers.

Whoever follows him, Walters said, "will have an awful, awful, awful job on his hands."

He said the system has improved in recent weeks, but will require a full shakedown of a year or more.

Barry held two high-level closed meetings on the city's financial situation yesterday -- with Rogers and other top administrators in the morning, and with key City Council members and his finance and budget directors in the afternoon. No decisions were announced.

Rogers said the Mayor told Budget Director Gladys Mack and Finance Director Carolyn L. Smith to prepare a new plan to overcome the financial shortage and to submit it to him.

Barry, talking to reporters, said the budget situation "is not a crisis yet." He said the city's potential net loss from the invalidation of the professional tax is estimated at $41 million in refunds plus about $4 million in interest -- a lower interest figure than originally estimated.