Fairfax City Treasurer Ray M. Birch resigned last night after less than three weeks on the job, claiming that he found the treasurer's books in such disarray he cannot perform the job.

Birch, who last November defeated veteran treasurer Frances L. Cox at the polls, said he will leave the office Jan. 31 "to call attention to the deplorable conditions" in the treasurer's office.

Birch's move brought a prompt rejoinder from Cox, who held the job for 28 years.

"In my opinion, Birch is not qualified for the job and he panicked," she said. "This is all a mountain out of a molehill."

In a memo to the City Council, Birch said he found $1.5 million in discrepancies between bank deposits and treasurer's reports from July to November 1981.

He said he also found more than $14,000 in checks and cash in desk drawers that were not recorded by Cox when she left office Dec. 31.

Mayor Frederick Silverthorne said at last night's council meeting that an independent audit by an Arlington accounting firm corroborated Birch's findings. He said no evidence of illegality has been found.

Cox, who lost the treasurer's race by 192 votes out of 5,679 cast in the three-way nonpartisan contest, denied she left the office in disarray. She said Birch's assertion that bank deposits do not match records is untrue.

"If the money was there it was not hidden," she said. "It was just part of the funds that were not deposited on Dec. 31 because they were just turned in."

Birch, 56, worked for the Fairfax County government for 29 years before retiring in 1977 as assistant director of its finance office. His election to the $30,000-a-year treasurer's post came on his second try.

Cox's management of the treasurer's office was the subject of criticism in earlier years. Some city council members and audit reports criticized accounting procedures and methods of handling the city's money and auditors cited Cox several times for delays in depositing large incoming checks.

Last night the council accepted Burch's resignation and initiated a search for an acting treasurer to serve until a new election next November.