A friend of newly elected Fairfax City Treasurer Ray M. Birch was just about to sit down in Birch's office recently when he found a penny on the floor. "Here," Birch remembers his saying jokingly, "you might need this to balance the books."

Then the man spotted another penny. "We both got down on our hands and knees," Birch recalled. "You know, we found 42 cents on the floor!"

In the three weeks since he took office, Birch says he has found a lot more than 42 cents in unaccounted-for monies lying around. In fact, so far he says he has stumbled across--in some cases literally--$14,929 in checks, cash and coins, most of it squirreled away in desk drawers or simply dropped into trays that were left lying on the floor.

That, says the 56-year-old Birch, isn't quite what he bargained for when he decided to run for treasurer last year. So Tuesday night, he told the City Council he was resigning his $30,000-a-year job, effective Jan. 31.

"I sure misjudged the depth of the problem," said Birch, whose campaign for the office was based on his claim that he was the only one who could straighten out the problems there. "My estimate is that it will take maybe six months of 15 hours a day of work to get things back in order, and I don't feel I can work 15 hours a day anymore."

While Birch lamented, seated at a desk in an adjoining room was the person he and others at City Hall are blaming for the current state of the city treasury: Frances L. Cox, the longtime city treasurer whom Birch defeated in November. Even though she left office Jan. 1, she is still trying to balance the city books for the month of December so that Birch can take them over. The City Council has given her until Friday.

"I wish that this wouldn't be dragged out like a soap opera," said Cox, 56, who claims that whatever problems exist, they are common to any transition. "I lost the election. I would have thought that would have been enough."

It is, however, not the first time that Cox, Fairfax's treasurer since 1959, has caught the public eye. In 1965 and in 1978 auditors found discrepancies in her city accounts.

Birch said he found:

* There were $1.5 million in discrepancies between bank deposits and treasurer's reports from July to November.

* Not since Aug. 31 had anyone bothered to record checks in the appropriate ledger.

* And keys to a city safety deposit box that Birch had feared were lost were in fact in Cox's possession. She turned them in last week, he said.

There has never been any hint that dishonesty was responsible for the discrepancies, according to Fairfax Mayor Frederick W. Silverthorne, who has criticized Cox's management. "It's just that I felt that Mrs. Cox was just unable to keep on top of it, whether it was because the city was growing too fast for her, or what," he said.

When Cox was first appointed town clerk-treasurer, Fairfax had a population of 13,585 and a yearly budget of $530,000. Today, Fairfax, which became a city in 1961, has a population of 19,390 and a $22.3 million budget. The city treasurer receives all payments for city taxes and services, puts that money in the bank and invests it, and pays all city debts.

Cox says city politics are behind all the accusations of poor management. "The big one was the audit in 1978. That was the year I supported an opponent of Silverthorne's for mayor and he never forgot it," she said. "If you stay in office as long as I have, this sort of thing will happen to you eventually."

Birch says that it wasn't really until his first couple of days behind the desk that the true enormity of his task became apparent.

"Right away, I recognized there were going to be problems," he said. "We started finding batches of checks in desk drawers and trays, like these." Birch reached under a table and pulled out a tray of checks for city taxes that dated back as far as 1978--bundles of them, wrapped with rubber bands. None had been deposited, said Birch.

Then Birch said he was notified by a bank handling city funds that the city's account was $156,000 overdrawn. "I hadn't even written any checks yet," he said.

What can be done? "I really don't have the answers," said Birch, who is leaving that issue to whomever the council appoints to serve as interim treasurer until a new election can be held next November.

"During the campaign, he went around with yellow sheets saying that my office was in a big mess," said an angry Cox last week. "The thing that puzzles me is, why, after he said that, is he so surprised at what he found that he has to quit?"