Every weekday morning, a balding elderly man leaves the big yellow house near the Dumbarton Oaks formal gardens and makes his way past Georgetown's fancy homes to M Street, where he enters an unmarked door behind a modest two-story bank.

Upstairs is his law firm, Fortas & Koven, a five-lawyer enterprise with none of the glamor of his former practice.

It has been 10 years since Abe Fortas resigned from the U.S. Supreme Court in a scandal over the $20,000 payment he took from securities swindler Louis Wolfson -- and 10 years since Arnold & Porter, the law firm he helped found, spurned him.

That unobtrusive office down the street from his Georgetown home marks a dramatic shrinking of his sphere of legal activities.

Fortas, who will turn 70 in June, no longer runs in and out of the White House during crises as he did when Lyndon Johnson was president. He never grants interviews. He isn't called on to argue important Supreme Court cases, like the one in which he established for Clarence Gideon and all poor people charged with serious crimes the right to court-paid counsel. In fact, court records show he has not argued before the high court since he resigned.

But for all that, Fortas is doing about the same kind of work he did before taking the bench -- only on a smaller, quieter scale.

"He has all the enthusiam for his cases that a young associate would," said Joseph Berl, a former Securities and Exchange Commission lawyer who is unofficially the second in command at Fortas & Koven.

"He's in by 9 every morning, he puts in a full day, he travels all the time and I've never heard him say anything about slowing down," Berl said. "His work seems to be what he cares most about."

That's a refrain that is repeated by many of his friends and associates. Despite the scandal that clouded his name and the old age that has crept up on him and taken away or incapacitated the three lawyers he was closet to, William Douglas, Thurman Arnold and Paul Porter, Fortas remains a man who lives to work.

He is no longer a trial lawyer because the resources of a small firm would make it impossible to take on extended litigation. Most often Fortas has a behind-the-scenes role dispensing advice and strategy.

He is retained to consult on complicated antitrust and securities matters for many of his old clients, especially New York financial figures like John Loeb, the Lazarus family of Federated Department Stores and Arnold Bernhard of Value Line.

From time to time, he drops in on Stanley Sporkin, the enforcement director of the SEC, to talk about clients like Vincenzo Cazaniga, the managing director of the Exxon Corp. subsidiary in Italy who was involved in the SEC investigation of Exxon's questionable overseas payments.

Fortas' fastidiousness about legal matters is legendary and contributes to his reputation as a difficult man to work for, a stickler for legal writing who would order endless drafts of every document he supervised.

In his years at Arnold & Porter, where he functioned informally as a kind of managing partner, he was seen as something of an ogre to the younger lawyers. Many of them, now prominent Washington lawyers in their own right, recall him as utterly indifferent to their emotional needs and supremely demanding professionally.

"He has a brilliant legal mind and that's all he looks for in other lawyers," said one Arnold & Porter alumnus. "I don't know anyone who has a very human relationship with him."

"He stood out as the most demanding lawyer in the office," said Charles Reich, who worked at Arnold & Porter for five years before moving into academia and book writing.

Fortas' resignation from the Supreme Court sent shock waves through the firm, where his wife, Carolyn Agger, 70 is a tax partner. And the firm's reluctance to let its founder back into the partnership sent shock waves through the Washington legal community.

It was not the taint of the scandal alone that bothered most lawyers at the firm. Rather, it was a combination of Fortas' reputation for being hard to get along with, which had assumed almost mythic proportions during his absence, and what many former Arnold & Porter lawyers said was nothing more dignified than greed and ego. Many of the lawyers simply didn't want their former boss to come back because they thought it would put a crimp in their standing within the firm.

According to a number of lawyers who were at the firm at the time, none of whom would talk on the record, the three key players in keeping Fortas out were Abe Krash and Dennis Lyons, two young partners whose stars were rising rapidly and who have remained at the firm, and Joseph Califano, who was hired shortly after Fortas was appointed to the Supreme Court to try to maintain the firm's big name reputation.

"There was very strong feeling about Abe at the time, so much so that everyone knew there was a good chance of a lot of the best lawyers splitting off into a new firm if Abe came back," said a former Arnold & Porter partner.

"Fortas had his boys, guys he'd personally taught, who were in their late 30s when he left. They figured the Supreme Court was a lifetime appointment, and they began to play a more important part in dealing with the clients, Fortas' clients.

"Four years later, there's the Wolfson scandal, and the lawyers have to confront the possibility of Fortas coming back," the former partner said. "They know he's not great about sharing power, and they think he'll stultify their professional growth if he comes back and takes the clients over again."

It was an exceddingly messy situation, made all the more difficult by Agger's presence at the firm. A tough, quick-minded lawyer who smokes long black cigars and once admitted owning 150 pairs of shoes, Agger had always been very careful to maintain her own profession identity. Still, many observers were surprised that she could remain at a firm that had given her husband such a slap in the face.

The Fortas question never came to a vote because Fortas announced that he was setting up the new firm, Fortas & Koven. He never made any statement about Arnold & Porter.

The youngest son of Jewish immigrants from England, Fortas grew up in Memphis, where he had, by 16, won a scholarship to Southwestern College.

He went on to Yale Law School, again on scholarship, in 1930. He was named editor-in-chief of the Yale Law Journal, and his law review article subject was suggested to him by William Douglas, who had arrived to teach at Yale the year before Fortas enrolled.

Upon graduation, Fortas turned down an offer to teach at Yale to join his other law school mentor, Thurman Arnold in Washington, where the government was expanding at an unprecedented rate with the establishment of an alphabet soup of new regulatory agencies.

For the next 16 years, Fortas worked his way through the Agricultural Adjustment Administration the SEC, the Public Works Administration and the War Relocation Authority.

Thurman Arnold, meantime, had gone from the AAA to the Justice Department's antitrust division to the U.S. Court of Appeals. But finding judging dull, Arnold quit the bench to set up a new law firm with Fortas in 1946. Soon after, Porter joined the firm, whose influence grew as Arnold, Fortas & Porter.

The firm has thrived ever since. With the deaths of his two partners, Fortas has no strong social ties there anymore. But he remains on good terms with his old firm and comfortable enought so that he and his wife could, for example, attend a firm party last month.

Since he left the court, Fortas also hasn't socialized with any of his former Supreme Court colleagues except for his former law professor, the ailing Justice Douglas, whom he still visits.

Now he works in his new firm and is of counsel to Chicago's Friedman & Koven. Originally, Howard Koven, who is a principal in both firms, anticipated moving to Washington to work with Fortas, but for what he said are family reasons, he stayed in Chicago.

The relationship between the two firms is fairly loose, Koven said.

Though Fortas concentrates on securities and antitrust matters, he occasionally ventures into other fields. For example, he's been retained to help the American Trucking Associations oppose deregulation and to aid the National Home Study Council in a Federal Trade Commission rule making proceeding on vocational training.

"Abe chooses what he wants to handle," Koven said. "The essential thing about him is that he can take any legal question, not just one about securities, and reduce it to understandable language."

Fortas seems to take a special pleasure in the most complicated questions. In last October, he filed voluminous documents in the Court of Claims here, including a 1,000-page, two-volume appendix filled with the most technical, scientific data, about a patent application Hughes Aircraft filed for an apparatus used to control the orientation of a satellite.

"He's the lawyer's laywer, the brain surgeon, the guy you call in when all else fails," said Sheldon Karon of Chicago's Karon, Morrison & Savikas, who has worked with Fortas on the Hughes case and on others.

"He still has the stamina of an Olympic athlete," Karon said, "he was out here working on the Hughes case last week and he'd been diagnosed as having Brazillian flu. But he worked until 9 at night, even when all the rest of us were getting crosseyed with exhaustion. Then he sent the secretaries flowers and candy the next day.

"I don't know where people get this idea of his brusqueness. He's been very good to me. About five years ago, I was in New York working very hard on a case, and he called and asked if I was free on a particular evening. He came and took me to the opening of a Tennessee Williams play, and they to Lincoln Center.

"I realized in the middle of it all that it was my birthday, and he had come to town especially to see that I wasn't lonely by myself so far away from home," he said. "It was a wonderful evening. We even went backstage to see Issac Stern after the concert."

Fortas has long been involved with a number of musician friends. He has played the violin since his childhood days in Memphis and is still an accomplished musician. He has a longstanding tradition on Sunday night concerts in his Georgetown home, where musicians like Stern, Rudolf Serkin, David Oistrakh and Pablo Casals would sit in when they were in town.

Fortas still sees his musician friends. Most of the lawyers who have been his friends, the last of the New Dealers, said they now see him once a year at best.

"Whatever he says, I know the Supreme Court affair left him crushed," said one old friend. "He felt very ill-used. It doesn't surprise me that he would have a much quieter life now. People don't make new friends so easily in their late 60s, especially people like Abe, and many of his old friends have passed on. I guess he's kind of isolated now. He really had a fall from glory."

David Ginsburg, a friend of both Fortas' and Douglas', disagrees.

"He may not see too many lawyers now, but a lot of lawyers don't like to hang out with other lawyers," he said. "He has his musician friends, and I'd say he was a happy man with an enviable life. I don't see any pathos in it."