Joe Hirshhorn, hands in his pockets, stares at two small Durer engravings perched in a bookcase in his Washington study. "They're the first things I bought, and they're going into the box with me. They're going to be buried with me."

Hirshhorn, the uranium mogul who gave $50 million in art and $1 million in cash to establish a national museum of modern art in Washington five years ago, will be 80 in August, and he spends a lot of time wrestling with that fact. He rarely lets it go.

"I can't believe it, that I'm gonna be 80. Except my memory isn't so good anymore."

But ask him where he bought the Durer prints when he was 17 and had just made his first $167,000 on the curb exchanged, and he'll spell out the answer: "Assenheim & Sons, 37 New Street, near Wall Street. That's A-s-s-e-n . . ." Or ask how much he paid ("75 bucks apiece") or how many works of art he has given to the Hirshhorn Museum and Sculpture Garden ("6,502 pieces, with 23 more last September"). Or, by mistake, ask him the same question twice. "You just asked me that question 20 minutes ago, "he snaps.

One thing Joe Hirshhorn has tried to forget is the pain he endured in the process of giving his collection to the nation - the hostile congressional hearings, the personal harassment.

"That's all over now, finished, "says Hirshhorn. I'm not bitter anymore. I was, but I'm going to be 80 years old. What's the point of wasting time on that? "

Time is now the only real enemy. And Joe Hirshhorn, whom critic Aline Saarinen long ago dubbed "a little man in a big hurry," isn't in a hurry any more. Take Our Picture! '

He still has the mobile and elusive face of a tragi-comic actor, the demeanor and gait of a former song-and-dance man. In their spacious front hall, Olga, his wife of the past 15 years, is apt to join in on the act, impulsively grabbing a straw hat, derby and cane off the coat rack and perching one on Joe's head and another on her own, as the two of them launch into a buck-and-wing. "Take our picture! " she'll say joyfully. And like the pros they are, they'll freeze-frame the routine in mid-air of the camera.

Other snapshots from the family album:

Click - Joe and Olga in intimate conversation on the escalator at the Hirshhorn Museum opening for Saul Steinberg Where Joe had been seated some distance from the head table), as she tenderly mops his brow and straightens his clip-on bow tie.

Click - Olga and Joe snuggled over catalogue at Sloane's Auction House, passing up an overpriced Chinese ivory and zeroing in on several pieces of early American furniture for the new winter house in Florida.

Click - Joe climbing out of his limousine the night the Hirshhorn Museum opened to the public in October 1974, singing: "Look at the lights! I'm getting excited. I'm getting delighted. I'm in love! "

Click - Joe, earlier this week, being honored at a benefit for the Skowhegan School of Art with a gold statuette for his long-time support of young artists, and announcing: "I'm gonna be 80 years old, and except for my wife Olga, this is the biggest prize I've had in my lifetime. " Every Day Is Sunday'

Now that the museum is launched, life for Joe and Olga Hirshhorn is a process piece filled with friends, travel, gallery hopping, auction going, house moving and just plain having a good time. "After all, every day is Sunday for us, " says the handsome, unspoiled Olga.

The only real sign of Hirshhorn's years is his constant awareness of it, and his acknowledged dependence, at least in part, upon somebody else. "Ask Olga, she's in charge of everything, " he says. "I just live here."

"Olga, where are you! "

A velvety-voiced Olga burst in, still pinning up her lush, dark hair, and looking very Connecticut suburban in black turtleneck, plaid skirt and pumps. "Sorry to be late, but we were up late watching a movie last night, " she apologizes.

Then Olga will tell you, right off the bat, "I'm 58, Mr Hirshhorn's fourth wife, and we have nine children between us. That's what everybody always wants to know, so I just tell them. " A Girl Friday Before they met, Olga Zatorsky Cunningham was running Services Unlimited, which she had begun as baby-sitting service, and which had grown into a successful employment agnecy, providing chauffeurs and cooks to the wealthy residents of Greenwich, Conn., Joe Hirshhorn among them. Eventually, Olga began working for Joe herself, first as a girl Friday, then Saturday and Sunday too, showing thousands of visitors through the house on charity tours and driving him to visit various artists and museums.

Finally, after 2 1/2 years, he popped the question - in a characteristic way: "Lose 10 pounds and I'll marry you. "

"It's a deal, " said Olga.

Now Olga plays the bright, capable, loving, energetic mother-wife-manager, and Joe plays Joe. "It's been a whole new life for me," says Olga. "I discovered art." Friends say it 's been a whole new life for Joe, too.

"He's been transformed from a lion into a pussycat. Or at least tamed." Move to Washington

Leaving Greenwich was wrenching for both of them. Olga was born there and had raised three sons there, after marrying her high school English teacher.The marriage lasted 23 years.

The youngest of three children of Greek immigrant parents - a laundress and a gardener - Olga was baptized in St. Mary's Roman Church in Greenwich.She was a big wheel at Greenwich High, editor of the newspaper, mile swimmer and tennis singles champ. She still plays tennis three times a week.

In June 1977, the Hirshhorns finally moved to Washington, to a house they had bought eight years before. It was a tentative move at first. They had, in fact, put the house back on the market, while Joe confided that he was angry and bitter about having been entirely cut off from the museum. He wasn't even on the board. The Hirshhorns remained in the stripped Tudor mansion in Greenwich for three years after helicopters and 55 trucks had carried away the last of the 2,000 sculptures and 3,000 paintings to the Hirshhorn Museum in August 1974.

"During all those years we used to turn up in Washington once or twice a year to sleep in the place," recalls Olga, "arriving in the middle of the night in evening gown and tux and suitcase and getting swallowed up in the darkness. The neighbours must have thought we were crazy."

Finally, Olga undertook what were to be "limited" renovation - "just the bare essential," Joe told her. "But it was like eating peanuts, and we couldn't stop," she says.

Nor could the public. From the moment it opened, the Hirshhorn Museum was a roaring success. Within two years it pulled out ahead of the Museum of Modern Art in New York, clocking in 1.5 million visitors a year to become the fourth most popular art museum in the country.

Confident, the Hirshhorns surrounded the house with a wooden fence, creating a small brick garden in the front - "a place for at least a few flowers and sculptures, and where we could sit in the sun," said Olga. A large seated figure by Henry Moore and two by de Kooning were promptly installed.

"That Moore cost me 5 1/2 times what I paid for the cast I gave to the museum, but I missed it so much I bought another as soon as one turned up in Baltimore," says Joe.

"It cost more than the whole renovation," reminds Olga.

"Yeah, but that's different," says Joe. Hirshhorns Are Coming!

"Before we had a chance to be homesick, we fell in love with Washington," says Olga. Once we found the grocery store, the drug store and the movies, we were set. It's a great city, and so easy to get about. Also, the tennis is great."

The Hirshhorns also have found all the auction and antique establishment within miles of the city, and rush around from one to another buying, though in relative moderation - mostly for the new house in Naples, Florida, where the spend longer and longer winters.

That old street cry, "The Hirshhorns are coming, the Hirshhorns are coming," now resounds through Washington galleries, and though dealer's hearts still flutter, they do not pound the way they used to. The legendary pace has slacked.

"Colect more? I'm gonna be 80 years old. Did you know that?"

On occasion, however, Joe does buy a major work, no doubt with the museum in mind, though he stead-fastly denies it. "I've given them enough," he insists. But he recently set an auction record for a major Gorky painting, for which he paid $140,000.

The Hirshhorn Museum admittedly would like to have another major Gorky. For now, the painting hangs over Joe's living room fireplace.

Otherwise, Hirshhorn is more likely to pick up a small Haitian sculpture at the Bader Gallery or two works by young Washington unknowns purchased from the citywide free-for-all exhibition held at the Museum of Temporary Art last winter. Olga, in fact, entered a sculpture of hers in that show, a simplified "Christmas Tree" made of three interlocking triangles of steel.

"Joe won't let me have a real Christmas tree, but he likes me to put this up because, as he says, 'it's different - it's a work of art.'"

Olga converted to Judaism six years ago. "She wants to buried beside me," explains Joe. A House Tour

The Hirshhorns don't give house tours anymore - "We just can't," says Olga. "But we certainly couldn't say no the Hirshhorn docents." So on a sunny morning last fall, while Joe, re-splendent in red vest, watched proudly with his nose pressed against the screen door, Olga began showing some grateful visitors around. A reporter tagged along.

The biggest surprise is the range of Hirshhorn's collecting interests. The paneled entrance hall is filled with everything from a headless Greco-Roman torso to an early American rooster weathervane and a painting by Richard Lindner. Nearby, Hirshhorn's study is filled with pre-Columbian sculpture and what he calls "My Girls" - a matlepiece covered with small sculptures by Giacometti, Nadelman, Degas, Picasso and others.

While Olga's desk,across the hall, is laden with invitations and other mail to be dealt with, Joes's desk groans with a burden of memorabilia and favorite "toys," everything from a mason jar full of marbles to a small piece by Nikidi Set. Phalle and a tiny vial of uranium, from whence his most recent fortune came. Hard by his chair is a bust of Hirshhorn by Manzu, and across the room, another bust, in process for years by Olga Hirshhorn. Shelf-images fill the house, painted portraits, busts and photographs, confirming the impression that Hirshhorn, in spite of everyting, still needs constant self-validation, and reassurance that this fabulous life has been his.

The phone rings and Joe grabs it - his umbilicus to the world. "Yeah? Hullo?" Pause. "You can do anything you want as long as you don't ask me for money." He hangs up. "That's the second request this morning," he says, shaking his head. It is only 10 a.m.

One flight up, past several Mother-well collages and a David Smith sculpture, the formal living room comes into view, centred with one of many exquisite oriented rugs - this a silk tabriz - and surrounded with fine American antique furniture, including matching Queen Anne lowboy and highboy, and a Simon Willard Grandfather clock. If he had to choose, the clock would probably be the one thing Hirshhorn would haul out himself in case of fire. He had it sent to storage during three divorce proceedings, which he explains by saying, "I just didn't want anyone to take it away. It is a wonder, you know."

Nearby, a small, book-filled sitting room, one of his favorite spots, spills over with Benin bronzes, while one more flight up is the Hirshhorn bed-room, devoted entirely (except for a few Calders) to the paintings of Eilshemius, the eccentric American painter whom Hirshhorn began collecting back in the '30s.

Incongruously, not far from the antique fourposter bed, draped with a pink silk swagger, is a small sofa which holds several stuffed toys, a patchwork rabbit among them. "We have lots of dolls," says Olga. "These we bought one time at a Howard Johnson's when we stopped for lunch. Joe saw them in a glass case and just went and bought them. These were the first; that's why I keep them here. You know Joe never had any toys when he was young. He was always working. It's hard for me to imagine him ever having any toys at all. 'Love and XXX's Henry'

Throughout the tour, Olga had pointed out various gifts she had made or found for her husband. "After all, he's very hard to buy presents for, as you might imagine." None, however, was more inventive than the collection of photographs of famous artists and art world figures, all with personal dedications to Joe. "I began this years back, and he liked it so much I've kept it up. It will go to the museum eventually."

Having downed his daily noontime bloody mary, Hirshhorn joined the group in the hallway of photographs to do a final number for the fans. "Look at that one ," he said, pointing to a photograph of Henry Moore, inscribed: "For Joe, with love and xxx's Henry."

"That's Henry Moore, he announced, unnecessarily.

There were others, frame to frame. "To Joe, for art and friendship, Raphael Soyer" and "For Joseph Hirsthorn, Lady Bird and Lyndon Johnson." Joe always stops to look once again at the photo of him with Picasso taken when Hirshhorn still had his place at Cap d'Antibes. And he'll always say, "I taught him the buck-and-wing. He admired that coat, and I gave it to him. It fitted him perfectly."

Joe has a vivid sense of his own history, much of which has been assembled and reported in a new book called "Hirshhorn: Medici from Brooklyn, by Barry Hyams, to be published later this month. All in a Name

The museum was well underway when the flood of protest came, and though there was inevitable criticism of the collection (few besides Hirsthorn and his curator, Abram Lerner, really knew what was in it) most of the mud-slinging seemed to arise from Hirshhorn's insistence that the building be named after him.

"I told him, 'Joe, it will go easier for you if you don't insist upon having your name on the museum'," says Lerner, now director of the Hirshhorn. "And he knew that. But he said, "Dammit, this was my life's work, and I want my name on it." Hirshhorn had seen the Guggenheim, the Frick, the Whitney, the Freer, the Corcoran and the Smithsonian itself, all named after their donors. He wanted the same treatment. He finally got it, but at a bitter price.

To this day he has not quite gotten over his ordeal, though he insists, "I am not bitter. Look, what did they want with a little Jew from Latvia and Brooklyn on the Mall?Of course there was some anti-Semitism involved. There always has been and there always will be. But I'm what I am, and that's not going to change."

In the aftermatrh of the controversy, Hirshhorn was separated entirely from the museum and its activities. He was not even a trustee.

"There was so much fuss that we all decided it would be best for Joe to detach himself for a while so that the museum could take on a life of its own, which it has now done," says Lerner. "People were afraid it would be just an extention of his own personal collection, which of course it is not. This is a national museum of modern art, but that probably won't be clear to people until the name Hirshhorn becomes as anonymous as "The Freer,' 'The Corcoran,' or 'The Whitney'. It will all take time."

Finally, in December 1977, one trustee finally said, "for God's sake, let's put Joe on the board." And there he now sits.

Meanwhile, is Hirshhorn planning further gifts for the museum? An endowment, perhaps?

"They're not getting any more," says Hirshhorn, though no one really believes it. "I gave them 6,502 pieces, 23 more last September, and $100,000 more to match a purchase fund gift. I don't want to go broke and sell apples." CAPTION: Picture 1, Joe Olga Hirshhorn in a favorite family snapshot; photo by Anomi Caryl; Picture 2, at the Hirshhorn Museum before its opening in 1974. and above by Harry Naltchayan-The Washington Post; Picture 3, Hirshhorn with President Johnson in1966.