The Maryland governor's mansion stood empty today, as Acting Gov. Blair Lee III moved out, Gov. Marvin Mandel continued thinking about whether to reclaim the ull powers of the office and Gov.-elect Harry R. Hughes awaited his chance to move in on Wednesday.

Suitcase in hand, Lee left the mansion this morning and took a final look at the building where he had lived for more than a year. "Maybe," he said before slipping into the back seat of a state trooper's car for the ride to his family home in Silver Spring, "I should put up a 'vacancy' sign and see if there are any takers."

Mandel, still savoring the court reversal of his corruption conviction, spent the day at his suburban Annapolis consulting office where he replayed tape recordings of congratulatory telephone calls, and at his favorite delicatessen where he received old friends.

"There's no hurry," said Mandel, whose term of office ends Wednesday. "I'm governor now. As far as the powers are concerned, there's nothing important [to do ice]."

Lee, who had evidently had his fill of this town and of Mandel's leisurely attitude, not only abandoned the mansion this morning but also moved out of the executive office on the second floor of the State House. He stripped the walls of personally owned maps and paintings and headed back home with a moving van that carried the office's only desk and green swivel chair.

"Would you believe I really don't know or care what Marvin decides to do at this point?" said Lee, who had served as acting governor since October 1977. "I've finished my work around here."

Hughes, the governor-elect whose preparations for office have been overshadowed this week by the resurrection of Mandel, stayed a safe distance from Annapolis, spending the entire day on the 15th floor of the state office building in Baltimore where he examined state fiscal papers with budget director Thomas W. Schmidt.

"We're not coming down (to Annapolis) until Wednesday," said Michael Canning, Hughes' press secretary. "There's no reason for Harry to move in any earlier than that."

Mandel has had little time to himself since the victory parties began 48 hours ago, little chance to catch his breath since striding into his old State House office yesterday, a colorful retinue of aides at his side, and taking back his cherished title with the words "I am governor."

Today, after a two-hour breakfast at Chic and Ruth's Delly, he took a crisp $20 bill from his wife, Jeanne, made one stop at a Main Street tobacco shop for "a quarter pound of Unique t'backa," and repaired to the quiet of his consulting office down Ritchie Highway in Arnold.

Inside the narrow two-room office, the man who governed Maryland for eight years sat at the receptionist's desk, a felt hunting cap planted on his head, and listened to a cassette tape recording of the phone calls that started streaming in shortly after the appellate court decision was announced Thursday evening.

The calls of congratulations came in from old friends and political supporters, an associate judge of Maryland's Court of Appeals and the chairman of the state Public Service Commission. None of the messages seemed to move him more than the simple pleas from people he didn't know who asked him to pretend he had never been convicted in the first place.

"I would like very much to see you move back in the governor's office," said one caller, who identified herself as a Gaithersburg woman. "Please for all those constituents who voted for you, go back and let 'em have it good."

Another woman caller was sobbing when she finished her message. "You don't have to call back," she said. "I just want to add my name to the people who are so happy."

Mandel, now leaning over the recorder to hear each word, jotted down the names and phone numbers of every well-wisher. "This is the fifth one I had telling me to go back," he said, puffing his ever-present pipe. "It almost makes you want to cry."

The day began for Mandel with a more public display of support for him. For two hours, he perched himself in the "Governor's Office" at Chic and Ruth's, the first booth in a crowded row, leaning forward to shake a hand or nodding his appreciation for a kind word.

Jeanne Mandel, the governor's wife and constant companion, was hardly able to supprrss her excitement two days after the event."I woke up this morning and said, 'Marvin Mandel, is this really happening, or is this a dream?" she reported.

The procession of visitors had continued for more than an hour when Mandel's 18-year-old stepson, John Michael Dorsey, entered the delicatessen, a surprise returned from Fork Union (Va.) Military Academy, a strict school where no television, newspapers or telephone calls are allowed.

"John Michael?" Mrs. Mandel called. "Is that you? How did they let you out? John Michael, God was sitting on your shoulder. Here dear, sitdown next to the governor."

The 6-foot 2-inch, 212-pounder slid into the booth and kissed his diminutive stepfather, hardly distracting him from a political discussion he was having with a reporter. Then, John Michael explained how he learned of the court reversal and jumped on the next possible bus home.

"I was sitting in my room on Thursday cleaning up the barracks for Saturday inspection," he said. "All of a sudden I looked at my radio and it says, 'Governor Mandel... we went nuts."

The State House, so charged with electricity yesterday when Mandel and his roguish aides returned to the building they once controlled, was quiet and desolate today as Lee moved out and janitors cleaned up the paper cups ad cigar butts left from the Mandel clan's second-floor celebration.

The few reporters and politicians who ventured into the building on this rainy day found little to do except whisper into the telephones and develop rumors about whether Mandel might reassume power.

For every source, there was a different report. Mandel's friends said "yes," the takeover would come on Sunday. Mandel's aides said "no," he's not going to do it at all.

The Associated Press quoted authoritative sources as saying Mandel would not reassume power. The Baltimore newspapers, with authoritative sources of their own, said that Mandel would be back Sunday.

Moving day was more pleasurable for Lee than he might have expected two days ago. At that time, he was enraged over a letter sent from the Hughes transition office requesting an immediate inventory of State House furniture and equipment, including the furnishings in Lee's office.

"I called up there and I couldn't find anybody," he said. "I asked for Mr. Hughes, Mr. McWilliams, Mr. Canning, and Mr. Feinsten (members of Hughes' transition team). Nobody was home.

"I finally got a nice young lady who was almost whimpering by the time I finished with her. I felt kinda bad about that. I pointed out that I was not in the habit of stealing state furniture."

Before closing the door to his office, Lee threw a Baltimore telephone directory into one of the six boxes a state trooper was lining up to carry downstairs. "This," joked Lee as he grabbed the directory, "is my only offense against the state of Maryland."

His press secretary, Thom Burden, was seen trying to stuff a brown swivel chair into the front seat of his white Chevrolet. "It's personal property," said Burden, who went out of his way to round up a few reporters to watch him make off with the piece of furniture. "My wife gave it to me for my 35th birthday."

Minutes later, two men began marching around the circular State House walkway, holding a sign that read: "The Fourth Circt Court of Appeals was Bought -- Mandel Did It." The men refused to identify themselves or discuss their protest. "I have no comment," said one.