With his customary wit and some pertinent public relations advice, White House press secretary James S. Brady returned to his desk in the West Wing yesterday for the first time since he was shot in the March 30, 1981, assassination attempt on President Reagan.

Arriving in a van, Brady was taken out in a wheelchair, but then got up and walked, with the help of an assistant and a cane, into the White House West Lobby. Later, he was ready with some advice on dealing with the press when White House chief of staff James A. Baker III dropped in at his office.

"Jim, how should we handle the unemployment story?" Baker asked.

"Your resignation story will knock it off the front page," Brady joked.

Brady spent three hours at work, dictating answers to personal mail and talking with White House aides. Sitting by a crackling fireplace in the press secretary's office, Brady listened by intercom as deputy press secretary Larry Speakes fended off questions in the daily press briefing.

"It was a tower of babble," he punned afterward.

Brady's appearance was the first in what Speakes said would be a return to work on "a regular basis," perhaps weekly. Speakes said Brady would come to the office in lieu of morning sessions at George Washington University Hospital, where he has been undergoing speech, occupational and physical therapy.

"He will do whatever we do," said Speakes, who has been using Brady's West Wing office. "He's going to work with us and be part of the team. I am, more than anyone, glad to see him back. I think it's the greatest thing for him and for us."

Brady was shot in the brain 19 months ago by John W. Hinckley Jr., who also wounded Reagan in the lung, and hit a Secret Service agent and a Washington policeman in the attack outside the Washington Hilton. A jury later found Hinckley not guilty by reason of insanity.

Although he was the most seriously injured, Brady has made a remarkable recovery. After several operations he still suffers from some paralysis on his left side and difficulties with speech modulation.

But as he returned to work yesterday, Brady was in high spirits and demonstrated the wit that earned him many friends during Reagan's 1980 presidential campaign.

Reporters gathered around the door to his office as Brady arrived, and the press secretary traded some old jokes.

"Mr. Felker, how are you?" he greeted Bruce Drake of The New York Daily News, a jesting reference to Clay Felker, who was editing the newspaper's "Tonight" edition during the 1980 Reagan transition, for which Brady was chief press spokesman.

In his White House office, Brady, whose nickname is "the Bear," took note of a stuffed bear on the fireplace mantle wearing a cap from his beloved Chicago Cubs. He greeted Speakes, who is from Mississippi, by his nickname, "Catfish."

Brady, still confined to a wheelchair, was at the White House on election night, watching the returns with other staffers in the Roosevelt Room.

Shortly before that he returned to the White House for lunch and stopped in to talk with reporters in the briefing room. Friends say that Brady's speech has improved markedly and that his grasp of politics and current events remains sharp.

Yesterday, chief of staff Baker filled him in on the latest details of Illinois Gov. James R. Thompson's post-election battle for ballots.