Rose Kennedy celebrated her 96th birthday yesterday in Hyannis Port with a mass conducted by the Rev. Edward Duffy, who has known her for 40 years. A family gathering is planned for the coming weekend. A birthday party had been scheduled for last Sunday with most of the Kennedy clan gathered for Caroline Kennedy's wedding to Edwin Schlossberg, but the party was canceled because, a family spokesman said, "she wanted to let it be Caroline's weekend."

Kennedy, who was bedridden for some time, has recently been seen in a wheelchair being taken for walks around the compound. She suffered a stroke in 1984, but family members say she is up and about and doing well. Schlossberg and Caroline Kennedy won't be at this weekend's party. The couple is spending a two-week honeymoon in Hawaii and then will travel to Tokyo and Osaka before returning home. Jean Harris Drops a Claim

Convicted murderer Jean Harris has renounced her claim to the $220,000 left to her by the man she killed, Scarsdale Diet doctor Herman Tarnower. She has also written a personal letter to New York Gov. Mario Cuomo asking for clemency. Serving a sentence of 15 years to life for killing her longtime lover, the former Madeira School headmistress has been pressing hard in recent weeks to be released from the Bedford Hills Correctional Facility for women, where she has served five years.

Her son James has gathered some 15,000 signatures on a clemency petition and Harris' attorney, Michael Kennedy, said he has more than 1,000 letters of support to file. Kennedy would not reveal the contents of her letter to Cuomo. There was no comment from Cuomo, who makes a rule of not commenting on a clemency topic unless he is taking some action on the request. Out and About

Nobody ever pays much attention to a delivery boy. Rep. Thomas Downey, whose wife Chris owns Prima Pizza, a thriving carryout on Capitol Hill, was in the restaurant Friday when a call came in for a delivery for nine pizzas. Downey decided to help his wife deliver the pizzas. When they arrived at the Fairchild Building on Capitol Hill, Downey, a New York Democrat, noticed they were delivering to the Campaign for Prosperity, a conservative Republican political action committee. A picture of Rep. Jack Kemp (R-N.Y.) was prominently displayed on the wall. Downey said no one recognized him so he took the money and left. A spokesman for the Campaign for Prosperity said yesterday they hadn't noticed the pizza delivery boy and probably didn't like the service, so they hadn't even given him a tip . . .

The family of former New York governor W. Averell Harriman gathered at his bedside in his Yorktown Heights, N.Y., home, where the 94-year-old former adviser to presidents lay suffering from kidney failure. A spokesperson in his Washington office described the former ambassador to the Soviet Union as being "very, very ill. We're taking it hour by hour, day by day." His wife, the politically active Pamela Harriman, was with him, as were his daughters, Kathleen Mortimer and Shirley Fisk. His daughters came in from Alaska, where they were on a trip to the same area Harriman had toured in 1899 on an expedition that included famed naturalist John Muir . . .

Broadway star Patti LuPone has been making the rounds of the city's nightspots while she's here playing Lady Bird Johnson in the made-for-television movie "LBJ." Over the weekend she sang at the Fairfax Bar, danced at Deja Vu and on Sunday night had dinner at the Dancing Crab, where she surprised all the people crushing crab claws by standing up to sing "As Long as He Needs Me," from "Oliver!" . . .

National Gallery of Art Director J. Carter Brown has lost a bet to San Francisco Mayor Dianne Feinstein. She bet more visitors would view "The New Painting: Impressionism: 1874-1886" at the M.H. de Young Memorial Museum in San Francisco than saw it at the National Gallery. In a letter to Brown, Feinstein pointed out that more than 550,000 saw the exhibition in San Francisco. About 407,000 viewed it here during the 80 days it was at the gallery. The show ran 93 days in San Francisco, but Brown says he will treat Feinstein to dinner at the Washington restaurant of her choice . . .

Even baseball teams don't like employes posing nude for Playboy magazine. Marla Collins, a ballgirl for the Chicago Cubs, is featured partially nude in an eight-page pictorial in the September issue of the magazine. She no longer will be on the field providing baseballs to the plate umpire, the team said. A terse Cubs statement said: "The Chicago Cubs' organization did not authorize nor does it condone or approve of Marla Collins appearing in Playboy magazine." There was no comment from team members or the fans. They may not be as upset . . .