Mabel (Muffie) Brandon yesterday resigned her $52,000-a-year post as White House social secretary, effective June 1, to become president of the newly established Washington office of Rogers and Cowan Inc., a public relations firm. She will start work on July 1 at an undisclosed salary.

In announcing the resignation, the White House said no successor had been chosen.

Brandon's resignation ended months of speculation that she would be leaving. Citing a desire to return to private life and spend more time with her family, Brandon wrote a letter of resignation to First Lady Nancy Reagan dated yesterday. In it she said that "serving you and the president has been the highest honor of my life . . . My admiration and affection for you both will not cease once I cross this threshold."

Though frequently rumored last fall to be leaving her White House post, Brandon, 47, said yesterday those rumors had been incorrect. She said she was first approached by Rogers and Cowan shortly after the first of the year.

An international firm primarily involved with public relations for the entertainment industry, the California-based company also serves corporate clients. Its Washington connections reach well into the Reagan administration but it has never before maintained a Washington office. Former White House aide Morgan Mason, son of actor James Mason, joined its Los Angeles office when he left Washington.

Brandon was hired by the Reagans in February 1981 after an exhaustive search in which Mrs. Reagan brought in Letitia Baldrige, social secretary in the Kennedy White House, to assist with screening more than two dozen candidates.

Hailed at the time as a valuable link to establishment Washington, Brandon's connections spanned both political parties. She as much as anyone on the predominantly California staff helped the newcomer Reagans assimilate into the social scene.

In accepting Brandon's resignation, Mrs. Reagan said that "during our time together you have proved yourself to be an extremely talented, creative and energetic social secretary . . . I can certainly understand your interest in returning to private life and the priorities of your husband and family."

In that same vein, the first lady said that "the long hours in a job like this don't always allow the kind of dedication I know you would like to give your family."

Brandon headed her own firm, Washington Corporative Arts Inc., when she left it for the White House. She said yesterday that Rogers and Cowan was familiar with her work there.

"They are interested in the visual arts' receiving backing from the private sector, which is what I did as a trained museum curator," she said. "So it's a natural relationship."

She said that she announced her resignation two months before its effective date "because I care so much about a smooth transition. I'll stay on through the economic summit and be here to help train my successor and to make the transition easy."

She is married to Henry Brandon, chief diplomatic correspondent of The Times of London; they have four children. Her husband is resigning his position after 33 years in Washington to write a book and also serve as an international financial consultant.