Mark J. Meagher, former president of The Washington Post Co., announced today that he will leave the presidency of Penthouse International Ltd. to form his own investment firm.

Meagher, who left the Post last January, joined Penthouse as president and chief operating officer in April. He will leave Penthouse Dec. 31. Penthouse Chairman Robert Guccione will assume the title of president, which he gave up when Meagher was hired.

Meagher and Sash A. Spencer will form Meagher, Spencer Associates, an investment firm that will seek investment opportunities for institutions such as pension funds. Spencer is chairman of Holding Capital Management Corp.

In a telephone interview Meagher, 49, said that he has long thought of heading his own firm and spoke with Spencer, an old friend, "several months ago."

Meagher said the new company, which will be formed Jan. 1, will look for small- to medium-sized companies where the owners want to sell some of their stock, but remain involved in managing the company.

He said the new company would try to develop deals in which owners can "cash in their chips," and his clients can have an equity involvement with the companies in which they invest.

There were no indications that Meagher, who received a large and continuing financial settlement when he left The Washington Post Co., had any financial agreements with Penthouse. "He had all sorts of contractual guarantees if he were fired," but not if he resigned, one source said.

Guccione launched Penthouse in Great Britain in 1965 and introduced his magazine as a more risque alternative to Playboy, the United State's leading men's magazine. Penthouse also publishes other sexually oriented magazines--the digest-sized Forum and Variations--and Omni, a science and science fiction magazine.

Meagher and Spencer once were colleagues at the management consultant firm of McKinsey & Co.