The Maryland Court of Appeals ruled yesterday that picketing a private home is a constitutionally protected action that a state legislature can regulate but cannot prohibit.

In an unanimous decision, the court said two young men who were convicted last summer on charges of illegal picketing had been within their rights when they held a peaceful three-day vigil outside the Bethesda home of then-Secretary of Defense Donald H. Rumsfeld.

Saying that "we believe that the statutory ban on all residential picketing is invalid on its face as violative of the right to freedom of speech," the judges struck down key portions of a 1971 law barring everyone except labor groups from picketing near residences.

In the opinion, written by Judge John C. Eldridge, the seven-judges agreed that, "while picketing and parading and the use of streets for such purpose is subject to reasonable time, manner and place regulations, such activity may not be wholly denied."

Gary Simpson, the American Civil Liberities Union attorney who pressed the case said yesterday that the decision "reaffirms the principle that the country is founded on - that people may freely express their views in what are normally considered public forums."

Simpson pointed out, however, that judicial decisions in other jurisdictions have been sharply divided on this issue, with the highest court of Wisconsin and the federal 10th Circuit Court of Appeals in the southern United States upholding anti-picketing statutes almost identical to Maryland's.

After the two men, Phillip Schuller and Sean Ozzie Simpkins, were convicted in Montgomery County District Court of violating the Maryland anti-picketing law. Simpson appealed the case to the County Circuit Court. There, Judge H. Ralph Miller overturned the convictions.

The Court of Appeals ruling yesterday upheld Judge Miller's action.

Schuller, Simpkins, and six friends went to Rumsfeld's house in early April, 1976, to hold a vital protesting nuclear proliferation. Both prosecutors and police agreed that the three-day vigil had been peaceful, with the demonstrators genially cooperating with the police who came to warn them and finally to arrest them.

After their conviction in June, Schuller and Simpkins, both college students, received suspended sentences from District Court Judge Stanley Klavin.The judge had asked attorney Simpson during the trial, "When is Mr. Rumsfeld supposed to get his rest?"

Two other students arrested with Schuller and Simpkins were convicted in the case did not join the appeal.

Simpkins, 23, a student at Goshen College in Indiana, and Schuller, 23, a senior at Slippery Rock State College in Pennsylvania, could not be reached yesterday for comment on the Court's decision.

In addition to ruling that the Maryland anti-picketing law violated First Amendment freedoms, the Court of Appeals also noted that, in exempting labor picketers from its prohibition, the law also violated the equal protection amendment of the Constitution.