The Supreme Court ruled unanimously yesterday that an Amish farmer must pay Social Security taxes for his employes even though it violates his religious beliefs.

Chief Justice Warren E. Burger, writing for the court, said that the federal government's need to maintain the "fiscal vitality" of the Social Security system through mandatory participation outweighed the farmer's religious claim.

The decision was part of a continuing and troublesome case-by-case attempt to determine when government must accommodate religion and when religion must accommodate government. Last year the court said it was unconstitutional to deny unemployment benefits to a Jehovah's Witness who, for religious reasons, quit a job in a factory that made military tank parts. In 1972 the court allowed the Amish an exemption from compulsory education laws.

Self-employed persons may claim a religious exemption from Social Security taxes, but yesterday the court ruled that such a claim may not extend to an employer or employes. The court said Edwin D. Lee must obey the Social Security laws requiring employers to file quarterly tax returns, withhold Social Security tax from employes and pay the employer's share of Social Security taxes.

Lee, who employed several other Amish on his Pennsylvania farm and in a carpentry shop, refused to participate in Social Security because he said it would force him to violate the Amish religion's strict doctrines of self-reliance.

The high court reversed a U.S. District Court ruling in Lee's favor. Burger conceded in U.S. vs. Lee that "compulsory participation in the Social Security system interferes" with the Amish First Amendment rights of free religious exercise.

But "not all burdens on religion are unconstitutional," he said. Some may be justified as serving some "overriding governmental interest."

Mandatory participation in Social Security is such an overriding interest, the court said, noting that nothing compels the Amish to accept Social Security benefits.

Justice John Paul Stevens wrote a separate statement concurring in the judgment.

In another action yesterday, the court went to unusual lengths to avoid ruling on whether teen-agers may be barred from coin-operated video game arcades. The decision, in a case from Mesquite, Tex., had been awaited by hundreds of local governments across the country that are experimenting with ordinances designed to prevent teen-agers from idling away their time at arcades.

The 5th U.S. Circuit Court of Appeals, creating a new "right of social association," had struck down a Mesquite ordinance barring persons under 17 from arcades.

Yesterday, the justices returned the question to the lower court for clarification. Stevens, joined by six other justices, said the appellate court had not specified how much of its ruling was based on the Texas constitution and how much on the 14th Amendment to the U.S. Constitution. Both were mentioned in the decision.

Stevens said the Supreme Court's power to review questions of state law is limited.

Justices Lewis F. Powell Jr. and Byron R. White, writing separately, called Stevens' approach too rigid and a misinterpretation of the 5th Circuit holding and the jurisdictional limitations on the Supreme Court.

The court did rule on a lesser issue in City of Mesquite vs. Aladdin's Castle Inc. A provision in the Mesquite ordinance required an inquiry into "connections with criminal elements" of any applicant for a video arcade license. The lower courts had held the provision unconstitutionally vague. Yesterday, however, the court said that the provision was merely an instruction to the local police chief, not a condition for the granting of a license. Therefore, it said, its vagueness was tolerable.