The Supreme Court ruled 9 to 10 yesterday that Congress violated constitutional protections against sex discrimination when it made Social Security benefits available to needy children of unemployed fathers but denied the same benefits to children whose mothers are jobless.

The decision affects 26 states, including Maryland, and the District of Columbia and Guam. They participate in a 1968 program called Aid to Families with Dependent Children, Unemployed Father. Currently, AFDC-UF helps about 300,000 children.

If no jurisdiction pulls out of the program, the federal government's costs would be increased substantially by yesterday's ruling. An initial estimate of $510.7 million in fiscal 1980 and a larger sum each year thereafter, may be high, Social Security Administrator Stanford G. Ross said.

While unanimous in holding that the program denies the equal protection of the laws guaranteed by the Constitution, the majority dwindled to 5-to-4 for affirmance of a lower court's order that benefits be paid to families in which either the mother or father is unemployed, within the meaning of the law, even if the other parent holds a job.

By contrast, the dissenters and Department of Health, Education and Welfare both had urged a less costly gender-neutral application of AFDC-UF in which benefits would be paid if the parent in question shows that he or she not only is unemployed but also the "principal wage-earner" - the family breadwinner.

Commissioner Ross said HEW has been asking Congress to deal with the sex-discrimination problem by legislating a family-breadwinner solution. Such a solution would be sounder than the more sweeping one in yesterday's decision, he said in a phone interview.

In addition, the costs would be "significantly lower," Ross said, though he immediately could specify neither how much nor the number of additional children to be helped.

Justice Harry A. Blackmun wrote the opinion for the court. The Dissenters only on the question of remedy were Chief Justice Warren Burger and three associate justices who signed his opinion: Lewis F. Powell Jr., Potter Stewart and William H. Rehnquist.

The court acted in two similar cases from Massachussets. In one, William and Cindy Westcott, parents of an infant son, were held ineligible for AFDC-UF because the father, unable to find work, hadn't worked sufficiently previously to be legally classified as "unemployed." Cindy has been the breadwinner. She was laid off and would have been legally unemployed, as defined by the law, had she been male. The second family, John and Susan Westwood, was denied benefits for similar reasons.

The two families filed what became a class action, and they prevailed before a special panel of judges in U.S. District Court. In affirming the court, Blackmun wrote that AFDC-UF, rather than paying "fringe benefits," provides "subsistence payments made available as a last resort to families that would otherwise lack basic necessities." Thus, he said, the program is important enough to invoke equal-protection guarantees.

Blackmun rejected arguments that the sex distinction is related to achievement of any important governmental objective. "Congress, with an image of the 'traditionally family' in mind, simply assumed that the father would be the family breadwinner, and that the mother's employment role, if any, would be secondary," he said.

Quoting from Supreme Court decisions in other cases, he said the gender classification is "part of the 'baggage of sexual stereotypes' that presumes the father has the 'primary responsibility to provide a home and its essentials,' while the mother is the 'center of the home and family life.'"

The court took other actions.

RELIGIOUS-SCHOOL BUSING

Voting 6 to 3, the justices dismissed a challenged to a 1972 Pennsylvania law requiring every school district to provide in a 10-mile belt beyond its boundaries the same free public busing to students attending nonprofit private or religious schools that it provides within the boundaries for public school students.

The 10-mile belt doesn't violate the constitutional ban on an establishment of religion, Judge Robert N. C. Nix wrote for the state Supreme Court majority that upheld the law. The belt isn't sufficient to amount to an impermissible aid to religion, he said.

By contrast, the Pittsburgh and Pequea Valley school boards, which had refused to transport any students beyond their boundaries, argued that a state order to bus parochial students into the 10-mile zone amounted to "favors" in violation of the First Amendment.

Four votes were needed to set the case for argument. The justices voting to hear it were Blackmun, Powell and John Paul Stevens.

JUVENILE MIRANDA RIGHTS

The justices directed a California appeals court to reconsider a ruling that sheriff's deputies had deprived Patrick "W," 13, of his Miranda rights to remain silent in the absence of an attorney by omitting to tell him that his maternal grandparents were standing by, ready to help him.

The boy had sought to talk neither to a lawyer nor to the mother he had just widowed by murdering his stepfather, Los Angeles police officer Edward W. Bullis.

The justices directed the reconsideration in light of last week's ruling that a 16-year-old hadn't invoked Miranda protections by asking police interrogators to call for his probation officer rather than a lawyer. In the opinion for a divided court, Blackmun emphasized a lawyer's "unique" ability to protect a client from self-incrimination.

In the case of Patrick "W," the appeals court found "no reason for the . . . deputies not seeking the presence of the grandparents" before obtaining a waiver of his Miranda rights and a detailed confession of the murder.

In the four years before Bullis was shot to death, he often had quarreled with, and sometimes physically mistreated, his stepson, the appeals court said. He had choked Patrick into unconciousness shortly before being slain in 1977.