A group of Indians who contend that their ancestral lands were illegally taken from them more than a century ago came into federal court today seeking to recover what they say is rightfully theirs.

Their suit, believed to be the first such claim to reach the courts, is the focus of national attention because similar claims amounting to billions of dollars worth of land are pending around the country. Other claims have already been filed in Maine, New York, South Carolina, North and South Dakota, Nebraska, Montana and Wyoming.

In the trial that opened today, the Wampanoag Indians took the first of a series of legal steps they hope will lead to the recovery of 11,000 acres, or nearly the entire town of Mashpee, Mass., valued at $200 million.

In their suit, filed in August, 1976, the Indians alleged that the land, which the Plymouth colonists granted them in perpetuity in 1685, was taken from them illegally in 1870 when the town as incorporated into the Commonwealth of Massachusetts. The Wampanoags assert that this act violated the Non-Intercourse Act f 1790, which held that no land could be taken from Indians without the consent of Congress.

Since the suit was instituted, virtually no real estate transactions have taken place in Mashpee, a fast-developing resort area, because no financial institutions will grant mortgages on properties with clouded titles. Last August the Wampanoags offered to drop their claim to properties containnoag Indian Council, backed by the ing homes of non-Indians, but a judge ruled this could be done only by Congress.

The trial pits the Mqshpee Wampa-Native American Rights Fund, supported in part by the Ford Foundation, against New Seabury Corp., the largest developer in Mashpee, the state, utility, and several national title insurance companies.

Chief counsel for the plaintiffs is Thomas Tureen, who has also been active in a Maine Indians' claim. For the defense it is James St. Clair, who is best remembered in Washington as counsel for President Nixon during Watergate.

A jury of eight men and four women, most of them middleaged, was selected today. Opening arguments begin Tuesday before Judge Walter Jay Skinner.

At issue in this trial are three basic questions: Did the Wampanoags own the land in question in 1870? Were they a tribe then? Are they a tribe now? If the Indians are successful, the way will be paved for them to sue in the U.S. Court of Claims for the return of their land, or compensation for it: If St. Clair can convince a jury the Wampanoags are not legally a tribe, this may put an end to their claim.

Both sides intend to cite a 1901 Supreme Court decision on what constitutes a tribe. It defines a tribe as "a body of Indians or a similar race, united in a community under one leadership or government, and inhabiting a particular, though sometimes ill-defined, territory."

One hundred witnesses, including anthropologists and museum curators, historians and Indian leaders, are expected to be called during the trial, which may last six to eight weeks.

Efforts to settle the suit out of court have thus far been unsuccessful.

Last week some Massachusetts legislators, Sen. Edward M. Kennedy and Rep. Gerry E. Studds, both Democrats, and Sen. Edward W. Brooke, a Republican, introduced an emergency bill that would end all Indian Claims to occupied home sites of less than one acre.

However the bill also subsituted a federal liability of $100 million of $200 million. That means, should the Indians' claim be recognized, the federal government would be open to a potential liability in that amount.

The Carter administration is said to be unwilling to set such a precedent by assuming potential liability where none was admitted before. Last Friday Robert J. Lipshutz, counsel to the President, the Massachusetts legislators and others met to try to arrange an out-of-court settlement.

The Wampanoags would be willing to settle for $3 million to $5 million from the federal government in exchange for abandoning their claim to only the home sites. The administration wants an overall package solution, though.

Mashpee council leader Russell Peters said today the Wampanoags would welcome emergency legislation granting them $5 million before the end of this session. However, if none is passed - as is possible given Congress' busy final days - and if the Indians win their trial before the legislators reconvene, Peters warned that the terms for a settlement might escalate.