A treaty is to come into force today among 11 nations of the Western Hemisphere establishing an Inter-American Court of Human rights and committing the participants to increase protection of human rights.

In a ceremony at the headquarters of the Organization of American States this morning, Grenada is to deposit its ratification, bringing the total of participating states to the necessary minimum almost nine years after the American Convention on Human Rights was opened to signature - and one year after the United States, long a holdout, signed the sweeping treaty.

President Carter told the OAS general assembly last month that he will seek prompt action by the Senate, which must approve the convention by a two-thirds majority.

Carter signed the treaty last year as a demonstration of his commitment to human rights in an areas where military governments have been widely accused of systematic abuses.

In theory, after the U.S. government ratifies the treaty, it could be hauled before the Inter-American Court as a result of a citizen's complaint. Secretary of State Cyrus Vance specified in submitting the pact to the Senate, however, that any U.S. acceptance of the jurisdiction of the court would have to be approved separately by the Senate.

A staff member of the Senate Foreign Relations Committee said that because of a backlog the committee is unlikely to take up the treaty until next year.

Of the 11 current signatories, only Costa Rica has accepted without reservation the court's authority in rights cases. Hence the court's impact is expected to be limited, at least initially.

Peru has announced that it will ratify the treaty shortly and three others among the OAS' 25 active members are close to joining.

The court is to be patterned after the 17-judge European Court of Human Rights.

With seven judges, to be elected at an OAS assembly, the Inter-American court normally will hear cases brought by the already existing Inter-American Commission on Human Rights. But the nations accepting the court's jurisdiction also can initiate cases.

The court will sit, at least 10 days per year, in the Costa Rican capital of San Jose, where the convention was negotiated at a special OAS assembly in 1969.

Although creation of the court is the more dramatic result of the convention taking effect, its main impact is expected to be strengthening of the existing rights commission.

Since the commission's founding in 1906 it has operated on the basis of OAS resolutions without the underpinning of a treaty.

The treaty 'specifies that enumerated rights are to be incorporated into the laws of the states themselves," said Vargas.

Until now, the seven-member human rights commission could make public the abuses that it was able to document only after a painstaking consultation process with an accused state. Under the statutes of the convention "the commission definitely will be strengthened," said its executive secretary, Chilean Edmundo Vargas Carreno.

If the European experience holds true, this new strength will come in the commission's ability to bargain quietly with an offending government to halt a practice rather than be subjected to a process that could end in the Inter-American Court.

A major weakness of the court, however, is that three major Latin American countries - Brazil, Mexico and Argentina - have shown no inclination to accept it. Two other countries instrumental in drawing up the convention nine years ago, Uruguay and Chile, had signed the pact and were on the point of ratifying it in 1973 when their democratic governments fell in military coups.

Countries that have ratified the pact are: Colombia, Costa Rica, Ecuador, El Salvador, Guatemala, Haiti, Honduras, the Dominican Republic, Venezuela and Grenada. Barbados, Jamaica, and Surinam, along with Peru, are expected to ratify soon.