The Senate Foreign Relations Committee voted yesterday to send a 36-year-old treaty outlawing genocide to the Senate for ratification after approving provisions that would limit the jurisdiction of the World Court in cases involving the United States.

The panel's vote was 10 to 0, with nine Republicans joining Edward Zorinsky (D-Neb.). Eight Democrats who favor ratification of the treaty but oppose the limitations voted "present."

The limitations were proposed by Sen. Jesse Helms (R-N.C.) and endorsed by committee Chairman Richard G. Lugar (R-Ind.).

The Genocide Convention faces a tough fight in the Senate, which has postponed ratification since the treaty was drafted after World War II, even though more than 80 other nations have approved it. The Senate brought up the treaty last October after President Reagan urged ratification, but backed away from voting on it. It settled for a nonbinding resolution when conservative opponents threatened a filibuster.

Helms said after the committee vote yesterday that he would not oppose the treaty now that his proviso is included, but he indicated that others might.

The likely leader of the opposition is Sen. Steve Symms (R-Idaho), who has indicated that he might filibuster. Symms objects to the treaty's definition of genocide as "the intent to destroy, in whole or in part, a national, ethical, racial or religious group as such."

Symms argues that this does not include political genocide and therefore does not apply to Soviet aggression in Afghanistan or atrocities by other communist regimes, such as the Khmer Rouge in Cambodia. He said yesterday that the Helms-Lugar restrictions changed nothing in this respect.

The Helms-Lugar proviso states that "the specific consent of the United States is required in each case" to which the United States is a party. It originally read that "nothing in the convention requires or authorizes legislation or other action by the United States prohibited by the Constitution of the United States as determined by the United States." Those last six words have been a sticking point for years, however, and the committee softened them slightly by substituting "interpreted" for "determined."

The treaty, which was a reaction to the Nazi Holocaust, originally was opposed by conservatives who feared that the United States would be accused of genocide because of segregation, and then because of similar fears concerning U.S. policy in Vietnam. Now it is opposed primarily by conservatives who argue that the World Court is dominated by Third World nations that will use it as a forum for anti-American and anti-Israeli sentiments.

The World Court is authorized to hear accusations of genocide but is not a criminal tribunal and cannot try or punish violators of the treaty. It must rely on the U.N. Security Council to provide sanctions; the United States, as a Security Council member, can block action there with its veto.

Sen. Alan Cranston (D-Calif.) complained that the Helms-Lugar proviso "really strips the treaty of any significance." Sen. Christopher J. Dodd (D-Conn.) argued that "the United States is a nation of law" that should show less fear of being mistreated in court and more confidence "in the legal process and our own strength and sovereignty."

Lugar, however, contended that the treaty's only chance of passage is with the Helms-Lugar restrictions.

"This is not a symbolic act but an opportunity that may not be seen again," he said.