President Carter joined families of American hostages held in Tehran at a subdued but solemn prayer service yesterday at Washington Cathedral.

Attending with the president were Vice President Walter F. Mondale, Secretary of Satate Cyrus R. Vance and 2,000 State Department employes plus members of the diplomatic corps.

Some knelt, others like the president sat head bowed as leaders from the Catholic, Protestant, Jewish and Islamic faiths read passages on mercy and justice from the Bible and the Koran.

"We as a religious people have nothing other to do than pray that Almighty God help our brothers remember the teachings of Islam and release the hostages," prayed Abdel Rahman Osman, the Egyptian deputy director of Washington Islamic Center.

"There was a real fellowship here. . . a sense of brotherhood. . ." said one of the worshippers after the service. He identified himself as a foreign service officer recently returned from South Africa, but declined to give his name.

Many others at the service declined to be quoted or identified, saying they had been warned that any statement could endanger the hostages.

The service was hastily organized Wednesday and Thursday by Louisa Kennedy, wife of one of the hostages, with the help of State Department officials. "I knew it would make everybody feel better. It certainly did," Kennedy said yesterday after the service.

Notices of the service were posted yesterday morning around the State Department. Others at the service said they had learned of it over the radio.

Shortly before the 3 p.m. service, 22 buses carrying State Department employes and relatives of the hostages began lining up in front of the cathedral. Diplomats arrived in chauffeur-driven limousines. Members of the general public, who said they had no relationship to the hostages, but just wanted to pray for them, also straggled in.

The president, looking tired and drawn, was the last to arrive, flanked by Mondale and Vance. He stared straight ahead and walked swiftly to his front-row seat alongside Penny Laingen, wife of the U.S. charge d'affaires in Tehran, and her son.

"Faith of our fathers living still," they sang, "In spite of dungeon, fire and sword. . ."

Osman, the first religious leader to speak, spoke slowly and deliberately calling Judaism, Christianity and Islam "divine sisters" of the human faith.

"Let not your hatred of others lead you to depart from justice," he exhorted the Iranian captors half a world a way.

"Let us from our diverse cultural backgrounds, religious traditions and walks of life draw near to the Supreme Creator. . . in whom there is no East or West," said the right Rev. John T. Walker, Episcopal Bishop of Washington.

The Most Rev. William Baum, Roman Catholic Archbishop of Washington, prayed God to "lead them and us from prejudice to truth: deliver them and us from hatred and cruelty. . ."

Rabbi Tzvi H. Porath read specially adapted passages from the Jewish "liturgy on hostages," which he said is read every Monday and Thursday in synagogues all over the world "because many times our people have been held captive.

"Our hearts are with the families of the captives," prayed Porath, representing the Jewish Community Council here. "We share their anxiety nd their concern."

"This was so organized and so spontaneous," said Helen Dudman, friend of one of the hostages. "It made me cry. When you see the chaos over there are then see this -- so touching. It shows you the difference between the two cultures."

Dudman said she and several other friends of the Laingens came to the service to offer "moral support' to the family.

Laingen had just wiped her eyes with a white handkerchief, but managed a broad smile when foreign service officers began to greet her.

"We're very hopeful," she said "We're going to get them out it's just going to take time."

She spoke of a phone conversation she had with her husband, L. Bruce Laingen, yesterday in which he said he was feeling "strong." Technically, Laingen is not a hostage in the embassy, but is being held "in protective custody" by the Iranians in Tehran's Foreign Ministry, trying to negotiate the hostages' release.

"He knows about the service today and he told me to sing loud and clear," she said.

Anita Schaefer, whose defense attache husband, Thomas, is a hostage, was visibly moved by the service, and said she was touched by the presence of the president, Vance and Mondale.

The president, she said, is "supporting us in every way possible.He exudes warmth. I understand he was deeply moved when he was told about the service and wanted to be with us. . . ask everyone to pray for him."