A federal court jury deliberated for 5 1/2 hours yesterday on the merits of the $50 million libel case brought by Mobil Oil Corp. president William P. Tavoulareas and his son Peter against The Washington Post, but went home without reaching a verdict.

The six-member jury began its deliberations after listening to 16 days of testimony in the complex case that centers on articles the newspaper published on Nov. 30 and Dec. 1, 1979, about the Tavoulareases' business dealings. In addition to recalling the testimony of 39 witnesses, the jury was faced with sifting through a small mountain of documents, about 160 exhibits that lawyers introduced as evidence in the case.

Shortly after beginning its deliberations, the jury asked for a written set of U.S. District Judge Oliver Gasch's legal instructions in the case. The judge denied the request, but a short time later reread two instructions the jury requested.

One related to the definition of slander and the other to Sandy Golden, a special correspondent for the Nov. 30 story who led Post reporter Patrick Tyler to an initial source. The slander question pertained to the $20 million slander suit brought by the Tavoulareases against Dr. Philip Piro, a Baltimore eye surgeon, a former Tavoulareas son-in-law and the source Golden introduced to Tyler. The jury is deciding the slander case against Piro as well as the libel case against the other defendants--The Post, its executive editor Benjamin C. Bradlee, assistant managing editor Bob Woodward, Tyler, and Golden.

Except for the jury's two requests there was no indication of how the jury was progressing in its deliberations. The jury is composed of three women and three men--a Library of Congress librarian, a retired Pan American Sanitary Bureau worker, an Agriculture Department secretary, a Commerce Department oceanographer, a D.C. public school system cook and a Western Electric Co. storekeeper.

Before handing the case to the jury, Gasch spent an hour and 15 minutes reading a lengthy list of legal instructions on the complexities of libel law and the evidence in the case.

The 76-year-old judge told the jurors "to place yourself in the place of the average newspaper reader" as they deliberated.

Gasch outlined the various conditions under which the jurors should rule for the newspaper or the Tavoulareases and also the conditions for deciding the slander suit.

"Truth is a complete defense, no matter how defamatory the articles," Gasch said, adding that if the jurors only found the articles "substantially true," they still should rule that The Post had no liability in the case.

He said the burden of proving the libel was the responsibility of the Tavoulareases and three conditions must exist in order for damages to be awarded: that the articles were defamatory, "false in some material way" and "published with fault."

Gasch set different terms of proof for each of the Tavoulareases as to whether the articles were "published with fault." The judge ruled that the Mobil president is a public figure and therefore the jury, if it believes the stories untrue and defamatory, was faced with determining whether the newspaper "knew the articles were false" or demonstrated a "reckless disregard" about the accuracy of the articles.

"It's not enough to show the defendants did not conduct a thorough investigation," Gasch said.

But the judge said that the younger Tavoulareas "has not attained the rank or grade of a public figure" and should be considered a private figure. As a result, Gasch told the jurors that if they believe the stories untrue and defamatory, they would only have to find that the newspaper was negligent in its preparation of the articles in order to award damages.