Samuel I. (S. I.) Newhouse, 84, who rose from the depths of childhood poverty to establish a multimillion-dollar publishing and broadcasting empire, died yesterday at Doctor's Hospital in New York following a stroke.

Over the years he had spent more than a half billion dollars to acquire his properties, which at one time included 31 newspapers, seven magazines, six television stations, five radio stations and 20 cable television stations.

Mr. Newhouse, who left school at the age of 12 to start work as an office boy, was foremost a businessman who went on to become what Business Week magazine once described as "America's most profitable publisher."

As he rapidly acquired newspapers all around the country, many of them financially ailing, he brought them back to monetary health.

At the same time, he permitted the newspapers to keep their editorial autonomy. He left the editorial decisions to the individual editors of the newspapers involved.

"Whatever political interest each newspaper publisher or editor has, is based on what he thinks is best for the community in which he operates and he expresses himself freely without submitting anything to me," he once said.

In one of his early purchases, Mr. Newhouse acquired the Staten Island (N.Y.) Advance. It became a privately-owned family operation, as did later acquisitions.

In early 1977, shortly after he had purchased the Booth papers in Michigan along with Parade magazine for a reported $305 million, Mr. Newhouse transferred the day-to-day operations of all of the family holdings to his brothers, Norman and Theodore, and to his two sons, Samuel I. Jr., known as Sy, and Donald E. All had been executives of the operations for many years.

In between, the powerful Newhouse chain (he disliked the word chain and preferred that the enterprise be called the Newhouse "family") had picked up such well-known papers as the Portland Oregonian, the St. Louis Globe-Democrat, the Cleveland Plain Dealer and the New Orleans Times-Picayune.

There were other newspapers in New Jersey, New York, Pennsylvania, Alabama, Massachusetts, and Mississippi.

In 1959, Mr. Newhouse bought the Conde Naste and Street and Smith magazines for $9.4 million. The U.S. editions included Vogue, Glamour, House and Garden and Mademoiselle. There were overseas editions in Great Britain, France, Italy and Australia.

"I wanted to give Mitzi 'Vogue' for a 35th wedding anniversary gift," he joked at the time, referring to his wife.

Mr. Newhouse was not given to joking or speaking out in public. A dimunitive 5 feet 2, he was modest and shy and loath to give interviews.

His biography in "Who's Who in America" is one of the shortest among those of business tycoons. It contains only a handful of the newspapers he acquired. He reportedly never replied to requests for a detailed account of his accomplishments.

Mr. Newhouse was born in a tenement on Manhattan's Lower East Side, the son of immigrants, a Russian father and an Austrian mother.

The oldest of a family of eight, he grew up in Bayonne, N.J. After leaving grammar school, when his father became ill, he took a brief business course and went to work for a Bayonne lawyer and police magistrate, who several years later acquired the faltering Bayonne Times.

Although he was only 16, Mr. Newhouse was asked to take over the paper and close it. Instead, he built it into a profitable paper and was given a share in the profits.

He also began studying law. He passed the bar but never practiced. He found a career in publishing more to his liking. Brothers and sisters began to assist in his enterprises.

Over the years, only two of the Newhouse properties failed. The Long Island Daily Press, which he purchased in 1932 in his early years of expansion, closed in 1977. The Long Island Star Journal had folded in 1968.

There were radio stations in Syracuse, N.Y., Birmingham, Ala. and Harrisburg, Pa. There also were six television stations in the Newhouse broadcasting group. Five of them were sold in 1978 to the Times Mirror Co. of Los Angeles. A sixth, in Portland, Ore., had been sold earlier.

Mr. Newhouse was as quiet about his vast philanthropic efforts as he was about his private and business life. Huge sums were given through the family foundation. His interests included the S. I. Newhouse School of Public Communications at Syracuse University and the Mitzi Newhouse Theater at the Lincoln Center in New York.

In addition to his wife, two sons, and two brothers, Mr. Newhouse is survived by two sisters, Gertrude Diamond and Estelle Miron, and six grandchildren.