Mathematician Felix Browder. (Nick Romanenko/Rutgers University)

Felix Browder, a mathematics prodigy who graduated from MIT at 18 and received his PhD at 20 but who struggled to gain a foothold in the academic world of the 1950s because his father was a longtime leader of the U.S. Communist Party, died Dec. 10 at his home in Princeton, N.J. He was 89.

The cause was a stroke, said a son, Bill Browder.

Dr. Browder was renowned in the field of nonlinear functional analysis — a branch of mathematics with wide applications to such fields as physics, engineering and finance. He later chaired the mathematics department at the University of Chicago and in 2000 was awarded the National Medal of Science by President Bill Clinton.

But during the heart of the communist-hunting McCarthy era of the early 1950s, Dr. Browder couldn’t find a steady job. He had a doctorate from Princeton University and major research papers to his name and was a scholar at the Institute for Advanced Study in Princeton at the same time as Albert Einstein.

The institute’s director, J. Robert Oppenheimer — who led the U.S. atomic bomb research program at Los Alamos, N.M., during World War II — was wary of being tarred as a communist sympathizer and, according to Dr. Browder’s family, declined to sign a draft deferment for the promising young mathematician. As a result, Dr. Browder was conscripted into the Army at 25.

Felix Browder receives the National Medal of Science from President Bill Clinton in 2000. (Reuters)

He was recommended for the intelligence service until the military brass found out about his father, Earl Browder, who twice ran for president as the Communist Party candidate. The elder Browder had been the subject of hearings of the U.S. House Un-American Activities Committee since the 1930s. He served several stints in prison.

During a 1953 hearing by the committee, the question arose whether Felix Browder, who was born in Moscow, was also affiliated with the Communist Party. One of his professors at the Massachusetts Institute of Technology testified that Dr. Browder had never joined the party and, moreover, was “the best student we had ever had in mathematics in MIT in the 90 years of existence of the institution.”

Tainted by the association with his father, Dr. Browder spent most of his two-year hitch in the Army pumping gas at Fort Bragg, N.C. It took the personal intercession of Eleanor Roosevelt for him to gain a faculty appointment in 1955 to Brandeis University in Waltham, Mass., where the former first lady was a trustee.

According to his son Bill Browder’s 2015 book “Red Notice,” Roosevelt said it would be the most “un-American thing we could do to deny a great scientist his profession because of who his father was.”

After Brandeis, Dr. Browder taught for seven years at Yale University before moving on to the University of Chicago, where he spent more than 20 years on the mathematics faculty, including 11 as department chairman.

In 1986, he joined Rutgers University in New Jersey in the new position of vice president of research, and was instrumental in establishing a science and technology center in conjunction with Princeton and Bell Labs. As president of the American Mathematical Society in 1999 and 2000, he lobbied Congress for additional funding for math education.

“I am devoted to carrying forth the cause of mathematics,” Dr. Browder said in 2000. “Computers are fundamentally mathematical, as is biotechnology. The problems of physics are increasingly mathematical in nature, and finance, in its global complexity, is mathematical as well.”

In 1986, Felix Browder became the Rutgers University vice president of research. (Nick Romanenko/Rutgers University)

As a scholar, Dr. Browder “achieved remarkable depth and generality” in his research, Jerry Bona, a mathematician at the University of Illinois at Chicago, said in an email. His specialty of nonlinear functional analysis, Bona added, “has become woven into the modern fabric of mathematical analysis and is used in their research by mathematicians, physicists, engineers and even economists all over the world.”

Dr. Browder’s father, who was born in Kansas, became a socialist in his teens, later joined the Communist Party and went to the Soviet Union as a trade representative in the 1920s. After divorcing his first wife, he married Raissa Berkmann, a Russian Jewish lawyer, in 1926.

They settled in Moscow, where Felix Earl Browder was born July 31, 1927. His first language was Russian.

In 1932, the Browders moved to Yonkers, N.Y., where Felix grew up. During the Great Depression, the Communist Party reached the height of its influence in the United States, with Earl Browder as its leader. He was the party’s candidate for president in 1936 and 1940.

He was sent to prison in 1941 for passport fraud, but his four-year sentence was commuted by President Franklin D. Roosevelt after 14 months. In 1946, Earl Browder was expelled from the Communist Party during an internal dispute in which he was rebuked for an insufficient Marxist zeal. U.S. officials repeatedly tried to deport his ­Russian-born wife before her death in 1955.

“These were traumatic events for us,” Dr. Browder told the Newark Star-Ledger in 1998. “This went far beyond politics.” (Earl Browder died in 1973.)

From an early age, Dr. Browder and his two younger brothers were immersed in a fervid intellectual and political atmosphere. He entered MIT at 16 and graduated in two years. He went to Princeton for graduate study in mathematics, receiving a master’s degree in 1947 and doctorate one year later.

At Rutgers, where Dr. Browder was still an active faculty member at the time of his death, he helped recruit many leading mathematicians and scientists. His brothers also became eminent mathematicians — William Browder at Princeton and Andrew Browder at Brown University.

Beginning at age 5, Dr. Browder read an average of a book a day. His personal library contained more than 35,000 volumes, each of which he had read at least once.

“Look,” his brother William Browder told the Star-Ledger in 1998. “Felix really does know everything. He can talk at length about anything — French literature, Buddhist philosophy, the best price for dog food — because he reads and retains so much. It gives him a confidence that few possess.”

Dr. Browder’s wife of 66 years, Eva Tislowitz, an Austrian-born Holocaust refu­gee, died in 2015. In addition to his brothers, survivors include two sons, Thomas Browder of Honolulu and Bill Browder of London; and five grandchildren.

Thomas Browder is a physics professor at the University of Hawaii. Bill Browder, a onetime hedge-fund manager, moved in the 1990s to Moscow, where his grandfather had been a part of the idealistic spirit surrounding the early promise of the ­Communist-led Russian Revolution.

Bill Browder amassed a fortune investing in emerging Russian businesses, but after raising questions about official corruption in the regime of President Vladi­mir Putin, he was denied reentry to the country in 2005 and declared a “threat to national security.” His companies’ ­offices were raided and their assets seized.

In 2009, his Russian attorney, Sergei Magnitsky, who had challenged Russian authorities with charges of theft, was beaten to death in a Moscow jail, where he had been held without trial for almost a year.

Bill Browder led efforts to pass the Sergei Magnitsky Rule of Law Accountability Act of 2012, which has led to sanctions against Russian officials. Last year, his book “Red Notice,” which describes his experiences in Russia — and his family’s tangled history with the country over three generations — became an international bestseller.