Thomas J. Watson Jr., retired board chairman and son of the founder of International Business Machines Corp., has been chosen by President Carter as ambassador to the Soviet Union, informed sources in the U.S. delegation said today.

The choice of Watson, 65, a major figure in American business for three decades, to succeed career diplomat Malcolm E. Toon later this summer had been rumored for several weeks.

Watson, whose late brother Arthur K. Watson served a controversial term as ambassador to France during the first Nixon administration, has no diplomatic experience.

If the Soviets accept him and the U.S. Senate confirms the nomination, Watson will be the first noncareer diplomat in the post since Gov. Averell Harriman was ambassador to Moscow during World War II.

Watson, a friend of Secretary of State Cyrus Vance, was strongly supported by Harriman on grounds that a prestigious outsider, as he himself was, can have more success with the Soviets than a career diplomat.

Toon, 61 a Soviet specialist who had served as ambassador to Yugoslavia, Czechoslovakia and Israel, went to Moscow at the end of 1976 determined to gain access to top Soviet leaders. But he failed to achieve his goal of matching in Moscow the kind of wide access to American leaders long enjoyed in Washington by Soviet Ambassador Anatoliy Dobrynin.

Ambassador Toon took a tough stance on U.S. Soviet relations and has strongly supported appointment of a professional diplomat to succeed him after his retirement this summer. He has said that an outsider unversed in Soviet methods cannot be a highly effective envoy.

Isolation of Toon deepened last summer after he unsuccessfully urged Vance to cancel a SALT meeting last July in Geneva with Soviet Foreign Minister Andrei Gromyko as retaliation for the political trials of Soviet dissidents Anatoly Scharansky and Alexander Ginzburg.

The Geneva session, which Toon participated in, later was described as one of the administration's most successful. He did not participate in subsequent SALT meetings between Gromyko and Vance.

Watson served as IBM president from 1952 to 1971 and chairman of the board from 1961 to 1971. He resigned both posts that year after a heart attack but continued as chairman of the giant computer company's executive committee several years more before retiring to devote time to charitable purusits. During his tenure as president, the company became the world's dominant producer of computers.

A tall, white-haired man, Watson has been here ostensibly in the role of interested onlooker at the summit as chairman of the General Advisory Committee on Arms Control.

Watson declined to be interviewed pending official announcement by the White House of his selection. He has been a largely inactive member of the Council on Foreign Relations, and has served on the board with Vance of Pan American Airlines and IBM.

Watson's brother, Arthur, was chosen by Nixon as ambassador to France in 1969 after he had contributed $300,000 to Nixon's reelection campaign in 1972. That same year he became a center of controversy when columnist Jack Anderson reported that Watson was "gloriously drunk" on a Paris-Washington flight. He later conveyed his regrets to inquiring senators and soon resigned.

Thomas Watson was not the first prospect for succeeding Toon. The Carter administration first sounded out Lawrence Eagleburger, a former associate of secretary of state Henry Kissinger and SALT expert who is now ambassador to Yugoslavia.

Others mentioned as possible choices included Richard Davies, former ambassador to Poland, and Harry Barnes, an East Europe specialist who is now director general of the Foreign Service.

Watson's selection is part reflects a diminished pool of senior Sovietologists in the Foreign Service. Toon is now finishing his third tour in Moscow since entering the Foreign Service in 1946.

Watson is married and has six grown children. CAPTION: Picture, THOMAS J. WATSON JR. . . . no diplomatic experience