Prescott S. Bush, in a stunning reversal, withdrew from the U.S. Senate race in Connecticut today after conceding that he probably would have lost to Toby Moffett in November even if he beat incumbent Republican Sen. Lowell P. Weicker Jr. in the primary.

Bush, the brother of Vice President Bush, said he made the decision Tuesday morning, and that there was no nudge from the White House to persuade him to go either way.

"There was no pressure from anybody in Washington, from the Senate side or the White House side, to get out of the race," Bush told a crowd of reporters and stunned observers at a state capital press conference in Hartford.

"We evaluated it on practical grounds and came to the conclusion, finally, that it wouldn't work, so I called my brother and told him what I was going to do," Bush said.

A statement issued by the office of the vice president later Tuesday said that the vice president had not influenced his brother's decision either to enter or to leave the race.

Prescott Bush had made dozens of assertions in past weeks that a primary was absolutely certain, claiming as late as Monday evening that an NBC-TV report that he would withdraw was completely false. Therefore, his announcement came as something of a shock.

A Weicker spokesman, Martin Moore, attended the Bush press conference and, after it was over, apologized for having to face the television cameras in shirtsleeves. He explained that the Weicker camp had no idea that Bush would withdraw and that he would be called upon to make a statement.

"We were ready for Round Two and now it's not there," Moore said.

Weicker won the Republican endorsement at the statewide party convention last Saturday with 65 percent of the delegate vote but would have had to run in the primary because Bush got more than 20 percent of the delegates.

Until Tuesday Bush had insisted he would run in the primary and win the nomination by aligning himself with the Reagan administration's programs.

He and the conservative wing of the Connecticut Republican Party believe that Weicker has betrayed Reagan and traditional Republican positions too many times.

Bush, a 59-year-old business executive from Greenwich, had been seen as the candidate of the party establishment until a week before the convention when the GOP state chairman, Ralph E. Capecelatro, made a surprising endorsement of Weicker.

Bush said Tuesday he still thought he could have beaten Weicker but decided not to run because of the polls, which showed him losing badly to Moffett.

"It looked as though we might win the primary, but winning the primary and then losing the Republican Senate seat would be sort of a hollow situation," said Bush.

Moffett's estimate of Weicker's relationship to Reagan and his conservative supporters was considerably at variance with Bush's.

"This entire episode makes clear just how important this Senate seat is to the White House and to right-wingers in control of the Republican Party," Moffett said after hearing the news. "Mr. Bush's withdrawal from the race is a 'thank you' to the likes of such reactionaries as Sen. Strom Thurmond R-S.C. , Sen. Orrin Hatch R-Utah and the rest of the group whose initiatives Mr. Weicker has supported in the Senate."