While President Reagan vacations here out of public view, the administration fired off three sharply worded challenges to the Soviet Union this week that signal a hard-nosed approach to the November summit with Soviet leader Mikhail Gorbachev.

Each of these messages suggests that senior White House officials are preparing for an autumn of confrontation with the Soviets to climax at the Geneva summit, and that they are increasingly pessimistic about the prospects for an agreement to slow the arms race.

One reason for this pessimism is that the administration is still rent by internal disputes over arms-control policy toward the Soviet Union, disputes that have characterized Reagan's presidency from its early days.

A sharp split exists between top Pentagon officials and others in the State Department and White House over a possible trade-off that would limit Reagan's Strategic Defense Initiative in exchange for deep cuts in the Soviet offensive missile force. If the internal disputes are not reconciled, officials said, it may be impossible to arrive at any agreement with Gorbachev on reducing nuclear arsenals.

But, officials said, an effort would be made to forge a consensus in the next few months in a summit planning group to be cochaired by national security affairs adviser Robert C. McFarlane and White House chief of staff Donald T. Regan.

White House officials said this week's signals to Moscow were prompted by different demands and events within the administration. But each was accompanied by a rhetorical challenge to the Soviets, including today's disclosure that the Soviets had used possibly harmful chemical substances against U.S. diplomats in Moscow. The most deliberate and detailed message to the Soviets was delivered Monday by McFarlane. He gave an address to a Santa Barbara club entitled "U.S.-Soviet Relations in the Late 20th Century."

McFarlane's speech was written as if he were speaking directly to Soviet leaders, and White House officials said the remarks were purposely aimed at the Kremlin leadership, not at a U.S. audience.

McFarlane declared that "we're having a lot of trouble establishing a real dialogue" with Moscow. He called for change in the Soviet approach to military and international security issues and "in the thinking that underlies it." Without such change, he said it would be "extremely hard" to make even slight improvements in relations.

McFarlane also said Soviet support for Cuba and Libya and the continuing invasion of Afghanistan "sends us loud messages that can't be ignored about the motivations of Soviet policy. It makes improvements in other areas more difficult. It all but guarantees that any small steps forward that we may be able to take will be isolated, hard to preserve and perhaps devalued in advance by both sides."

McFarlane also pointedly criticized Soviet actions in recent years on chemical weapons production and intermediate-range missiles in Europe, apparently in response to Soviet diplomatic overtures. In both cases, he said, superpower competition was sparked by Soviet decisions to revive weapons-building.

After noting the United States responded by building weapons of its own, McFarlane asked rhetorically, "What can the Soviet Union imagine that it got out of reigniting this competition?"

McFarlane suggested that "the area in which the most momentous changes could take place" in the Geneva summit would be human rights, an area where he acknowledged the Soviets have always turned aside U.S. complaints by saying it is an "internal matter."

In addition to McFarlane's address, the White House also challenged Moscow this week in announcing plans for the first test of an antisatellite weapon against a target in space, which the Soviets have tried to stop. Presidential spokesman Larry Speakes said the test was necessary to counter a "clear threat" from Soviet military space programs.

Today, as well, the White House responded to the Soviet use of chemical substances against Americans in Moscow by accusing the Kremlin of letting its military and security services "act as if they are under no control."

A White House official said the timing of this development was "coincidental" with the others. But officials said it was a purposeful decision to again signal the Soviets of U.S. displeasure.

Both superpowers appear to be gearing up for a competition for world opinion before the summit; Speakes complained last week that the Soviets have become regulars on American television networks.