Prodded by congressional critics, outside advisers, First Lady Nancy Reagan and political reality, White House chief of staff Donald T. Regan is changing his ways.

Faulted for an authoritarian, self-promoting approach and for political insensitivity, Regan in recent weeks has adopted more conciliatory tactics. Though his relations with some White House colleagues remain strained and criticisms of his "chairman of the board" attitudes can still be heard, Regan has weathered a crisis of confidence with Nancy Reagan, informed sources report, and appears to enjoy considerable job security.

In Regan's dealings with Congress and outside constitutencies, compromise increasingly has supplanted confrontation and consultation has replaced isolation. Longtime California political advisers have once more been welcomed in the White House.

"If you want a hackneyed analogy, I'm in the middle of the forest and all I see are trees," Regan said about the outside advisers in an interview. "Occasionally, they can point out that there is a forest, and that helps keep things on track."

Informed sources give much of the credit for the new approach to Nancy Reagan and to Stuart K. Spencer, the California political consultant who is one of the Reagans' closest advisers. Spencer visited the Reagans at their California ranch in mid-August, and he has talked with Nancy Reagan and Regan by telephone from time to time since. The sources say that the first lady was unhappy with Regan's high-visibility performance following the president's colon cancer surgery, when the chief of staff pushed himself to the forefront and became "a sort of deputy president," in the words of one Republican close to the administration.

At many moments of staff crisis in the past, Nancy Reagan has turned to Spencer for counsel. She did so again this time and was advised by Spencer that the wisest course would be to buttress Regan rather than remove him, sources said. Spencer's view was that the White House needed stability rather than change and that no one was available who would be quickly capable of replacing Regan. Spencer, like Nancy Reagan, recognizes the importance of staff to a president who delegates a great deal of his day-to-day authority.

Regan now holds periodic meetings with a group that includes Spencer and another California political consultant, Kenneth L. Khachigian, former political adviser Lyn Nofziger, Washington lobbyists Bob Gray and William Timmons and Kenneth Duberstein, the first-term White House congressional liaison who is now a member of the Timmons firm.

Khachigian, a longtime Reagan campaign speechwriter, is the principal draftsman of the major address the president will deliver at the United Nations Thursday.

In addition, Reagan friend Michael K. Deaver, who left the White House staff in May, has been brought back to give public relations advice on the summit meeting with Soviet leader Mikhail Gorbachev. Regan also has asked presidential pollster Richard B. Wirthlin for ways to translate the president's high popularity into support for his programs.

Regan's decision to reach out beyond his tightly knit circle of aides followed a period of personal stock-taking in early August, according to informed officials. Sources say he recognized that he was vulnerable because his critics included Nancy Reagan plus Vice President Bush, national security affairs adviser Robert C. McFarlane and the Republican congressional leadership. Regan's aides were worried about what was happening; they advised their boss he needed a legislative strategy and assistance from old-time Reagan political hands.

Regan asked and received assurances from the president that his job was secure. In the view of Regan's most loyal deputies, Spencer's backing signified that the Reagans were using him as a "chosen messenger" to say that Regan was not leaving and that others should cooperate with him.

Edward J. Rollins, White House political director until he left the staff to join a political consultants' firm this month, thinks that Regan has learned valuable lessons from his bruising break-in period.

"He may not have the best political antennae," Rollins said in an interview, "but he's a bright man and he is oriented to success. He's been a success all his life, and he'll keep coming back to a problem until he's satisfied it's been handled successfully."

Regan demonstrated this quality with what one aide referred to as a series of "marriages" with other key officials in the administration and Congress:

* Despite the president's strong inclinations to the contrary, Regan cooperated with McFarlane to persuade the president on the necessity of imposing limited sanctions against South Africa and heading off stronger congressional action.

* Regan and Treasury Secretary James A. Baker III teamed up to defuse the intensifying demands from Capitol Hill for protectionist legislation. Regan oversaw preparation of a presidential speech that included tough new warnings that the United States would fight unfair trading practices. Baker orchestrated a new initiative with other important capitalist nations to try to force down the value of the dollar, which has seriously aggravated the U.S. trade problem.

* Responding to instructions from the president to try to find a way to balance the budget in the next few years, Regan told one of his chief lieutenants, Dennis Thomas, and acting budget director Joseph R. Wright Jr. to come up with a plan to accomplish this.

As they were working, the White House received word that Sens. Phil Gramm (R-Tex.) and Warren B. Rudman (R-N.H.) were developing a similar plan. By arrangement, the White House waited until the bill had 25 sponsors, then backed the plan as part of a package that raised the debt ceiling. This gave the administration at least a short-range reprieve from a congressional challenge on deficit reduction.

Not all Regan's "marriages" produced political success. Responsibility for the president's ballyhooed tax-overhaul plan, the supposed centerpiece of his "fall offensive," was divided between the White House and Treasury. The White House was to run a Reagan roadshow while Baker and his deputy, Richard Darman, tried to push the tax bill through the House Ways and Means Committee. The combination did not work, and tax revision appears to be in trouble.

And despite Regan's efforts to be more conciliatory and a better politician, he is faulted by some of the administration's allies. A Republican senator involved in deficit-reduction efforts who felt he had an important reason to talk to the president tried repeatedly to reach him earlier this month through the White House staff. His calls were returned by Regan and Thomas but never by the president. Finally, in frustration, the senator gave Vice President Bush a letter to take to the president. The next day Reagan called the senator.

"It's a helluva note when you have to use the vice president as a messenger because you can't get through the White House staff," the senator complained.

And Regan's previously publicized tensions with McFarlane have continued.

The tension between Regan and McFarlane has been highlighted by the strain of preparing Reagan for the forthcoming summit. Sources on both sides say that Regan views McFarlane as unnecessarily aggressive in presenting his views while the national security affairs adviser considers the chief of staff too meddlesome in foreign affairs. Each seems to thinks that the other is overly anxious to take credit for presidential success.

At a staff meeting on the day that the president endorsed the Gramm-Rudman budget proposals, McFarlane was sitting at one end of the conference table and Regan at the other. The president was not present. McFarlane asked a series of critical rhetorical questions about the package, with his message evident to the room. "Does the president know this will mean the end of the defense buildup?" McFarlane asked. "Does the president know this will mean the end of SDI Strategic Defense Initiative ?"

Regan fired back bluntly, according to a source who was present.

"Oh, come on Bud," he said. "You know it's not that bad."

Despite the Regan-McFarlane conflict, both men have attempted to work together, officials say.

"I don't care who cares for whom," said a Regan deputy last week. "Tell me," he said, "does it work? It does," he asserted.