Dr. Carolyn Robertson Payton, who only last spring said her goal was "to restore the Peace Corps to health," abruptly resigned yesterday as director of the struggling overseas volunteer assistance organization.

Payton, 53, was the first woman and the first black to head the Peace Corps.

Details of her departure were unclear yesterday. Neither she nor ACTION Director Sam Brown could be reached for comment.

A spokesman for ACTION, the agency that administers the Peace Corps, VISTA (Volunteers in Service to America) and other volunteer service programs, issued a terse statement saying:

"Dr. Carolyn Payton informed Sam Brown today (Wednesday) that she would be resigning from the Peace Corps. But we haven't yet received her letter of resignation."

However, several administration sources said that Brown requested Payton's resignation because of "policy differences."

One of those who claimed familiarity with the events surrounding the resignation and who was willing to speak on the record was Randall Robinson, executive director of Transafrica, a Washington-based lobby group for African and Caribbean interests.

Payton's resignation is, "in effect, an ouster," Robinson said.

"To put it bluntly, her resignation was asked for by Sam Brown, who got authorization to do so from President Carter," he said.

Robinson, repeating a statement made by other sources, said Brown met yesterday morning with Carter to request permission to fire Payton, a presidential appointee.

The White House deferred any comment on the resignation to ACTION, whose spokesman confirmed that Brown went to the White House yesterday. "But he often does go to the White House to discuss various things," the spokesman said.

Robinson charged that Payton's resignation "is just another example of the administration's downgrading of black input in the foreign policy process." he said there "has never been a complaint about the quality of Dr. Payton's work."

Payton, a psychologist and former director of the Howard University counseling service, joined the Peace Corps in 1965. She had served as director of its operations in the eastern Caribbean.