Dr. Carolyn R. Payton, forced to resign two weeks ago as director of the Peace Corps, yesterday accused federal volunteer program administrators of trying to turn the corps into an "arrogant, elitist" political organization designed "to meddle in the affairs of foreign governments."

Payton said she believes the Peace Corps has "strayed away from its mission" of "promoting world peace and friendship" and is trying to impose American intellectual fads -- political and cultural -- on host countries.

For example, she said, "it is wrong to tell a government in the Third World that its efforts to teach its citizens a world language -- be it English or French -- is an 'elitist idea.'" And it is "arrogant and neocolonialist for the American Peace Corps to say to a nation, 'We will no longer teach your children mathematics and science' so that some secrets of western technology will become accessible to them but that 'we will teach your peasants numeracy and literacy'" so they can count their cows or print their names on a wall, she said.

"I believe it is wrong to use the Peace Corps as a means of delivering a message to particular constituencies in the United States, or to export a particular political ideology," Payton said in a speech here before the conference of the Eastern Association of College Deans.

"Those now responsible for the Peace Corps seem to wish the organization to be engaged in a kind of political activism and advocacy. They would be pleased to have Peace Corps volunteers demonstrate overseas against corporations that engage in practices with which they disagree, or that market products they see as harmful.

"They would see the Peace Corps as a vehicle to allow unemployed black ghetto youth, as short-term volunteers, learn about life in a black socialist country."

Payton, described by some as an "establishment" black liberal, was the first black and first woman to head the Peace Corps, the government's overseas volunteer organization. Her 13-month tenure ended Nov. 24 after a long-running conflict between herself and ACTION Director Sam Brown, a former antiwar activist, who had jurisdiction over the Peace Corps and other federal volunteer service programs.

Brown demanded Payton's resignation because of what were officially described as "policy differences." Payton initially refused, but relented at the request of President Carter, who said the "unresolvable policy differences" between the two administrators were hurting ACTION.

Payton's speech yesterday was her first public comment on her resignation.

"The Peace Corps has strayed away from its mission," she said. "As director, I could not -- because of the peculiar administrative structure under which the Peace Corps operates -- do anything about this situation. As an ex-director, I am free to sound the alarm."

Brown could not be reached for direct comment, just as he could not be reached for direct comment on Payton's resignation.

Some ACTION officials said privately that Payton's statements were "unfortunate" and "Outlandish." However, Marylou Batt, an agency spokeswoman, said: "We are carrying out the policies which the president wanted and which the Congress supported. We are talking about differences of policy, not of politics, as implied in Dr. Payton's remarks."

Batt said Congress has given the Peace Corps a vote of confidence by increasing its budget by $9 million, from $86 in fiscal 1978 to $95 million in fiscal 1979.