An American nun is being held without bail and three members of the Roman Catholic Commission for Justice and Peace are facing trial here as the result of a police investigation into a commission report, scheduled to be published soon, alleging a campaign of brutality and torture against black civilians by Rhodesian security forces.

Rhodesian officials said today that a police search of the commission's offices led to the discovery of other documents that they said violate Rhodesian law. Authorities have not given any indication what those documents contain.

The American is Sister Janet MacLaughline, 39, of the Maryknoll order, who served in Tanzania and Kenya before arriving in Rhodesia about three months ago. She has been serving as press officer for the Catholic Bishop's Conference here.

Authorities said she is being held under the emergency powers regulations, which allow detention for 60 days, and will be charged with "extremely serious" multiple offenses.

The commission officials are Father Deiter Vernard Rodolph Scholtz, Brother Joseph Ferdinand Dupois and lay chairman John Deary. They apperared in court last week, and were released on $1,600 bail.

The commission previously has published three reports and several fact sheets on alleged police brutality against blacks, describing specific incidents in detail. Each time the government has refused to permit an impartial inquiry into the allegations, saying they were baseless.

The commission has long irritated the government. A previous head of the commission, Bishop Donal Lamont, was found guilty earlier this year of failing to disclose the presence of guerrillas. He was deported and stripped of his Rhodesian citizenship.

The report triggering the police probe charges that "Reports of torture at the hands of government security forces continues to be the rule rather than the exception . . . The practice of torture has become a common event in the lives of people in the rural areas . . . Their brutality has succeeded in driving many people into joining the guerrillas, where at least they can fight back.

"The government accuses the guerrillas of victimizing and intimidating the people. Most of the people would turn and point a finger at the government and declare that it is responsible for most of the terror in the country."

The report concludes that "the army and police have become 'the great enemies' and are hated and feared by the majority (of black Rhodesians)."

Specially, the commission charges that, following a guerrilla attack, the army "quarrillas."

The report describes methods of torture allegedly used by police and army officers against students, potential recruits for guerrillas and "frequent targets of (police) interrogation campaigns," as the "towel and hose" treatment in which youths are stripped, a towel placed over their faces, and running water sprayed into their mouths and noses through a hose.

The commission has prepared two other fact papers, not yet released dealing with the high rate of civilian deaths during five years of war, with its system of "protected willages" built along the Mozambique and Zambian borders. Authorities claim the villages are necessary to protect the population against guerrilla attacks, while critics have described them as concentration camps designed to cut of blacks from providing the shelter, food and information to guerrilla bands.

The fact paper on civilian deaths charges the government does not report all civilian and army deaths, adding "it is believed that the government fears morale would be lowered if the true figures are known."

The paper claims the government has acknowledged the deaths of only 532 civilians since January, or roughly four times the number of soldiers killed in the same period. Even this ratio, the paper suggests, "raise vital questions about the conduct of the war."

Law and order Minister Hillary Squires today called the commission's charges, "patently absurd." He said that when the government investigated earlier commission reports, "another side to the story had been revealed."