Peace talks with the Western powers on Namibia (Southwest Africa) collapsed yesterday as the guerrilla organization fighting for the territory's independence quit New York, citing the "grave situation" created by the South African paratroop attack on its bases in Angola.

The abrupt cancellation by the Southwest Africa People's Organization of discussions on the Western plan to convert the South African-ruled territory of Namibia into an independent, black-ruled nation this year disrupted nearly a year of complex diplomacy.

Until the South African raid into Angola last Thursday, the United States Britain, Canada, France and West Germany counted Namibia as the brightest prospect for a settlement in Africa's racial conflicts.

"Of course it's serious," a senior U.S. official said of this latest blow to the Carter administration's foreign policy goals. "But I think everyone involved realizes that the risks are so great," if the diplomatic path is abandoned, "that SWAPO will come back to the talks eventually."

"We think that the raid obviously had an unfortunate effect on the talks," said State Department spokesman Hodding Carter III. South Africa has told the State Department that its air and ground thrust 150 miles into Angola was solely designed to retaliate against "terrorism" by SWAPO guerrilla forces. Carter said, however, "We do not feel that the raid was justified under the circumstances."

Indications mounted, meanwhile, that the raiding party's battle plan went awry, and that the fighting was fierce, causing severe manpower losses to SWAPO.

"We came up against touch resistance, much stronger than we expected," said South Africa's chief of operations for the cross-border raid, Brig. Hannes Botha. "SWAPO continued fighting to the last breath, so seriously that they were shot dead four deep in the trenches."

South Africa has not disclosed the number of troops it sent across the border but unofficial reports have estimated that 700 men were inolved.

Botha said his forces had expected to finish their mission inside Angola in about 90 minutes but because of guerrilla resistance the battle dragged on for about 12 hours, according to an interview published in the Johnannesburg Sunday newspaper Rapport.

South African officials have said their units suffered five dead and fewer than 12 wounded. The newspaper quoted SWAPO sources in Lusaka, Zambia, as saying that at least 1,000 of some 2,500 to 3,000 guerrillas in the border area were killed. If correct, that would be a major military setback for SWAPO.

In New York, in a statement announcing the cancellation of talks set to begin yesterday between SWAPO leader Sam Nujoma and the five Western U.N. members, the nationalist group said:

"As a result of this invasion by several South African battalions, hundreds of Namibian men, women and children were killed and wounded and valuable property destroyed at our refugee camp at Cassinga, in southern Angola."

Because of the "grave situation" created with the assault by "fascist troops" of South Africa, SWAPO said, its central committee "has decided to urgently recall SWAPO's negotiating delegation" from New York. The Western nations were privately informed of the cancellation by letter late Sunday, when Nujoma left New York to return to his headquarters in Lusaka. No date was set for renewed talks, but U.S. officials stressed that "talks have not been broken off," only suspended.

The status of the now-dangling Western formula for Namibia is that South Africa officially accepted it, on April 25 while SWAPO questioned several of its provisions - among them the pace of South African troop withdrawal from Namibia and replacement by a U.N. peacekeeping force. SWAPO's misgivings are bound to be intensified now American sources ruefully admit.

"The United States continues to hope that SWAPO will accept the Western plan for a settlement" said State Department spokesman Carter. Another official conceded. "We're sort of in a holding pattern now," but the West must somehow "recapture the momentum we had built up on our proposal."

That plan contains a Dec. 31 deadline for transfer of power in Namibia, and South Africa has said it is committed to action by that date. There is black African suspicion, now intensified by the raid, that South Africa may be bent on sabotaging the Western plan, in order to supplant it with South Africa's own version of an "internal settlement." State Department officials generally discount that suspicion, but admitthat they are puzzled, as well as dismayed, by South Africa's latest action.

While SWAPO claims that the major South Africa target was a Namibian refugee camp in Angola, South Africa's Brig. Botha contended in his interview that uniformed women, and children no older than 13, fought his troops with anti-aircraft weapons, machine guns, and Communist-made anti-tank missiles.

Botha also claimed that some of the blacks in the base had been kidnaped from Ovambo in the northern sector of Namibia, and welcomed the invaders.

"They clapped their hands for us, he said. "Several of them could speak Afrikaans. They asked to be taken back to Ovambo, but because of transport problems we could not do so."

There were some insinuations raised in the U.N. Security Council debate on Namibia Friday night that Western powers may have colluded in the South Africa raid.

"I think we effectively denied them," a U.S. official said yesterday. The Security Council condemned South Africa's action Saturday and demanded "immediate and unconditional withdrawal" of its troops from Angola. South Africa claims they are all out.

The SWAP statement issued yesterday said ambiguously that recent "sinister developments" suggested that "intrigues are afoot to try to prevent Namibia attaining authentic independence."