The Carter administration's first attempt to appoint a black as a federal judge in the Deep South suffered a setback yesterday when the Senate Judiciary Committee failed on a tie vote to approve the nomination of Robert F. Collins, a New Orleans magistrate.

The committee split 5 to 5 on approving the nomination, with the dissenting members clearly following the lead of Sen. Dennis DeConcini (D-N.M.), who conducted several days of closed hearings on allegations about Collins' fitness.

Sen. James O. Eastland (D-Miss.), the committee chairman, said afterward that the nomination will be voted on again. And a knowledgeable Justice Department official said that no thought had been given to withdrawing Collin's name.

Contacted in New Orleans yesterday, Collins said only that he was "totally shocked and nearly speechless" at the news. He said he thought the problems were over and that the approval would be routine.

DeConcini asked for a roll-call vote on the Collins nomination and was the first "no" vote. He was joined by Sens. Orrin G. Hatch (R-Utan), Malcom Wallop (R-Wyo.), James B. Allen (D-Ala.) and Paul Laxalt (R-Nev.).

The closed hearings on Collins' nomination focused on charges that he traded judicial favors for a local lawyer in return for prostitutes and accepted cash from the campaign of Louisiana Gov. Edwin Edwards, apparently to drum up black support, according to sources.

DeConcini said after the vote that he "could not conclude there was any basis for the complaints." But he said he "still wasn't comfortable" about voting for Collins.

Wallop and Hatch said they, too, were concerned with the evidence they saw in the closed hearings and commended DeConcini for his work.

"From what I saw I felt we could have a more highly qualified nominee," Hatch said.