James D. Briley, who was a member of a gang responsible for 12 killings and who helped mastermind the largest death-row escape in U.S. history last year, was electrocuted in the Virginia Penitentiary here tonight after his last legal appeals were turned aside.

He was pronounced dead at 11:07 p.m. after being subjected to two 55-second bursts of 2,300 volts of electricity. Prison officials said he "just smiled" instead of making a final statement.

A crowd estimated at about 600 had gathered behind barricades a half block from the prison's death row block. At 11:20 p.m., police announced that "the court order has been carried out" and there was cheering and a few racial taunts from the faction -- about 350 strong -- supporting the execution.

About 250 anti-death penalty demonstraters holding a candlelight vigil sang hymns.

Police said there were no major incidents, and fewer than a dozen disorderly conduct arrests.

Seventeen maximum-security prisoners at the penitentiary were involved in a melee early today that prison officials said was an attempt to disrupt the execution. Nine guards and one inmate were injured in the fighting, which broke out about 7:45 a.m. as four inmates, wearing pillowcases over their heads and armed with clubs and screwdrivers, tried to drag a guard into a cell.

"All hell broke loose," said Assistant Warden Rufus Fleming.

It took nearly 30 minutes to quell the violence; some of the injuries were described as serious, but none as life-threatening.

Officials said they had been tipped that a disturbance was likely and it had not affected their plans for Briley's electrocution.

Briley, 28, died in the same electric chair where his brother Linwood, 30, was executed last Oct. 12.

They and a third brother were members of a Richmond gang that was active in the late 1970s.

James Briley was condemned for raping a pregnant woman and then murdering her and her 5-year-old son. Linwood Briley was electrocuted for murdering a Richmond disk jockey. A third brother, Anthony, is serving a life sentence for his role in another murder.

James and Linwood Briley masterminded the breakout last May 31 from the maximum security Mecklenburg Correctional Center in Southside Virginia in which they and four other condemned men escaped. All were recaptured, the last being the Briley brothers, who were apprehended 19 days later in Philadelphia.

Briley's attorneys were turned aside by a federal judge tonight, as the hour of execution approached. Judge D. Dortch Warriner rejected as unreliable the testimony of a woman prisoner who supposedly said that another member of the gang to which Briley belonged committed the murders for which he was sentenced to die.

Gov. Charles S. Robb remained in his office with an open phone line to the prison tonight, but did not intervene.

Richmond prosecutor Warren Von Schuch, who was involved in the Briley trials, said today that James Briley had been "the more volatile, less predictable" of the two brothers.

"What Linwood did, Linwood thought about. James would do whatever occurred to him."

James Briley was first involved with police at age 16 when he robbed a convenience store and fired shots at a pursuing officer, according to court records.

"They are not from a poor family," Von Schuch said. "Yet their father slept with padlocks on his door. Even he was afraid of them. I don't know why it turned out like it did."

The execution of Briley was the third in Virginia since the death penalty was restored after a 1977 U.S. Supreme Court ruling. Three other executions are scheduled within the next two months, officials said. There are 27 other men on death row in Virginia.

Richmond Mayor Roy A. West, who served as one of six official witnesses at Linwood Briley's execution, also witnessed tonight's execution. He said the crimes of the Brileys, who were notorious in their northside Richmond neighborhood, were also wrenching. ". . . I'll never forget the pictures of the victims. It was the most gruesome sight I have ever seen. The victims looked like animals when they got through with them. The gruesomeness of an execution doesn't even compare. I felt like there had to be some sort of accounting. I felt like the citizens of Richmond should be represented."

Corrections spokesman Wayne Farrar, describing the morning prison melee, said, "These few inmates were intent on taking hostages for the purpose of disrupting the execution and the operation of the institution. The hostage-taking was prevented, that's the important thing."

Briley's bride of about a month, Evangeline, visited her husband for about two hours this afternoon, and praised the inmates who initiated the uprising "because you care about James."

"We're very sorry you were hurt in the process, but this is what happens when a state acts in a violent manner," she said. "They are planning on commiting a violent crime against James Briley tonight. Violence breeds violence."

Officials said that for his last meal, Briley requested and was given fried shrimp with cocktail sauce and a lemon-lime-flavored soft drink.