A state court judge yesterday denied a request by Arlington prosecutor William S. Burroughs to convene a special grand jury to investigate possible misconduct by Virginia Attorney General J. Marshall Coleman and the state police.

It was the third time in eight months that the prosecutor has been rebuffed in attempts to press an investigation of why Coleman last year ordered a state police probe of Burroughs' handling of a crontroversial double murder case.

Yesterday's decision by Arlington Circuit Court Judge Paul D. Brown came after only two members of the county's five-member regular grand jury said they would serve on the special investigative panel.

Judge Brown asked the grand jurors if they would serve after the jurors heard testimony yesterday from both Burroughs and State Crime Commission Executive Director Lewis Hurst. Brown said that under "the court's discretion," he refused to convene a special panel. Special grand juries can issue reports but cannot return indictments.

As he left the courtroom, Burroughs said, "as far as I'm concerned, the matter is not closed." He said he is considering making a fourth request for an investigation when the next regular grand jury convenes in December.

Burroughs, a Democrat seeking reelection as Arlington commonwealth's attorney, in next month's election, and Coleman, a Republican, have been feuding over Burroughs' handling of the murder trial of Richard Lee Earman, who was acquitted of murdering a man and woman in 1977.

In unusual court papers filed last week, Coleman said he ordered a state police probe of Burroughs after receiving "corroborated information" that Burroughs was suspected of bribery and a cover-up in connection with the murder case.

Coleman said the plice investigation produced insufficient evidence to warrant criminal charges and that "Mr. Burroughs' conduct could merely have been the result of poor judgement."

In Richmond yesterday, Coleman released a statement saying he was "glad the grand jury showed to the good sense to refuse" Burroughs' request for an investigation.