Arlington County's chief prosecutor William S. Burroughs Jr. yesterday was rebuffed for the fourth time this year in his effort to secure a special grand jury investigation into his complaints of misconduct by Virginia Attorney General J. Marshall Coleman and Virginia State Police.

Burroughs, who has been engaged in a continuing debate with Coleman, failed to obtain the investigation during what was his final scheduled appearance before an Arlington grand jury. He was defeated in the November elections and will leave office at the end of the month.

Yesterday four members of a seven-member Arlington County grand jury voted against impaneling a special investigative body and Circuit Court Judge Charles S. Russell also refused to convene the panel.

"This matter has been previously explored on a number of occasions," said Russell, who last February rejected the prosecutor's first request to impanel a special grand jury. "At the discretion of the court no special grand jury will be impaneled."

As he left the courtroom after Russell's announcement, Burroughs, smiling tightly, turned to reporters and said he would have no comment. Coleman could not be reached.

Burroughs, a Democrat, and Coleman, a Republican, have been feuding publicly for the last 18 months over a 1978 state police investigation that Coleman ordered into the prosecutor's handling of a spectacular double murder case.

Coleman has said in papers-filed in Arlington that he ordered the probe after receiving "corroborated information" that Burroughs was suspected of bribery and a cover-up in connection with the case. Burroughs has denied those allegations and has suggested that Coleman's motivations for the probe may have been political.

Coleman said the investigation failed to produce evidence of criminal wrongdoing and said Burroughs' actions "could merely have been the result of poor judgment." The furor over Burroughs' handling of the case and Coleman's charges were central issues in Burroughs' defeat last month by Henry E. Hudson, a Republican-backed candidate whom Coleman supported.

Although he soon will leave office, Burroughs could return to the next regularly scheduled grand jury session in February as a private citizen to continue his efforts to have Coleman's actions investigated. Court officials said yesterday that instances of private citizens winning such investigations are extremely rare in Virginia.

Three previous grand juries and three circuit court judges have rejected Burroughs requests for the Coleman probe. The murder cases involved Richard Lee Earman and Joseph N. Martin, who are awaiting sentencing on charges growing out of a 1977 murder-for-hire scheme that left a young Arlington couple dead.