A military judge in the case of Marine Pvt. Lindsey Scott refused yesterday to allow prosecutors to introduce testimony that said Scott's most important witness in his attempt to get a new trial in an attempted murder case had once lied.

Scott, a black former military investigator, was convicted 16 months ago of raping, sodomizing, kidnaping and attempting to murder a white woman at the Quantico Marine Base. He is currently serving a sentence of 30 years' hard labor.

His key witness is Cynthia Ausby, a former Zayre store detective who testified in a fact-finding hearing last month that Scott was at a department store miles away from the scene where the victim was abducted the night of April 20, 1983.

Yesterday, military prosecutors submitted affidavits from two employes of the Prince William County police department, where Ausby applied for a job as a police officer in 1981. Both men -- one a polygraph examiner and the other a lieutenant -- said in the statements that Ausby lied to them. There was virtually no explanation of what Ausby lied about.

The military judge, Lt. Col. H.S. Atkins, refused to let the prosecution call the two men and reopen the fact-finding hearing, which was originally held to determine whether Scott's attorney had represented him effectively in his October 1983 court-martial.

Ausby, a licensed private investigator in Maryland and Virginia, threatened immediately after the proceeding yesterday to sue Prince William County for slander. She said there is no basis for the allegations that she had lied while applying to the police department there.

Ausby also said -- and prosecutors confirmed -- that she had passed a polygraph test administered by Prince William police.

"My life is an open book and I have nothing to be ashamed of," Ausby said after the proceeding yesterday.

Despite the judge's decision, prosecutors said they had succeeded in introducing doubt about Ausby's credibility into the record.

The case now goes to the commander of Quantico, Lt. Gen. David M. Twomey, who may order a retrial, dismiss the charges or send the case to the Navy-Marine Corps Court of Military Review in Washington. It is considered unlikely that he would dismiss the charges against Scott.

Scott's case has attracted attention nationally from civil rights groups, which charge that Scott was victimized by the military because of his race.

In the court-martial, Scott was convicted largely on the strength of circumstantial evidence; the victim had at first failed to identify him positively in a police line-up.