A U.S. District judge ruled yesterday that FBI agents violated a fundamental rule of criminal interrogation when they continued to question John W. Hinckley Jr. after he shot President Reagan despite Hinckley's repeated requests to speak to a lawyer.

As a result, Judge Barrington D. Parker said, statements that Hinckley made during that period of questioning, including a detailed account of his activities prior to the March 30 shooting, cannot be used by the prosecution as evidence during Hinckley's criminal trial.

In a 34-page ruling, Parker also concluded that federal prison officials last summer violated Hinckley's right to privacy when they seized personal notes and a diary from his cell at the federal correctional institution at Butner, N.C.

Those documents, which reportedly included a fictional conspiracy plot contrived by Hinckley, are also banned from use at Hinckley's trial, Parker said.

Hinckley, who is recuperating from an attempt to hang himself last Sunday, is scheduled to stand trial on Jan. 4 and will contend he was insane when he shot President Reagan and three other persons.

Since Hinckley's lawyers have conceded in court papers that their client committed the shootings, the only issue at his trial will be his mental state at the time. Teams of prosecution and defense doctors have been examining Hinckley in preparation for what is expected to be a complex courtroom battle over whether Hinckley should be found insane and thus not criminally responsible.

In his ruling yesterday, Parker rejected the prosecution's contention that the FBI agents simply sought harmless background information from Hinckley when they questioned him for hours after his arrest. In addition, Parker found that law enforcement personnel "made no genuine or sincere effort" to comply with Hinckley's desire from the outset to talk to a lawyer.

Parker's ruling hinged on the safeguards set down by the U.S. Supreme Court in 1966 in the case of Miranda v. Arizona, which established the now familiar rule that police must inform a defendant of his rights to remain silent or to have a lawyer present during questioning. Interrogation must cease when a suspect says he wants to remain silent or consult a lawyer before making any statements.

The Miranda rule, now routinely read to criminal suspects as soon as they are arrested, was intended to carry out the Fifth Amendment's protections against self-incrimination.

The prosecution had contended that Hinckley, who was repeatedly advised of his so-called Miranda rights, had voluntarily talked to the FBI agents. But Parker noted that the agents had initiated the conversations, not Hinckley. Even if Hinckley spoke voluntarily, that did not mean he was giving up his right to consult a lawyer and did not diminish his constitutional rights against self-incrimination, Parker asserted.

In granting the defense motion to prohibit use of the personal notes and diaries, Parker cited rules that hold that even though prisoners lose some rights of privacy when they are incarcerated, they are still entitled to some of the Fourth Amendment's protections against unreasonable searches and seizures.

Parker found that the scope of the searches conducted at the prison were an exaggerated response to the government's concern that Hinckley would commit suicide. Hinckley had taken an overdose of an aspirin substitute while he was confined in Butner in May. He was subjected to "extraordinary security precautions" that made it unnecessary to take the further step of reading his personal papers to help prevent a possible suicide attempt, Parker said.

In the ruling yesterday, Parker turned down Hinckley's contention that statements he made during court-ordered mental examinations should be excluded from evidence since Hinckley did not have a lawyer present.

Meanwhile, the Justice Department reported that Hinckley is in satisfactory condition at the base hospital at Fort Meade, Md., where he has been confined since he tried to hang himself Sunday.