A radical toughening of California's criminal justice system, passed at voter demand and expected to influence other states, has begun to crumble under the weight of contradictory interpretations, rising court caseloads and resistance from judges.

Prosecutors who hailed the passage last June of Proposition 8, the "victims' bill of rights," say that it has helped to increase slightly the prison sentences handed out in state courts.

But some of the proposition's strongest anti-crime weapons--such as use of illegally seized evidence and prior criminal records of defendants--have been blunted by trial judges or held back by prosecutors fearing reversal in the state's supreme court.

The language of the proposition, endorsed by a 56 percent majority of the state's crime-conscious voters, was so broad that it inadvertently wiped out much of the state's evidence code and gave defense attorneys some unexpected ammunition to use on behalf of criminal clients, directly counter to the intent of its supporters.

In one case, a defense attorney has managed to introduce a lie detector test favorable to his client under Proposition 8's new rule that "relevant evidence shall not be excluded in any criminal proceeding." Because of an apparent weakening of past law, Proposition 8 also appears now to allow defense attorneys to demand psychiatric examinations of rape victims.

"The impact has been less than its proponents had hoped and less than its opponents had feared," said Steve White, executive director of the California District Attorneys Association.

Edward J. Garcia, a former prosecutor who is now presiding judge of the Sacramento Municipal Court, said that "from my viewpoint, it has had little or no impact." Paul Gann, who initiated both Proposition 8 and the earlier tax-cutting Proposition 13, said he thought the crime-fighting measure has succeeded because the state crime rate is dropping.

Since the proposition is so broad and complex, prosecutors and defense attorneys say, it may take years for the state supreme court to rule on all its provisions and end the contradictory way it is now applied across the state.

Considered the most sudden and extensive revision of any state's justice system in decades, Proposition 8 made several changes in state law and the state constitution.

It allowed use of "relevant" evidence even if declared illegal in previous court decisions. In serious cases, it forbade plea bargaining, a process in which lighter sentences are traded for guilty pleas to avoid expensive, time-consuming trials. It allowed use of prior convictions to discredit testimony of a defendant or other witnesses in the eyes of a jury and established a victim's right to restitution from a criminal.

It also allowed victims or their families to testify at sentencing and parole hearings, added five years to the prison terms of second offenders and weakened or abolished various insanity defenses.

John J. (Jack) Meehan, district attorney for Alameda County, which includes Oakland, applauded the plea bargaining limits, the longer prison terms and the support for victims rights, all policies he and other prosecutors tried to pursue before the proposition was passed.

But he said that despite Proposition 8 many judges in his county were still not allowing prosecutors to tell juries about prior records of defendants who are testifying. The judges say another section of state law prohibits such evidence as "prejudicial to defendants," Meehan said.

In Sacramento, Garcia said, the judges in his court are allowing prior records to be heard unless they are so old "as to have no realistic relationship to the honesty or veracity" of the defendant who has testified.

Other examples of contradictory use of Proposition 8 abound. In Orange County, Chief Deputy Public Defender Thomas McDonald said the new rules against plea bargaining and the tougher sentences have persuaded defendants to fight their cases in court and created "a 50 percent increase in Superior Court trials." This "taxes our manpower tremendously" and gives prosecutors and defense attorneys less time to investigate and prepare their cases, he said.

In Alameda County, however, Meehan said an all-out effort to persuade defendants to plead guilty at the early stages of their cases in Municipal Court, where plea bargaining is still allowed, reduced the number of cases awaiting in Superior Court from 1,000 to 600.

California politicians such as Republican Gov. George Deukmejian have called for years for a reform of the state's "exclusionary rule," which kept improperly seized evidence from being used to convict persons of serious crimes.

Proposition 8's validation of all "relevant evidence" was designed to solve the problem, but many attorneys wonder if it will work and are awaiting a state supreme court ruling on the 1981 arrest of comedian Flip Wilson for cocaine and hashish possession.

Los Angeles Deputy District Attorney Roderick W. Leonard has argued that even if the search of Wilson's attache case at Los Angeles International Airport was illegal under old state rules it is legal under looser federal rules accepted by Proposition 8.

Meehan and other prosecutors, however, are reluctant to base their cases on the new rules until a decision comes down. "I feel our supreme court will always abide by the exclusionary rule," Meehan said.