THE LAW of libel, as it is now developing, works increasingly badly. Ideally, the law strikes a fine balance between freedom of expression and the rights of a person defamed. Currently it is protecting neither of them adequately. The next conspicuous demonstration of its shortcomings will probably appear in Gen. William Westmoreland's suit against CBS, now scheduled to come to trial in New York this month.

In retrospect, it is evident that the Supreme Court set the law moving on the wrong course in the memorable Sullivan case two decades ago. There the court tightened the standard that a public official (later expanded to public figure) has to meet, by declaring that he must prove malice. Malice, as the court defined it, means that the people who published a statement knew that it was wrong, or that they recklessly disregarded that possibility. At first, the press applauded the decision. But that applause turns out to have been premature.

Because the plaintiff bears the burden of proving malice, the courts have granted the plaintiff's lawyers the power to interrogate widely all the people connected with the disputed statements. Who knew exactly what -- and who showed which biases? Gen. Westmoreland's case involves a CBS documentary in early 1982 suggesting a "conspiracy" in the American military command during the Vietnam War to conceal the enemy's full strength. Interrogating the witnesses to determine their states of mind, as they worked on the documentary, comes perilously close to examining their political beliefs and opinions.

Whether this process of interrogation serves the interest of the plaintiff or the defendant will vary from one libel case to another. One purpose of a libel suit -- and usually the most important -- is to obtain in court a verdict that tells all the world that an accusation was false. But in the present state of the libel law, that does not reliably happen. Steven Brill, writing recently in The American Lawyer, argued persuasively that the law now frequently produces verdicts that are actually perverse.

If a plaintiff can use interrogation skillfully to show a political bias against him, the jury may favor him even if the story was substantially true. If a plaintiff cannot prove that the publication was deliberately lying, or was absolutely reckless with the truth, the suit may fail even though the charge actually was false.

The present pattern in libel litigation is that juries generally give the verdict to the plaintiff, and that verdict is very often reversed on appeal. Evidently juries' sense of what they are supposed to be doing is at odds with the judges' sense of it. That in itself is a signal that something's going wrong. It's time for the appellate courts to reconsider the whole question of libel. The point at which to begin is by asking whether the concept of malice is not turning into a diversion from what most people, in these cases, would consider justice.