The Supreme Court ruled yesterday that a prosecutor can threaten to file more serious charges against a defendant in order to pressure the defendant to plead guilty to other charges.

The case divided the court five to four. The central issue was whether the court was validating the kind of vindictiveness against defendants that it earlier had outlawed.

The defendant in the case, Paul L. Hayes, was indicted in 1973 on a charge of forging a check for $88.30. He faced a possible sentence of two to 10 years. In plea-bargaining, the Lexington, Ky., prosecutor in the case offered to recommend a five year term in charge for a guilty plea.

But, the prosecutor warned, if Hayes declined to "save the court the inconvinience any necessity of a trial" by pleading guilty, he would seek a reindict him under the Kentucky Habitual Criminal Act.

This law, when invoked, made a life sentence mandatory on conviction for a third offsense; it allowed parole after 15 years. The state repealed it in Hayes could not have been convicted as a habitual criminal.

The new law takes into account only offset committed by a defendant 18 or older. Hayes was 17 when he was charged with rape, pleaded guilty to detaining a female and was sentenced to five years - in a reformatory, not a prison. In 1970, he was convicted a robbery and sentenced to five years, but immediately was put on probation. Thus he had served no time in prison.

Claiming to be innocent of the bad-check charge, Hayes insisted on a trial. The result was that a jury not only convicted him, but also found that he had been convicted of the two other felonies. This automatically subjected him to a life sentence.

The conviction was upheld in state courts but struck down by the Sixth U.S. Circuit Court of APpeals. "In this case, a vindictive motive need not be inferred," Judge Wade H. McCree, now solicitor general of the United States, wrote. "The prosecutor had admitted it."

The Supreme Court termed the ruling "mistaken" and reversed it. The prosecutor's conduct, "which no more than openly presented the defendant with the unpleasant alternatives of forgoing trial or facing charges on which he was painly subject to prosecution of due process of law, Justice Potter Stewart wrote for the court.

In the "give and take" of plea-bargaining, the opinion said, there is no violative "element of punishment or reject the prosecution's offer."

In a dissenting opinion, Justice Harry A. Blackmum wrote that the prosecutor's admitted threat to bring a new indictment solely to deter Hayes from exercising his right to a trial was the kind of retaliation the court had barred in decisions going back to 1969.

"Prosecutional vindictiveness in any context is still prosecutorial vindictiveness," Blackmum said, Justices William J. Brennan Jr. and Thrugood Marshall joined his opinion.

In an additional dissent, Justice Lewis F. Powell Jr. wrote that the prosecutor denied due process to Hayes by admittedly setting out to "penalize with unique severity his exercise of consitutional rights." He said he was "not satisfied that the result in this case is just . . . "

The offer of five years for passing a bad $88 check "hardly could be characterized as . . . generous," Powell said.

In another decision, the court ruled that the right to marry is of such "fundamental importance" that a state can't deny a marriage license to a parent just because he fails to make court-ordered payments for the support of his child.

The court voted 8 to 1 to overturn a Wisconsin law that requires judges to withold permission to marry - in Wisconsin or elsewhere - unless the parent is making the payments, and unless the child is not, or is not likely to become, a public charge, The dissenter was Justice William H. Rehnquist.