A divided Supreme Court tightened its one-man, one-vote rule yesterday, saying population differences of less than a single percentage point among congressional districts in a state may be unconstitutional.

The dissenting justices said the new standard could lead to irrationally drawn districts and actually more gerrymandering by state legislatures, not less.

The 5-to-4 ruling declared New Jersey's 1982 redistricting plan unconstitutional because its largest congressional district has 3,674 more people than the smallest--a deviation of about one seventh of 1 percent from absolute equality. The court said the state can bring about even greater equality.

The decision raised legal questions about plans in at least 16 states, including Virginia, which have deviations larger or about the same as New Jersey's.

The ruling does not mean these plans are necessarily invalid. But if court challenges demonstrate that greater equality was feasible, the states must then assume the heavy burden of offering specific, legitimate policy justifications for retaining their plans, the court said.

The opinion, written by Justice William J. Brennan Jr., guaranteed an even greater future role in the redistricting process for the federal courts, which many politicians think are already exercising too much authority. It imposed new levels of mathematical precision on state legislatures, at a time when some are complaining that computers have already taken over the process.

The one-man, one-vote rule laid down in the 1960s requires congressional districts to be equal in population. The question in yesterday's case was how equal.

Advocates of a reduced federal role said a deviation of about 1 percent or less within a state should be looked on as the "functional equivalent of zero." Anything more demanding, they said, would be nitpicking, especially in light of the fact that census figures that are the basis for redistricting plans may be high or low by as much as 2.5 percent.

But Brennan, a New Jersey native who wrote one of the earlier redistricting opinions, said there can be no standard other than pure equality. "Adopting any standard other than population equality . . . would subtly erode the Constitution's ideal of equal representation," he said in Karcher vs. Daggett.

Justice Byron R. White, joined by Justices Lewis F. Powell Jr., William H. Rehnquist and Chief Justice Warren E. Burger, said the majority opinion was an "unreasonable insistence on an unattainable perfection" that will encourage gerrymandering by making mathematics more important than geographic common sense and local political history.

They also said it will assure "extensive intrusion of the judiciary into legislative business." Justice John Paul Stevens said he agreed with the opinion, but wrote a separate concurrence.

Legislatures are given much greater flexibility when realigning their own state legislative districts. The court underscored that point yesterday in a separate ruling, Brown vs. Thompson.

It upheld a Wyoming state redistricting plan in which one district deviated by 89 percent from the ideal. The justices, with Powell writing for the 5-to-4 court, let that plan stand because of unique circumstances: it saved one tiny county from loss of all representation in the state legislature and had only a minor impact on the rest of the state.

Justices Brennan, White, Thurgood Marshall and Harry A. Blackmun dissented. Justice Sandra Day O'Connor, who was in the majority, wrote separately to explain the difference between the two opinions and say that "equal representation is not just a matter of numbers."

Kenneth J. Guido Jr., who represented the speaker of the New Jersey Assembly in the New Jersey case, said the decision will not require a new congressional election there. He said a district judge is expected to set a timetable for the redrawing of the state's districts by the legislature. Few major changes will be required, Guido said.

New Jersey, because of population changes between 1970 and 1980, lost a seat and its House delegation was reduced from 15 to 14. The legislative redistricting plan contained 14 districts with an average population of 526,059. That average figure is considered the "ideal" for reapportionment purposes.

The largest district, which includes Trenton, had a population of 527,472. The population of the smallest, which encompassed most of Middlesex County, was 523,798. The difference was 3,674 people, or .6984 percent of the ideal, Brennan said.

The plan, enacted by a Democratic legislature, was unsatisfactory to Republicans, who challenged it in court. They demonstrated with the use of a computer that they could reduce the deviation to .4514 percent.

A U.S. District Court judge agreed, rejecting the legislature's contention that its deviation was "the functional equivalent of zero." A good-faith effort, the lower court said, could have reduced the discrepancy. The lower court also rejected the state's explanation for the deviations: that it was trying to preserve minority voting strength in the Newark area.

Brennan said that in such a situation, the burden falls upon the legislature to explain its less equal plan. "Any number of consistently applied legislative policies might justify some variance," he said, "including making districts compact, respecting municipal boundaries, preserving the cores of prior districts and avoiding contests between incumbent representatives . . . . The state must, however, show with some specificity that a particular objective required the specific deviations in its plan, rather than simply relying on general assertions."

According to the National Conference of State Legislatures, states with maximum deviations larger than New Jersey's are: Indiana (2.96 percent); Alabama (2.45 percent); Tennessee (2.4 percent); Georgia (2 percent); Virginia (1.81 percent); North Carolina (1.76 percent); New York (1.64 percent); Kentucky (1.39 percent); Washington (1.3 percent); Massachusetts (1.09 percent); New Mexico (.87 percent), and Arkansas (.78 percent).

States with similar deviations are: Ohio (.68 percent); Nevada (.6 percent); Oklahoma (.58 percent), and West Virginia (.5 percent).