On the night of Nov. 11, 1977, Helen Butler, a Montclair, N.J., woman in her early 60s, was mugged in a supermarket parking lot. The incident set off a string of litigation that resulted two months ago in a ground-breaking ruling that adds expensive new responsibilities to the job of running a store.

The New Jersey Supreme Court ordered Acme Markets Inc., where Butler was shopping, to pay her $3,600 for injuries to her head, face, chest and ankle from the attack, and for pay lost on the days afterward when she couldn't go to work. Acme escaped relatively easily; there is little doubt that in other cases, the awards could be a hundred times that amount.

The Jersey high court ruling seems to be the first major decision about a merchant's duty to ensure the safety of his parking lot. But it "is consistent with the law as it is developing throughout the United States," says Michael J. Hutter of the New York State Law Revision Commission. Hutter chairs the American Bar Association committee on business tort litigation.

The new development is based on the very old legal concepts of the duties a property owner owes to those he invited to come onto his property.

Any businessman who has an outlet open to the public is assumed to have issued such an invitation. And he is, therefore, liable, as a standard legal text puts it, "for physical harm caused by the accidental, negligent, or intentional harmful acts of third persons or animals." The obligation: "to exercise reasonable care" in discovering what sort of hazards are likely, and in protecting customers against them or giving them enough warning so they can avoid the harm.

The law does not require merchants to protect shoppers against unexpected dangers--that would not be "reasonable"--but it does demand safeguards against the kind of hazards that are, in the word the lawyers use, "foreseeable." That's why a customer who slips on a wet floor has the right to sue.

What has been changing is the way that general rule is interpreted. Judges in recent years, says Hutter, "have essentially overruled an older line of cases that said: 'Criminal act? There's no duty to foresee or guard against.' "

In finding for Helen Butler, the New Jersey justices demonstrated how thinking has altered since 1962, when that same court went the other way in a similar case: a housing authority, the decision ran then, had no legal obligation to a milkman who was beaten and robbed while making deliveries to a project owned by the authority. In the intervening two decades, Justice Daniel J. O'Hern notes:

* "Trained private security services have now become widespread," so the former concept that policing is the sole job of government is no longer valid.

* "In the modern context of merchandising" it is deemed fair to force business owners and, indirectly, their patrons "to bear the costs of avoiding negligence." Twenty years ago such imposition of extra expenses seemed unjust.

* Criminal attacks are much more predictable.

Neither the Butler decision nor any of its predecessors suggest that a business owner is always liable when a customer gets attacked. There is still the question of whether the merchant has acted reasonably, and that is up to a jury to decide.

In the case of the Montclair Acme, there had been seven parking lot muggings in the previous year, five of them during the four months before Butler was hit. The supermarket had responded by making sure the lot was well lighted, and by hiring off-duty town police officers as store guards.

But only one guard was on duty at a time, and his duties were so varied--spotting shoplifters, preventing bad checks, and watching parcels as well as patrolling the store and parking lot--that he was inside the store more than he was out. The store had posted no signs warning about the parking lot attacks.

The jury admitted that Acme "did not cause the crime," but concluded that the market just had not exercised enough care for its patrons. he new trend in decisions does not give merchants any firm guide to just how much protection they must buy--or how explicitly they must warn customers about perils lurking on the premises. Juries are likely to be sympathetic to injured innocents, and to look for someone with a big enough bank account to pick up the bills.

"All recognize the serious concern that this new burden may place upon already hard-pressed businessmen," Justice O'Hern admits. "Still there remains sufficient flexibility in the standard of reasonable care applied for juries to avoid harshness through application of 'common sense.' "

But wise businessmen will also check to make sure their insurance policies cover incidents outside the store.