Madeira School headmistress Jean S. Harris was romantically involved with Dr. Herman Tarnower, the "Scarsdale Diet" author she is accused of murdering, and had been brooding over his involvement with another woman, according to friends.

Harris, who acquaintances said had once hoped to marry the wealthy New York physician, was released yesterday from a jail in Westchester County, N.Y., on $40,000 bail.

Friends said Harris, 56, had been romantically involved with Tarnower for years but had been recently rejected by the doctor in favor of a younger woman who worked on his medical staff.

"It was the eternal triangle," said one of the doctor's neighbors in fashionable Westchester. Until the new woman, identified by neighbors as Lynne Tryforos, a tall, blond nurse in her 40s, appeared, Harris would spend her weekends with Tarnower, the friends said.

Tarnower, 69, a bachelor who told friends that he would never marry, was shot four times late Monday night in the second-floor bedroom of his house on a six-acre estate in Harrison, N.Y. He was found unconscious in his pajamas, lying between twin beds and bleeding heavily, in a room that showed signs of a scuffle. He died about an hour later in a nearby hospital.

Harris, who was arrested shortly after the shooting at Tarnower's home, was described yesterday by friends as a highly intelligent and driven woman. That said she was devoted to her job as headmistress of the exclusive Madeira School in suburban Washington, but sometimes took her work too seriously, becoming outraged over minor problems.

"She was very intense, very intense," said one student at the private girls school in McLean. "Even the littlest things seemed to set her off."

Harris, who frequently admonished her students on the need for integrity in public and private life, recently banned oranges from the campus after spotting too many orange peels on the school grounds, which occupy 400 acres of rolling land and woods along the Potomac River.

The headmistress, a trim, blond woman who has worked in private schools most of her life, was released from jail shortly after 1 p.m. yesterday. Judge John C. Couzens stipulated that she not leave Westchester County, north of New York City, without his permission.

In Westchester County Court yesterday, Assistant District Attorney Jospeh Rakacky asked the judge to deny bail for Harris, arguing that she shot Tarnower four times and brought the murder weapon -- a .32-caliber revolver -- to his home.

"The killing apparently followed a dispute that arose out of a personal relationship," Rakacky said.

Arguing for the release of the headmistress yesterday, lawyer Joel Aurnou said that Harris had suffered bruises on her lip, left arm and her right eye while inside Tarnower's house on Monday night. Harris did not appear in court yesterday because her laywer said she was in a "severe state of shock" and needed immediate psychiatric care.

Harris, who was divorced 15 years ago and has two sons in their 20s, had known Tarnower for about 14 years. Her sons, two sisters and a brother were in court yesterday to testify as character witnesses, but they were not called upon.

Aurnou indicated that his client will plead self-defense to the murder charge. She is scheduled to appear in a county courtroom today at 4 p.m. for a hearing on the charge.

Tarnower was described by the coauthor of "The Complete Scarsdale Medical Diet" as a "very sedate, private and austere man."

Tarnower's most recent girlfriend, Lynne Tryforos, is mentioned in the foreword to his best-selling diet book, as is Harris. The doctor's friends said yesterday that Tryforos has been seen with him frequently in recent weeks and had dined at the physician's home the night of the shooting.

But Tryforos and two other dinner guests had left Tarnower's Japanese-style home before Harris showed up at about 11 p.m.

Yesterday, as Harris' lawyer argued for her release on bail, a separate dispute arose between Tarnower's coauthor, Samm Sinclair Baker, and the lawyer over what role the Washington educator had played in the diet book, the second best selling hard cover book of 1979.

Aurnou, the lawyer, said Harris had written much of the book and had recently revised the final chapter for another printing of the book. But the book's publisher, Rawson Wade Inc., and coauthor Baker said they were unaware of any major contributions Harris made to the book.

The controversial book prescribes a 14-day, high-protein diet that is supposed to take off a pound of weight a day. Tarnower claimed weight loss from the diet was caused by changes in the body brought on by restricting intake of carbohydrates, but critics of his diet claimed it was ineffective and caused only a temporary weight loss.

Harris had been expected at a Monday night dinner party for 14 at the Northwest Washington home of John and Kiku Haines. The headmistress called Mrs. Haines on Sunday to say that she was "looking forward" to the dinner, Mrs. Haines said yesterday.

"When she didn't show up for my dinner, my first instinct was that something had come up at the school," said Mrs. Haines.

Harris, who lives in a one-story, brick house on the Madeira School Campus, attended meetings there until midafternoon Monday, according to school authorities. Then, the headmistress, who had told friends she planned to remain on campus during the school's three-week vacation beginning Monday, left in her blue-and-white, 1973 Chrysler for Westchester County.

Harris, who was picked in 1976 to be headmistress of Madeira from among 100 applicants for the job, had worked as a sales manager for 18 months at Allied Maintenance Corp. in New York. After she took the Washington job, she kept her own home in Putnam County -- about 45 minutes' drive from Tarnower's house.

Before taking the sales job in New York, Harris had been headmistress for four years at the Thomas School in Rowayton, Conn., where she was "not terribly popular," according to several people familiar with the private school.

Harris did not get along well with the staff of the girls' school that closed in 1975, one said. "She was given to unexplainable emotional outbursts" and had "a violent temper and would shout and scream at the students," the acquaintance said.

James R. Worsley Jr., a Washington lawyer and vice president of the Madeira School, said yesterday that the school had attempted to check into Harris' record at the Thomas School before hiring her.

"She received very strong support," Worsley said, but he added that extensive checks were difficult because the Connecticut school had closed and its faculty members were difficult to locate.

Harris, who graduated magna cum laude in economics from Smith College in 1945 and holds a master's degree in education from Wayne State University, went to the Thomas School from the private Springside School in surburban Philadelphia, where she had been director for five years. Before that, she had worked for 16 years at the University Liggett School in Grosse Pointe, Mich.

Harris reared her children in Michigan and was divorced there in the early 1960s. Her son, James, 27, is a lieutenant in the Marine Corps. David, 29, her other son, works for a bank in Yonkers.

At the Madeira School yesterday, administrative officials said they continued to expect Harris' resignation as headmistress, but that she had not formally been relieved of her job.

Although some students at the school yesterday criticized Harris for being too critical and overbearing, parents said the headmistress had done a great deal in three years to improve the quality of education at the school and increase school spirit.

The motto of the 74-year-old Madeira School, where administrators said yesterday the murder charges against its headmistress continue to shock faculty and the 325 students, is "festina lente," which is Latin for "Make Haste Slowly."