The owner of the Denny's restaurant chain yesterday agreed to be acquired by a New York-based food services firm after turning down an offer by Marriott Corp. that would have made Marriott the country's largest operator of family restaurants.

TW Services Corp. said it will buy California-based DHI Corp., the owner of the 1,200 Denny's restaurants, in a deal valued at $843 million. DHI turned down an earlier offer by Marriott that was reported to be worth $854 million. Neither Marriott nor DHI officials would confirm the figure.

The $843 million to be paid by TW Services includes $218 million in cash and the assumption of $625 million in DHI's outstanding debt, said Philip J. Reilly, assistant comptroller of TW Services. Reilly declined to comment on how the deal would be structured.

The announcement surprised some industry observers, who quesioned why DHI would accept the offer from TW Services while spurning Marriott's. DHI officials said last week that talks with Marriott had been called off because the price Marriott offered was lower than they originally expected.

TW Services' plan to permit Denny's to operate as an independent chain may have been a deciding factor in the deal, industry analysts said. Marriott would have combined the Denny's coffee-shop restaurants with its 900 Big Boy restaurants and merged the two staffs.

"It was a cleaner deal," said one source close to the negotiations for the TW Services offer. "They're taking the whole package. It makes it very difficult to compare the deals. What we don't know is how the debt was valued, how it's being assumed."

The businesses TW Services will acquire had 1986 revenues of about $1.3 billion and include the Denny's restaurants, a 42 percent stake in the chain of more than 750 Winchell's donut shops, and nearly 70 El Pollo Loco chicken restaurants. The Marriott offer did not include the chicken restaurants.

A Marriott spokesman declined to comment on the Denny's sale yesterday, but added, "We're still interested in the coffee-shop business."

The closest competitor of the Denny's and Big Boy restaurants is the 550-restaurant Shoney's chain.

Marriott "still has opportunities to grow in the Big Boy system," said John J. Rohs, an analyst with Wertheim & Co. "They're turning it around from its previous weak results. But clearly Marriott felt this was an interesting and unique opportunity. It looks like an opportunity lost."

Marriott made its original offer for Denny's in May. It was the latest aggressive move by the Bethesda-based company to acquire major competitors in the lodging and food services businesses.

In 1985, for example, Marriott purchased 350 Howard Johnson restaurants, and in 1986 it acquired Saga Corp., a major provider of contract food services. Marriott recently completed the acquisition of Residence Inns, a chain of budget-priced all-suite hotels.

Denny's was publicly held until 1985, when it was acquired for more than $750 million by an investment group that included Merrill Lynch & Co.'s investment banking arm and some of the company's top managers.

The restaurant chain's s chairman, Vern Curtis, will step down after the TW Services buyout is completed in September, Reilly said. However, Donald Pierce, the company's president, will remain.

TW Services was formed last December after TransWorld Corp. sold its Hilton International hotels to UAL Inc., the diversified travel services corporation now known as Allegis.

TW Services, which had 1986 sales of $1.9 billion, owns Canteen Corp., Spartan Food Systems Inc. and American Medical Services. Through its Spartan subsidiary, it owns and operates 600 restaurants, including franchises for nearly 400 Hardee's restaurants and 200 Quincy's Family Steakhouses.

Canteen Corp. is a contract food-services company. American Medical Services owns 26 nursing facilities and three retirement centers.