Romance and thrills just don't mix, the Justice Department decided yesterday.

The department's Antitrust Division came to that conclusion when it advised Harlequin Enterprises, Ltd., of Toronto, Canada, that it will file a civil suit to challenge Harlequin's proposed takeover of Pinnacle Books. Inc., of Los Angeles.

Both firms are mass market paperback book publishers, with Harlequin specializing in Gothic Romance novels and Pinnacle holding a large share of the thriller market.

The Justice Department claims the acquisition would violate Section 7 of the Clayton Act, which prohibits mergers that are anticompetitive in nature.

Although there has been no public announcement by Harlequin that it is planning to buy Pinnacle, John H. Shenefield, the assistant attorney general in charge of the Antitrust Division, said his staff learned of the takeover plan several months ago.

After reviewing the facts surrounding such a takeover, including the market shares of each company, the department concluded that the result would be a lessening of competition in the industry.

Consequently, Shenefield said, the department decided to warn Harlequin that it would challenge in court any takeover attempt.

Pinnacle is a wholly owned subsidiary of Michigan General Corp. of Michigan and Texas.

Harlequin is a Canadian corporation, 52 percent of which is owned by Torstar, a holding company that also owns the Toronto Star, a leading Canadian newspaper.