The Justice Department moved yesterday to block the proposed merger of the second- and third-largest pay television networks into a company that could challenge the dominance of Home Box Office.

The government said it plans to file a civil antitrust suit opposing the merger of the pay TV networks Showtime and The Movie Channel into a company whose owners would include three major motion picture studios, a cable TV firm and Shearson/American Express Co. because of the possibility of decreased competition in the distribution of movies generally and in the sale of programs to cable TV networks.

The complex deal, announced last January, proposed joining Showtime, owned by Viacom International, a cable and programming firm, with The Movie Channel, whose owners include Shearson/American Express, Warner Communications, Paramount Pictures and Universal Studios.

Home Box Office, the Time Inc. company that is far and away the industry leader with 12 million subscribers, hailed the Justice Department decision as confirming "our earlier concerns of the potential for anticompetitive conduct by the parties to the venture." HBO has expressed fears that the deal would allow film studios to gain control of pay television and increase the price of their movies to the home viewer.

Viacom International, which has acted as spokesman for the proposed new company, said it had no comment on the Justice Department announcement.

Showtime, the number two pay TV service, has about 4 million subscribers, while The Movie Channel, number three, has about 2.5 million. Together, their subscribers amount to slightly more than half of HBO's viewers. Under the merger plan, Showtime and The Movie Channel would be run under one management, although they would continue as separate 24-hour-a-day services.

Acting Deputy Assistant Attorney General Wayne D. Collins said the Justice Department was concerned over the relationship between the film distributors and the companies that supply movies to cable TV stations.

The merger "would pose a serious threat to competition in the licensing of motion pictures to pay programming services" and could add to the loss of competitiveness among motion picture distributors who have joined in the deal, Collins added.

The partners in the joint venture felt it could pass Justice Department muster because it would not have exclusive rights to films produced by the studios who are its part owners. In an earlier case three years ago, the government went to court to block a plan by three movie studios and Getty Oil Co. to form a pay TV firm that would have had exclusive rights to films produced by its owning studios.

In both cases, the companies were trying to mount a challenge to the phenomenally successful HBO operation, which recently joined with CBS and Columbia Pictures in a film production partnership. The Justice Department is looking into its antitrust implications.

In deciding to file a civil antitrust suit against the Showtime-The Movie Channel joint venture, the Justice Department followed the recommendation of staff attorneys in its antitrust division. Collins said this is the first time the Reagan administration has found vertical relations to have anticompetitive aspects. No date has been set for filing the suit.