The backers of the two major pay television competitors of Home Box Office, the industry leader, today announced plans to join forces in what promises to test both HBO's dominance of that market and the federal antitrust policy toward video technology.

In an annoucement that startled the cable television business, MCA Inc., Paramount Pictures, Warner Communications Inc., American Express Co. and Viacom International Inc. announced preliminary plans to set up a joint venture to operate both The Movie Channel and Showtime.

The move is the latest development in a mad rush by the pay and advertiser-supported television networks, the major film studios and the cable television business to carve joint ventures that link motion picture production with distribution networks. Lurking behind these ventures is the prospect of pay per view distribution of programming, in which subscribers would pay fees for individual showings of films, for instance, rather than monthly subscription fees.

Viacom now operates Showtime, which is the second leading pay service with four million subscribers, while The Movie Channel is a Warner-American Express venture with 2.3 million subscribers into which Paramount and MCA recently joined as partners.

HBO, which dominates the pay television business and is playing an increasingly important role in movie production, has 11 million subscribers. In a statement, an HBO representative said the company is "concerned" that the deal is part of an effort by the studios involved to "gain control of the future development of pay television and thereby increase the prices for their movies to the cable operator and the television viewer."

Meanwhile, CBS Inc., HBO, a Time Inc. subsidiary, and Columbia Pictures, a Coca Cola unit, recently formed their own partnership to produce films for a variety of distribution purposes.

The Justice Department is looking at the two earlier deals and presumably will merely add the newly announced linking of Showtime to The Movie Channel as it attempts to evaluate, perhaps for the first time, the competitive posture of the newly emerging pay television marketplace.

The new joint venture will operate the two pay television networks as separate 24 hour a day services, the firms said. Programming will be licensed from the "entire creative community." Theatrical films will be available to all pay services on a "non exclusive basis," the companies said.

The four entertainment companies will have equal shares in the new venture, while American Express will have a smaller interest. Viacom, in addition to remaining a partner in the venture, will receive a payment of about $75 million once the deal is completed.