In the first court challenge to President Carter's anti-inflation program, U.S. District Court Judge Robert C. Belloni today denied a paperworkers union's request for a preliminary injunction against adminstration efforts to encourage compliance with wage-price guidelines.

A group of 11 Republican members of Congress-including Sen. H. John Heinz III of Pennsylvania and Rep. Jack Kemp of New York-had filed a friend-of-the-court brief agreeing with a claim by the Association of Western Pulp and Paperworkers that the president had no legal authority to order what the union claimed are in effect mandatory wage controls.

Belloni, however, ruled the 22,000 member union had not exhausted its administrative remedies in its challenge to a finding by the Council on Wage and Price Stability that union demands for a 21 percent wage hike over two years violate presidential guidelines.

Government attorneys said the council would agree to hold a hearing to determine whether a "tandemrelationship" exists between paperworker wage settlements earlier this year and contracts still being neg-for the 21 percent increase. Traditionally, earlier contracts provide the pattern for later ones. A council decision that the relationship did not exist prompted the lawsuit.