The Justice Department yesterday challenged plans to create a private self-regulatory group to police the nation's commodity markets, contending the proposal "might well inhibit vigorous competition."

The Justice Department response was a stunning and unexpected setback for the Commodity Futures Trading Commission and the commodity industry, which have been working for five years to set up such an organization.

Called the National Futures Assn. (NFA), the group's job would be to oversee commodity trading practices in much the same way that the National Association of Securities Dealers polices stock brokers.

One of the leading advocates of the National Futures Associations has been Philip Johnson, the Chicago commodity lawyer who was chosen by President Reagan to be chairman of the CFTC. Because he helped draft the proposal, Johnson apparently will have to disqualify himself from deliberations about it.

Congress has authorized the CFTC to charter a self-regulatory organization. The private group's goals are to simplify federal regulation and at the same time establish some controls over futures brokers who are not subject to regulation by either the CFTC, state authorities or the commodity exchanges internal rules.

After years of haggling and rejection of its first proposal, the industry brought its NFA plan to the CFTC a few months ago. The CFTC endorsed the NFA and asked for comment from private and other government sources.

In a 32-page response delivered last yesterday, the Justice Department "strongly urged" the CFTC to reject the NFA proposal.

While NFA's objectives may be laudable in the abstract," the Justice Department's Antitrust Division said, "in practice such an association might well inhibit vigorous competition in the commodity futures trading industry."

The Antitrust Division said its major objection was that membership in NFA would be mandatory. Compulsory membership "seems both unnecessary and inappropriate," the agency said.

The NFA constitution creats a complex board of directors that would be largely dominated by the major commodity exchanges and their members. The Justice Department said that might give the exchanges the chance to work together to eliminate competition.

"We believe these markets may complete with one another," the Justice memo noted. "Permitting such an all-encompassing grouping may well eliminate a vital and dynamic force in this industry."

Aside from its objection to mandatory membership, the Justice Department also questioned whether the NFA would serve the Reagan administration's goal of reducing regulation.

"Industry self-regulation may be preferable to direct governmental intervention in the affairs of the private sector," the Antitrust Division said. "Here, however, a pervasive regulatory scheme already exists, and NFA's activities will add to rather than lessen that regulation."

Before creating a whole new layer of regulation, the CFTC ought to decide whether present rules are "failing to maximize consumer welfare," Justice suggested.

If new regulation is necessary, Justice added the CFTC "must use the least anticompetitive means available to achieve its goal. . . . NFA's application does not meet this test."