The New York Mercantile Exchange agreed yesterday to pay a $200,000 civil fine for failing to enforce its own rules for commodity trading.

To settle charges filed by the Commodity Futures Trading Commission, the exchange also signed a consent order promising not to be lax again and to tighten its enforcement.

The settlement ended the exchange's part in a CFTC investigation of irregularities in silver and gold trading during 1977, 1978 and 1979.

The federal commodity regulators still are considering possible action against 43 Mercantile Exchange members who were charged last August with violating the exchange's rules.

The commodity traders allegedly arrange in advance to buy and sell contracts for gold and silver. Under rules governing commodity markets, all transactions are supposed to be conducted in an open auction on the floor of the exchange with competitive bidding.

The prearranged trading apparently was done in part for tax purposes, to allow traders to set up artificial losses that could be deducted from their income taxes.

Under terms of the settlement, the Mercantile Exchange neither admitted nor denied it had failed to enforce its rules or supervise its members. Officials of the exchange said that they agreed to pay the fine and settle the case to avoid a protracted dispute with the CFTC.