A committee of citizens who oppose legalized gambling in the District and four Baptist minister yesterday challeged the decision of the D.C. Board of Elections to allow the gambling issue to be placed on the Nov. 4 ballot.

In a lawsuit filed yesterday in U.S. District Court, the antigambling forces contended that they were denied adequate time to challenge the validity of 24,100 signatures that gambling supporters had collected to get the question on the ballot.

A broader gambling initiative was rejected by voters last May. The new one would legalize daily numbers and lottery games, but does not include such activities as jai alai and dog racing, as the earlier one did.

The lawsuit filed yesterday, while directed at the board's procedural crusade by gambling opponents to keep such activity out of the nation's capital.

The suit contends that while challengers have only 10 days and eight hours a day in the Board of Elections office to research the validity of signatures, the board has 30 days and 24 hours a day in its own office to certify the validity of the same signatures. Such a procedure, the lawsuit contends, "puts on impermissibly heavy burden" on the challeners and violates their constitutional rights to equal treatment and free speech.

In addition, the antigambling forces said that while persons who circulate petitions to collect signatures are required by law to be registered voters, the board will not invalidate any signatures collected improperly. Instead, the case is referred to the U.S. Attorney's office.

Winifred R. Mundie, an attorney representing the groups who filed the lawsuit, said yesterday that if the challengers had been given extended time to challenge the signatures, and if signatures collected by unregistered voters had been discounted by the board, the gambling referendum would not have qualified for the November ballot.

Mundle is the former general counsel to the Board of Elections.

The lawsuit was filed against the board yesterday by Citizens Against Legalized Gambling and the Revs. Andrew Fowler, John D. Bussey, Raymond R. Robinson and James E. McCoy.

Last month, gambling opponents formally challenged the validity of the signatures collected to get the question on the ballot, but the board voted 2 to 0 on august 7 to certify the measure.

Attorneys for the board could not be reached for comment yesterday. The case has been assigned to Judge Gerhard A. Gesell.