A congressional task force will recommend to the House Administration Committee Wednesday that it dismiss Democrat Antonio B. Won Pat's challenge to the 1984 Guam election results that cost him his House seat, two task force members said yesterday.

Staff members yesterday noted that the full committee rarely rejects a recommendation by an election task force.

Rep. Robert E. Badham (R.-Calif.), one of three congressmen on the task force, said the panel last week voted unanimously to approve a motion to dismiss the complaint. The motion was filed by Republican Ben G. Blaz, the current nonvoting delegate from Guam who was declared the winner last year.

"We felt there was just not sufficient grounds for sustaining the contention that a challenge exists," Badham said in a telephone interview.

Another task force member, Rep. Samuel Gejdenson (D-Conn.), said from Connecticut that although Won Pat's complaint "was not frivolous," it "was not significant enough for Congress to get involved."

Blaz, 57, a former Marine brigadier general, was declared the winner by the Guam Election Commission last November by about 350 votes. Until that election, Won Pat had been the only delegate to hold the nonvoting seat from Guam.

After the election, Won Pat accused the territory of conducting an election fraught with "substantial irregularities." Lawyers for the 77-year-old Guamanian said the election problems included a power outage during the vote counting, lost and altered ballots, delayed absentee ballots, and confusing election forms.

Won Pat also said Blaz had failed to capture a majority of the votes. Unlike most other congressional elections, which require the winner to receive a plurality, federal law states that the delegate from Guam must receive a majority.

In oral arguments before the task force last month, Blaz's lawyer, Lawrence Halloran of the National Republican Congressional Committee, said Won Pat's complaint "completely failed to meet the threshold requirements" of the federal elections laws. He also alleged that the complaint had been filed too late.