The International Brotherhood of Teamsters was declared the winner yesterday of a hotly contested election to represent the 2,300 employes of the District's troubled prison system.

The Teamsters defeated the American Federation of Government Employees, AFL-CIO, by a three-vote margin, arbitrator Arnold Ordman of the American Arbitration Association ruled after opening 23 of 32 challenged ballots that had left the election outcome in doubt. The vote was 764 to 761.

The victory by the 1.9-million-member Teamsters, the nation's richest and most controversial labor union, marks the first time the union has succeeded in winning representation of Washington-area government workers.

The union already represents police, corrections officers and other public employes in other parts of the country, and also represents Metro transit and security police.

The Teamster victory also introduces a new element to the current controversies over D.C. Jail overcrowding and other prison conditions because the union has pledged an aggressive stance in pushing Mayor Marion Barry's administration for prison improvements and better, safer working conditions, which it contended AFGE had failed to pursue.

Donald H. Weinberg, chief of the D.C. Office of Labor Relations and Collective Bargaining, said he expected the city Public Employee Relations Board would shortly certify the Teamsters as the official bargaining agent for the Department of Corrections because there appeared no legal grounds for denying certification.

AFGE organizing director David Kushner described the balloting as a "well-run election . . . . This is the choice of the employes who voted." But he declined comment on whether the union would appeal the outcome to PERB or to the courts.

AFGE had represented the District's corrections department workers for three decades.

Corrections officers, whose base pay ranges from about $15,000 to $21,000 plus overtime, have the lowest salaries among area law-enforcement employes and were deeply dissatisfied with what they saw as AFGE's failure to aggressively pursue improvements, AFGE officials acknowledged.

More than 1,000 employes petitioned PERB to leave the union last year, prompting the union election.

AFGE tried to portray the Teamsters as corrupt by pointing to past criminal convictions of union officials, and by highlighting Teamster President Jackie Presser's $500,000 salary, $550,000 union-paid condominium, and other perquisites. But a majority of frustrated prison officers believed that the Teamsters could represent them more effectively, AFGE officials acknowledged.

"We are just elated, and we will be preparing to negotiate with the city," said Ernie V. Jumalon, secretary-treasurer of Teamster Local 246.

Regarding the Teamster image as a tougher and more aggressive union, Weinberg said, "I don't know about rougher and tougher. You deal with issues and prepare your case for negotiation. One union may present more difficult demands than another, but I am not presuming anything. Both unions have dealt with employes who do a very difficult and sensitive job . . . and get very frustrated."

The election outcome had been in doubt following balloting on Tuesday because the names of 32 employes who voted had been left off the master voter lists distributed to both unions.

The Teamsters led 752 to 750 before the 32 votes were counted.

AAA, which supervised the election, disqualified nine voters because they were not in union-eligible jobs, or were not on the payroll as of Feb. 8, the date the election was ordered. Of the 23 ballots opened yesterday, the Teamsters received 12 votes to AFGE's 11.