Balloting to determine which of three labor unions will act as bargaining agent for rank-and-file D.C. corrections department employes ended inconclusively last night as three competing labor unions failed to win a majority of employe votes cast.

As a result, a runoff election between the two top finishers -- the American Federation of Government Employees and the International Brotherhood of Teamsters -- will be held within the next few weeks, union and city officials said. The AFGE, which has represented corrections workers here for 30 years, received about 48 percent of last night's vote, while the Teamsters won about 29 percent.

The third contending union, the Fraternal Order of Police, received about 23 percent of the vote.

Officials said AFGE won 710 of the 1,535 votes cast, compared to 425 for the Teamsters and 333 for the FOP. They said 57 votes were disallowed, and 10 ballots were marked "no union."

Turnout yesterday was described as heavy with about 70 percent of the department's 2,200 officers and civilian workers casting ballots at D.C. Jail and Lorton Reformatory.

Last summer, more than half the city's corrections workers petitioned the Public Employees Relations Board for an election that would allow them to choose the FOP as their bargaining agent instead of the AFGE, which they claimed had not worked hard enough with the D.C. government for improvements in pay, benefits and working conditions.

The FOP represents the city's rank-and-file police officers.

The petition sparked a challenge by Mayor Marion Barry, who sought to prevent yesterday's election because the city believes that the same union should not represent both police and prison employes. The city argued that such an arrangement might lead to a conflict of interest if corrections officers were to go on strike and police officers were called in to replace them, as happened during a wildcat strike in 1981.

The Teamsters also filed a petition to take part in the election, after which the PERB overruled the mayor's protest and ordered yesterday's three-way balloting.

In another strongly contested representation campaign, ballots will be counted Friday to decide who will represent the city's 3,500 police officers, who are voting by mail.

That election, too, pits the FOP against the AFGE and a third contestant, the National Association of Government Employees/International Brotherhood of Police Officers.