Virginia state Sen. Peter K. Babalas of Norfolk asked a Senate ethics panel today to investigate allegations that he improperly voted on legislation affecting a troubled investment firm that paid him as much as $8,000 a month in retainers.

Babalas, a Democrat and member of the legislature since 1968, becomes only the second sitting Virginia state senator to face a formal inquiry into his business and legislative relationships. He will be the first to face the ethics panel, which was established in 1983.

Babalas, 63, has been the subject of recent articles in The Virginian-Pilot that detail his work for Landbank Equity Corp., a Norfolk firm now in bankruptcy proceedings and under investigation for fraud. The firm also is the subject of suits in five states in connection with the sale of about $178 million in high-interest, second mortgages to banks and savings and loan associations.

In the 1985 session of the legislature, Babalas cast his vote and a proxy in committee that killed a bill that would have placed second-mortgage companies such as Landbank under greater state regulation.

Senate President Pro Tem Edward E. Willey, a Richmond Democrat, whose proxy against the regulation was cast by Babalas, told United Press International today that "had I been Sen. Babalas and had a private interest in the bill, not only would I refrain from voting on it, but I would not vote my colleagues on it."

Willey said Babalas "made a bad judgment voting himself and myself" because the disputed bill would have died on a 5-to-5 vote without Babalas' maneuver.

Babalas, a lawyer, previously has said he did not discuss a retainer with Landbank until after the vote and had properly disclosed being paid about $1,500 to write letters on behalf of the firm in the fall of 1984. He could not be reached for comment today.

The president of Landbank, appearing in bankruptcy court in Norfolk Monday, said she was mistaken when she initially testified last week that Babalas discussed the retainer in the fall of 1984.

Landbank's president, Marika L. Runnells, amended her testimony today and said the firm did not discuss the Babalas retainer until this past spring -- after Babalas' vote. He was paid $8,000 a month for five months.

The timing of the retainer is important under Virginia's conflict-of-interest laws. Under the law, legislators are barred from voting on bills in which they have a direct personal interest in which they receive or expect to receive $10,000 a year.

Babalas was paid about $1,500 last fall to write letters on behalf of Landbank to an insurance firm that was questioning Landbank's financial dealings; but that fell well under the $10,000 level, Babalas has said.

"I request that appropriate steps be taken . . . ," Babalas said in a letter to state legislative officials today. The letter will trigger the first session of the Senate Ethics Advisory Panel, whose members have never met to elect a chairman or establish rules.

Under the law, the panel must complete its work in four months. It can recommend no action, refer the case to the Senate Privileges and Elections Committee for investigation or refer the case to the state attorney general's office.

Legislative officials said today it would be unlikely the panel would hold even an organizational meeting until January.

Members of the panel include former governor Linwood Holton of McLean and Albertis Harrison, a retired state Supreme Court justice, as well Robert Fitzgerald, an attorney and former state senator from Fairfax, and Raynor Sneed, a circuit court judge in Rappahannock County. A fifth seat is vacant.

The panel was established after a dispute involving state Sen. Nathan H. Miller of Harrisonburg, the 1981 GOP candidate for lieutenant governor. Miller was accused of improperly acting on legislation that benefitted his clients, including the state's electric cooperatives. Miller, who lost the election, was investigated by the Senate Rules Committee and the Senate Privileges and Elections Committee. They determined Miller should have disclosed his interest but took no action against him.