A group of Northern Virginia investors said yesterday it has settled a lawsuit and will shelve its efforts to oust the board of HBO & Co., an Atlanta hospital computer firm, after losing a crucial shareholder vote.

The Andover Group, which had been trying for nearly a year to gain control of HBO, said it was backing off because HBO's board agreed to seek a buyer for the firm.

Andover had tried unsuccessfully to vote out HBO's board. After a series of legal maneuvers the Andover candidates for the board lost during an election at HBO's June 3 stockholders' meeting. HBO had tried to delay the meeting, but Andover obtained a court order forcing the vote.

Under the terms of a new agreement revealed yesterday in a filing with the Securities and Exchange Commission, Andover said it would dismiss its pending litigation against HBO, and will not "conduct or encourage or participate in any proxy contest for the election of directors" until June 15, 1988.

Andover also agreed not to acquire any more stock in HBO for six months, except in connection with a complete takeover of the company. The group holds about 2 million shares, or 8.7 percent of HBO stock, according to Austin E. Kearney, HBO's assistant treasurer, and papers filed with the SEC. Its offer for the company of $15 a share last August was rejected by HBO's directors.

HBO's stock, which is traded over the counter, closed yesterday at $13 a share, up 25 cents, on volume of 396,400 shares.

The hospital information systems company is one of the nation's largest. Ronald V. Aprahamian and Ronald L. Holden, both former executives of Compucare Inc. of Reston, a hospital data processing firm that was sold in 1985, control Andover.

"The company is proceeding to work with Goldman, Sachs {& Co.} to find a buyer and maximize shareholder value," said Kearney. HBO, which is not connected with Home Box office television network, formed a committee in April to study restructuring through sale or recapitalization under pressure from shareholders.

"If our lawsuit had still been pending, the directors would have been seriously hampered from selling the company," said Aprahamian.

Aprahamian said Andover dropped its litigation because HBO's directors "switched their position and adopted our program" to sell the company. "The commitment they have is to sell the company as soon as possible," he said.