Hostility between the city and Maryland suburbs broke into the open during a Metro board meeting yesterday, as Maryland members vetoed a proposed fare-increase package on the grounds that it would subsidize short-haul rail riders in the District at the expense of long-distance commuters living in the suburbs.

The meeting broke up in confusion after the Metro staff took members aside to plead for a compromise and D.C. representatives, some of them openly angry, walked out of the crowded boardroom at Metro headquarters. Discussion is to continue at a retreat in Maryland today and Saturday.

The board had been told earlier that it had to make a decision by today if Farecard machines were to be reprogrammed in time to raise fares when the Red Line is opened to Van Ness on Dec. 6. Yesterday, General Manager Richard Page said that each week of delay now might cost Metro $250,000 in lost revenues.

There is a general agreement among Metro board members that a fare increase is unavoidable, due to accelerating operating costs and the expected loss of federal subsidies. But long-smoldering resentment has flared up over how the increase will be distributed among Metro's eight jurisdictions, complicating efforts to cover growing deficits in the transit system.

The Metro staff had proposed a package raising the cost of an average rail and bus ride by about 12 percent. The board agreed with the proposed bus fares but came to an impasse yesterday over a 10-cent bus transfer charge and the formula used to compute rush-hour rail fares. The dispute pitted D.C. against Maryland, with Virginia backing the District.

The Metro staff says the transfer charge would reduce fare evasion accomplished through illicit exchange of transfers and would raise $1.3 million in new revenues.But D.C., where use of transfers is concentrated, has long refused to accept such a charge, saying it would be equivalent to raising base fares by 10 cents for transfer passengers.

Maryland strongly favors the charge. It argues that this type of fare evasion occurs mainly in the District and creates a need for extra subsidies.

From the beginning, Maryland has criticized the Metro staff's rush-hour rail fare schedule, which would charge passengers 65 cents to board a train and travel up to three miles, adding 13.5 cents to the fare for each mile after that. This makes fares for long trips higher than justified by the costs Metro incurs on those trips and short-haul fares too cheap, Maryland says.

At Metro headquarters yesterday morning, David Hartlove of Prince George's County and Cleatus Barnett of Montgomery County unveiled a new counterproposal to the staff's plan: a 70-cent boarding charge and 12.5 cents for each mile after three.

This would mean the short trips favored by D.C. residents would cost a nickel more than under the staff's plan. It would make any trip of more than eight miles cheaper than under the staff's rate. Most of Prince George's rail commuters board at the New Carrollton station, which is more than eight miles from Metro Center, according to Metro planner Robert Pickett.

Montgomery County would get no immediate benefit from the plan. But with the Red Line scheduled to open to Shady Grove late in 1983, the county apparently wants to set a precedent for favorable long-distance fares.

Virginia, meanwhile, favors a bus transfer charge, but does not insist on it. It backs the staff's rail fares -- stations far enough out to give Virginia commuters a real break under Maryland's plan will not open until 1986 at the earliest, Metro officials said.

Voting yesterday, board members lined up squarely along jurisdictional lines. A motion to approve the staff plan, but delete the transfer charge and make other minor amendments, was supported by the District's two voting members and Virginia's two.

It was opposed by Maryland's Barnett and Hartlove. To be approved, a fare increase must receive four votes in favor with at least one from each of the three jurisdictions.

The meeting then dissolved without a formal recess. General Manager Page caucused with District delegates, who angrily charged that Maryland had a history of making last-minute, nonnegotiable demands. The charged atmosphere was evident when the Rev. Jerry Moore, a board member from D.C., said bitingly to board chairman Joe Alexander, "Montgomery County just has to be stopped somewhere."

Alexander later said that he learned of Maryland's position only yesterday morning, allowing no time to work out a compromise informally. But Maryland representatives said their county governments had made their decision only this week and said District inflexibility on the transfers was also holding up accord.

Today board and staff members are to resume fare discussions at a previously scheduled retreat in Elkridge, Md. The retreat had originally been planned to discuss long-range objectives of Metro.