The Metro board broke a weeks-long deadlock yesterday and tentatively decided how the Washington area will pay for the estimated $21 million more it will cost to operate the subway next year than will be collected in fares.

The decision, negotiated by board members during a recess in the back room, came after public debate ended in a deadlock. After the recess, board members, all smiles, returned to the meeting room and unanimously approved a three-year formula to apportion the subway operating deficit among the area's jurisdictions.

Final approval of that or some other formula must come quickly if Metro is to expand its subway operations to a new 12-mile line running from RFK Stadium through downtown to National airport.

The scheduled date for opening that line is July 1. If financial questions are not solved quickly -- including such puzzlers as how much the riders of the subway should pay to board and thus how much the various area jurisdictions must subsidize -- the computerized fare collecting equipment will not be programmed and the system simply won't open.

May 19 is about the last date that leaves Metro enough time to get the fare program into all the computers for a July 1 opening, Metro General Manager Theodore C. Lutz said yesterday. There is some flexibility there, but not much, he said.

With yesterday's action, both a proposed fare schedule and a formula for dividing the deficit now have tentative Metro board approval. Board members will not vote final approval on either of those matters, however, until the area jurisdictions also review and indicate approval of the formulas.

Formal approval from the area's jurisdictions is not technically required before the Metro Board can take final vote. As a political matter on something as sensitive as transit subsidies, however, board members will make sure they have approval from their local jurisdictions before making final commitments. "The board is a child of the local jurisdictions," as member Joseph S. Wholey said yesterday.

Metro board members voting approval of the deficit formula yesterday included representatives from Montgomery, Prince George's, Fairfax and Arlington counties and the D..C mayor's office.

The formula is a complicated process that charges area governments for the Metrorail subsidy based on how many stations are located in their jurisdiction, how many people live in their jurisdiction, and how many of their residents ride the train.

The haggling has come over such seemingly arcane points as how many people might reside in one part of Prince George's County five years from now or whether Arlington should be penalized because the construction schedule puts subway stations there months before Montgomery County gets them.

The District was particularly anxious to get a longer-term agreement. Because D.C. will have most of the Metro stations in the early years and has heavy population and ridership, it will pay a larger share of the deficit --more than half in the first year. As the system extends into the suburbs, the District's share will decline.

The final tentative agreement was struck because "I think everybody disliked it enough," said Douglas Schneider, director of the D.C. Department of Transportation.

Francis W. White, Metro board chairman and a member of the Prince George's County Council, said the formula adopted yesterday "goes a long way toward solving the Prince George's County problem." All officials contacted said they believed that most local jurisdictions would finally approve the proposal.

The percentage of the total operating deficit that each jurisdiction would pay shifts slightly in each of the three years as construction on the system continues. The District of Columbia, for example, drops from 55.03 per cent in fiscal year 1978 to 48.55 per cent in fiscal 1980; Prince George's increases from 10.76 per cent to 14.76 per cent.

In the first year under the formula, subsidies from each jurisdiction would be as follows, assuming a $21 million deficit D.C., $11.6 million; Montgomery County, $2.1 million; Prince George's County, $2.3 million; Arlington County, $2.4 million; Alexandria, $827,000; Fairfax County, $1.7 million; Falls Church, $44,100, and Fairfax City, $92,400. (NOTE: Total exceeds $21 million due to rounding.)

When the board met yesterday, five alternatives and one basic formula were on the table. After considerable discussion, one alternative was voted down. Maryland representative Carlton Sickles called for a recess and board members retreated to the back room.

There they agreed on a proposal "It was time for everybody to understand everybody else's problems so they can go back to their home folks and say honestly that this agreement is what is acceptable," Sickles said later.