The Reagan administration endorsed yesterday a long-debated plan to expand the Metro subway system to 89.5 miles, including a Green Line branch to Greenbelt in Prince George's County and a Yellow Line spur to a Van Dorn Street station in Alexandria's West End.

In a letter to the transit authority, Ralph L. Stanley, the administration's mass transit chief, said that federal officials had approved the plan "in principle" and are prepared to "move forward with federal funding of the construction program envisioned" by Washington area officials as soon as terms of a contract are worked out.

The administration previously has limited Metro subway construction to 76.4 miles, sparking protests from Metro's advocates. The 89.5-mile plan was adopted by county and city governments last year in an attempt to persuade the administration to lift its construction limit.

Metro's long-range plans call for 102.7 miles. But transit officials have concluded that no more than 89.5 miles can be completed with funds currently authorized by Congress. To extend the rail network to 102.7 miles, officials say, they will have to seek another $2 billion from Congress or other sources.

The administration's action was hailed yesterday by Prince George's officials, who have long pressed for construction of the Greenbelt route. "It's a significant step in the right direction," said County Executive Parris N. Glendening. "For us, that means the Greenbelt line, and we have put an immense priority on that."

Rep. Steny H. Hoyer (D-Md.), who spearheaded a congressional move last year to force the administration to lift its 76.4-mile limit, described Stanley's action as a "very positive sign." An aide said Hoyer now expects federal officials to release construction funds that were withheld last year because of the limit.

Gladys W. Mack, chairman of Metro's board of directors, responded cautiously, saying she needed to discuss the issue with other board members. "I'm going to wait for some kind of analysis," said Mack, a general assistant to D.C. Mayor Marion Barry. She termed Stanley's action an "initial step."

The 89.5-mile plan includes two key extensions outside the previously approved 76.4-mile network. One is the branch between Greenbelt and Fort Totten in Northeast Washington, scheduled to open in 1992. The other is the section expected to open in 1990 between Alexandria's King Street and Van Dorn Street stations.

The rail system now encompasses 60.5 miles, and several major extensions are scheduled to be completed by 1990 as part of the 76.4-mile plan. The Orange Line is to be extended to Vienna in Fairfax County in mid-1986. Also under way are a Red Line section between Silver Spring and Wheaton and a long-delayed Green Line section between Anacostia and Northwest.

The Reagan administration has indicated it will oppose further federal appropriations for Metro construction beyond 89.5 miles, and Metro officials have not devised a plan to obtain funds to complete the proposed 102.7-mile system.

The rail sections for which no funds have been set aside include a proposed Glenmont terminus in Montgomery County, a planned Franconia-Springfield terminus in Fairfax and two major parts of the Green Line, one connecting a U Street station in Northwest with Fort Totten and the other linking Anacostia with a proposed Branch Avenue terminus in Prince George's.

One key issue expected to be a focus of negotiations among federal and local officials is a proposal by Stanley to conclude what is known as a "full funding contract." In his letter, Stanley described this form of contract as a "necessity."

Under such a contract, Metro would be required to complete the proposed 89.5-mile system with the $960 million remaining in its current congressional authorization. If construction costs exceed the authorized amount, Metro would be required to finance the overruns with local or other nonfederal funds.

Although Washington area officials have expressed concern about this proposal, key Metro officials said recently a "full funding" pact may be acceptable if it contains adequate safeguards against effects of a possible recurrence of severe inflation and other unforeseen problems.

In the past, Metro's construction costs have outstripped initial estimates by billions of dollars. The proposed 102.7-mile system is now expected to cost more than $9 billion.

Stanley appeared to soften his proposal by urging Metro to draft its own version of a "full funding" contract for consideration by the Urban Mass Transportation Administration. He said that "some modification and variation from the 'standard' full funding contract will be necessary" because of Metro's "unique" situation.

The Office of Management and Budget cleared the way last November for Stanley to approve the 89.5-mile plan by recommending federal spending levels of $250 million a year for Metro over the next four years. Stanley's action appeared timed to coincide with congressional hearings on Metro this week and next.