An attempt to shift the route of Metrorail's Green Line from its planned terminus at Rosecroft Raceway north through Suitland along Branch Avenue was defeated yesterday by the Prince George's County Council.

The alternative alignment, proposed by council member Sue V. Mills, failed even though the vote was 4 to 3 in its favor. Five votes were needed for passage. Two members abstained from the vote.

"Have a funeral for Metro, that's it, it's done," said Mills. She explained that she feared continued opposition from supporters of the more northerly route, combined with the Reagan administration's unwillingness to fund the entire proposed 101-mile system, ultimately may doom any Metro service to southern Prince George's.

As the result of yesterday's action, the last hope for Branch Avenue supporters may be a challenge in federal court in Baltimore. Judge Norman Ramsey already has blocked Metro from further work on the Rosecroft line until he hears arguments about whether the route decision was made properly. Last November the Metro board reaffirmed its commitment to the Rosecroft line.

Before the outcome of yesterday's tense vote in Upper Marlboro was known, Mills laid her head on the council table and shed tears of joy when council member Anthony Cicoria, whose position had been a secret, voted to move the line to Branch Avenue. But a minute later her platinum bouffant hit the table in disgust when council member Hilda Pemberton, whose constituents would be most served by the Branch Avenue line, abstained from voting, as did council vice chairman Floyd Wilson.

After the meeting, Pemberton said she still supports the Branch Avenue alignment, but believes the controversy, which has been going on for 14 years, should be resolved in federal court.

Lorraine Sheehan, Maryland's new secretary of state and a Branch Avenue supporter, charged that Pemberton and Wilson "wanted to take a walk" to avoid taking a stand on the issue. Sheehan further charged that Peter O'Malley, one-time Democratic political strategist and Rosecroft lawyer, and remnants of the erstwhile Prince George's Democratic "machine" were responsible for the abstentions that killed Mills' bill.

"Mrs. Pemberton has been under a lot of pressure from Mr. O'Malley and Mr. State Sen. Tommie Broadwater--I guess she just caved in," said Sheehan, who vowed to continue the fight from her office in Annapolis, if necessary by holding up road projects necessary for the Rosecroft line.

Pemberton denied that she had been pressured to vote against the Mills resolution and said that she did not know O'Malley.

"This is a decision I made all alone. It was a very difficult one. I'm sure it's not going to be a very popular one," said Pemberton, who was elected to the council last fall with strong support from Democratic party regulars.

Metro officials say the Rosecroft line would cost $110 million less to build, but would displace more homes, cost slightly more to operate and would serve 10,000 fewer people than a Branch Avenue alignment.

Wilson declared that the issue was not one of costs and ridership projections but about the image of the council, which voted for Branch Avenue in 1978 but switched to Rosecroft one month later.