President Carter ordered an immediate halt to the nation's crippling three-day-old rail strike yesteday after union and railroad negotiators failed to resolve their dispute in 26 hours of government-mandated bargaining.

But the Brotherhood of Railway and Airline Clerks (BRAC), whose picketing has shut down two-thirds of the country's rail service, refused to call its striking members back to work and was in the process of expanding the walkout late yesterday. The union said it will return to work only if the government guarantees to protect members from company reprisals.

Industry officials said they were preparing to seek a federal court injunction orderings all strikers back to work immediately.

There was no indication when the walkout, which has snarled freight and passenger traffic from coast to coast and is threatening massive job layoffs as well as crop losses, willend.

Officials said the government is prepared to go to court immediately at the first sign of resistance to the order. Even after the strikes return, it would take four days to get the railroads back to full operation, industry officials said.

In using his emergency powers to impose a 60-day "cooling-off" period during which a special board would recommend a settlement of BRAC's strike-causing dispute with the Norfolk and Western Railway, Carter said he will take legal action if necessary to force the strikers to return to work.

"This will take the railroad workers back to the job," Carter told a news conference after Labor Secretary Ray Marshall reported failure of the "last chance" bargaining effort and mediators urged that the emergency provisions of the National Railway Labor Act be invoked.

"If there is opposition to this action, then I would not hesitate to go to federal court to enforce it, " Carter added, noting that the strike has caused an "almost complete shutdown of rail service" in the country.

The strike was triggered by a two-year-old dispute between BRAC and the Virginia-based N&W over union jurisdiction and automation protections. It was confined to the N&W until Tuesday when BRAC, claiming that assistance from other railroads was helping the N&W weather its strike, spread picketing activity to 43 other railroads, crippling service in 42 states as other rail workers honored the clerks' picket lines.

Asserting that the nation's economic health was imperiled by the walkout. Marshall summoned top officials of BRAC and the N&W to the Labor Department Wednesday, appointed a special mediator and gave the two sides until noon yesterday to resolve the dispute without government intervention.

Despite reports of some progress as the talks continued through the night with only a short dinner break, the negotiators remained at loggerheads as the noon deadline approached, prompting a last-minute personal mediating attempt by Mrshall.

By 1:30 p.m., the labor secretary had to concede defeat. He reported that the National Mediation Board, which handles railway and airline labor problems, was recommending creation of an emergency board, a move that automaticallly triggers the 60-day "cooling-off" period.

Implicitly rejected by Carter and Marshall in opting for the emergency board was seeking special legislation from Congress to end the strike, an option outlined earlier by Marshall that has often been used in rail disputes. Marshall aides said congressional leaders, bogged down in an end-of-the-session logjam and uncertain that a national emergency exists, were discouraging the idea of a legislated strike end.

BRAC President Fred J. Kroll issued a statement shortly after Carter's press conference saying the union "will determine its course of future action on the basis of the nature of the governemnt's guarantee of a variety of protections for all railroad workers on the N&W."

An aide said Kroll was not sending BRAC members back to work but said he would do so if the government could guarantee no retribution against strikers by the N&W.

Marshall told reporters after Kroll's statement that "the Labor Department's interpretation is that the Railway Labor Act protects workers against retribution . . . Our view is that they are protected." There was no immediate response from Kroll.

BRAC has also raised questions about whether the emergency powers of the act, particularly the president's powre to order an end to the strike, apply to the N&W dispute. The emergency provisions normally come into play before a strike occurs.

But Marshall said department lawyers believe the president's power to act under the law is clear. "We think it applies, and we therefore expect to do what we have to do to enforce it," said the secretary. If there is evidence that the order is being ignored, he added, "our lawyers will be in court as fast as they get such evidence."

Under the law as interpreted by the administration, workers must stay on the job for 30 days while the emergency board prepares its report and for another 30 days after. If there is still no agreement after 60 days, the union is free to strike, although Congress has often acted to impose a settlement, often encompassing the board's recommendations, to avoid a strike.

The board was appointed yesterday and is expected to start work by Monday, according to Marshall. Its three members are Paul H. Hanlon, of Portland., Ore., Jerre S. Williams of Austin, Tex., and Jacob Seidenberg of Falls Church, Va., all of whom are professional arbitrators. Hanlon wil serve as chairman.

Both Carte and Marshall asserted that the issues dividing the 235,000 member BRAC and the N&W, the nation's seventh-largest railroad, had narrowed considerably and were not insurmountable.

"I think it is accurate to say that both sides do want a settlement," said Carter. "The differences between them are relatively small compared to what they were originally."

But BRAC legislative counsel James Kennedy said the progress had been exaggerated and accused the railroad of trying to void a negotiated settlement and trying to "destroy our union and all of railway labor."

This week's strike was the closest thing to a nationwide rail strike since 1971 when Congress imposed a settlement to end a two-day strike by signalmen that was causing major disruptions. The last time that emergency provisions of the Railway Labor Act were used in a rail dispute was when then-President Ford invoked the law to avoid a BRACK strike in 1975.