Britain's state-owned railroads today ordered almost 25,000 engineers who have been on strike since July 4 to return to work by next Tuesday or face firing when the nation's rail system is shut down the next day.

The announcement by British Rail forces a major test of strength between the leadership of this heavily subsidized industry and the powerful labor unions that work in it. The confrontation could have a profound impact on the Thatcher government's determination to improve productivity in nationalized industries, a key element in reviving Britain's economic vitality.

"This is one more painful chapter in the modernization of Britain," an aide to Prime Minister Margaret Thatcher said today, offering the government's full support for British Rail's decision to take drastic action.

The Associated Society of Locomotive Engineers and Firemen (ASLEF) called the strike to protest British Rail's plans to introduce flexible work schedules instead of the traditional eight-hour day. ASLEF contends that "flexible rostering," as the change is called, would mean a loss of jobs.

British Rail has attempted to maintain service throughout the 11-day strike, mainly relying on members of the larger National Union of Railwaymen (NUR), which is not on strike. But only a fraction of the usual rail traffic is running.

The British Railways Board decided yesterday, therefore, to order cancellation of the existing ASLEF contract and dismissal of all engineers who refuse to accept flexible rostering unless a "substantial" number return to work by Tuesday. The next day, the system will be closed down completely and pay for the 160,000 members of NUR will be reduced to the minimum wage.

If the strike lasts a week after that, a statement from the Railways Board said, consideration will be given to laying off the NUR workers. That would provoke a fierce legal dispute, as nonstriking railway employes are guaranteed weekly pay in an agreement that dates from 1919.

Announcement of these moves was postponed one day to give the government's conciliation service one last opportunity to head off the confrontation. When that failed, Labor Party leader Michael Foot, who has generally backed ASLEF in the dispute, attempted a last-minute intervention, which also was unsuccessful.

The firing of striking ASLEF members would be unprecedented for a state-owned industry. In 1980 British-Leyland, a nationalized automaker, threatened to dismiss strikers but they returned to work before the issue could be tested.

The ASLEF situation has been compared by some to President Reagan's decision last summer to fire striking air controllers. But there are important differences. The controllers were violating a federal ban against striking and virtually none was rehired. In this case, an engineer could win reinstatement by accepting flexible work hours.

Moreover, the fired controllers were replaced by military controllers and the U.S. air system continued to function, although at a reduced level. As part of its strategy to break this strike, British Rail plans to discontinue service, which it says is essential to save money. Government subsidies to the railway system amount to nearly $2 billion annually. The planned shutdown would save British Rail about $50 million a week.

The Thatcher government has no plans to bring in the military to run essential rail services, officials said, preferring to see pressure on the strikers increased by the prospect that the system will be permanently reduced as a result of the stoppage.

The government message was conveyed to Parliament by Transport Secretary David Howell. "The path ahead for Britain's railways is very dark," he said. "Vast resources are being bled away. Thousands of jobs could disappear for good."

Reaction from Britain's other labor unions is likely to build as the date for the dismissals draws closer. The secretary general of the Trade Unions Council, Len Murray, urged ASLEF to suspend the strike but also called on British Rail to postpone its productivity plans so that "serious and speedy" negotiations could begin on the issue.

Relations between ASLEF, considered one of Britain's more militant unions, and the bigger, moderate NUR are strained. Nonetheless, many NUR members find it difficult to cross an active picket line. The government and British Rail are counting on widespread public disapproval of the strike, which has caused considerable inconvenience to commuters, to bring still further pressure on ASLEF. In all, ASLEF has been on strike for 28 days this year on the rostering question.

Underlying the dispute is a general decline in the role of railroads in Britain's economy. In 1955, the last time there was a nationwide stoppage, about 48 percent of the country's freight was carried by rail. Today the figure is 14 percent. The continued decline explains worker concern over loss of jobs.

However, some important industries, such as coal and steel, still rely heavily on the rail system and the country's economy would suffer from a prolonged shutdown. There has been talk that the militant mineworkers' union might strike in sympathy with ASLEF, but a general work stoppage is thought unlikely.