A federal judge in Houston yesterday took the extraordinary step of ordering a public television station to air a program -- "Death of a Princess" -- that had been canceled after protests from the Saudi Arabian government.

U.S. District Court Judge Gabrielle McDonald ruled that the Texas decision to cancel the program on state-owned KUHT was politically motivated and constituted government censorship.

She issued a temporary order, which station lawyers said late yesterday they would appeal. A spokesman for KUHT said the station would comply with the order as long as it lasted.

Lawyers contacted yesterday said they knew of no previous court decision ordering a station to run a particular program anywhere in the country. Such orders, even when viewed hypothetically, are considered potential violations of First Amendment free speech and press guarantees.

Judge McDonald's opinion, however, turned on the fact that KUHT was not just a private broadcaster but an agency of a government.

The program was originally to air Monday night in Houston and on 100 other Public Broadcasting Service affiliates. Judge McDonald's order requires the station to air it as scheduled before the cancellation.

"Death of a Princess" is a dramatization of the execution of a Saudi princess in 1977 for acts of adultery. The Saudi government has vehemently protested its showing to the State Department, just as it protested its airing in Great Britain, arguing that the program is biased, inaccurate and a slur on the Islamic people.

Despite these protests, as well as strong expressions of concern from oil companies, members of Congress, and hundreds of viewers and small contributors, the Public Broadcasting Service intends to feed "Death of a Princess" to about 100 stations Monday night.

Only eight stations had canceled in response to the protests. KUHT, located in a center of Arab-American oil activity, was one of those canceling.

After the cancellation decision, Gertrude Barnstone, a former Houston school board member, filed suit against the station. She and her lawyer, David Berg, argued that state educational TV officials canceled the program for political reasons.

Since the station is state-owned, Berg contended, the cancellation decision represented government censorship, and impermissibly infringed on the rights of viewers guaranteed by the Constitution.

Berg said yesterday he had heard of no precedent that would support an order. In fact, courts have repeatedly declined for constitutional reasons to order newspapers or television stations to publish or broadcast anything.

But McDonald ruled from the bench yesterday that "this is not a case of a privately owned and operated TV station making programing decisions.

"Here, the government, the University of Houston, is already involved in programming decisions. It was the government which decided not to program 'Death of a Princess.'

"That decision was based on the political belief of its vice president. . .

"When the government gets involved in broadcasting, it has an obligation, at a minimum, to establish procedures that assure the programming decisions are not based on the political beliefs of its programers and are not made arbitrarily and without due process of law.

"The programming decisions of state-owned and operated TV stations cannot be based on constitutionally impermissible reasons," she ruled.

"I find that the decision not to air 'Death of a Princess,'" she concluded, " was reached in violation and contrary to these constitutional proscriptions.

"Therefore, the request for a temporary restraining order is granted. The defendants are hereby directed to air 'The Death of a Princess' as previously scheduled, on May 12, 1980, at 8 o'clock p.m."

The judge's order made even sticker one of the stickiest issues ever to confront public broadcasting in this country.

The Public Broadcasting Service already faced protests from the Saudi government, relayed by the State Department, from Mobil Oil, one of public broadcasting's largest underwriters, and hundreds of 10- and 20-dollar contributors who flooded public broadcast switchboards yesterday with objections and threats to withhold further contributions.

In the face of the barrage, eight stations including Houston cancelled. Two more announced plans to delay. But the bulk of the stations and PBS were holding firm yesterday.

PBS announced plans yesterday to expand to an hour a panel discussion originally set to last a half-hour following the program. The Saudi Arabian ambassador was also invited to comment following airing of the program.

All of those protesting -- from the oil companies to the viewers -- expressed concern that America's major oil supplier might retaliate for the showing, as it retaliated against Britain by recalling diplomats.

American sources close to the Saudis speculated that retaliation was a very real possibility.

If the protests were not orchestrated by the Saudis, there was strong evidence that the Saudis stimulated many of them -- directly or indirectly.

Sources said the Saudis informed Exxon -- a partner in Aramco in Saudi Arabia -- that they expected a good faith effort from the firm to urge cancellation. Exxon, a spokesman said, expressed its concern to the State Department but did not contact TV officials.

Mobil said it did not consult Saudi Arabia before running ads in major newspapers yesterday and Wednesday urging a review of the decision to run the program. But it did inform the Saudi government of the ads, a spokesman said, just as it informed numerous other agencies, including the White House. s

Mobil and Exxon are major public-television benefactors. Both firms said, however, that their actions on "Death of a Princess" were unrelated to their support of public television.

Meanwhile, WETA in Washington and the Center for Public Broadcasting in Maryland said they were flooded with phone calls.