Conor Cruise O'Brien, historian, literary critic and stormy Irish politician was tonight named as editor-in-chief of The Observer, Britain's distinguished Sunday newspaper.

The 60-year-old O'Brien, a brilliant writer with a talent for touching off political dynamite, will direct the weekly's editorial policy. Donald Trelford remains as the paper's editor, in charge of gathering its special blend of news and commentary.

O'Brien's selection is seen by insiders as another example of the remarkable self-denying ordinance imposed by the paper's American owners, the Atlantic Richfield Company or ARCO. ARCO's chairman, Robert O. Anderson, rescued the Observer 13 months ago, promising to underwrite its losses for a term of years still undisclosed.

Anderson installed as his man on the scene Douglas Cater, a former aide to President Lyndon B. Johnson. Cater was given the title of vice chairman.

But the paper's individualistic and independent staff has not found Cater a congenial influence and Cater's poor health limited his ability to contribute. Strong pressures developed to move Cater from The Observer and they have culminated tonight with O'Brien's appointment.

Cater, who played a central role in bringing Anderson to purchase The Observer, is now expected to return to the United States to serve another Anderson project, the Aspen Institute of Humanistic Studies.

This palace coup at a historic British institution which first published in 1791, will further confound the cynics who thought ARCO bought the liberal paper to make it an organ for oil industry views. In fact, staff members agree that ARCO has kept its hand completely away from editorial direction. O'Brien, a tempestuous man of fierce independence, is an unlikely candidate to change this pattern.

More realistic analysts have understood that ARCO bought The Observer partly because of Anderson's wide ranging personal interests and partly because it guarantees ARCO licenses in the oil-rich North Sea.

ARCO refines more crude than it produces and therefore seeks new sources. Britain's energy ministry favors firms with British ties in awarding licenses to exploit tracts in the North Sea. Shortly after ARCO bought The Observer, it won its second North Sea license.

Editor Trelford said tonight "I am delighted to welcome" O'Brien-abroad. He is exactly our sort of man and the nearest thing to a modern George Orwell."

The team of O'Brien and Trelford will give The Observer a structure more like that of an American than a British paper. It is common in the United States but rare in Britain for one editor to direct the editorial and comment pages and another to be in charge of everything else in the paper.

O'Brien was an Irish diplomat from 1944 until 1960. Then he went to the Congo as a special representative of the United Nations Security General in an effort to prevent the secession of the mineral-rich Katanga province.

O'Brien won few marks for diplomacy but his eventual book, "To Katanka and Back," is a classic account of cynical, big-power maneuvers to grab minerals and undermine the U.N.

After a teaching spell at New York University, O'Brien turned to Irish politics and served as minister for posts and telegraphs in the coalition government that was beaten last June. He infuriated his constituency by insisting that Irish Catholic Nationalists have traditionally ignored the feelings of Ulser Protestants and perpetuated a bloody and mythic cult.