An unusual meeting of U.S. and European bishops and high-ranking Vatican officials on nuclear arms ended tonight with a cryptic statement indicating that they had failed to reach agreement after two days of deliberation.

The closed-door conference represented a first step in an attempt by the Vatican to build a consensus on nuclear weapons and disarmament.

The meeting, called to discuss a controversial pastoral letter on nuclear weapons being drafted by American bishops, was attended by 32 high-ranking prelates and several Vatican experts. A number of Western European prelates reportedly have taken exception to the American letter, parts of which also have been strongly criticized by the Reagan administration.

The closing statement included a pledge "to continue a dialogue designed to better understand the issues posed by violence and the threat of aggression," an indication that whatever differences brought the churchmen together had not been fully resolved.

Msgr. Daniel Hoye, secretary of the U.S. National Conference of Catholic Bishops, said that there was a "basic consensus on principle" among European and U.S. bishops that the church has the authority to speak out against nuclear war but that there were some disagreements on specifics, The Associated Press reported.

"We agreed to disagree on some basic applications," Hoye said. He did not specify what the disagreements were but said that the Americans were not pressured by the Vatican or European bishops to change their position.

The conference, which discussed the issue of deterrence, the relationship between conventional and nuclear arms and the moral and theological questions involved in the use of nuclear weapons, took place as the governments of the European countries involved--West Germany, Italy, Belgium, Britain, the Netherlands and France--were embroiled in an increasingly complex debate over implementing a NATO plan to deploy 572 Pershing II and cruise missiles.

So far among Catholics the lead on the antinuclear front has been taken by the U.S. conference of bishops represented at this week's meeting by Hoye; Archbishop John Roach of Minneapolis-St. Paul, president of the national conference; Archbishop Joseph L. Bernardin of Chicago and the Rev. Brian Hehir.

In contrast, some of the Western European bishops are known to have begun the meeting with concern over some aspects of the American document, which, in its present form, substantially rules out the use of nuclear weapons in most instances and seriously questions the validity of deterrence unless it is coupled with serious bilateral disarmament efforts.

The document issued at the close of the meeting stressed the importance of reaching agreement. "It seemed necessary that these episcopal conferences, particularly involved for different reasons in the problem of nuclear armaments, should get together to exchange information on the experiences of their different countries and to examine them in the context of the church's tradition and the teachings of Pope John Paul II," the statement said.

Before the meeting a Vatican spokesman had said it was strictly "informal and private," but the presence of top Vatican officials and theologians made it clear that the Holy See has decided to give great importance to the subject.

The meeting was chaired by Vatican Secretary of State Cardinal Agostino Casaroli and Cardinal Joseph Ratzinger, head of the Congregation for the Doctrine of the Faith.

As recently as Saturday the pontiff reiterated his concern over nuclear arms, urging that "those involved must make reciprocal efforts to go through the stages of disarmament together, trying to reach the maximum possible reductions without delay."

The White House has not been pleased with the American bishops' firm stance and last October sent special envoy Gen. Vernon Walters to Rome to explain U.S. strategic policy to the pope.

Although the pontiff has never commented publicly on the American pastoral letter, church sources here in Rome say the pope's recent decision to make Chicago's Bernardin a cardinal showed he was not displeased with the American document. Archbishop Bernardin, who will receive his cardinal's hat on Feb. 2--the only American among 18 selected--is the head of the bishops' commission charged with drafting the document on nuclear weapons.

The European bishops kept an extremely low profile here during the meeting but are known to represent various shades of opinion. The West Germans are now preparing a pastoral letter on the subject that is expected to be published at the end of February. Cardinal Joseph Hoeffner of West Germany said recently that he did not expect the German document to differ in substance from the American letter.

A British statement on nuclear weapons issued about a month ago was weaker and far more general than the American letter, but the British bishops, like the Dutch, are believed to be sympathetic to the American point of view.

The French, perhaps because of their government's commitment to the nuclear force de frappe, have not taken as strong a stand as the Americans, and in Italy, the only official bishops' statement on a related topic was devoted to the issue of a projected nuclear base in Sicily.