When Brazilian authorities seized a convoy of Libyan planes carrying arms to Nicaragua two months ago, U.S. Ambassador Langhorne Anthony Motley, the former businessman and lobbyist recently named to the State Department's top Latin American post, learned one of the consequences of his high-profile diplomacy here.

The afternoon the Brazilian action was announced to reporters at Planalto Palace, the seat of Brazil's military government, Motley found himself inside, carrying on his habit of taking U.S. issues directly to top officials rather than through traditional diplomatic channels.

His meeting, he says, had nothing to do with the confiscation of the Libyan planes. But when Brazilian reporters saw him leave the palace that evening with several government officials, they drew a predictable conclusion. Motley, it was soon reported, had once again used his fluid contacts and aggressive style to push the government into an action favorable to the United States.

The incident exacerbated Motley's already tense relations with Brazilian Foreign Ministry officials, and left the 45-year-old ambassador convinced that his image had gotten out of hand. "I can't go over there anymore," he said, "without some guy writing a story saying that I'm there to dictate something."

After 19 months as ambassador to Brazil, such reports have become a regular spinoff of Motley's style as one of the most aggressive--and occasionally controversial--U.S. diplomats here in recent times.

Whether negotiating sensitive trade and political issues, arranging President Reagan's visit to Brazil in December or meeting Brazilian reporters, Motley consistently has emerged as a freewheeling and occasionally blunt advocate of Reagan administration policies.

Using a fluent knowledge of Portuguese and Brazilian customs, he has won nearly unrivaled influence within the top circles of the governing Brazilian military and frequently is given credit for improving relations with Brazil after they reached a low point during the Carter administration.

At the same time, his hands-on approach and frequently colorful public and private declarations have led to increasing tensions with some Foreign Ministry professionals and political sectors and produced a spate of media exposure that at times appeared to work against him.

It is a record in some ways discordant with Motley's projected role as a loyal implementer of policy in replacing Thomas O. Enders as assistant secretary of state for inter-American affairs. Motley, one diplomat here said, is "not taking this job to go up there and stamp papers."

For his part, however, Motley has grown uncharacteristically discreet in describing his view of the post or the shift it may reflect in U.S. policy in Latin America. In an interview this week, he turned aside questions on his duties or U.S. policy, explaining, "Some senators get their nose bent out of joint if they don't hear it from you directly."

But while saving policy statements for his confirmation hearings later this month, Motley said he had few regrets about the unvarnished diplomatic style that has earned him such notoriety in Brazil.

"The high profile thing inhibits. I might fine-tune it a bit," he said. "But I think my ability to talk to all sides has helped me more than anything here. And if you do that, you're going to have a high profile."

Motley's prominent role came as a surprise to some State Department professionals, who viewed his arrival in the fall of 1981 as another instance of the Reagan administration's habit of replacing career diplomats with relatively unqualified political appointees in ambassadorial posts.

Motley, however, had several advantages in penetrating the often intricate politics of Brasilia and Brazilian foreign policy. Born in Rio de Janeiro to an American father and a half-Brazilian mother, Motley lived and studied in Brazil for 17 years and frequently used his command of Portuguese slang and Brazilian customs to charm military officers and local political reporters.

As a former developer and Republican organizer in Alaska and a congressional lobbyist on Alaskan land issues in Washington, Motley also had won the personal confidence of Reagan and several of his key aides.

As a result, Motley has rarely hesitated to slip the normal bounds of an ambassador and advocate the positions of the Reagan administration or local U.S. business interests--most commonly by bypassing Foreign Ministry contacts and meeting directly with President Joao Figueiredo and other top officials.

In the process, Motley increasingly has become involved in the delicate Brazilian political division between the mostly civilian Foreign Ministry officials traditionally responsible for Brazilian policy and the more conservative military leaders who wield ultimate power.

Last year, for example, Motley was faced with the task of negotiating the future of a group of U.S. military sales advisers stranded by cancellation of the military cooperation treaty between the United States and Brazil several years earlier.

When the Foreign Ministry abruptly notified U.S. officials that the advisers should be withdrawn and their office closed, Motley took his case directly to his friends in the military and won a new agreement that kept the advisers in Brazil.

Since then, Motley has circumvented the Foreign Ministry on several other issues, calculating, according to sources here, that its recent shifts of Brazilian policy toward some nonaligned positions and closer identification with the Third World are not always enthusiastically backed by the military leaders.

Such tactics have helped earn Motley his reputation for effectiveness among U.S. business leaders and Reagan administration officials, but the cost has been an increasingly contentious relationship with the foreign policy establishment and its political allies.

Motley, the highly respected Brazilian magazine Veja summarized this month, was "the most efficient and active American ambassador in Brasilia since Lincoln Gordon" in the 1960s. But, it added, when word of Motley's promotion came, "the rejoicing of the top ranks of the Foreign Ministry was . . . considerable."

In recent months, Motley's differences with some top officials and his candid public style have led some of his colleagues to worry that his effectiveness was eroding. After he challenged Brazilian foreign policy statements on Central America earlier this year, critical articles appeared in the Brazilian press reflecting official views that his aggressiveness had become irritating.

Motley characteristically fired back. Brazilian-U.S. relations, he told a group of Brazilian reporters, were good "despite the efforts of the column of 'bearded ones' around Foreign Minister Saraiva Guerreiro," a reference to a local nickname for the Foreign Ministry's inner circle.

In his new post, however, Motley expects that his relations with Latin American governments will not become so intricate. Despite his experience in the 1960s as an aide to the U.S. Air Force commander in Panama, Motley said, "I obviously don't have the rapport anywhere else that I have here . . . . One thing I have learned is that what we are talking about is 33 different countries, and 33 different sets of facts."