With ony 2.2 million people and a land area no bigger than Iowa, the republic of Nicaragua is the sort of country that normally gets dumped into the dusty, attic corners of Washington's storehouse of overseas concerns.

Yet, Nicaragua - and some other tiny backwaters of Central America and the Carribean - has become a proving ground where the Carter administration is undergoing a trial-and-error test of its ability to translate a concern for human rights into an effective instrument of U.S. foreign policy.

Essentially, it's a carrot-and-stick approach that tries to use U.S. financial aid as a lever of pressure and persuasion on the military regimes that control many of the region's countries. In theory, a government that violates the rights of its people will find the spigot turned off, while those who ease up on repression can expect a sympathetic hearing for their aid requests.

Since Central America is a collection of client states heavily dependent on U.S. trade and aid, it would seem to be the perfect laboratory for the successful use of these tactics. But, as Washington has been finding out, what looks logical when spelled out in a position paper isn't really that easy in practice.

That was vividly underscored late last month by a pair of State Department decisions involving Nicaragua - decisions so outwardly confusing and contradictory that they seemed to border on the bizarre.

First, the administration decided to move ahead with approval of a military-assistance agreement whose practical effects would be to strengthen the forces holding the reins of dictatorship in Nicaragua.

Then, it turned around and withheld a sizable chunk of nonmilitary aid whose purpose was to help ease the poverty in Nicaragua.

To diplomats and human rights activists, such decisions are the tea leaves clues to Washington's foreign policy direction. In this instance, though, even the most astute State Department watchers were left bewildered about what had happened.

Some concluded that the decisions signaled a retreat from human rights activism, one dictated by pressures from Nicaraguan government supporters in Congress. Others simply saw it as proof that the administration, for all its hymns to the human rights cause, doesn't really have a policy to match its rhetoric.

However, extensive interviews with many of the officials involved tell a different story. Basically, it's one of the difficulties involved in trying to crank a new, still loosely defined concept like human rifhts into traditional diplomatic decision-making.

Attempts to apply political pressure did figure in the Nicaragua situation. So, too, did the rivalries of government agencies and their conflicting ideas about what best serves U.S. interests.

There were traces of the "clientism" that sometimes makes some diplomats identify more with the country they are dealing with than their own government. There were philo-ophical disagreements about the meaning and purpose of foreign aid, and there was the attitude of many in Congress and the foreign service establishment that human rights considerations are an unworkable intrusion into traditional diplomacy.

But, while all these factors were present in the Nicaragua decision, they weren't the most important elements. In the end, the decision basically was dictated by the rules that Congress has decreed for playing the aid game.

These rules establish different procedures for the ways military aid and nonmilitary aid are approved. As a result, when the administration tried to apply its carrot-and stick methods to the Nicaragua case, it unexpectedly found itself in what one participant describes as "a situation where what should have a straight line got bent into a pretzel."

"Our intention was to use aid to send a warning message to the Nicaraguan government," the source says. "But we suddenly discovered that the rules prevented us from using the logical stick to beat Nicaraguans - military aid - in that way."

"So," he concluded, "we were left trying to dance our way out of the situation by reversing the logical order - using military aid as the carrot and humanitarian aid as the stick. There's no question that the message we were trying to send got garbled as a result, but we did the best we could given the constraints under which we had to work."

Complicating the situation was Nicaragua's position as the touchstone of human rights controversies in Central America. For 44 years, the country has been the personal fiedom of the family of current president, Anastasio Somoza; and, as the man behind the longest running show among Latin American dictatorships, he excites very strong passions in Washington.

Human rights partisans charge that Somoza and his main instrument of power, the Nicaraguan National Guard, have conducted a systematic repression campaign that has included the killings of thousands of opponents.

However Somoza also has a lot of friends in Congress and other influential places in Washington. His supporters have been so outspoken in his defense that the anti-Somoza forces refer to them collectively as the "Nicaragua lobby."

Last summer, the two groups collided when congressional foes of Somoza tried to block a $2.5 million authorization to sell U.S arms to Nicaragua on favorable credit terms. The effort failed, partly because of lobbying by Somoza's partisans and partly because the State Department promised not to sign the agreement until the human rights situation in Nicaragua showed signs of improvement.

That then put the ball in the hands of the State Department, or more specifically of Deputy Secretary of State Warren M. Christopher. In the words of a National Security Council official. "Christopher is Mr. Human Rights - the guy who basically makes almost all the final decisions in these cases."

Technically, Christopher's writ extends only to cases involving nonmilitary, economic and financial assistance. Decisions on military aid are supposed to be made by Secretary of State Cyrus R. Vance.

However, because of Vance's frequent absences from Washington, many of the military decisions - some sources estimate 40 to 50 per cent - fall to Christopher. Nicaragua was an instance where Christopher had to make the calls.

The mechanisms that Christopher uses in making a decision are different in the military and economic assistance areas. On the nonmilitary side, he has set up a body known in the State Department as the Christopher Group.

Its function is to consider the granting or withholding aid to countries with human rights problems. Basically says one participant, "it's a means of sorting out the guerrilla warfare disputes between the human rights people, the area officers who deal with the country on a geographic basis and the representatives of other agencies with an interest in the matter."

Where military aid is concerned, there is no parallel formal mechanism - primarily because of the need to involve agencies and considerations that don't figure in nonmilitary aid. While rights are factors, they have to compete with national defense, military alliance commitments and arms sale and export policy considerations.

These factors are shepherded up the State Department bureaucracy under the direction of Lucy Wilson Benson, under secretary for security assistance. Ultimately, they are referred either to Vance or Christopher for a decision.

By late September, both mechanisms had brought to Christopher the need for decisions on the military pact and other aid to Nicaragua.

Reinforcing the need for some action was the fact that Somoza had just loosened the screws a notch in his country by lifting a three-year "state of siege" that had suspended all constitutional guarantees. The problem was to find some means of recognizing the gesture while still making clear that the State Department wanted to see even more improvement.

Normally, department sources agree, that would have been done by holding back on military aid and going forward with economic assistance. The problem, though, was that, under Congress' mandate, the military aid authorization, which applied to the 1977 fiscal year, would expire unless the agreement was signed by the end of September.

As Benson notes: "We had to decide whether we were going to throw this weapon away or try to preserve it as an instrument for future leverage."

The result was Christopher's decision to sign the agreement as a means of keeping it alive. But he added a stipulation that no sales or transfers of arms would be made until there was further evidence of human rights improvement.

"That's something a lot of people have failed to grasp." Benson says. "All we did was keep the agreement on the burner as a basis for further negotiation. But Nicaragua hasn't received so much as a single bullet under that agreement."

The next step came on Sept. 27 when the Christopher Group met on what to do about a proposed package of more than $12 million in nonmilitary loans and grants for Nicaragua. Information provided by reliable sources has produced this picture of what happened in the meeting:

Sally Shelton, a deputy assistant secretary for Latin America, described Nicaragua's internal situation as "a mixed bag." She noted that while reports of killings and torture continued to come out of Nicaragua, the lifting of the state of siege also had produced some improvements such as greater freedom for the press.

Shelton and others noted that, unlike the military aid authorization, the funds for economic assistance could be carried over into the next fiscal year without being lost.

She, Robert Pastor of the National Security Council and Mark Schneider, deputy assistant sectary for human rights, recommended that part of the aid package be granted as an encouragement to Somoza and part be withheld for the time being to spur further improvements.

That drew a strong objection from Jessica Tuchman, also of the National Security Council. She argued that it was administration policy not to use aid aimed at directly helping the needy as a political weapon.

After hearing everyong out. Christopher said he was "very troubled" by the Nicaragua situation and added: "I feel we are out as far as we want to be in Nicaragua." In the end, he decided that, except for some small grant to nongovernmental organizations, the bulk of the aid package would be held back for the time being.

Tuchman interjected that she wished to be recorded as opposing the decision because she felt that "in the name of promoting human rights, we are denying assistance to the needy."

Christopher responded that we were "postponing" assistance to the needy, not denying it. With that action, he added, he felt that the United States had given the Somoza regime "a big enough signal for the time being."

However, his words seem more than a little over-optimistic in the light of the confusion subsequently generated by the decision. If no one else seems to have understood the point of the message, one participant concedes, there are grounds for supposing that Somoza didn't either.