Vice Admiral Oscar Montes unexpectedly resigned yesterday as Argentina's foreign minister, saying that his ability to conduct the country's foreign relations had been undercut by rumors that he would soon be replaced as part of a major cabinet shakeup under way within the government of President Jorge Videla.

After more than 2 1/2 years as president of Argentina's military government, Videla is in the process of trying to broaden his government's base by bringing in more civilian ministers. It now appears that when the reorganization is completed within the next few weeks, three civilians will sit in the Cabinet, according to sources close to the president.

The sources said that Montes had been notified earlier this week that he would be replaced but had been asked to stay in his post until Argentina winds up sensitive negotiations with Chile over a territorial dispute focusing on the southermost beaches of both countries near Tierra del Fuego.

The negotiations are scheduled to end next Thursday.

However, it was learned that Montes, angry that he would not keep his position within the reorganized Cabinet, prematurely announced his resignation yesterday.

Defense Minister jose Maria Klix will assume Montes' position Monday until a new foreign minister is named, according to government sources.

In addition to the civilian economy minister, Jose Martinez de Hoz, who has been in the cabinet since the military overthrew the government of Isabel Peron in March 1976, the ministers of both education and justice will be cililians.

Alberto Rodriguez Varela, currently an attorney working for the province of Buenos Aires, is expected to become minister of justice and Juan Rafael Llerena Amadeo, a former under-secretary of education, is expected to become minister of education and culture.

Several candidates have been mentioned in the press here for the foreign ministry position. Sources close to Videla said that Carlos Washington Pastor, a retired air force officer, is most likely to be named.

The Argentine Air Force is generally thought to advocate tougher positions than the navy toward the country's human rights critics and to favor closer ties with the governments of South Africa and Rhodesia. However, foreign policy here is often decided within the military junta, where the army has a decisive voice.

When the current shakeup is completed, only two of the cabinet's eight ministers are expected to retain their posts, Martinez de Hoz and Interior Minister Albano Harguindeguy. The number of civilian ministers will increase by one because the former education minister, who resigned two months ago, was also a civilian.

Sources close to Videla had indicated a month ago that the president had hoped to name four or five civilians to the cabinet but it now appears that the country's three armed services, with which Videla shares power, were unwilling to give up more than one additional ministry at this time.

The idea is to move Argentina slowly toward civilian rule, six or seven years from now. The first step was taken in August when Videla retired as commander-in-chief of the army, keeping his position as president and head of state. The second step is bringing more civilians into the government.