Prime Minister Ian Smith yesterday declared martial law in parts of Rhodesia and a new crackdown on organizations here representing the black nationalist guerrillas fighting to overthrow the biracial transitional government.

In his anxiously awaited speech on a promised "new course" in the escalating war, the Rhodesian leader rejected a general mobilization for now but promised "tougher, stronger measures" to wipe out the foreign-based guerrillas and "liquidate" the groups supporting them inside Rhodesia.

Telling white Rhodesians they face "the greatest crisis of our lives," he appealed to them to "keep our cool." He added, "Providing we close our ranks and retain the unity which we have shown over the past 15 years, we will continue to confound our enemies."

Smith, in announcing a crackdown on guerrilla backers, apparently referred to the reported arrests starting Saturday night of 20 followers of guerrilla leader Joshua Nkomo.

The 28-minute address did not seem to spell any other change in Smith's policy of trying to cope with the growing guerrilla threat.

He reaffirmed his commitment to the "objectives" of the March agreement among himself and three moderate black leaders now in the transitional government providing for the election of a black majority government. But he did not say that this objective could still be reached according to schedule, by Dec. 31. It is now widely doubted here that the vote could be held then.

He also said there was "no real difference" among the four members of the ruling Executive Council on the question of meeting with the guerrilla leaders, as suggested by the British and Americans, Smith said his government colleagues still had to be convinced "there was a genuine prospect of success" and that the purpose of the proposed meeting was not solely a handover of power to the Patriotic Front, the alliance of the two guerrilla factions waging the war.

Smith expressed "deep resentment" against Britain and the United States for what he said was their support for "a gang of Marxist terrorists" - a reference to the Patriotic Front - and charged that the two Western nations bore the "major shar of the blame" for the escalation of the war.

But his strongest attack on any outside power involved in the 13-year-old Rhodesian conflict was reserved for President Julius Nyerere of Tanzania, whom he termed "the evil genius" and "a major stumbling block" to a peaceful settlement.

Smith warned Rhodesia's neighbors harboring Patriotic Front guerrillas that if they continue to allow infiltration of Rhodesia "they must bear the consequences of any defensive strikes we may undertake." He did not announce any retaliation for the guerrillas' downing by missile of an Air Rhodesia passenger plane last Sunday.

The crash resulted in the death of 38 whites and Indians. Ten of the 18 survivors later were gunned down at the site. The incident has enraged white Rhodesians and led to calls for revenge.

Smith warned South Africa and apparently Britain and the United States on Wednesday that he was planning "something positive and firm" that might not please them and hinted it might upset current Western efforts to arrange a general peace conference.

But the only thing he announced last night that appeared capable of compromising Western peace efforts was the crackdown on the internal opposition that apparently began with the reported arrest of 20 memebers of the Nkomo's Zimbabwe African People's Union, including four top officials.

The organization's vice president, Josiah Chinamano, called the crackdown "a very negative step indeed. This seems to be closing the gate to an all-party conference. One wonders where he is leading the nation."

"My prediction is that this step does not end the war but will escalate it," he said, adding that he expected to be arrested himself "anytime tonight."

Smith, however, did not rule out further contacts between himself and Nkomo, with whom he met secretly last month. Speaking to correspondents after recording his address yesterday morning, Smith said he thought there was still a chance for a settlement although another secret meeting with Nkomo was "just for the moment, not on."

The Rhodesian leader seemed anxious to calm the incensed white population. He told the 230,000 remaining whites he could not allow himself "to be influenced by emotion which would result in ill-conceived and unbalanced decisions.

"There are those amongst us on the one side who say that the position is so serious that we must resort to some dramatic, desperate action, a kind of 'do-or-die' attitude," he said. "On the other side there are those who claim that our salvation lies in capitulation to the forces of evil. Both smack of desperation and must be discarded," he said.

Instead, Smith proposed what he called "a modification of martial law" that would enable the government to streamline its machinery for fighting the war while leaving intact those civil authorities which are required to continue to play their part.

"It will be introduced to particular areas as and when required and not on a nationwide basis," he continued. "It will lead to tougher, stronger measures against our enemy. This is the precise intention."

He said the government would still seek to avoid injuring innocent civilians caught in the middle of the war. There has been intense speculation that the government planned virtually to abandon those border areas heavily infiltrated by guerrillas where no whites are still living. It would then concentrate its forces in defense of the white and black areas in the heartland and wage from there an allout war on the guerrillas in the outlying regions.

Martial law, it is thought, would be imposed in those zones where the guerrilas are concentrated, giving a free hand to the military to escalate its war against them there.

Smith said details of his plan for the use of martial law would be disclosed later. He left the impression that the government was running out of options, saying, "We will continue with our policy of fighting the enemy with all our energy, our resources, our well-known and well-acclaimed valor. And as there is no end to our reserves of these qualities. We will go on and on until final victory is achieved."

Smith also appealed to whites to "look on the positive side" of the now much maligned transitional government and give it a chance to carry out its program.

He said "each and every one of us" in the Executive Council and Cabinet, which has nine whites and nine blacks, was determined to overcome its problems and make it work. "We all know very well what the stark alternative is."

But several of his own white ministers have already said there is no hope of finishing the new constitution, holding a white referendum to approve it and carrying out general elections for a black majority government by the end of the year.

In fact, at least two white ministers have been telling whites to vote against the referendum on the new constitution as a way for showing their opposition to the transitional government and the March agreement.