Joseph A. Califano Jr. yesterday urged the National Urban League to temper its attack on the Carter administration and turn its attention toward Congress in the battle to retain and increase civil rights gains.

Califano, Secretary of Health, Education and Welfare, was the fifth Cabinet member to address the league's 67th annual convention, which became a setting for debate on what the administration is, and is not, doing for blacks and the cities.

The convention ended yesterday.

In response to criticisms made Sunday by Vernon E. Jordan, executive director of the league, Califano told the group members that he understands why they came to Washington to "Speak and shout and call out for President Carter."

"The reason I understand it is . . . there's someone in the White House who listens. There's some people in the administration who listen . . . There was no point in shouting out at the White House for the past eight years if you were a black American, because the people who lived there were deaf," Califano said.

"What I'm saying to you now is that you must speak to the Congress as well," Califano continued. "It is imperative (to speak to Congress) if you want to get your rights enforced. We cannot do this alone, and your voice is not heard on Capitol Hill . . . You must go to Capitol Hill and make your interests and your problems understood, and fight the kinds of legislation that has been going on."

Congress is in an "anti-desegregation" mood and "the kinds of legislation that has been going on" reflects that mood, the HEW Secretary said.

Specifically, he cited a measure adopted Tuesday that prohibits HEW's Office of Civil Rights from using federal funds to conduct surveys on the progress of school desegragation.

"As you sat here yesterday (Tuesday) in conference, that measure was agreed to by the House and Senate on Capitol Hill," Califano told the convention. He listed other examples of Congress' "anti-desegregation" mood, including an attempt to prevent HEW from spending money on minority hiring programs and a vote to bar the use of busing to desegregate schools.

Califano said nothing about Congress' action to restrict the use of federal funds to perform abortions. Carter supports some restrictions, which many civil rights leaders contend discriminate against low-income and black women.

Still, Califano insisted that the league has a friend in the White House. He asked the conferees to recall the 1960s, when the league worked closely with the Kennedy and Johnson administrations to make civil rights gains.

"I asks you to remember how it was and how essential it was for us to fight together in the 1960s to make that progress. And I tell you that it's imperative that we fight together in the 1970s to progress," he said.

Califano acknowledged that the Congressional Black Caucus and other sympathetic members of Congress are fighting the "anti-desegregation" trend. But their number is too small, and both they and President Carter need lots of help, he said.

At the convention's last press conference, held several hours after Califano's speech, Jordan was asked if he thought the HEW Secretary was trying to "take the heat off of Carter."

"I think that's a reasonable interpretation," answered Jordan, who - in what is viewed as the sharpest criticism to date by a national black leader - this week accused the President of ignoring the needs of blacks and the cities.

Carter addressed the convention one day after Jordan's attack and told the group that he has "no apologies" for the administration's record on dealing with blacks and the urban poor.

Jordan said yesterday that he, too, has "no apologies."

"President Carter is my friend and I am his friend," he said. "Contrary to news reports. I am not mad at the President and he is not mad at me. The President knows that true friendship means telling the truth, and I have no apologies to make for telling the truth."