Housing and Urban Development Secretary Samuel R. Pierce Jr., after a three-day effort to court black Republican leaders, said yesterday that the Reagan administration's poor image among blacks is based on widespread misunderstanding of the president's policies.

He announced that he has set up new channels for communication with 40 black Republican leaders in an effort to bolster the administration's sagging image in the black community.

Pierce told reporters that black discontent is not based on substantive administration policies. "It's perception, it's not reality," he said. "The reality is that the president really cares . . . . I believe the black community is ready to move toward the Republican Party."

For three days, Pierce has arranged for the black contingent to hold private sessions with all but two members of President Reagan's Cabinet. Only Labor Secretary Raymond J. Donovan and Defense Secretary Caspar W. Weinberger missed the meetings because of illness. Pierce said no Democrats or civil rights activists were invited because these meetings were "a political matter."

"Frankly, this is a Republican affair," Pierce said. "I'm trying to get the Republican Party in better touch with the black community and vice versa."

Pierce has been trying to dispel the impression that he has not been a forceful advocate for blacks in the administration. His press conference came shortly after four high-ranking blacks at HUD accused Pierce of racism for approving their demotions, a charge that Pierce dismissed as groundless.

The group invited by Pierce included former senator Edward Brooke (R-Mass.); LeGree Daniel, chairwoman of the National Black Republican Council; T.M. Alexander, vice president of E.F. Hutton; George Haley, a Maryland lawyer who is brother of "Roots" author Alex Haley, and Arthur McZier of the National Business Services Enterprises.

Pierce had few specifics to announce. He said his agency would work with the Labor Department to hire skilled minority youth for a housing rehabilitation program, but that no additional money was available for this effort.

He also said that each Cabinet secretary has named high-level contact people to discuss with black Republicans political appointments and contracts for minority firms. He stressed that the group had just begun work, sidestepping a question about why it took nearly two years to set up.

Timothy Jenkins, president of the Council of 100, a black political group, said he told Attorney General William French Smith that blacks were "very concerned" about his civil rights policies. He said Smith "acknowledged at the outset that there were some areas that should have been handled better."

Clarence Patterson, a California businessman, said he hoped to "eradicate some of the misgivings black people have about the Republican Party." Gorham Black, a state official in Pennsylvania, said he thought such meetings would lead to more government jobs, contracts and ambassadorships for blacks.

The sessions apparently did little to improve strained relations with many of the organized civil rights groups, such as the NAACP and the National Urban League, which were not consulted about this week's activities.

Jesse Jackson, director of Operation PUSH, said, "I didn't even know about this meeting or who was invited. They are trying to circumvent the acknowledged leadership of major black organizations, a process that creates further distance between the White House and the black community."

Ralph Neas of the Leadership Conference on Civil Rights said, "One of the president's problems has been his unwillingness to obtain advice from people who have experience in civil rights."