President Reagan tried and failed yesterday to rally support for his economic program from unionized construction workers whose industry has been hit with a 20 percent unemployment rate.

In a 30-minute speech here before 4,000 representatives of the Building and Construction Trades Department of the AFL-CIO, Reagan invoked the name of President John F. Kennedy, appealed to the strong anti-Communist sentiments held by many building trades union members, and predicted that the United States is on the verge of "an economic recovery that can last for years."

But in the end, the president walked away to a brief round of applause from the union members that left little doubt about their cool reaction to his message. There were scattered boos when he entered, and he offered no support for the building trades' legislative program.

Robert Bonitati, White House labor liaison, said "the theme of the president's speech was courage, and it took some courage for him to come in here and to speak to a group that has a 20 percent unemployment rate."

Last year, in an appearance before the same group at the same place, Reagan was received enthusiastically. But the event was marred by an assassination attempt in which Reagan and his press secretary, James Brady, were seriously wounded.

The president yesterday jokingly reminded his audience about the attempt on his life. "I know you understand how happy I am to be back, standing before you today. If it's all the same to you, though, I think I'll slip out of that back door after my speech," Reagan said.

That line produced a round of warm applause and some laughter. But it was hard going from there.

"We're in a tough period," Reagan told the delegates, who had been saying the same thing to one another all morning before the president's arrival.

"I want you to know our highest domestic priority is to revive this economy to spur employment in hard-hit industries like housing and construction, your stock in trade," the president said. But he gave few specifics. Instead, the president seemed to blame Congress for "the great tragedy of our recession."

That "tragedy . . . might have been avoided, or at least curtailed, if the first phase of the administration's tax cut had not been so little and so late," Reagan said.

Earlier yesterday, before Reagan's speech, Robert A. Georgine, president of the building and construction trades unit, told his delegates: "The administration and the Congress seem preoccupied with trying to pin the blame on one another, rather than doing the work that must be done, finding the jobs for the unemployed."

Construction union members are a largely conservative group, "hard hats" fond of displaying their affection for the American flag, as they did in elaborate form yesterday. They are anti-Communist by nature.

Reagan attempted to strike that chord by condemning Soviet intervention in Afghanistan and Poland. He said Americans will "not accept martial law in Poland" or other actions aimed at crushing the Solidarity free trade union movement there.

"Poland's government says it will crush democratic freedoms. Well, let us tell them: 'You can imprison your people . . . and smash their unions. But you can never destroy the love of God and freedom that burns in their hearts, they will triumph over you'," Reagan said.

There was only scattered applause for those words, and a number of union delegates said afterward that Reagan's stand on the treatment of the Polish workers was inconsistent with his handling of last year's strike by the Professional Air Traffic Controllers Organization, which resulted in the firings of 11,500 controllers and the decertification of their union.