House Speaker Paul D. Ryan on Tuesday played down President Trump’s threat to revoke security clearances of former top officials who have criticized him, saying he thinks the president is merely “trolling people.”

Ryan (R-Wis.) was asked about the possible White House action a day after press secretary Sarah Huckabee Sanders said Trump is “looking to take away” the clearances of half a dozen former senior national security and intelligence officials who served in the administrations of George W. Bush or Barack Obama.

“I think he’s trolling people, honestly,” Ryan said at a news conference with other House Republican leaders.

Ryan said the issue falls under “the purview of the executive branch,” not Congress, before saying again: “I think he’s just trolling people.”

The White House’s threat on Monday immediately prompted accusations of political retaliation by current and former officials, as well as security analysts, who said Trump would set a dangerous precedent by punishing political speech.

Several of the officials he cited have written books questioning his leadership and his affection for Russian President Vladi­mir Putin and harshly criticized Trump in television interviews.

Ryan also said Tuesday that Putin would not be invited to speak to Congress if he accepts Trump’s invitation to come to Washington for another round of talks.

“We will certainly not be giving him an invitation for a joint session,” Ryan said. “That’s something we do for allies.”

Ryan added that he’s comfortable with Putin coming to the United States if a message is delivered that Russia needs to stop interfering in U.S. elections.

“It’s the message that counts,” Ryan said.

Sanders announced the planned visit in a tweet last week, saying that national security adviser John Bolton extended the invitation to Putin and that “discussions are already underway.”

Trump and Putin met last week for a summit in Helsinki. Trump’s performance in a joint news conference with Putin was panned by lawmakers in both parties, who said Trump should have more aggressively confronted the Russian leader about interference in the U.S. election in 2016.

The officials who Sanders said might have their clearances revoked are former CIA director John O. Brennan, former FBI director James B. Comey, former CIA director Michael V. Hayden, former national security adviser Susan E. Rice, former director of national intelligence James R. Clapper Jr. and former FBI deputy director Andrew McCabe. (Comey and McCabe no longer have security clearances, according to their representatives, and it wasn’t clear why the White House put them on the list.)

It’s routine for the former directors of intelligence agencies and other senior officials to maintain their security clearances, so they can share their expertise with current leaders or be called in for consultations on how a prior administration handled an issue or crisis, current and former officials said. Some former officials also have jobs that require a security clearance.

Shane Harris and Felicia Sonmez contributed to this report.