Maryland Gov. Larry Hogan (R) (Jose Luis Magana/AP)

Maryland Gov. Larry Hogan (R) has dropped the idea of challenging President Trump in the 2020 primary, but he made clear this week that he will continue to speak out against the leader of his party when he thinks it is warranted.

At Pickles Pub in Baltimore on Tuesday evening, before the baseball game between the Orioles and the Nationals, Hogan strongly condemned Trump’s call for four Democratic members of the U.S. House of Representatives — all women of color — to “go back” to the “places from which they came.”

“I thought the comments were terrible and very unbecoming of a president,” Hogan told WJZ, the CBS affiliate in Baltimore. “Totally inappropriate.”

A divided U.S. House voted largely along party lines Tuesday to condemn Trump’s remarks about the congresswomen, saying the president had made “racist comments that have legitimized fear and hatred of new Americans and people of color.” Only four Republicans broke ranks and voted with the Democrats — Reps. Will Hurd (Tex.), the lone black Republican in the House; Brian Fitzpatrick (Pa.), Susan Brooks (Ind.) and Fred Upton (Mich.). Rep. Justin Amash (Mich.), an independent after quitting the GOP this month, also voted for it. Six Republicans did not vote.

Trump was referring to Democratic Reps. Alexandria Ocasio-Cortez (N.Y.), Rashida Tlaib (Mich.), Ayanna Pressley (Mass.), all of whom were born in the United States, and Ilhan Omar (Minn.), who became a U.S. citizen after fleeing civil war in Somalia when she was a child.

This week, facing harsh criticism from Democrats, Trump doubled down on his initial tweet Sunday, saying he doesn’t “have a Racist bone in my body!”

All four Republicans in Virginia’s congressional delegation — Reps. Ben Cline, H. Morgan Griffith, Denver Riggleman and Rob Wittman — voted against the resolution condemning Trump’s remarks on Tuesday.

Griffith had gently admonished the president on Twitter the day before. “I have previously said that the President should tone down his Twitter rhetoric, and I would say that again,” the lawmaker tweeted. He added: “I think he is correct to be unhappy when Members of Congress constantly talk of our country with disrespect.”

Representatives for Wittman and Riggleman did not respond to an email Wednesday asking whether their bosses made statements or comments about Trump’s tweets.

Cline’s spokesman noted that the congressman posted comments critical of House Speaker Nancy Pelosi (D-Calif.), in which he said the speaker had “allowed the people’s House to be turned into a three-ringed circus.”

Rep. Andy Harris, the sole Republican representing Maryland in Congress, also voted against the resolution. He defended Trump in a radio interview Monday, saying the comments were “clearly” not racist.

“He could have meant to go back to the district they came from, to the neighborhood they came from,” Harris told Baltimore talk radio WBAL. The president’s tweet began by referring to the countries that the congresswomen “originally” came from.

Hogan announced in June that he would not challenge Trump in 2020, dashing the hopes of Trump’s leading GOP critics, who for months had courted the governor as a moderate who had maintained top approval ratings in a deeply blue state.

Hogan will become the chair of the National Governors Association on July 26 and has said he will continue to engage in debates about the future of the Republican Party.

Jenna Portnoy contributed to this report.