Members of the Congressional Black Caucus yesterday stepped up their criticism of the invasion of Grenada, circulating a resolution calling for withdrawal of U.S. forces and demanding that President Reagan respond to their objections in person.

Hispanic legislators, however, were more restrained in their comments, with those from the Caribbean firmly supporting the administration.

Several Black Caucus members said they are disturbed because administration officials told them last week that no military action was planned, then did not invite any black legislator to briefings about the invasion.

"That left a bad taste in some members' mouths," said caucus Chairman Julian C. Dixon (D-Calif.). "Based on our close ties to the Caribbean, we should have been consulted."

Those ties to black Caribbean leaders, who sought the U.S. intervention, prompted some caucus members to mute their criticism.

The caucus had also expressed support for former Grenadan prime minister Maurice Bishop, had urged Reagan to recognize his government and, caucus officials said, did not want to support the military regime that succeeded Bishop, who was killed in the military's coup.

The strains on the caucus were evident when Rep. Gus Savage (D-Ill.) walked out of a caucus meeting this week with Prime Minister Eugenia Charles of Dominica. Savage said that he did not want to listen to "a pack of lies" and that Charles "was just a puppet" for U.S. interests.

He said it is not suprising that Grenada, in response to open U.S. hostility, turned to communist help to stockpile weapons.

Savage said that the administration "wanted to overthrow the regime and install a banana republic" and that most of the House leadership seems inclined to support Reagan's action.

Caucus member John Conyers Jr. (D-Mich.) demanded a meeting with Reagan yesterday. He told the president in a letter that the invasion was a clear violation of international law and complained that the administration had censored news coverage of fighting on Grenada.

Dixon, meanwhile, is pushing a caucus resolution in the House that seeks withdrawal of U.S. forces. He said that Grenada posed no clear threat to other Caribbean nations or to American students there and that U.S. troops "weren't defending anyone. They were the aggressors.

"The main concern among members of Congress . . . is don't let this thing drag out and don't prove you're a liar by setting up occupational forces there," Dixon said.

He said that the United States is trying to dictate the shape of a new government in Grenada and that "in the next two or three weeks there will be all kinds of excuses" about why U.S. forces must remain on the island.

"The president has done a good communications job by confusing the facts," Dixon said. "It's like the cop who breaks down the door without a warrant and finds a stash of cocaine and then tries to justify it to the judge. It's still an illegal search."

Hispanic Caucus Chairman Robert Garcia (D-N.Y.) said, "I think we may have blundered by going in. We're still just hearing one side of the story. It reminds me very much of the military briefings in Vietnam."

Nevertheless, he added, "At this moment, the jury is still out within the Hispanic Caucus."

But Baltasar Corrada, Puerto Rico's Democratic resident commissioner, said that Grenada had no established government to maintain order and that the invasion "was necessary to protect not only the lives of U.S. citizens . . . , but also to protect the lives and well-being of the citizens of Grenada."