TOKYO, OCT. 17 -- Japan's justice minister, struggling to keep his job amid rising political heat, sent a personal letter to the Congressional Black Caucus today apologizing for his recent comments comparing U.S. blacks to prostitutes.

But the storm surrounding Justice Minister Seiroku Kajiyama, Japan's equivalent of the U.S. attorney general, continued to grow as media criticism increased and a leader of the Socialists, Japan's second-largest party, demanded that Kajiyama be fired.

On the floor of the Diet, or parliament, opposition members used Kajiyama repeatedly as a political cudgel against his boss, Prime Minister Toshiki Kaifu. From his podium in the Diet chamber, Kaifu formally apologized for the justice minister's "regrettable" words but did not respond directly to the calls for Kajiyama's removal.

The prime minister indicated that he will send a personal letter of apology to the NAACP.

All this furor centers on a remark Kajiyama made almost a month ago -- a comment that initially drew minimal attention in the Japanese press. Now, because of harsh reaction from the United States and Africa, it is front-page news around the nation and the subject of heated editorials almost daily.

Kajiyama, a powerful 64-year-old elder statesman of the dominant Liberal Democratic Party who has held two prior cabinet-level jobs, was named to the justice post about four weeks ago.

A few days after taking office, Kajiyama joined local police on a vice raid in a neighborhood near Tokyo's lavish new city hall where prostitutes, many of them foreigners, ply their trade. A number of prostitutes, all foreigners, were arrested.

In a news conference the next day, the minister complained that prostitutes "ruin the atmosphere" of city neighborhoods. "They say that bad money drives out the good," he continued. "It's like in America when neighborhoods become mixed because blacks move in, and whites are driven out."

Although it is rarely stated so bluntly, that sentiment is not uncommon in Japan, a nation that openly prides itself on racial homogeneity. Many Japanese believe that racial, religious and national diversity is a cause of social instability and weakness in the United States.

In recent years, other Japanese government officials have gotten in political trouble for expressing out loud their views about America's ethnicity. In 1986, then-Prime Minister Yasuhiro Nakasone said the presence of blacks and Hispanics in the U.S. population tends to lower the average intelligence level.

Japanese popular media sometimes depict blacks according to primitive stereotypes. Last month, a nationally televised comedy show included a skit in which Japanese men with blackened faces, large white lips and bones tied in their hair danced around the stage in grass skirts and pretended to eat raw snakes.

When Kajiyama's remarks were reported in the United States, they sparked an uproar that has left him reeling. The minister personally called on the U.S. ambassador here to apologize. He apologized again today in a letter to Rep. Ronald V. Dellums (D-Calif.), chairman of the 24-member Congressional Black Caucus.

"My earlier comment was . . . wrong and totally improper," Kajiyama wrote in the Japanese text of the letter, as translated by The Washington Post. The letter was sent to Dellums in English, but Kajiyama's office has not yet released the English version. "Now that I understand how much my comment hurt the American people, especially African-Americans, I wish to convey directly to all Americans my feeling of deepest regret."

{In Washington, spokeswoman Amelia Parker said the Congressional Black Caucus received Kajiyama's letter Wednesday afternoon. Parker said the caucus had no immediate response to the letter, but she pointed out that the caucus had demanded Kajiyama's resignation in a sense of Congress resolution introduced in the House Oct. 3 and again in an Oct. 11 meeting with the Japanese ambassador to Washington, Ryouhei Murata. "This demand is non-negotiable," Rep. Kweisi Mfume (D-Md.), speaking for the caucus, told Murata.}

Kajiyama, who comes from a rural area not far from Tokyo, has gone out of his way in recent days to say nice things about the United States.

At a press conference Tuesday, he was asked if he shares the common Japanese opinion that "Japan-bashing" in the United States stems partly from American racial prejudice. "This has nothing to do with racist feelings," the minister replied. "I do not think there is racism in the U.S."

But in political terms here, this back-tracking may have come too late. The Kajiyama affair is "deeply depressing," said a tough editorial in the Mainichi Shimbun, a national newspaper. "It is hard to believe that other countries will accept these apologies as sincere."

The controversy comes as Prime Minister Kaifu is working hard to win passage of a bill permitting him to dispatch Japanese troops to assist the allied forces in the Persian Gulf.

A chief argument Kaifu has made in support of the bill is that Japan is under intense foreign pressure to offer help in the gulf crisis. In those circumstances, it has been hard for him to argue that Japan should ignore the foreign pressure pushing for Kajiyama's removal.

For the past two days, Diet debate on the gulf question has been regularly interrupted by caustic comments about Kajiyama. "In order to restore Japan's credibility in the world," a Socialist Party leader, Manzu Hamamoto, said today, "I demand that the prime minister do the right thing about Mr. Kajiyama's resignation." In Diet parlance, that means Kaifu should fire the justice minister.

The Kajiyama case is complicated by the intricate political networks within Kaifu's Liberal Democratic Party. Kaifu himself appears to have little personal investment in Kajiyama's future, since the justice minister was picked for the job by other party members and essentially forced upon the prime minister. But if Kaifu were to force Kajiyama out of the cabinet, it might anger senior power figures in the LDP and thus threaten Kaifu's stature as party leader.