Crossroads of the world and capital of nonchalance, New York is a hard city to rattle. Yet Fidel Castro, without even trying, has monopolized the Big Apple.

Every extra city policeman, on or off regularly scheduled duty, has been assigned to the Castro guard patrol. At the floodlit corner of 38th and Lexington, where four city blocks have been sealed off near Castro's lodgings, hundreds of glum-looking officers shiver in what has been a nearly constant cold drizzle since Castro arrived early Thursday morning.

Normally snail-like traffic in midtown Manhattan has been brought almost to a halt. Taxicabs, the essential lifeblood that carries citizens along urban arteries, are in short supply as many drivers have opted to stay home rather than fight the disrupted traffic patterns.

For those who have no direct contact with what police call the "frozen zone," the city's local television and radio stations have staked out police barricades and offer live, hourly "Fidel watches."

With the sole exception of his trip tgo the United Nations Friday, there has been little of Fidel to watch. On Thursday, when Castro did not emerge at all, eager broadcasters were reduced to interviews with irate residents of the "frozen one," who objected to being accompanied to their front doors by police and having groceries and laundry sniffed by bomb-detection dogs.

Castro says he does not worry about threats.

"I don't think anyone dies before his time," he told an interviewer on his flight from Havana. "I know I'm going to die, but I don't know when." Despite stories in the U.S. press, Castro said, he does not wear a bullet-proof vest. He opened his shirt for the interviewer to check.

Nineteen years ago, when he last visted the United States, Castro was a reporter's dream. Accompanied by a small band of bearded young revolutionaries, he traveled around town holding impromptu press conferences, and dined in Harlem with the staff of his hotel.

What was once a happening by choice or because of requests from security agents having nightmares about public hotels, Castro and his 200-man entourage -- all members of which wear suits except the leader -- are staying in the Cuban mission to the United Nations, a 12-story building in a row of undistinguished structures.

"I feel a little bit like a prisoner here," Castro said in a Thursday night interview in his quarters on an upper floor.

As Castro spoke, puffing on the butt end of a rapidly unrolling cigar he tried to keep intact, a monstrous console television on one wall was tuned to one of New York's Spanish-language stations.

On the mission's lower floors, silent groups of men stood in strategic corners or carried stacks of bedding through a heavy fog of pungent smoke from Cuban cigarettes.

Although he earlier had expressed a desire to view a World Series game, Castro said he probably would stay in New York during his U.S. visit, which he predicted might last as long as 10 days.

"I don't want to cause problems for the U.S. government," he said.

"I'm not in any hurry," he said. "A long list of people have asked to see me, and I have friends here I would like to see. But I don't want to spend the money of the New York police."

News services reported that Castro met for three hours yesterday with Congressional Black Caucus members Ronald Dellums (D-Calif.) and Mickey Leland (D-Tex.) to discuss East-West issues such as the SALT II pact and the question of Soviet troops in Cuba.

Castro reportedly planned to meet three of the four Puerto Rican nationalists recently released from U.S. prisons by President Carter, after a reception he hosted last night for U.S. news executives.

Castro's outing to the United Nations Friday, when he delivered a two-hour speech and was guest of honor at a lunch for 120 dignitaries hosted by Secretary General Kurt Waldheim, required closing off several blocks in front of the U.N. building and a 16-car motorcade made up mostly of police vehicles.