Leonel J. Castillo, the controversial head of the Immigration and Naturalization Service, announced his resignation yesterday after months of frustration in trying to change one of the most tangled and troubled agencies in the federal bureaucracy. Castillo said he will leave the INS by Oct. 1.

Although he frequently was at odds with his superiors in the Justice Department, Castillo, the first Mexican American to hold the INS post, timed his resignation to coincide with a meeting of Hispanic leaders at the White House so that the best possible light could be put on the move.

Castillo, who is considering running for mayor or the city council in his hometown of Houston, resigned quietly. "I'm not leaving with any fights," he said yesterday. "I'm not blasting anyone as I go out."

And when he spoke before more than 200 Hispanic leaders at the White House, he told them, "I have had no problems with Jimmy Carter. I think he's a decent, sensitive human being."

But Castillo said he was "still disappointed in the lack of change in national immigration policy, which has serious and glaring deficiencies."

"Hispanic groups in realistic terms are presented very little power here in Washington," he added.

Hispanic leaders greeted the long-expected resignation with disappointment. President Carter apparently alleviated some disappointment by announcing the appointment of Esteban E. Torres, ambassador to UNESCO (the United Nationsl Educational, Scientific and Cultural Organization), to a senior White House staff post recently vacated by another Hispanic, Joseph Aragon. The president also promised to name at least one Hispanic to one of four vacant Cabinet under secretary posts.

The Hispanics were far less pleased with their day-long series of meetings with senior White House staff members and Cabinet officials.

"A lot of us were very dissatisfied up until the time the president spoke," said New York Deputy Mayor Herman Badillo, a former congressman. "We thought the Cabinet members should have come prepared to discuss the real issues with us, and how they would deal with them. But they weren't prepared. They paraded before us with no new announcements of appointments or programs for Hispanics."

As for supporting Carter's reelection, Badillo said many of the leaders are taking "a wait-and-see attitude."

"The president displayed a sensitivity that we didn't find in his Cabinet or staff," said David Montoya, the president of IMAGE, a national group that promotes government employment of Hispanics.

When Castillo, grandson of a Mexican immigrant, took office 27 months ago he found the INS to be a thicket of red tape and a system supposedly designed to help people that was, in fact, adding to their problems. It was taking as long as four years to process immigration applications, and morale among the agency's 10,000 employes was poor.

Catillo, 40, lobbied for a form of "amnesty" for illegal aliens who have been in the United States for a number of years. He attempted to computerize the agency's records system and redirect its enforcement efforts.

He frequently found himself in the middle of controversy, alternately accused of being too hard or too soft on illegal immigrants.

For much of his term, he found that INS was an unwanted stepchild of the federal bureaucracy.

But when then-attorney general Griffin B. Bell suddenly became interested in the agency, Castillo found himself increasingly at odds with superiors at Justice. He was the only major appointee in the department not personally selected by Bell.

Yesterday Castillo said he came to feel "you have to go outside [the federal government] to change the system." As a private citizen, he said, he will work to see that the immigration laws, which have not had major revisions in 27 years, are changed to allow more legal immigrants.

Castillo would not speculate about a successor. INS Deputy commissioner Mario T. Noto, Deputy Associate Attorney General Doris Meissner and INS general counsel David Crosland are possibilities.

Administration sources have said that Carter is hoping to replace Castillo with another Hispanic, but few apparently want the job. "You catch heat from all directions," Castillo said, adding that he had asked several Hispanics mentioned as possible successors whether they would accept the post and that "they all said "forget it."" CAPTION: Picture, LEONEL J. CASTILLO..."I'm not blasting anyone..."