The cabinet-level Trade Policy Council has rejected a Florida company's petition that President Reagan use a little-known provision of American tax law to curb imports of machine tools from Japan by denying tax credits to the purchaser, administration sources report.

Instead, in one of the knottiest trade problems to come before the Reagan administration, officials are considering whether to use more traditional trade remedies to handle the complaint of Houdaille Industries Inc. of Fort Lauderdale that heavy subsidies give Japanese machine tool makers an unfair advantage in this country.

The Trade Policy Council, headed by U.S. Trade Representative William E. Brock, may submit its final recommendation to the president as early as this week, although past predictions of quick action have proved unfounded as wide differences have emerged among administration officials and the White House staff.

It is possible, although considered highly unlikely, that the original Houdaille idea of using tax law in a trade case will be resurrected then.

The complaint has tied up the administration's trade policy apparatus for months and has sent shivers of apprehension through major American trading partners, such as Canada and the European Community, as well as Japan, who fear that the Houdaille proposal could give the United States a new weapon in trade battles.

It also raises the ticklish problem of how the administration should deal with Japan's targeting of specific industries, such as machine tools, for export growth and whether present U.S. trade laws and the 88-nation General Agreement on Tariffs and Trade (GATT) are designed to handle it.

Houdaille asked the president to apply a 12-year-old, never-used provision of the tax code to deny the 10 percent investment tax credits to any American company that bought Japanese-made machine tools. Under that law, the president can deny tax credits to any U.S. company that buys foreign-made goods that benefit from cartels.

In effect, this would punish American companies for buying foreign products.

The Fort Lauderdale firm argued that Japanese machine tool companies have benefited from more than 25 years of government coddling aimed at helping them capture a major share of the world market. This included heavy subsidies, protection against import competition and orders to other industries to buy Japanese products. As a result, Japan's share of the American machine tool market has soared from about 4 percent in 1976 to almost half last year, Houdaille said.

The Houdaille petition, filed in May, is running on a parallel track with a complaint filed 10 days ago by the National Machine Tool Builders' Association asking the government to limit imports of machine tools for five years on national security grounds. The Commerce Department has one year to make its recommendation.

According to administration sources, the White House's Office of Management and Budget and the President's Council of Economic Advisers believe that the U.S. machine tool industry has been hurt more by the recession than by Japanese policies, and that its fortunes will improve as the American economy does. Those groups, along with some Treasury Department officials, thus advocate doing nothing on the Houdaille complaint.

Others, with Brock described as "the leader of the charge," believe that Japan's industrial targeting violates the spirit, if not the letter, of GATT and that U.S. machine tool makers should be helped with either import quotas or tariffs.

A third view within the trade policy group is that America's machine tool industry has been hurt, but that changes should be negotiated with the Japanese in discussions over the whole question of industrial targeting.