In a case that has split the nation's trade policy apparatus for nearly a year, the Reagan administration has rejected a Florida machine-tool company's request for protection from unfair trade practices by Japan.

But at the same time, the administration revealed that it would start immediate talks with Japan over the use of government subsidies to boost exports to the United States.

The abrupt Cabinet-level rejection of the plea for help from Houdaille of Fort Lauderdale, Fla., came Friday evening after strong lobbying efforts by the government of Prime Minister Yasuhiro Nakasone that reached into the White House, administration sources said yesterday.

U.S. Trade Representative William E. Brock, now in Europe, said he and Japanse Trade Minister Sadanori Yamanaka will set talks this week on Japan's industrial targeting policies that formed the nub of the Houdaille complaint. "It is important that these discussions effectively deal with these problems, making further action unnecessary," Brock said, hinting at the possibility of future administration moves against Japanese targeting practices if the talks fail.

Houdaille's innovative petition asked the Reagan administration to use a little-known provision of the tax code to deny investment tax credits to American companies that bought machine tools from the subsidized Japanese industry.

After months of internal debate, the administration decided early this year to reject the tax approach. But splits within the administration remained over whether Houdaille should get some help under more traditional trade-relief laws because of the massive Japanese subsidies--including about $100 million a year given to machine-tool makers from revenues from bicycle races.

Coincidently, a Japanese delegation will begin talks today with administration officials over reducing quotas for the sale of American beef and oranges in Japan. There have been reports in Tokyo that the Nakasone government is willing to agree to slight increases in imports of beef and oranges.