Secretary of Transportation Drew Lewis said yesterday that the Reagan administration task force on auto imports has agreed on the need for "voluntary restraints" by the Japanese and differs only in how to signal that decision to the Japanese government.

Lewis said "our committee and, I'm sure, the president, would like to see breathing room" provided for the domestic auto industry by the Japanese legislating a three-year curb on their exports to the United States. But he said there was no unanimity as yet on how to communicate that wish to the Japanese.

Administration sources said task force members are trying to decide on an import limit for Japanese autos that would be acceptable to Japan, and yet tough enough to halt the campaign in Congress for mandatory, legislated import quotas.

"I think you need some kind of clear signal from Bill Brock [the special trade representative] or somebody," Lewis said. "They need to hear it more clearly than they have so far."

Japanese Foreign Minister Masayoshi Ito, who is scheduled to meet with President Reagan today, said following talks yesterday with administration officials he hadn't received "the specific signal yet." During more than three hours of discussions with Secretary of State Alexander M. Haig Jr. and an hour with U.S. Trade Representative William E. Brock, Ito said the importance of the president's general economic recovery program was stressed. "There was no explanation of a specific policy direction on how to resolve" the auto question, Ito said through an interpreter during a press conference. Ito also met with Defense Secretary Caspar Weinberger.

In addition, such legislation ultimately could lead to worldwide unemployment and recession, Ito said.

When asked under what circumstances the Japanese government would ask its automakers to limit their exports to the United States, Ito said, "You have touched on a very delicate problem."

Ito said he asked administration officials what steps by Japan would contribute to restoring the strength of the American auto industry within the framework of the president's economic revitalization program.

Ito said he is aware of the plight of the auto industry and the "mood for protectionist legislation in the U.S. Congress."

Ito also said his government hasn't reached any conclusion on what policy it will pursue as far as a voluntary restraint agreement with the United States. He said he is awaiting the report from the president's task force, and an important consideration is protecting the principles of free trade.

Ito said Brock told him he hoped the auto issue would be resolved in the defense of free trade and avoid the pitfall of protectionism.

Lewis's comments on the controversial import issue came during a metting with reporters. Informed sources said Lewis' view was substantially shared by Haig, Treasury Secretary Donald Regan, Secretary of Labor Raymond J. Donovan, Commerce Secretary Malcolm Baldrige and Brock.

Lewis told reporters that no one on the task force favored legislation pending in Congress to set mandatory quotas nor did anyone favor a formally negotiated U.S.-Japan agreement.

But he said "it's in their [the Japanese] best interests" to restrict their exports rather than face congressional restraints. He said a cutback of 200,000 to 500,000 units from the 1.9 million cars Japan sold here last year would give U.S. industry "breathing room" to retool for the change in consumer tastes for small cars.

Arguing that the restraints would last no longer than three years, the Transportation secretary said, "If three years from now, we don't have a competitive product," neither he nor the president would favor further protection.

While Lewis referred to the desired Japanese action as "voluntary restraints," he conceded that the only way such an export curb could avoid legal challenge would be the passage of a law by the Japanese Diet setting the permissible quota.

The 67-year-old foreign minister also said he will continue his talks with Haig today because they didn't finish yesterday. Other issues discussed were, measures Japan is taking regarding the Soviet Union, cooperation in economic and defense matters, Asian problems, and energy.