A ranking Japanese trade official today rejected the Reagan administration's reported request for a 20 percent cut in auto exports to the U.S. market.

The official said the figure does not even provide a basis for negotiation on the troublesome issue, which has taken on overtones of becoming a major problem for the Reagan administration's domestic program as well as upsetting relations with a major ally.

The rejection followed the return of Foreign Minister Masayoshi Ito from talks in Washington that focused on U.S. hopes for "voluntary" Japanese restraint on car sales to help the troubled American auto industry.

"That is not negotiable at all," an official of the Ministry of International trade and Industry said of the U.S. request reported yesterday in The Washington Post and Japanese newspapers.

Because his ministry probably would be charged with enforcing any export reductions on Japan's auto industry, the official's remarks indicated that working out what the Reagan administration calls a "voluntary restriction" will get off to a rocky start.

He asserted that the ministry was not even prepared to begin talks if the U.S. opening position was a reduction that deep from the number of vehicles exported last year. When asked what figure would form a basis for negotiations, the official refused to respond.

Japanese automakers refused to comment on the report, saying they could not discuss a "hypothetical" reduction, but they also are expected to resist a cut that deep.

The request has not been announced publicly by either side and its accuracy is in part disputed by the U.S. Embassy, although the accounts leaking here and in Washington are generally similar.

The Washington Post reported yesterday that the Reagan administration has informally asked Japan to hold exports of passenger vehicles this year to about 1.6 million. According to statistics available here, Japan exported about 1.99 million cars and vans to the United States last yea.

The report said that the request was sent to the Japanese government through U.S. Ambassador Mike Mansfield before last week's meeting between Ito, President Reagan and Secretary of State Alexander M. Haig Jr.

One Japanese official corroborated that account today, but Mansfield, through a spokesman, denied forwarding any such request.

Reagan has yet to announce the specifics of his plan for obtaining a "voluntary restriction" from Japan and the accounts of his and Haig's talk with Ito last week did not spell out any specific target.

Two Japanese newspapers, quoting government officials, added more details of the reported U.S. request. The Yomiuri newspapers said the request has come through Mansfield from Haig and called for the restraint to be in force for three years at the longest. Representatives of both countries would meet annually to review the report levels.

It also said that the reduction would be formally stated either in an exchange of letters or in what was described as a "voluntary restraint agreement."

The 1.6 million export level is said to represent the average exports to the United States during the years 1978 to 1979. The Japanese are said to be favoring a significantly higher level -- about 1.8 million vehicles, representing the average of exports in 1979 and 1980.

The formal Japanese response to President Reagan so far has been a general promise to cooperate in helping the American auto industry. Officials said this week that some kind of agreement is preferable to the imposition of quotas, as it being threatened by some members of Congress.

But under the layer of amicability is a big dispute over how far to go -- and in what way -- to satisfy the White House.

Japanese say they are somewhat baffled by the Reagan administration's insistence that the restraint will be "voluntary." Attorney General William French Smith has made it clear that any voluntary agreement among Japanese automakers to restrain exports will be regarded as illegal collusion under U.S. antitrust law.

That would apparently mean that some of the Japanese government, proably the Trade and Industry Ministry would have to issue a legal order restraining exports. The usual device of "administrative quidance," which lacks the force of law, probably would not be used, one official said this week, because it is "too vague" to satisfy American congressmen. There is also a question of whether "administrative guidance" is strong enough legally to get around the anti-trust law.