A protege of Sen. Jesse Helms (R-N.C.) has become the first U.S. ambassadorial nominee rejected by a vote of the Senate Foreign Relations Committee, but Helms promises to continue the confirmation fight.

In a 9-to-7 vote, the committee refused Thursday to confirm James L. Malone as ambassador to Belize after testimony that he had lied to the panel.

Sen. Edward Zorinsky (D-Neb.), who led the opposition, said the State Department inspector general's office had told him that Malone was an "inadequate" manager who performed earlier jobs in a "less than satisfactory" way.

The committee has never voted down a presidential choice to serve abroad as an ambassador, according to the Senate librarian's office, although some nominations have been withdrawn to avoid a negative vote and the Senate has rejected others.

Helms, Malone's chief backer, "will absolutely pursue this nomination," an aide said yesterday. Helms will ask the committee to reconsider or seek Senate action to bring the nomination out of committee, the aide said.

Officials at the Central American bureau of the State Department, where Malone is working as a consultant, said he was traveling and unavailable for comment.

The department said in a statement that it is "disappointed" by the committee action and fully supports Malone's nomination, adding that he "performed responsibly" in carrying out President Reagan's foreign policy during his previous jobs there.

Malone has been in and out of difficulty at Foggy Bottom since first attracting notice as a hard-charging general counsel to the Arms Control and Disarmament Agency (ACDA) during the Nixon administration. He made enemies there for what conservatives called a housecleaning of pro-disarmament officials.

After working as an attorney on behalf of two Japanese power companies, a West German firm and a Taiwanese government power plant, Malone recommended to the Reagan transition team in early 1981 that controversial nuclear reprocessing technology be given to Japan.

His memo said there was "no concern" about spread of nuclear weapons by certain countries -- he named West Germany and Japan -- and proposed that controls over nuclear power exports be loosened.

When Malone was nominated in March 1981 as assistant secretary of state for oceans and international environment and scientific affairs, he refused for three weeks to promise the Foreign Relations Committee that in his new post he would not handle any matters relating to Japan, Taiwan or West Germany. When he did so, he was confirmed.

"Then we discovered that he was, in fact, dealing with his former clients about business matters," a former committee aide said yesterday. "It was a clear violation of what he had promised the committee."

Nuclear industry officials also expressed unhappiness with the way Malone handled his new job. "He promised all the right things but couldn't really deliver," one company spokesman said, referring to Reagan administration efforts to promote new reactor sales at home and abroad.

Malone advocated allowing exports to nations trying to make nuclear weapons, but Congress refused.

He headed the negotiating team that failed to persuade India to retain international safeguards at its Tarapur reactor after U.S. supplies were halted and was lambasted in Congress for saying the State Department should take over from Congress and the Nuclear Regulatory Commission responsibility for licensing nuclear sales.

Then-Secretary of State Alexander M. Haig Jr. took Malone out of the nuclear field and nominated him as ambassador at large to the Law of the Sea treaty talks. But the Foreign Relations Committee, irked about his $200,000 expenditure for consultants on the treaty while agency experts sat idle, balked again before approving him.

Last June, amid rumors Malone was about to be ousted, Helms put Malone's name on a list of diplomats to be "taken care of" before Helms allowed Senate consideration of 29 other State Department nominations. The nomination to Belize, an impoverished coastal nation between Mexico and Guatemala, followed.