The Reagan administration's senior official in the field of nuclear nonproliferation, James L. Malone, is being dropped as assistant secretary of state for scientific affairs because of alleged inability to work out a politically acceptable policy for increased sales of U.S. nuclear technology abroad, informed sources said yesterday.

These sources said that Malone, who has been one of Reagan's most controversial appointees at the State Department, will continue for now to head the U.S. delegation to the Law of the Sea Conference, which resumes today at the United Nations.

But they said he will be stripped of all responsibility for nuclear questions. The reason given is that some senior policy makers think too little progress has been made in reversing the Carter administration's restrictive nuclear export policy. The Carter administration looked askance at such exports on grounds they would help more countries build nuclear weapons. The Reagan administration argues, on the contrary, that increased sales will increase U.S. influence with other governments.

Some officials suggested the sidelining of Malone would be a victory for Richard T. Kennedy, undersecretary of state for management and a former member of the Nuclear Regulatory Commission. In recent weeks, these sources said, Kennedy increasingly has assumed de facto control over the nuclear policy functions of Malone's bureau of oceans and international environmental and scientific affairs, and has put them under personnel responsive to his direction.

The sources, all of whom spoke only on the condition that their names not be used, said Malone's ability to function effectively in the nuclear field has been virtually crippled since last October when The Washington Post revealed details of a memorandum on policy changes prepared under his direction.

The memo proposed transferring to the State Department all the export licensing functions of the NRC, which is an independent agency charged among other things with the maintenance of nuclear safety; repealing legislation that bars U.S. nuclear exports and military and economic aid to countries moving toward production of atomic weapons; and relaxing the current requirement that nations without nuclear weapons permit international inspection of their nuclear facilities if they wish to keep buying nuclear materials from the United States.

These proposals were quickly denounced by congressional critics who charged that they would mean a total reversal of past U.S. efforts to prevent the international spread of nuclear weaponry. In the face of this criticism, the administration quickly backed off Malone's plan, but, the sources said, the residue of congressional suspicion has made it virtually impossible for Malone to work with Congress on potential compromises.

Earlier, Malone had great difficulty winning Senate confirmation because of charges that his job at State might present him with a conflict of interest.

Malone had represented power companies in Taiwan and Japan while in private law practice, and then helped write a Reagan transition report saying there should be "no concern" about proliferation in industrialized countries including some where his former clients had plants. He was confirmed only after reluctantly promising to excuse himself from decisions involving the nuclear power industries in Japan and Taiwan.

Malone also has had problems because of the failure of negotiations with India on a formula for terminating the 1963 agreement under which the United States supplied atomic fuel and expertise for India's Tarapur reactor. Malone headed the U.S. negotiators; India has balked at continuing to submit to international safeguards on nuclear fuel already provided by the United States, a position that undermines efforts to police the use of such fuel in developing nuclear weapons.

When The Washington Post tried to talk with Malone, the caller was referred to a spokesman who said there would be "no comment" on Malone's future status as assistant secretary. The spokesman stressed, though, that Malone will be at the head of the U.S. Law of the Sea delegation today.

Informed sources say that that position is likely to be short-lived, since the Reagan administration has disavowed the stance, adopted under Carter, of supporting a broad sharing of seabed resources with Third World nations. Instead, the administration now has a negotiating stance that strongly reflects the private development views of U.S. business interests; and since the new U.S. position is so far from that of other countries, the Law of the Sea Conference is widely expected to bog down.

The sources said those under consideration to succeed Malone as assistant secretary include Charles Horner, Malone's deputy for environmental affairs; William C. Salmon, an assistant to undersecretary of state James L. Buckley, and Donald Fortier, deputy chief of the department's policy planning staff.

Some sources said there also was a good chance that the job would be left unfilled for an indefinite period, with Kennedy running the nuclear policy through his surrogates in the bureau, although it technically is supposed to be under Buckley's direction.