The president's Office of Management and Budget has proposed deep budget cuts in the National Institutes of Health, calling for reductions in personnel and the size of most research grants, officials said yesterday.

The OMB proposal would cut NIH funds for this fiscal year and next to about $5.1 billion a year -- $300 million to $400 million less than Congress appropriated for this fiscal year.

The OMB proposal, made about 12 days ago, is being appealed, sources in the Health and Human Services Department said. OMB may agree to reduce the proposed cuts, the sources said, noting that no final decision has been made.

The proposed cuts are part of the administration's efforts to pare about $50 billion or more out of domestic programs to meet the $144 billion deficit target for fiscal 1987 mandated by the new Gramm-Rudman-Hollings balanced-budget law.

The NIH reductions are among numerous cuts the White House will have to propose to meet the Gramm-Rudman target and fulfill its own pledges to protect defense spending and avoid tax increases.

NIH, which operates the world's largest biomedical research program, has always had tremendous support on Capitol Hill. It often has been able to resist efforts by OMB to pare it or hold down increases.

In a separate proposal on acquired immune deficiency syndrome (AIDS), OMB recommends total spending of about $213 million in fiscal 1987, about $30 million less than Congress appropriated for use in 1986. Part of the cut would result from a decision not to seek renewal of $16 million for five experimental treatment centers that Congress initiated this year.

In a new approach, OMB proposes that AIDS funds be appropriated to the HHS assistant secretary for health, who would then parcel the money out to various agencies.

Sources said NIH had initially asked for about $5.9 billion for fiscal 1987.

In fiscal 1986, which began on Oct. 1, Congress voted $5.5 billion for NIH after the Reagan administration initially proposed $4.9 billion.

Now, the OMB documents indicate, the budget office wants to rescind about $300 million to $400 million of the 1986 appropriation.

For fiscal 1987, it wants an appropriation of about $5.1 billion, which includes about $144 million for NIH research on AIDS that would come out of the $213 million total spending on AIDS.

The OMB plan for NIH would include the costs of establishing a new arthritis institute.

The NIH cuts would not come primarily from reducing the number of research grants but from telling NIH to be less generous with each grant. The OMB proposal would cut back "indirect," or overhead costs that are added to grants -- from 26 percent to 20 percent.

Overall, the OMB proposal calls for about 6,000 new and competing grants to be awarded by NIH each year from 1986 to 1989, plus about 12,000 continuations for grants awarded in previous years. Since grants would cost less under the proposal, the NIH would not have to reduce the 18,000 total scheduled for fiscal 1986.

The proposals, some of which were first reported in the newsletter Medicine and Health, are certain to fan growing fears in the research community that budget pressures could choke off promising research initiatives.

At a luncheon last week, officials of the National Institute of Mental Health said that possible cuts forced by deficit pressures could curb work on the homeless, child studies and the biology of schizophrenia.

NIMH Director Dr. Shervert Frazier said community mental health center funding also could be cut.

As Dr. Frederick K. Goodwin, NIMH director of intramural research programs, put it, "There is an explosion in brain sciences . . . and yet mental health is on the edge of not being included in that explosion."