U.S. workers who go into hospitals for treatment of mental or nervous disorders next year would have to pay up to $500 out-of-pocket before their insurance takes effect under a cost-cutting proposal advocated by the Reagan administration for the health-care program that covers 10 million active and retired employes and their families.

The administration proposal would affect the Federal Employe Health Benefits Program (FEHBP), the nation's largest "company" health care program. In the Washington area it helps pay the hospital, mental and dental benefits of about 950,000 people or one out of every two residents.

The Office of Personnel Management, which oversees the program, is currently negotiating with the 100 or so plans in the program over benefits and premiums they can charge starting in January. Those new benefits and rates will be officially announced sometime in October and employes and retirees will have a three-week "open enrollment" season starting in November to shop around for 1984 health insurance coverage.

Although the negotiations are confidential, The Washington Post has learned that OPM is proposing the following:

* More plans should offer catastrophic insurance coverage to pick up medical costs if individuals or families are hit with heavy bills in excess of $8,000 a year;

* Insurance plans should "restructure" benefits so that the overall cost of their health package will not be increased if they offer the so-called catastrophic coverage;

* A deductible of $500 should be established for in-patient treatment of mental or nervous disorders. The proposed deductible would not apply to other kinds of treatment, which would have a lower deductible.

Currently most health plans in the FEHBP have deductibles, but they are not limited to specific kinds of treatment.

Blue Cross-Blue Shield, the largest plan in the program, has a $50 deductible for hospital and clinic treatment in its high-option (most expensive) plan and a $100 deductible in its standard-option plan. The deductible for out-patient treatment is $200 in the high-option and $250 in the standard-option.

Government employes and retirees now pay from $300 to $1,500 a year in premiums for health insurance, depending on the plan they choose.

The federal health benefits program includes plans offered by nationwide giants such as Blue Cross-Blue Shield and Aetna, local health maintenance organizations (HMOs), unions and special groups of workers such as employes of the FBI, National Security Agency and the CIA. Most members of Congress and their staffs belong to the FEHBP, as do many high government officials. Former President Lyndon Johnson had Blue Cross-Blue Shield coverage in the Senate and White House.

In most cases, premiums paid by employes are more than matched by the government. Using a complicated formula, agencies pay anywhere from 40 percent to 75 percent of the total premium. That is why the OPM is anxious to hold down overall costs of the program.

Despite OPM-ordered cutbacks in benefits, premiums in the federal health program have jumped 50 percent in the past two years. Although the new premium rate will not be announced until October, insiders anticipate the average employe cost could rise another 20 percent in 1984.

Many key OPM officials believe that many people in the FEHBP over-utilize mental benefits. They think that assigning a large deductible to that benefit would discourage "unnecessary" treatment.

Many health plans in the program are balking at the OPM proposals. They are especially critical that the deductible would apply to in-patient treatment because that would most hurt people most in need of help.