The head of the $80 billion-a-year Medicare program says that there has been an "alarming" increase in federal costs in the nonhospital part of the program.

Nonhospital costs were 22 percent higher in the first six months of 1987 than in the corresponding period last year, according to Dr. William L. Roper, head of the program.

By contrast, the government's costs for the hospital part of the program rose only 3.9 percent over the same period, according to Medicare figures.

The numbers illustrate some problems and trade-offs inherent in managing a giant federal program like Medicare, which provides health insurance for 31 million elderly and disabled Social Security beneficiaries.

Roper told reporters Wednesday that the increase in outpatient and nonhospital costs is in part the result of policies adopted in recent years to induce patients to shift "from high-cost hospitals to low-cost outpatient" services to save money.

The shifts have taken place and overall have helped Medicare save on total outlays. But Roper said one consequence is that utilization of outpatient services has grown "dramatically," with attendant increases in outpatient costs.

Roper also announced that in 1988 Medicare will raise by 13.5 percent the fees it pays to Health Maintenance Organizations for Medicare patients who choose to enroll in them -- receiving all their medical services from the HMO rather than from private doctors, who are paid separately for each service.

Roper said it is difficult to help people assess quality of care, and that is one reason Medicare will soon publish a volume as thick as a telephone book listing death rates for each hospital in the nation as well as the death rates that could be expected at each hospital based on its mix of patients. People can use the data to compare hospital performance.

Roper said Medicare is developing a "totally new data system" to help it improve management of the program. Now, he said, the costs for inpatient and outpatient services for a given person are collected in separate data systems and it is extremely difficult to put the two halves together for a full view of services offered each person. He said the PRISM project (Project to Redesign Information Systems Management), a complete reworking of the data system, is an effort to correct this failing.