The General Accounting Office says the formula used by the government to determine how $1 billion in health insurance premium refunds will be split between Uncle Sam and government workers and retirees shortchanges the policyholders by $98 million.

The premium split proposed by the congressional watchdog agency would result in substantial premium refunds for some policyholders and cutbacks in proposed rebates for persons in other plans.

More than 250,000 people here are eligible for the refunds.

Eleven federal health plans, six of them available to workers and retirees in the Washington area, are offering premium refunds to 1985 policyholders. Some of the plans decided to make refunds after OPM earlier this year ordered health plans to reduce their reserves by trimming 1986 premiums or offering refunds for premium overpayments this year, or both. The buildup in premium reserves, federal officials say, resulted from a decline in health-plan usage by employes and retirees, or by policyholders paying more of their own medical costs.

The Reagan administration approved the refund proposal last summer. But the checks have been held up until Congress can change the federal health insurance law to make retirees eligible for the same refunds as workers. The Senate joined the House last week in approving a rebate bill, but a compromise must be reached on the two bills before the payments can be authorized.

Nationwide, more than 2 million workers and retirees in the plans are due some kind of refund.

The GAO report, prepared for Rep. Mary Rose Oakar (D-Ohio), looks at the method used to determine how the refund pie should be divided. GAO said the complex formula the government used doesn't produce a fair refund split based on premiums paid.

The exact amount of individual refunds depends on which plan and option an employe or retiree was enrolled in this year, and whether the individual is a white-collar civil servant or a postal employe.

Blue Cross-Blue Shield, the largest health plan, is offering refunds ranging from about $18 for postal employes with standard option, single coverage to nearly $400 for nonpostal employes or retirees with high option family coverage.

The GAO says that splitting refunds based on how premium payments are shared by government and employes would result in:

Bigger refunds going to federal workers and retirees enrolled in the high option of Blue Cross-Blue Shield (an additional $153), Aetna (an additional $173), Government Employees Hospital Association (GEHA), Foreign Service plan and in the Blue Cross-Blue Shield standard option family plan.

Smaller refunds for individuals enrolled in the American Federation of Government Employees plan, National Association of Letter Carriers plan and the Government Employees Benefit Association plan.

No change in the refund split proposed for persons enrolled in the standard option (single and family) Aetna plan or for refunds proposed for single enrollees in the Blue Cross-Blue Shield plan.

The GAO recommended that the Office of Personnel Management revise the proposed refunds. OPM officials, plus those representing several health plans, said they cannot comment on the recommendations until they see the report.