A House Ways and Means subcommittee yesterday began to probe the hornets' nest of life insurance taxation, an issue that has divided the industry into two hostile camps and an issue in which at least $1.3 billion a year is at stake.

The relevant section of the law is so complex that Congress repeatedly has abandoned its tax-writing responsibilities in the past and has let the industry write its own code, demanding only that certain revenue targets be reached.

The result is that an estimated 80 percent or more of the investment income of the top 20 mutual and stock life insurance companies goes untaxed, according to testimony yesterday by the General Accounting Office.

This year, however, there are a set of time bombs in the code in the form of temporary provisions that will expire on Dec. 31. If Congress takes no action, the burden of these provisions--about $1.3 billion in additional taxes--will fall almost entirely on one segment of the industry, the mutual life insurance companies.

In addition, Congress and the Treasury are pressing for a major overhaul of the law that would be likely to increase the industry's total tax burden significantly above the $2 billion a year it now provides the federal government.

These intense pressures are emerging at a time when the mutual companies and the stock companies have been unable to reach a private agreement on revising the law.

Initially, they formed a committee of 20 to work out a plan, then reduced the number to 8 and finally to 2, according to sources. But when the two-person committee reached agreement in principle, the recommendations were rejected by the original committee of 20.

Throughout these negotiations, distrust was so deep that, whenever a representative of one side of the industry saw a member of Congress or an aide, a "chaperone" from the other side was required to be present, sources said.

With the breakdown of negotiations, the tax-writing burden has been passed on to a reluctant Congress, where few members understand the law and even fewer want to charge into a politically loaded conflict between mutual and stock companies, not to mention their agents and policyholders.

In addition, some members of Congress are looking skeptically at provisions that allow the creation of policies that are as much tax shelters for investments as forms of life insurance.

At a hearing before the Ways and Means select revenue subcommittee yesterday, the Treasury Department took what was seen as a hard line, outlining a "conceptual framework" for the taxation of the companies that calls for a far broader corporate tax base than now exists.

In addition, the Treasury would restrict the tax breaks for policies that function more as tax shelters than insurance coverage, and reduce the ability for policy holders to get loans and other tax-free or tax-deferred benefits from policies, similar to restrictions passed last year on pension plans.

In an effort to describe existing law, John E. Chapoton, assistant Treasury secretary for tax policy, quoted a U.S. Court of Claims opinion that declared: "These complex and obscure provisions bear all the earmarks of a conspiracy in restraint of understanding."

Referring to some of the "special deductions" available to insurance companies under the law, Chapoton questioned whether the "justification for these provisions withstands serious scrutiny. Moreover, insurance companies have entered into non-economic arrangements primarily to take advantage of these provisions."

Chapoton was critical of provisions currently benefiting both the mutual and stock companies, suggesting that, if specific reform legislation is proposed, it will fall on both segments of the industry.

The current conflict grows out the delicate and complex distribution of the tax burden between the mutual and stock companies following passage of the last major change in the code. in 1959.

The two segments shared the tax burden in rough proportion to the share of the total assets in the industry, as mutuals were taxed largely on the basis of investment income, while the stock companies were taxed largely on the basis of underwriting income.

Investment income is far more sensitive to interest rates than underwriting income, however, and when interest rates began to shoot up in the late 1970s, so did the share of the tax burden paid by the mutuals.

The mutual companies were able to use a special provision allowing them to recharacterize much of their investment income as underwriting income, and consequently significantly reduce their tax burden, however.

The Joint Committee on Taxation found that, when the mutual companies began to use this special provision extensively in 1980, their share of the industry tax burden fell from 60 percent to 49 percent, while the stock companies' share rose from 40 to 51 percent.

Last year, however, Congress eliminated the special provision, in a step that would have increased the tax burden on the mutuals by $2.3 billion. This was considered to be too large, and temporary provisions were added, reducing the increase to about $1 billion.

These temporary provisions end this year, leaving the mutual companies in danger of a major tax hike if Congress fails to act. "The mutuals are desperate; the stocks companies are sort of chuckling," one participant in the controversy noted.