Three self-styled "citizen" lobbying groups yesterday accused the oil industry of reaping higher profits than it reports as the Senate prepared to open debate on a scaled-down version of the administration's proposed "windfall profits" tax on oil.

In a joint study, the Citizen-Labor Energy Coalition, the Public Citizens Reform Research Group and the Energy Action Educational Foundation charged that third-quarter profit increases for the 18 largest oil companies were "substantially" higher than the average 103 percent increase they reported.

They said the oil companies understated their profits by up to 50 percent through several kinds of accounting "chenanigans," including counting as paid taxes they have really deferred on income earned abroad and using artifically fast tax write-offs for unsuccessful drilling.

The groups also disputed claims by the industry that most of its reported profit increase came from foreign operations, saying a "substantial" portion of foreign-earnings profits came from the sale of imported oil to domestic consumers.

In any case, they said, domestic crude oil producers are averaging a 23 percent return on their equity, or investment, considerably more than the average for all American industries.

In defense of the oil industry, the American Petroleum Institute said the accounting methods used by the companies are legal and commonly accepted ones in industry.

The "citizen" lobby report was timed to strenghen the case in the Senate for increasing the "windfall" tax, which is designed to capture some of the extra money that oil companies will take in as a result of President Carter's decision to phase out oil price controls by September 1981.

The Senate Finance Committee performed major surgery on the administration's tax proposal. The committee-approved bill would raise only half as much as a bill passed earlier by the House and preferred by the administration.

Under current estimates the Finance Committee bill would take about $138 billion of the estimated $1 trillion that oil companies are expected to get from decontrol over the next 10 years. The House bill would cost the oil industry about $273 billion.

The administration, in an odd alliance with Sen. Edward M. Kennedy (D-Mass.), Carter's chief challenger for the Democratic presidential nomination, is expected to join the "ciitizen" lobby in pushing the Senate to strengthen the measure.

Kennedy issued a statement while campaigning in Minnesota yesterday, saying that he and Sens. Dale Bumpers (D-Ark.) and Howard M. Metzenbaum (D-Ohio) will offer an amendment today to restore the bill to its House strength. They would use the extra revenue to help pay Social Security benefits and avert scheduled Social Security tax increases over the next several years, which Kennedy said would be "one of the most effective steps Congress can take to fight inflation," since a lower tax would mean lower labor costs for business.

Kennedy also said if Congress does not strengthen the windfall tax he will renew his call to Carter to reimpose price controls. And in a separate development, 102 members of the House called on Carter to reimpose controls and allocation rules for home heating oil.