In a move toward a compromise on national health insurance, Sen. Edward M. Kennedy (D-Mass.) said yesterday he could accept a plan that expanded coverage step-by-step only if skyrocketing health costs were successfully controlled.

Kennedy and the liberal coalition behind him have until now urged Congress to pass a $40 billion comprehensive plan that would order step-by-step implementation without qualification.

Kennedy's move was seen as a concession to the growing feeling that Congress is unwilling to approve any such plan. It also brings the Massachusetts senator closer to President Carter and Sen. Russell B. Long (D-La.), chairman of the Senate Finance Committee and the sponsor of a limited health insurance program that seems to be gathering support.

Kennedy said yesterday, in testimony before Long's committee and in a later interview, that he could accept a $10-billion-a-year "first step" toward an overall plan.

But the beginning phase must include strict cost controls and health system reforms, Kennedy said, and there should be advancement to other phases without new action by Congress if the plan stayed within its budget "objectives."

Coverage in this first step would include help for "youth and the elderly" and "preventive services," he said, including prenatal, obstetric and pediatric care for mothers and infants. He said other details will be announced after July 4.

His $10 billion price tag seemed significant, as part of what seems to be a sudden series of moves in the Senate toward a possible health insurance compromise.

President Carter has proposed a $24.3-billion-a-year plan for the aged, poor mothers and infants and the very ill who outrun their ordinary insurance coverage. But Carter is expected by many observers to strike a deal sooner or later that might bring him close to $10 billion to $12 billion a year limit suggested by Sen. Long.

Long would target his plan toward the "catastrophically ill" - those who outrun other coverage - and the poor and the aged.

In a similar vein, several other Finance Committee Democrats and Republicans have agreed in work sessions this week that coverage for the unprotected catastrophically ill should be a first priority in any new insurance bill.

Kennedy and backers urged other priorities. They said a flood of dollars to pay for catastrophic ills would skew the nation's costly health system even further toward the costliest care, unless the dollars were limited by strong controls and there was help for other groups too.

But Kennedy also said aid for the elderly under his first step could be geared to the expensive, catastrophic illnesses where even Medicare runs out. Also, he said, it should help the elderly "with modest rather than generous incomes."

He expanded on these points after a four-hour Finance Committee session, at the end of which Sen. Abraham A. Ribicoff (D-Conn.) said he believes that a health insurance compromise "isn't going to work, frankly, unless Sen. Kennedy has a piece of the action" - and unless Long, the president and Sen. Bob Dole (R-Kan.) each has a piece too.

Dole and some fellow Republicans are backing a $6-billion-a-year plan mainly for catastrophic illnesses, though Dole has indicated he is willing to negotiate to do more for the needy. CAPTION: Picture, Sen. Edward M. Kennedy waiting to testify on health insurance proposals. By James K. W. Atherton - The Washington Post