Gov. George Deukmejian today launched the nation's first significant incentive pay system for gifted teachers, a cornerstone of President Reagan's plan to improve American education.

The plan, part of a new $26 billion state budget, will pay as many as 5 percent of the state's 200,000 teachers $4,000 extra a year.

Deukmejian initially had wanted to delay financing a wide range of educational overhauls, including the "master teacher" plan, but heavy lobbying by parents and teachers and last-minute compromises saved most of the package.

Bill Honig, California superintendent of public instruction, hailed the plan as "the strongest, most comprehensive educational reform bill in the country." The $800 million increase over last year's $7.4 billion education budget will raise new teachers' salaries, which now average $13,500, by up to 10 percent. It also will provide local school boards with enough funds to increase overall teacher salaries by 5 percent.

State officials said master teachers would be nominated by school district committees with a majority of teachers as members, a key provision that avoided organized teacher opposition. School boards will make the final selection. Master teachers will help train new teachers.

By signing the state budget today, Deukmejian ended a record 21-day delay caused by a pitched battle with the Democratic-controlled legislature. He guaranteed future wrangling, and possible court action, however, by cutting a record $1.1 billion from the budget, more than double the $500 million in cuts Ronald Reagan made as governor in 1971.

Democrats in the legislature have already sued the governor over his calling a Dec. 13 special election for a redistricting plan favorable to Republicans. Deukmejian said today he might scrap the election if Democrats let an independent commission redistrict the state. Today he struck language from a budget companion measure that he said unconstitutionally restricted his authority, once more inviting a suit.

Until the budget dispute was solved, state worker paychecks had been frozen and welfare and unemployment recipients had been paid only because of court orders.

At a news conference, Deukmejian said that he had wielded "a scalpel, not a meat ax" to make cuts. But agencies he has criticized for interfering with the state's economy and crime fighting suffered the deepest cuts. These include the state Coastal Commission, Energy Commission, Agricultural Labor Relations Board and Public Defender's office.