Republican candidates in state races generally had trouble latching onto President Reagan's coattails Tuesday, although the GOP made a net gain of one governorship and control of five legislative houses, including the House in Minnesota, Walter F. Mondale's home state.

And while many Republicans credited their victories to concern about high taxes, sweeping initiatives to restrict state taxing authority were defeated in Michigan, California and Nevada. A property tax cap was trailing in Oregon but remained too close to call.

Among more than 200 other state and local ballot issues, gambling initiatives fared well nationwide with California, Oregon, Missouri and West Virginia approving state lotteries.

In taking over four governorships from Democrats, Republicans elected their first chief executive in Utah in 20 years, their first in Rhode Island in 16 years and their first in North Carolina in 12 years. They also regained the West Virginia statehouse.

But Republicans also lost three governors' offices to Democrats -- Washington, North Dakota and Vermont.

This left Democrats in charge of 34 statehouses and Republicans in charge of 16.

In Minnesota, where a 12-seat gain gave the Republicans control of the state House for the first time since 1972, all the Democratic candidates had their pictures taken with Mondale, but few used the photos in their campaigns.

"My guess is they weren't confident enough of a Mondale victory here in Minnesota that they wanted to tie themselves to him," said state House Republican Leader David Jennings, who will become speaker. Jennings said the Republicans won by promising to use a big state budget surplus to cut taxes across the board. He said Mondale's call for a national tax increase nearly cost him his only state victory.

In North Carolina, conservative Rep. James G. Martin (R) handily defeated Attorney General Rufus Edmisten (D) to replace Gov. James B. Hunt Jr. (D), who lost his race for the Senate. Without Reagan's strong showing, said Martin campaign manager Jack Hawke, "we Republicans would have had a problem in North Carolina. It's a traditional Democratic state, and the president had a great deal of effect in convincing people that their grandfathers won't roll over in their graves if they vote Republican."

In Rhode Island, Cranston Mayor Edward DiPrete (R), who, like Martin, spoke of cutting taxes, beat state Treasurer Anthony J. Solomon (D) in the race to replace retiring Gov. J. Joseph Garrahy (D).

In Utah, House Speaker Norman H. Bangerter (R), promising no tax increases for two years, bested former representative Wayne Owens to succeed Gov. Scott M. Matheson (D).

And in West Virginia, former governor Arch A. Moore Jr. (R) beat House Speaker Clyde M. See Jr. (D) to reclaim his old job from Gov. John D. Rockefeller IV (D), who was elected to the Senate.

But the 13 gubernatorial races also saw the ouster of two GOP incumbents. Washington Gov. John Spellman (R) fell to Pierce County (Tacoma) Executive Booth Gardner (D), an heir to the Weyerhaeuser fortune, despite attacks that he was too close to labor.

North Dakota Gov. Allen I. Olson (R) lost his job to state Rep. George Sinner (D) in a bitter contest typified by a Sinner ad that showed a farmer slinging cow manure.

And in Vermont, former lieutenant governor Madeleine M. Kunin (D) squeaked by Attorney General John J. Easton Jr. (D), barely winning the 50 percent majority she needed to keep the election out of the Republican House and become the state's first woman governor.

Kunin is the country's second woman governor, with Kentucky Gov. Martha Layne Collins (D).

She succeeds retiring Gov. Richard Snelling (R), whom she challenged unsuccessfully in 1982.

Elsewhere, Indiana Gov. Robert D. Orr (R) defeated state Sen. W. Wayne Townsend (D). New Hampshire Gov. John H. Sununu (R) turned back House Minority Leader Chris Spirou (D). In Delaware, Lt. Gov. Michael N. Castle (R) defeated former judge William T. Quillen (D) to replace Gov. Pierre S. du Pont IV. And in Missouri, conservative Attorney General John Ashcroft (R) beat Lt. Gov. Kenneth J. Rothman (D) to succeed Gov. Christopher S. (Kit) Bond (R).

On the Democratic side, Arkansas Gov. Bill Clinton (D), a rising party star who lost his job in 1980 but reclaimed it two years later, trounced contractor Woody Freeman (R). And popular Montana Gov. Ted Schwinden (D) easily beat state Sen. Pat M. Goodover (R).

Republicans scored a double victory in wresting control of both houses of the North Dakota and Connecticut legislatures. Part of the reason, said Connecticut Senate Republican Leader Philip Robertson, "is that Ronald Reagan won 61 percent of the vote in Connecticut. Mondale and his tax increase did not turn on the people of Connecticut." Robertson also said that voters blamed the ruling Democrats for a tax hike last year that was the largest in state history.

Republicans also took control of the Ohio Senate, Delaware House and North Dakota House, while losing control of the Vermont and Alaska legislatures. Thus, the GOP continued to control 11 legislatures, but reduced those totally run by the Democrats from 33 to 27.

While Republican candidates made considerable political hay out of Democratic tax increases, voters in three states were unwilling to accept more radical anti-tax measures.

A Michigan initiative to strip state lawmakers of the power to raise taxes was trounced, 2 to 1. A huge early lead in the polls vanished as Gov. James J. Blanchard (D) was joined by two former Republican governors in a television ad warning that the measure would cripple the state.

The Voters Choice measure would have rolled back Blanchard's 1983 income tax increase and required a referendum for future state tax hikes. The state's political establishment, from the AFL-CIO to the big auto makers, mounted an expensive campaign against it.

Blanchard spokesman Richard Cole said the vote showed that the measure "went too far" and that voters "didn't want to kill Michigan's comeback."

Nevada's anti-tax measure, which also would have required a public vote on future tax increases, lost narrowly. A property tax cap in Oregon was trailing, but considered too close to call.

In California, voters handily defeated a tax-cutting initiative written by Howard Jarvis to extend his successful 1978 referendum called Proposition 13. It would have required a two-thirds legislative vote to impose new state taxes and a two-thirds popular vote to boost local fees and taxes.

Unlike the earlier Jarvis measure, however, Tuesday's proposition was complicated and uneven, promising a large tax rebate to some homeowners while others would be saddled with a property tax increase.

In addition to the states that approved lotteries, Missouri also approved pari-mutuel betting. But voters in Arkansas and Colorado rejected proposals to authorize casino gambling in designated communities.

In Arizona, voters pulled the plug on several measures to control hospital costs. One set, sponsored by major Arizona corporations, would have allowed the state to regulate health facilities and oversee health planning. Competing initiatives backed by the Arizona Hospital Association would have placed a ceiling on hospital revenue and frozen construction of new hospital beds.

In California, voters defeated initiatives to cut sharply health and welfare benefits, limit campaign contributions and transfer authority for political redistricting from the legislature to a panel of retired judges. Residents of West Hollywood, where homosexuals hold political power, voted to incorporate as a city.

In Maine, church and antiabortion leaders helped defeat a proposed equal rights amendment. In Missouri, a measure to restrict utility rate increases and disposal of nuclear waste failed after an opposition campaign funded by state utilities. In West Virginia, a measure to allow voluntary prayer in schools passed overwhelmingly. And in Utah, a proposed ban on pornography on cable television was defeated.

Voters in Washington state approved continued state funding of abortions, while such funding was narrowly banned in Colorado.

Several communities in California, Oregon and Washington voted to bar nuclear weapons within their borders, but a similar measure was rejected in Ann Arbor, Mich.

In East Detroit, residents rejected a proposal to change the Michigan suburb's name to Erin Heights and weaken its identification with Detroit's high crime rate.