After spending $1.7 million on controversial ads featuring actors that look like Jimmy Carter and House Speaker Thomas P. (Tip) O'Neill Jr., Republicans have decided to take a more positive approach to television advertising.

Instead of blaming the nation's economic ills on the Democrats, as the first ads did, a new set of commercials shown the Republican National Committee in closed meetings yesterday is built around the theme that Ronald Reagan and his party "have made a beginning," and should be given a chance to finish the job.

They picture folksy-looking older Americans talking about receiving a 7.4 percent cost-of-living increase in their Social Security checks and a 10 percent cut in income taxes July 1.

In one ad, a postman with a bushy white mustache says, "I'm probably one of the most popular people in town today," as he delivers Social Security and paychecks.

The changes aren't much, he admits, "but President Reagan has made a beginning. For gosh sakes, let's give him a chance."

The ads are the work of Rep. Carroll A. Campbell Jr. (R-S.C.), who produced them himself because, he said, "We Republicans have some things we want to say from a positive point of view, and nobody was saying them."

They are expected to air in about two weeks after testing and minor revisions. They represent a major change in advertising strategy by Republicans, an obvious attempt to neutralize the sensitive Social Security issue.

Perhaps more intriguing than the ads themselves is a spirited debate, fought behind the scenes on Capitol Hill and in the White House in the past month, over the approach the party's $10 million ad campaign should take and who should produce it.

It involves bruised egos, subtle power plays, snide second-guessing, and some hardball politics between the Democratic and Republican parties and among the nation's three major television networks.

The stakes were exceedingly high. "If this ad program is done effectively it could mean the margin of victory next fall," says White House political director Edward J. Rollins. "Ten million dollars in the advertising budget is a lot of money. It could mean a 5 or 10 percent difference in the vote."

Most key players in the conflict are reticent about details. But as pieced together in more than a dozen interviews, the outlines look like this:

Earlier this year, the Republican National Committee and the National Republican Congressional Committee contracted with Korey, Kay & Partners, a New York advertising agency that normally doesn't do political assignments, to produce party ads. The ads it delivered were slick, negative and cutesy.

They borrowed heavily from the highly effective ad campaign the GOP mounted in 1980 that used an actor to portray O'Neill running out of gas on a highway. The most controversial 1982 ad, released with great fanfare May 17, used the same actor, Edwin Steffe of New York, to play O'Neill, and Ed Behler, of Waco, Tex., to play Jimmy Carter.

The two actors sat silently as a "last will and testament" was read by a third actor, portraying a lawyer, who said, "To Ronald Reagan, we leave a recession."

The ads got a mixed reception from White House political advisers and key Republicans.

"A group of congressmen gave them a standing ovation. They loved them," recalls one strategist. "But the only reaction was a stony silence when we showed them to a group of Republican political consultants."

The public reaction was worse. The day the ad was released, Rep. Tony Coelho (D-Calif.), chairman of the Democratic Congressional Campaign Committee, denounced it as a lie, saying the economy was growing when Carter left office. In telegrams to networks and station managers, he noted that the National Bureau of Economic Research had said the recession began in July, 1981.

Coelho urged Democratic leaders around the country to demand equal time from television station managers to reply to the ad. This was successful in a few cases. In Tennessee, for example, one station gave the party a half-hour to reply to the ad.

ABC, CBS and NBC television all initially refused to run the ad, although it did run later on ABC. The ads also produced a host of unfavorable editorials around the country.

"The ad was an insult," says Robert Squier, a respected Democratic political advertising consultant. "They reverted to old product advertising techniques and ignored everything we've learned in this field. Instead of doing deodorant ads they were using ads as deodorant."

The reaction wasn't much better among many GOP leaders.

"The ad was aimed at a second-grade level. It was ineffective," said Jennifer Dunne, Washington state Republican chairman. "I was looking for a message that was clear and hard-hitting that could help our candidates. I didn't find it very creative."

Some party leaders objected to the idea of mimicking a former president. Others didn't like the idea of a negative ad campaign. Still others felt the message in the ad was murky--a view White House political operatives shared.

The RNC and the NRCC, the party's two most effective fund-raising organizations, quietly decided to quit using the ad and sever its relationship with Korey, Kay & Partners after a month-long run.

The whole controversy probably would have passed unnoticed had Rollins, the White House's top political operative, not told reporters on a presidential trip to Texas last week that the party had dropped the ad and fired the ad agency because the ad was ineffective.

The next day, Rep. Guy Vander Jagt (R-Mich.), chairman of NRCC, issued a statement that Rollins was incorrect. "The ads that have been referred to have run their course--totally and completely," he said.

Meanwhile, Campbell, an ambitious young congressman and Reagan loyalist, had been working quietly on his own. He said he had become concerned that all four of South Carolina's Republican congressmen might not survive next fall's election unless the political dialogue shifted, so he wrote the new ads and used his own campaign funds to produce them.

He said he had gotten a favorable reaction from White House political advisers, about 30 congressmen, and party leaders from around the country to whom he has shown the ads.

The RNC is expected to begin financing the ads within several weeks. "What we're looking for is the best story, and Campbell has got it," Republican National Chairman Richard Richards said yesterday.

The NRCC, which controls $6 million of the party's $10 million advertising budget, has yet to make a decision on them.