President Reagan is trying to lure Stuart K. Spencer back from California to shape up the White House political office and quarterback the 1982 midterm campaign, informed White House and Republican Party sources said yesterday.

Spencer, who ran Reagan's gubernatorial campaigns in California and served as top strategy adviser aboard the Reagan campaign plane last fall, is expected to give his answer on the offer this weekend.

Officials said Spencer had declined at first to consider leaving his California political consulting firm but appeared in recent conversations to be yielding to White House pressure. One source said he might sign on initially as a part-time consultant to Reagan's chief of staff, James A. Baker III, rather than as a full-time staff member.

The post has been held for the past nine months by Lyn Nofziger, a longtime Reagan press secretary and political aide. Nofziger confirmed yesterday his long-stated intention to return to his own political consulting business by the end of the year.

There have been growing expressions of concern in recent weeks from senior White House staff members, Republican congressional leaders and party officials about the apparent lack of a coordinated campaign plan for 1982.

While the GOP's three major campaign arms, the Republican National Committee, the National Republican Senatorial and National Republican Congressional campaign committees, are far ahead of any previous year in fund-raising, upsets and narrow victories in several special elections have triggered concern about political preparations for 1982.

There also have been public conflicts about the stance the GOP would adopt in 1982 toward conservative Democratic congressmen who have been supporting Reagan's program, and about the relationship among the White House, the campaign committees and independent conservative political action groups.

Much of the criticism from the White House has been focused on party chairman Richard Richards. But informed officials said yesterday that while no decision has been made to replace Richards, changes are being made in his staff.

The first came Monday when William I. Greener left his post as public affairs director of the Department of Energy to become communications director of the Republican National Committee.

Spencer is a somewhat controversial figure to some Reagan supporters because in 1976 he played a key role, along with Baker, in helping then-President Ford turn back Reagan's challenge for the Republican presidential nomination. But Spencer was invited back into Reagan's circle in the summer of 1980 and is credited by the president and senior aides with playing a vital role in the fall campaign.

Nofziger is reported to have recommended Edward Rollins, his deputy and fellow Californian as a successor. Unless plans change, Rollins is scheduled to leave in the middle of next month to take over Lt. Gov. Mike Curb's campaign for the California Republican gubernatorial nomination.

Reports also have persisted that some of Nofziger's duties might be given to Richard Williamson, Reagan's assistant for intergovernmental relations.

Nofziger enjoys the trust and strong support of conservative activists in the GOP because of his unbroken loyalty to Reagan since the president's first campaign for governor in 1966.

But Republican officials outside the White House have complained that Nofziger's apparent lack of a close working relationship with Baker and presidential assistant Michael K. Deaver have led to frustrating delays and confusion.

Nofziger said yesterday that he had experienced "no more than the usual frustration" in the job and had "no problems with anyone" on the senior White House staff.

"My problem," he said, "is with government as an entity, and I don't want to stay here so long I become a part of it."