President Carter plans to nominate Pittsburgh Mayor Peter F. Flaherty to fill the second-ranking position in the Department of Justice, a White House source said yesterday.

The source said Flaherty's nomination to the position of deputy attorney general would be sent to Capitol Hill this week.

Flaherty, 50, one of the first big-city officials to endorse Carter's presidential campaign, would not confirm yesterday that he had been offered the post, but said he would accept the nomination.

"It's not exactly correct to say that I have the nomination," he said. "I'm under serious consideration at this point . . . I would accept it. It's a great and challenging post."

Flaherty, an attorney for 25 years and a two-term mayor, spent most of yesterday meeting here with Attorney General Griffin B. Bell and H. John Heinz III and Richard S. Schweiker, Pennsylvania's Republican senators.

The Associated Press reported yesterday that Heinz and Schweiker said they would support Flaherty's nomination on the condition that he would not use the post as a springboard to federal elective office.

Flaherty said that only Heinz questioned him in that regard.

"These were just courtesy calls," he said. "Sen. Heinz asked me how long I would stay" as deputy attorney general "and I told him that if I am nominated. I would stay four years -- the full term."

Flaherty's nomination and confirmation would solve a major problem for Bell, who has been trying for several weeks to fill the department's No. 2 post. The deputy, responsible for the day-to-day running of the department, also plays a major role in the selection of U.S. District Court judges.

The Carter administration earlier attempted to recruit U.S. District Court Judge Frank M. Johnson Jr. of Montgomery, Ala., for the deputy's job. But Johnson unequivocally refused the position, according to reliable sources.

Bell then appointed Richard L. Thornburgh, a Republican who had been serving as assistant attorney general in charge of the Criminal Division, to temporarily take over the deputy's post.

Flaherty was a city councilman in 1969 when he challenged Pittsburgh's Democratic regulars and successfully ran for mayor.

He is not regarded in Pittsburgh as having an extensive legal background. Sources there said he had a modest law practice and that he served six years as assistant district attorney for Allegheny County -- a part-time job --before seeking elective office.

Bell cited part of that record yesterday in a formal announcement that Flaherty "is under serious consideration" for the job of deputy attorney general. Bell praised the mayor for prosecuting "hundreds of cases" and for having a "broad background in criminal justice matters." The Attorney General also said that Flaherty was one of the leading public administrators in the United States.