Republican National Chairman Richard Richards yesterday denied trying to "stack" or "manipulate" the impending White House Conference on Aging to prevent it from criticizing President Reagan and his Social Security and budget programs.

Richards conceded, however, that the Republican National Committee financed a poll of the delegates to the conference and revealed that the list of delegates used for the telephone poll was obtained from Secretary of Health and Human Services Richard S. Schweiker.

Richards' statements came at a tumultuous meeting of the House Committee on Aging at which Chairman Claude Pepper (D-Fla.) accused the RNC and administration of attempting to "pervert and prostitute" the late-November conference in order to prevent it from taking a strong stand against "the disastrous impact that President Reagan's Social Security and budget proposals would have on the elderly."

Rep. Pepper released a letter asking the attorney general to investigate whether, by giving the names to Richards when no one else seemed to be able to get them, Schweiker might have violated the Freedom of Information Act. The letter also asks an investigation of whether the pollster, Campaign Marketing Group, acted illegally by not revealing to delegates that it was working for the Republican National Committee.

Several delegates testified that the pollster had led people to believe that the poll was being taken on behalf of the conference itself.

Witnesses who are delegates, such as Eisenhower administration Social Security commissioner Charles Schottland and several others, said the poll-takers had asked what they thought of Reagan's policies and performance to date, what party they belonged to, what organizations they belonged to and whether they favored Social Security cuts.

Pepper, Mary Rose Oakar (D-Ohio), Tom Lantos (D-Calif.) and other Democrats expressed the belief that one possible aim of the poll was to intimidate the delegates and that another was to identify pro- and anti-Reagan delegates, so the administration could stack key positions with administration supporters.

Pepper also told of reports that the conference might be allowed only one mass vote on the recommendations of all 14 panels combined, to avoid opportunities for amendments and for taking positions against Reagan.

Betty Brake, executive director of the conference, told the committee that she had no connection with the poll, that she was willing to give the delegate list to organizations of the aging as soon as it was in better form, and that she had no intention of packing key slots.

Richards, in a caustic statement, said the poll was paid for by the Republican National Committee, covered 916 delegates by phone, cost the committee around $9,000, and was a perfectly open and aboveboard attempt to see if the president's positions were known to older Americans. Once the results were tabulated, he said, he might mail the president's views to delegates. He said there was no attempt to stack key positions or intimidate the delegates.

Richards released a copy of the questionnaire used in the phone poll and said it did not ask people their party affiliation. But Paul Kerschner of the American Association of Retired Persons said some of his members had been asked that.

David Newhall, Schweiker's top executive aid, said later that the national's committee's request for the delegate list was the only one made directly to Schweiker, and that the secretary would have approved giving the list to others, too, had there been direct requests. Newhall said he thought the conference staff was giving the list to others.