Sen. Charles McC. Mathias Jr. (R-Md.) unleashed a rare rhetorical broadside yesterday against the "appalling cost" of congressional mailings, which he said has more than doubled in five years to $111 million.

But in his gentlemanly style, Mathias declined to reveal which of his colleagues are stuffing constituents' mailboxes at taxpayer expense.

"It would be like talking about other members of the medical profession," spokesman Ann Pincus explained. "It's not done."

Mathias, chairman of the Senate Rules and Administration Committee, released a committee study of unsolicited mass mailings, which lawmakers can send out at no cost to themselves under the congressional franking privilege.

Twelve senators were responsible for half the costs of last year's Senate mailings, with a lone senator accounting for more than 10 percent, Mathias said.

Although senators from large states would be expected to incur higher costs in sending out newsletters and press releases, Mathias said, there were exceptions. Two of the 12 biggest Senate mailers hail from states that rank between 25th and 30th in population. And mailing costs for six senators from the six most populous states ranked from 16th to 95th. At the same time, Mathias said, eight senators had no mailing costs last year, one of whom was from a state whose population ranks in the top 10.

Although public money is being spent on the mailings, the figures for individual lawmakers' offices are not made public. Nor can they be obtained under the Freedom of Information Act, from which Congress exempted itself.

The figures leaked out in 1982, when it was revealed that Sen. John Heinz (R-Pa.) sent out 15 million pieces of mail at an estimated cost of $2.25 million. Heinz was followed by Sens. Alfonse M. D'Amato (R-N.Y.), 14.4 million pieces; Charles H. Percy (R-Ill.), 13.5 million; Lloyd Bentsen (D-Tex.), 11.8 million; and Daniel Patrick Moynihan (D-N.Y.), 11.6 million.

Spokesmen for some of those offices said yesterday they had cut back in recent years because their allotment of paper has been limited. But Mathias said that the limit has been raised from 1 to 1 1/3 sheets of paper for each voting-age citizen in the state, and that some senators are exceeding the higher ceiling.

The unidentified champion Senate mailer eclipsed Heinz's total by spending $3.8 million last year. Mathias said that equals the income taxes paid by 2,188 families of four earning $20,000 a year.

Congress must pass an $11.9 million supplemental appropriation to cover this year's huge volume of mail, Mathias said. He said the cost will jump to an estimated $144 million in 1986.