Standing inside the high arched windows of Dolley Madison's historic house, Sen. Charles McC. Mathias Jr. (R-Md.) looked out upon Lafayette Park's flowery green and recalled that once there were heavy pressures to make the park a canyon of high-rises government office buildings, leaving St. John's Church, as a said, "a peanut in scale."

In order to save the Lafayette historic houses, Mathias proposed to make them residences for Cabinet members. Well, that didn't come about, but the houses were saved, to be used as offices, and new office buildings were tucked in behind. Now Mathias thinks the problem is "how to spreach such successes as the Lafayette Square environs to other parts of this city and other parts of America . . . too many of us blindly accept deadly dull buildings and places as the way things are."

Mathias spoke yesterday at the first annual meeting of the Partners for Livable Places, a coalition of 29 organizations, from the America the Beautiful Fund to Vision, Inc., all dedicated to the proposition that the cities can be beautiful, exciting and healthful places to live. About 100 delegates and guests heard a battery of speakers from government departments and private organizations who put in their $2 billion worth of ideas.

Rep. John Brademas (D-Ind.) cited as evidence of federal concern President Carter's new proposal to spend $20 million for a 'Livable Cities' arts program.

Brademas pointed out that not all federal programs have to be big spenders, citing the Arts and Artifacs Indemnity Act, which he said, "now makes it possible for exhibitions of art to be brought to the United States from other countries for showing in American museums without the requirement that our museums pay extraordinarily high insurance premiums."

The government has insured, for instance, the King Tut tour, as well as the Dresden art exhibit coming to the National Gallery next month.

Jay Solomon, General Services administrator, Mathias and Brademas spoke of their support for the Building Building, the proposed National Museum of the Building Arts to be established in the old Pension Building.

Solomon said GSA plans to rehabilitate the Old Customs House in New York whose murals alone are said to be worth $1 million. "And in the last year we have commissioned 125 major pieces of art for the country. We have been able to do this because of the encouragement from President Carter, Rosslyn, and Rep. Brademas and Sen. Mathias."

Solomon himself came in for a great deal of praise from the various speakers, including Brademas, who said he had never expected a GSA administrator as sympathetic to the causes of architecture and city livability as Solomon. "Esthetically speaking, he can read and write without moving his lips, unlike some people who have held his job."