For the past two days, more than 20 pianists have been playing some of the most challenging music of this century in the Coolidge Auditorium of the Library of Congress. Most of the time they have been heard by fewer than a dozen listeners, including their five judges, even though the sessions have been open to the public without charge.

The pianists, who have come from across this country as well as one each from Argentina and Brazil, are taking part in preliminary auditions leading up to the finals of the Kennedy Center's first international competition for excellence in the performance of American music. Those finals preceded by a week of semifinals, will take place in the center next Sept. 16 and 17. The following year the competition will be for singers.

At the library on Thursday and Friday, the judges conducted some of the most unusual auditions in the history of contests.Much of the time they listened to music by George Crumb, John Cage, Curtis Curtis-Smith, Aaron Copland, Dane Rudhyar and other contemporary composers. Often that music came from "prepared" pianos that had been equipped - by each pianist in his turn - with the requisite assortment of chains, wires (for pulling underneath the strings), glass tumblers, and other material needed for some of the most recent compositions.

After hearing each competitor in various examples of these American works, the judges then asked for samples from the more standard repertoire: Mozart, Brahms, Beethoven, Mussorgsky.

The judges, nearly all of them distinguished for their playing of contemporary American music, are pianists Robert Black, Robert Miller, George Pappastavrou, Beveridge Webster, and composer-piansit Ulysses Kay. They have now completed preliminary rounds in Chicago, Los Angeles, New York and Washington, which leaves only the European auditions to be heard in Vienna, Austria, on Monday and Tuesday. Entries for those sessions have come from many Western European countries and also Poland and Czechoslovakia.

The Soviet Union sent respectful regards but said there would be no Russian entries because they require a minimum of one year for preparation for entrants in any competition. This year's contestants have had about nine months to prepare.

When the judges have completed the final auditions in Austria, they will choose eight pianists for the semi-finals and finals, which will be heard in the Kennedy Center beginning Sept. 11.

With prizes totaling over $30,000, plus concerts, recordings and other engagements sure to go to the top winners - the first prize being $10,000 - it is not surprising that 120 pianists from around the world have entered. Each semifinalist will receive not less than $1,500.

Meanwhile, over at Lisner Auditorium, judges are hearing nearly 80 contestants entered in the 20th annual International Bach Competition. This year's contest is, for the first time, for violinists and cellists rather than pianists. Today's sessions begin at 9 a.m., with the finals scheduled tomorrow afternoon at 2. All of these auditions are open to the public without charge.