TWO WEEKS ago, when Rep. Fred Richmond announced the formation of a congressional arts caucus, he had 48 House members signed up. A few days later he had 53. Last Thursday he had 73 at midmorning, and by the time he emerged from a noontime meeting, he had 77.

"It's caused a tremendous stir," said Richmond, a 58-year-old Brooklyn Democrat, art collector, millionaire, former Carnegie Hall chairman and creator of the Freddie Richmond Swing Band. "Every time I walk down to the floor, someone will say, 'Hey, I heard about your arts caucus, I think I better join.'"

It has taken very little effort: "I haven't personally called a soul," he said. "If I call, then they have to join. I've got 100 guys on that floor who owe me favors. But I don't want people to join because they owe me favors. I want them to join because they're interested in the arts."

After the initial announcement, "Joan Stratton, the wife of Congressman Sam Stratton [D-N.Y.], got so excited about the arts caucus when she heard about it that she called him in Japan to tell him to join," said Richmond. "Sam and I meet every few days because he's dean of the New York delegation. One day Sam said, 'What's this arts caucus you've got?' So I told him, 'It's a really good thing.' He said, 'I think I better join. I'm a member of the mushroom caucus. I might as well join this.'" Richmond chuckled at this story in his deep, gravelly voice.

Richmond created the caucus in response to anticipated Republican budget-cutting fever and the fiscal conservatism that defeated several longtime congressional arts advocates and threatens to slash at federal arts agencies' budgets. "A caucus hasn't been necessary up until now," said Richmond. "We've had tremendously powerful voices for the arts in Congress -- John Brademas [the former Democratic congressman from Indiana] and Frank Thompson [former Democratic congressman from New Jersey]. Now the only person left is Sid Yates -- who's wonderful -- but who couldn't join because he had a conflict of interest. He's chairman of the Appropriations Committee. So the next person in line is me."

Despite worries about the future of arts funding under the Reagan administration, Richmond says there's no reason why -- with a little advocacy from the caucus and other members -- the requested $175 million budget for the National Endowment for the Arts and the $169 million budget for the National Endowment for the Humanities couldn't pass Congress. "We think Mr. Reagan's bark is maybe 100 times worse than his bite," said Richmond.

But what about the bite of David Stockman, the new director of the Office of Management and Budget? "What's he going to do?" asked Richmond. o"Reduce the $175 million and get every arts agency in America mad at him? How much is he going to reduce it by? $30 million?"

The caucus' agenda "will be to support the budgets of the NEA and the NEH," said Richmond, "and to support such items of legislation as the caucus may vote to support." The caucus will hold meetings, elect officers (Richmond hopes to be elected chairman) and put out a newsletter.

If Richmond does not yet have the reputation -- or the chairmanship -- of a Sidney Yates, he has had a longtime interest in the arts.

Trained as a pianist, he formed the Freddie Richmond Swing Band in the '40s to earn money while he was in college. After college, from 1949 to 1960, he headed an investment group that bought companies and modernized them. "Everytime I made a deal, I went out and bought a piece of 18th-century furniture, Chinese porcelain or a painting."

One of his acquisitions was a 1906 Picasso crayon that he bought for about $10,000. "I had to sell it to pay campaign expenses," said Richmond, who is now entering his seventh year in the House. "That's when my friends knew I was really serious." He now owns more than 100 pictures -- "lots of Chagall." Dali watercolors, a Mary Cassatt watercolor, five Pasquin oils. No Picasso. "The latest things I'm buying are Greek and Roman sculpture of the second and the fourth centuries B.C.," he said.

A friend and designer, New Yorker Helen Fioratti, finds things for him at auctions. When he has vacation time, he travels to Europe -- at his own expense, he quickly points out -- haunting museums, looking, sometimes buying.

"Carter Brown [director of the National Gallery of Art] and I are trying to get the Alexander the Great sarcophagus from an Istanbul museum for a traveling show around the country," Richmond said. "It's probably the single most mind-boggling piece of art I've ever seen."

But the art on his office walls reflects his Brooklyn district and the city for which he used to work. It was December of 1959 when he went to see New York Mayor Robert Wagner about saving Carnegie Hall. At the time, he was in Wagner's cabinet. "I ran up to see the mayor," he said. "I told him, 'Hey, Bob, they're going to tear down Carnegie Hall.' Bob said, 'I don't think that's a very good idea. I took violin lessons there.' That was a Saturday. We got an order preventing demolition. On Monday we went to see Gov. Rockefeller. Within two weeks there was a bill approved by the New York City Council to purchase the building for $5 million. The state legislature approved a bond issue and transferred money to the owner." Richmond, violinist Isaac Stern and businessman Jack Kaplan were the founders of the nonprofit Carnegie Hall Corp.

In 1960, Richmond founded Walco National Corp., a manufacturer of heavy machinery that established his wealth. He ran for Congress in 1974 because "where else can you really make a contribution?" he said, shrugging.

Since he's been in Congress, he has become the chairman of the Agriculture Subcommittee on Domestic Marketing, Consumer Relations and Nutrition. Each year he spends a week working on a farm. Although he does not sit on any of the arts subcommittees, he has a special assistant for the arts, Barry Nickelsberg, on his staff. Among other things, Nickelsberg watches the arts legislation, like the 30 pieces of legislation in the 96th Congress that didn't pass last year -- many of them introduced by Richmond. One was a proposed checkoff box on IRS forms allowing a taxpayer to donate a certain amount to his county arts agency. Another would allow artists to deduct the fair market value of their art work when they donate it to an institution. Now, they can only deduct the cost of materials. d

"Americans are 100 percent behind the arts," Richmond said. "Every dollar you spend on arts generates $5 more. The arts are good for revitalizing downtown areas."

For all Richmond's enthusiasm, some of his colleagues are less optimistic about the arts caucus. "I wonder if unintentionally they're not counterproductive," said Rep. William Ford (D-Mich.), chairman of the Subcommittee on Post-Secondary Education, which overseas reauthoriztion of the NEA and NEH. "They may be taking attention away from Sidney Yates' committee and my education committee, where the action is. Sidney Yates is going to have all the work he can handle with appropriations. I don't want anyone else piecing us off, telling us another way to do appropriations."

Couldn't the caucus testify on behalf of the arts before his or Yates' subcommittee? "Fine," Ford said. "I don't need that advice."

John Brademas, who lost his last reelection bid, said the caucus might be successful. "It's not possible to say. We've seen a proliferation of caucuses in the last few years," said Brademas. "Some are better than others. Naturally I'm pleased to see members of both parties involved in this. It could be helpful. It could reinforce the efforts of those who have legislative responsibilities."

Some of the House members who have joined Richmond's caucus include: Morris Udall (D-Ariz.), Millicent Fenwick (R-N.J.), Toby Moffett (D-Conn.), William Natcher (D-Ky.), James Jeffords (R-Vt.), Julian Dixon (D-Calif.), Barney Frank (D-Mass.), Parren Mitchell (D-Md.), Charles Rangel (D-N.Y.), Peter Rodino (D-N.J.), Geraldine Ferraro (D-N.Y.) and Margaret Heckler (r-Mass.).

"The average member of Congress is interested in arts and has many art forms in his district," said Richmond. "We have 1,000 symphonies, 2,000 operas, 35,000 little theaters and thousands of dance companies in this country."

Richmond wants to be the man at the door to whom other members come for advice about how to vote on arts legislation. And he is counting on the support of his colleagues. "As you gather seniority, expertise and notoriety, you can do almost anything around here."