Secretary of Transportation Brock Adams, holding the first of the Carter administration's promised "town meetings" today, said he came to listen, and he got an earful.

"Consider the economic environment of the working man and not the environment of the snakes and frogs," demanded William J. Huebner of the Connecticut Construction Industries Association, complaining of the impact of environmentalists on road construction.

"In connection with highway projects, the Department of Transportation has, in my opinion, the worst record of any federal department in responding to the environmental impact statement requirement," charged Perry R. Hagenstein, director of the New England Natural Resources Center.

And so it went for more than three hours as a stream of about 25 New England business, labor, civic and environmental leaders, local officials and two governors trooped to the microphone in a cavernous hotel ballroom here to pour out their transportation pleas and complaints.

The new Transportation Secretary, sitting alone on the stage, appeared to listen intently as he took notes and occasionally interrupted with a question. He was getting the "flavor" of the New England transportation scene, he said.

About the only position taken by the secretary was a pledge when he arrived by train Tuesday to begin work on the overhaul of the Northeast corridor rail system between Washington and Boston by April 1 - "if I have to go out there myself with a pick and a shovel."

Gov. Michael Dukakis, there to greet Adams, promptly presented the former Washington congressman with a pick and a shovel.

Adams elaborated in a news conference today that he is accelerating the $1.9 billion project to begin some of the track repairs in order to put laborers to work while the engineering planning still continues. He said that his predecessor, William T. Coleman Jr., had been delaying the project, which was first approved by Congress three years ago.

Otherwise, Adams offered no solutions and few opinions while many of the speakers, despite their competing positions, nonetheless said they were happy just to be heard.

"You can't learn too much sitting in a Washington office," Adams has said the day before in a dilapidated East Boston neighborhood as he talked with residents complaining of the noise from nearby Logan Airport. "You have to go out and see the people."

If there was a main theme sounded by the New England leaders today it was that the six states want more federal help for mass transportation and the railroads, more flexibility in that funding and less paperwork.

Connecticut Gov. Ella Grasso and other speakers stressed that states need the flexibility to apply highway construction funds to highway repairs and what mass transportation funds they receive to operating costs, two expenditures which federal funds by regulation often do not cover.

Mayor Gordon Paquette of Burlington, Vt., said that across the country there are 313 public transportation systems in rural areas and small cities under the 50,000 population mark, which the federal government designates as the dividing line to be considered "urban" and quality for many transportation funds.