Energy specialists who agree on little else managed to concur yesterday that energy conservation would help the nation's economy more than it would hurt.

The question had arisen over the fears of oil company and nuclear power advocates that too much stress on conservation would require nation-wide industry cutbacks and cause unemployment to rise. Some labor groups and other organizations, including the National Association for the Advancement of Colored People, have adopted that view.

Yesterday's symposium at the American Association for the Advancement of Science annual meeting, however, reconciled some aspects of the views of "hard" energy advocates such as Walt W. Rostow, former adviser to President Lyndon Johnson, who is now at the University of Texas, with "soft," or renewable, energy advocates such as Denis Hayes, director of Solar Action Inc. and the organizer of Earth Day.

"The investmesnt required to get oil imports down" to levels recommended by President Carter "would take us a long way toward full employment," Rostow told a news conference. He said industry would have to increase its energy-related investments, now more than one-third of total annual investmen, by up to 3 percent in order to reduce oil imports to 6 million barrels a day by 1985. The country now imports 9 million barrels a day, and demand is rising by about 3 percent a year.

"A great deal of talk about conservation is as if it were free," Rostow said later. "Most of it is very capital-intensive and takes time."

Incentives to such investment would have to include marginal cost pricing regulations, a strong tax on excess profits to force industries to re-invest, and a firm government policy statement on what environmental costs would be acceptable, he continued. He added that the effort would have to be coupled with greatly expanded energy production in order to avoid massive economic dislocation.

Hayes disagreed. Any funding now going to spur energy production would produce "a bigger bang for the buck" if invested in energy conservation, he said, because conservation, in his view, is labor-intensive. As they now stand, President Carter's energy proposals, "will be more of a hindrance than a help to national energy development and ought to be withdrawn, he said.

Hayes attacked the Carter administration for failing to promote conservation.

Joel Darnstadter of Resources for the Future told the gathering that energy use never has moved in lockstep with economic fluctuations in any case.

Historically, energy use per unit increase in the gross national product was rising before World War II and the declined through the mid-1950s.