The U.S. construction industry is huge, with an estimated 650,000 firms in the business employing more than six million people to create nearly $1 trillion worth of structures each year, according to data from the Associated General Contractors of America.

Small businesses are, without argument, the driving force behind this industry. The Small Business Administration reports that no less than 92 percent of construction firms in this country have less than 20 workers. The industry has been good for small firms too. Despite 2009’s real estate collapse, six of the 10 fastest-growing industries here are tied to construction and include firms doing everything from contracting and carpentry to architectural services and real estate sales.

But it’s a tough business with a high failure rate. Only 36 percent of firms in this industry survive after five years, according to research on start-ups around the country. And those survivors can at best hope to see a net profit (before taxes) of about five percent. “Man, I should have never have listened to my dad when he told me to get into the construction business,” one 50-year-old client recently told me.

But will this change? We’re hearing a lot about the Trump administration’s plans for significant spending on infrastructure over the next few years–$1 trillion in spending has been promised. The plan remains short on details, relies primarily on tax credits to private industry and would need congressional approval. But say everything falls into place. Does this mean good times are ahead for the small firms in this industry?

Unfortunately, I don’t think so. At least not in the short term. Of course, some will prosper. But for most, the construction business, regardless of any new infrastructure spending initiatives, will continue to be a challenging area for small businesses to make money–and for these reasons.

Both millennials and boomers are getting in the way of new construction.

Millennials are continuing their migration to the cities and their income levels are still not growing significantly. As a result, we’ve seen a proliferation not of new homes in the suburbs but new downtown high rises for renters. In the meantime, the boomer generation – with the help of the healthcare industry – is living longer and better than any generation before it and, instead of uprooting their lives many are choosing to stay in the homes they purchased decades ago. These two trends have contributed to less homebuilding (and higher real estate prices) which means less work for construction-related companies.

Anti-immigration sentiment will affect both demand and costs.

No one’s certain if that wall will ever be built. But most agree that there’s a stronger-than-ever anti-immigrant sentiment here. A trend towards curtailing immigration in the United States means that an entire population of customers whose dreams of owning a home here will be less than ever before, meaning slower demand for new construction. Compounding this revenue problem are cost pressures brought on by less immigrant workers available to do lower skilled construction jobs which is already causing a shortage of experienced employees in the industry and driving up the costs for construction companies who need bodies.

Higher interest rates and less government support will inhibit home ownership.

As the economy has continued its growth, it is a foregone conclusion that the Fed will increase interest rates shortly and will likely plan additional increases throughout 2017. Higher rates mean more costlier finance which means less incentives to buy new homes in lieu of renting. Also, the Trump administration’s pick for Treasury Secretary, Steven Mnuchin, has expressed his desire for the government to spin off both the Federal National Mortgage Association (or Fannie Mae) and the Federal Home Loan Mortgage Corp. (or Freddie Mac), to the private sector–a move supported by some in the Republican-led Congress. While relieving the government of a controversial and long held economic burden, such action may make it harder for some to get much-needed loans to purchase a new home.

Government spending on infrastructure will benefit a minority of firms.

The backbone of the Trump administration’s plan to fund its infrastructure investments is offering tax credits to “any corporation that invests in public infrastructure.” That means big firms with deep pockets, strong relationships with large financiers and government approvals to do the work. Sure, some smaller firms will indirectly benefit from sub-contracted services but it will be the Bechtels, Fluors and Turners who will likely benefit the most. Regardless, as anyone in the construction industry can tell you, there’s a long way between project approval and money actually changing hands so at best the flow of most work likely won’t be seen until well after 2017. This is why The American Road & Transportation Builders Association is calling for transportation construction and “related market activity” to grow just 1.3 percent next year, with most of the increase expected to come from the private sector.

There are some bright sides for smaller firms in the industry.

For example, construction activity in certain regions (Texas, Florida, parts of California) are projected to be better than others. But I’m keeping an eye on a trend towards more “social infrastructure” projects. The American Institute of Architects is pushing for more spending on the country’s crumbling infrastructure of public buildings, government offices, schools and libraries that need better air quality, lighting, systems and accessibility. Russ Davidson, the president of the organization recently held a “Build America” summit to promote this initiative with the hopes of diverting some of the government’s spending on roads and bridges.

“It’s all about improving the health of the country,” he said.

As the leader of an organization of almost 9,000 smaller firms, I’m sure he’s also hoping to improve the health of his industry too.