On Friday, President Trump tweeted that ranchers around the U.S.-Mexico border have found prayer rugs in the desert, insinuating that Muslims are sneaking into the United States from Mexico. It’s not the first time Trump has invoked the idea of Middle Easterners sneaking across the border, an effort to stoke public fear with anti-Muslim stereotypes.

While the threat is imaginary, the presence of Middle Easterners living and working on the Rio Grande is not. In fact, they have a long history of living in the U.S.-Mexico borderlands and have helped shape that region and the country. The booms, busts and daily lives of these Middle Eastern communities have contributed to the story of North America, all the while enduring persistent American xenophobia as they searched for opportunity and safety.

Since the late 19th century, people from Ottoman Syria, what is now the countries of Lebanon, Syria, Israel/Palestine and parts of Jordan and Iraq, traveled to the U.S.-Mexico border in search of opportunity. They were part of a diaspora known as the mahjar. The exact number of immigrants is hard to determine, but it is estimated that between 1880 and 1950, around 35,000 Syrians immigrated to Mexico and around 800,000 to the United States.

Many settled along the U.S.-Mexico border, a porous boundary they regularly crossed. They came to the borderlands to work as peddlers and laborers, supplying Mexicans, Europeans and Chinese living in the area with dry goods, silk and specialty products from the Middle East. Many set up stores, and through their mahjari contacts across the Americas, built businesses and trading networks that connected places as far apart as Montana, New York and El Paso.

And it wasn’t just Syrians. A century ago, the U.S.-Mexico borderlands were full of multiracial, thriving communities. Black, Mexican, Chinese, Syrian and European people lived and worked together, unlike in the segregated North and South. Many intermarried, and Syrians sold goods to Mexicans and Chinese merchants living in Mexico. African Americans, fleeing the confines of the American South, created new lives for themselves in the borderlands.

This intermingling left some surprising legacies. The dish tacos al pastor was created through the fusion of Lebanese spit-roasting and Mexican tortilla-making. Burlesque developed at the Bird Cage theater in Tombstone, Ariz., when a Syrian immigrant, Farheda Mazar, entertained European and American prospectors with belly dancing.

When the federal government took over border inspection in the late 19th century, these communities came under attack. The xenophobia and white nationalism that created Jim Crow and mandated Asian exclusion in the United States soon reached the borderlands. Syrians, who straddled the line between white and nonwhite, became an object of anxiety, then a target. “Syrians are the trash of the Mediterranean,” one U.S. senator wrote. Other than East Asians, Syrians were the group likeliest to be barred from entry into the country by 1900.

Much like today, the government accused Syrians of disguising themselves as Mexicans to get into the United States. The New York Times lamented in 1927 that “only 750 men stand guard … to halt the invasion of aliens who slip through in violation of law. … A conservative estimate places the number at 100,000 a year.” (Historians now believe it is a fraction of that). In 1906, President Theodore Roosevelt even personally assigned immigration inspector Marcus Braun to investigate Syrian migrants in Mexico and report on their “devious” migration. Braun’s report caused the government to install more police at the border, dole out harsher punishments to anyone found without documents and create steeper hurdles for people who were trying to cross the border legally.

These legal obstacles came into force on the eve of the Mexican Revolution of 1910, which drove refugees fleeing violence northward. Much like asylum seekers today, the people who showed up at the U.S.-Mexico border were there to seek safety and asylum. In fact, the first detention camps at the border were full of Chinese migrants, fleeing slaughter in Mexico. And just as now, the American government was so convinced that these refugees must be hiding something that they sent many back, often to their deaths.

This racist fearmongering worked, gaining mainstream acceptance in American society. In 1917, Congress passed the Asiatic Barred Zone Act, the harshest immigration policy to that point, which made white paranoia and nativist restrictions the law. The act made it almost impossible for people from Eastern Europe, the Middle East and Asia, many of them fleeing violence brought on by World War I, to enter the United States. Armenians attempting to escape genocide were rejected at the border.

The borderland residents suffered the consequences of these policies. African Americans faced harsher restrictions when trying to cross between the United States and Mexico. Chinese migrants became wholly restricted from entering the United States and at times were deported to Mexico if they could not prove they were here legally.

And just like today, the racist laws encouraged racial violence. In the 1920s and 1930s, the Ku Klux Klan was revived along the border and threatened Syrians living in Texas and the Mexican state of Coahuila, right across the border from Eagle Pass. The attacks on these thriving communities of the borderlands began through paranoid reports and investigations, then evolved into racist laws and racial violence. By the 1930s, their way of life became untenable.

Racial panic doesn’t just happen — it is stirred up by policymakers to pursue their ends, to the real harm of desperate people. Racial and ethnic diversity is not part of a plot to overthrow the country. Rather, it reflects the reality of our connected histories.

We need to devise policies on the basis of the real people who come to our borders, not on the fear of a boogeymen in disguise. Right now, migrants seeking asylum are for the most part men, women and children fleeing violence. On this information, it is clear we need a compassionate migration policy to accommodate this new population. And sure, maybe there are Middle Eastern descendants in the crowd. But that’s not something to be afraid of — it’s a reminder that our world is one of motion and movement, and we need sound policies that reflect that reality.