I was a young Senate summer intern, watching from the gallery on June 19, 1964, when that still-august body convened to vote on the landmark civil rights act that would desegregate public accommodations, especially in the South.

President Lyndon B. Johnson had pushed hard for this legislation, in the aftermath of President John F. Kennedy’s assassination seven months earlier, and he had lined up significant support on both sides of the aisle. After an earlier cloture vote, the outcome (73 to 27 in favor) was well-assured. But still there were moments of high drama — including when Sen. Clair Engle, a California Democrat, near death and unable to speak, was wheeled into the chamber to lift his hand to deliver a silent “aye” vote for the bill.

Few political hands were surprised when a bloc of 21 Southern Democrats refused to support the measure. Their number included Sen. Albert Gore Sr. of Tennessee, father of future vice president Al Gore; Sen. Sam Ervin of North Carolina, later the stalwart leader of the Watergate hearings; and Sen. J. William Fulbright of Arkansas.

This one startled me and my friends the most. He was known for standing up to demagogues like Sen. Joseph McCarthy of Wisconsin and for challenging key elements of U.S. foreign policy. Indeed, we had an image of him as one of the leading intellectuals of the Senate, a progressive internationalist and the namesake of the iconic exchange program that sends U.S. scholars abroad and brings the best and the brightest from around the world to the United States. How could Fulbright possibly endorse the brutal segregation that was not just morally wrong, but was sullying the United States’ global reputation?

The answer: Fulbright had risen to office with the support of segregationist voters in the Old South. And his vote was apparently the kind of thing he had to do to sustain their support while acting more independently on other issues. In fact, he had already voted against a number of civil rights bills in the 1950s and ’60s and would continue to do so. We assumed this was a moral dilemma for him that he resolved with difficulty, but a part of his record that could, ultimately, be forgiven by most integrationists and his fellow liberals.

In hindsight, perhaps we were wrong. I’ve thought of Fulbright often as I’ve wondered whether and when revisionists and name-changers would get around to him, as they have to other American icons.

People of the left are busy re-examining the legacy of key figures in U.S. history who may have had landmarks or buildings or programs named for them; and deciding, with all the benefit of hindsight, whether these honors should be stripped away in order to register our (sometimes late-blooming) disapproval of the lives they lived and the positions they took.

On campuses, students have demanded the renaming of Princeton’s Woodrow Wilson School of Public and International Affairs, which pays tribute to a man who is arguably the university’s most famous alumnus and who eventually became its president and then governor of New Jersey and the 28th president of the United States. The idea of taking the image of Andrew Jackson, our seventh president, off the $20 bill has emerged in recent months. In December, New Orleans’s city council voted to take down several of the city’s Confederate monuments.

It is too facile to dismiss this as just another round of political correctness; sometimes the reasoning has been compelling and even tempting, as slaveholders, avowed racists and bigots of various stripes have come under scrutiny. But these cases are harder than they look.

Take Wilson. He was clearly a white supremacist: Particularly odious is the fact that he sent an envoy to France during World War I to warn African American troops fighting in Europe that despite their courage and sacrifice for our country, they should not expect their circumstances to improve when they came back home. Wilson also indulged the first Red Scare and empowered his attorney general, A. Mitchell Palmer, to conduct raids on allegedly subversive immigrants and activists. It’s easy to imagine the sense of justice — even payback — some might anticipate from effacing his name.

But Wilson, like most, was a man of many dimensions. Writing for Politico, Stephen Kinzer has made a fiendishly clever argument that Wilson’s name should remain in lights and on walls, if only to remind us of his towering hypocrisy — waxing on about the right to self-determination for all people, for example, while intervening militarily in many Latin American and other countries.

Others might point to the fact that Wilson established the Federal Reserve System and the Federal Trade Commission. He signed the Clayton Antitrust Act, lowered tariffs and revived the federal income tax. He took significant steps to control child labor (only to be rebuffed by the Supreme Court). He wrote the idealistic Fourteen Points to guide the Allies in World War I and proposed the League of Nations (which his own country refused to join). He appointed Louis Brandeis as the first Jewish associate justice on the Supreme Court.

Princeton is a private institution, of course, and will ultimately do as it sees fit. But the idea of a posthumous trial to decide what to do about the Woodrow Wilson School, not to mention other institutions carrying his name, seems like a slippery slope. Horrifyingly, 12 U.S. presidents owned slaves, including George Washington, the father of our country; Thomas Jefferson, author of the Declaration of Independence; and Jackson, a war hero who is also the villain of the Trail of Tears. Do their various good deeds and policies outweigh their evil ones or should we be racing to strip their likenesses from our currency?

As The Washington Post has reported, there are nearly 200 public schools across the country named for Confederate leaders, and in Virginia, the most popular street name is Lee, as in Robert E., leader of the Confederate army. What to do about these reminders?

Which is why I’ve been dwelling on Fulbright, whose case is instructive. The moral of his story is that sometimes good people feel compelled to do wrong for the sake of political survival. How is all this to be measured decades later, and what latitude do we have in fashioning his reputation for posterity? Should one of America’s most respected brand names — the Fulbright scholarships — now be changed? And who’s to decide when a historical figure has crossed a line that may have been blurry in the past but seems clear and bright to us today? Who’s to decide whether a historical figure contributed more evil than good?

What is at stake, in the end, is an understanding of our own history. We certainly must confront the reality that many of our greatest public figures did not always live up to American ideals. But wiping out the names, Soviet-style, of key actors who shaped the society, and world, that we live in today, will not help that process.