Millions who live in U.S. territories don't have full voting rights, like being able to vote for president or elect a voting member of Congress to represent them, a topic John Oliver discussed on "Last Week Tonight" Sunday.

"More than four million people live in U.S. territories, more than 98 percent of them are racial or ethnic minorities, and the more you look into the history of why their voting rights are restricted, the harder it is to justify," Oliver said.

Residents of America's island territories can't vote because the Supreme Court found in a series of early 20th century decisions that they belonged to the United States but were "not a part" of the United States. The decisions also found the territories were inhabited by "alien races" who might not be able to understand Anglo-Saxon laws, so the U.S. Constitution didn't have to apply. The lead decision in one of the rulings was written by the justice who wrote the "separate but equal" decision in Plessy v. Ferguson, Henry Billings Brown, and was intended to be temporary.

Oliver also pointed out some of the odd ways we refer to the territories and their residents as if they were foreign, like news reports that Supreme Court Justice Sonia Sotomayor is "the daughter of Puerto Rican immigrants," and an instance when American Samoan Del. Eni Faleomavaega was introduced in Congress as the delegate from "American Somolia."

"There are a lot of complicated issues surrounding what the precise status of all the U.S. territories should be and what the people who live there would prefer, but surely, when it comes to denying Americans the right to vote, we have to find a better reason than citing a 100-year-old legal decision written by a racist that was always supposed to be temporary," Oliver said.