Former Defense Secretary Donald Rumsfeld suggests in a new interview with Larry King that legalizing gay marriage could one day lead to polygamy.

"I listen to some of the Supreme Court justices and one of them said, well, what's next after that? Is it two people? Three people?" Rumsfeld said after being asked whether he supports gay marriage.

The justice who brought up polygamy at a March Supreme Court hearing was not one of the court's conservatives, but rather Justice Sonia Sotomayor, an appointee of President Obama.

"If you say that marriage is a fundamental right, what state restrictions could ever exist?" Sotomayor asked former U.S. solicitor general Ted Olson, who was arguing for gay marriage. In that context, she mentioned polygamy and incest.

Rumsfeld also agreed in the interview that many great soldiers were likely gay or lesbian, and King noted that Rumsfeld supported civil rights legislation during his tenure in Congress in the 1960s.

But Rumsfeld said he doesn't view the gay rights movement in the same light as the civil rights movement.

"I was proud of the work that the Congress did in the 1964 legislation and 1965 legislation," Rumsfeld said. "I guess I just don't equate the two. And it's not a subject I'm knowledgeable about, and I guess the Rumsfeld Rule here is, 'I don't know.'"