History, they called it. There was Patricia R. Harris, the first black woman Cabinet member, taking the oath of her new office at the White House yesterday from Thurgood Marshall, the first black Supreme Court justice.

"To see her and Thurgood Marshall stand there is in a sense at special moment in history,' said black activist Jesse Jackson, one of 500 people who crowded the East Room to see Harris sworn in as Health, Education and Welfare secretary.

"We never thought we'd live to see it," said Jesse Dedmon, a black Washington lawyer who said she knew Harris from her days in law practice.

Even Harris, as many of the old friends and black leaders there, seemed particularly moved by what in substance was nothing more than another White House swearing-in ceremony. "I looked around and I saw Thurgood standing there," she said, "and I thought to myself, 'We've got a ways to go, but we've come a long way, too.'"

Harris leaves her post as secretary of Housing and Urban Developement to lead the $199 billion department succeeding Joseph Califano, whom President Carter fired in last month's Cabinet shakeup.

Like Califano, Harris is known for her feistiness and outspoken independence -- qualities Carter acknowledged in remarks before her installation.

"Anybody who looks to Pat Harris as a yes-woman would be both foolish and ill-advised," Carter said. "I have never been that folish and I have never been ill-advised."

In contrast to Califano, however, Harris kept a spot within the Cabinet by remaining publicly loyal to Carter. That loyalty was evident yesterday when she told the crowd she would help the president with his pledge to "restore confidence in government."

She also appeared to speak in Carter's defense when she added, "Judgments and controversies are part of public life. They go with the turf -- they come with the territory."