A smooth-talking young man claiming to be the son of actor Sidney Poitier has pulled the cashmere over the eyes of at least a dozen well-to-do families in Manhattan, who gave the impostor money, clothes and room and board.

"I think an awful lot of people have been conned," Inger Elliott, wife of Osborn Elliott, dean of the Columbia University Graduate School of Journalism, said yesterday. "It was very cleverly done."

Beside the Elliotts, the young man hoodwinked WNET president John Jay Iselin, as well as a wealthy urologist and several other families whose names and numbers the hoaxer may have found in a stolen address book.

Identifying himself as "David Poitier," the con man called the Elliotts on Oct. 2, saying he had been mugged and that he had no money and no place to stay. He said he was enrolled at Harvard and that he was a friend of the Elliotts' daughter, a student at Yale. He added that his father would be flying in from Los Angeles the next day to start rehearsals for the film version of "Dreamgirls."

"I remember saying, 'I don't give a damn who your father is, come on over,' " said Inger Elliott.

When the young man arrived at the Elliotts' East Side home, he acted upset, complaining that the muggers had also taken his term paper. The subject, he said, was "Injustices in the Criminal Justice System."

"He was very pleasant. He said he had gone to Andover and Exeter and that he was now a student at Harvard," Inger Elliott recalled. "He was dressed perfectly nicely. We had a long discussion of what it was like to be the son of a famous actor. Then he started talking about how awful Malibu was. He said it was like Fire Island."

That statement raised some doubt in Inger Elliott's mind, who said Malibu is nothing like Fire Island, but still she believed the young man's identity. "I still didn't know that Poitier had no son."

Poitier, 60-year-old star of "Lilies of the Field," "Raisin in the Sun" and "Guess Who's Coming to Dinner," has six daughters but no sons. He could not be reached for comment yesterday.

The young man said he was the son of actress Diahann Carroll and Poitier, who is now married to Joanna Shimkus.

The scam was successful largely due to the impostor's familiarity with the upper-crust Manhattan set and his ability to drop the right names.

"He described the inside of Diane von Furstenberg's apartment, which I've been in," Inger Elliott recalled. "He knew where John Kennedy Jr. lived. He talked of Nina and Lenny Bernstein."

He also seemed intimately familiar with the Harvard campus and other haunts of the Ivy League.

Osborn Elliott gave "David Poitier" $50 and some clothes.

But the next morning, when Inger Elliott went in to wake the young man--who had said the night before that he wanted to go jogging--she found him in bed with another man, a stranger to the house.

"I knew then something crazy was going on," she said. "By that time I was very suspicious."

The young man eventually left when the Elliotts reached their daughter at college, who denied knowing him, and Inger Elliott placed a call to her friend Lea Iselin, wife of John Jay Iselin. "I was telling her about this young man. She immediately said, 'His name wouldn't be 'David Poitier' by any chance.' "

The day before, the impostor had called the Iselins, given them the same story and had gone to their apartment. The Iselins had given him money and invited him to stay, but then they had grown suspicious and had asked him to leave.

Inger Elliott finally reached Sidney Poitier in California, who said he had no son. "He was terribly nice about it," Elliott said.

At that point, Elliott began her own investigation and discovered other families who were taken in by the hoaxer, including one hostess who found "David Poitier" sleeping in her nightgown.

No one had bothered to call the police.

"They're ashamed and embarrassed they were taken in," Lt. Edward Shea of the Manhattan police department's special fraud squad said yesterday. "These people are worldly. They're affluent. To think that this guy could just walk into their houses this way is pretty embarrassing."

According to Shea, police are looking for a man who calls himself David Hampton, from Buffalo, age 19 to 23, with a history of minor violations, including credit card fraud and destruction of property. Now he's suspected of engaging in a con game and stealing at least $350 from his unwitting hosts. Shea said the man was using an address book lost by or stolen from Robert Stammers, a Connecticut College student who is friendly with both the Iselins and the Elliotts.

"This is a guy who's living by his wits," Shea said. "It's a way of life for him."