John W. Hinckley Jr.'s older brother and sister told a federal jury yesterday that they grew up in different worlds from their younger brother, watching from a distance as he drifted into isolation and apparent depression in the years before he shot and wounded President Reagan.

"We come from different ends of the spectrum," said Scott Hinckley, 31, a petroleum engineer and president of the family oil and gas exploration business.

"I am a professional. My friends are young professionals," he added, describing his younger brother as "a loner, a fellow who liked to keep to himself. John had a different way of doing things. His ideas and ways of doing things may have been different from mine."

Diane Hinckley Sims, 29, told the jury yesterday that "my social activities were very much of a different nature" from what her brother would have liked. She was a cheerleader at the high school they attended together and she had been in the choir, she testified.

Scott Hinckley told the jury that by early March 1981 he and his sister had agreed that John should be institutionalized.

"I just thought the situation was out of control," he told the jury.

He said the plan was dropped after he discussed it with his father, who was concerned "about the emotional scars that would be left on John once he was cured."

Three weeks later, John Hinckley was arrested outside the Washington Hilton Hotel and charged with the attempted assassination of Reagan. Defense lawyers at Hinckley's trial, now in its third week, are trying to convince the jury that Hinckley was legally insane when he wounded Reagan, White House press secretary James Brady, a U.S. Secret Service agent and a D.C. police officer.

Much of yesterday's court session was taken up with testimony from Scott Hinckley and Sims about their younger brother's adolescent years, when he spent long hours in his room strumming his guitar and had few friends and a limited social life.

In May 1974, Sims said, John Hinckley was an usher at her wedding. From then until February 1980, Hinckley made four visits to her home in Dallas, getting an apartment each time and, he had told her, looking for work.mer of 1980, Hinckley offered advice about a "lovelorn creep" who was pursuing her. He also described in detail a mail-order business he set up in Texas, saying "I'll certainly be a millionaire the next time you see me."

Sims told the jury that the only times she saw her brother happy in recent years were those Hinckley spent with his nephew Christopher, now 5 years old. She affectionately described the long hours Hinckley spent "playing cars" with the little boy and the postcard in which Hinckley had asked his nephew to "Remember me in your prayers."

"My son could make him laugh," Sims said. Hinckley, who had a picture of the child in his wallet when he was arrested, drew his hand up to his face and rubbed his eyes as his sister testified.

Scott Hinckley testified that when his brother began traveling back and forth to Hollywood, trying to become a rock star, his family wondered "just how well he would be able to do." But, Hinckley said, they weren't ready to dismiss the possibility that his younger brother might "be the next Barry Manilow."

At that, John Hinckley dropped his head into his hands, then smiled and shook his head in apparent embarrassment.

A suburban Denver psychiatrist, who had been treating Hinckley in the months before the shooting, also completed more than six hours of testimony yesterday. Dr. John Hopper told the jury that while Hinckley showed a "pattern of depression and disillusionment" he did not observe symptoms typical of a mental illness.

Hopper testified yesterday that he had an opinion about whether Hinckley suffered from a mental illness, but he was not permitted to tell the jury about it.

It was later learned that during a bench conference, the transcript of which is sealed, Judge Barrington D. Parker upheld the prosecution's objection to such testimony from Hopper, since the defense had not included him on its list of psychiatric expert witnesses.

Both the defense and the prosecution are expected to call as expert witnesses psychiatrists who specialize in diagnosing mental illness and evaluating its relationship to criminal activity. Those experts were hired by the prosecution and defense to examine Hinckley after he was charged with trying to assassinate Reagan.

When Hopper completed his testimony, he abruptly turned to Parker and said "May I talk with you? I think there is something that would be helpful to the psychiatrists," apparently referring to the expert witnesses.

After another private bench conference, Hopper was sent out of the courtroom but was later called into Parker's chambers for a meeting with the judge, transcribed by a court reporter. The transcript of that meeting was sealed by Parker.

In September 1980, when Hinckley enrolled in a writing course at Yale University, Scott Hinckley said the family thought it was "a heck of a way for him to get on the right track and help him lead . . . a more productive life."

Hinckley's defense lawyers have said he went to Yale to contact the young actress Jodie Foster, who was a student there. When Foster rebuffed his attempts, Hinckley returned to suburban Denver.

From Washington 10 days later, Hinckley sent a postcard to his sister depicting the Capitol Hilton, which is located a short distance from the White House. "I'm staying a couple of blocks from Carter's fortress," Hinckley said on the postcard, which was introduced into evidence.

"The past two weeks have been strange times. I keep getting hit over the head by reality. It doesn't feel very good," he wrote.

The prosecution contends that Hinckley stalked both Presidents Carter and Reagan before firing on Reagan.

In February 1981, a month before the shooting, when Hinckley said he had a job at a suburban Denver newspaper, his brother Scott testified, "We thought: 'Gee, this is just what he needs to put him on the right track, make him happy.' "

Hinckley quit the job and left home after two days, his brother testified, leaving behind a note that said he had gone to "exorcise some demons."

"Well, you can imagine my parents' reaction to that," Scott Hinckley said. A week later, Hinckley said, his brother called home, "tired, sick and broke" in New York City.

He came home again, on March 7, but not to the family house. Hinckley's parents, consulting with Dr. Hopper, had put him out on his own, in a motel.

It was then, Scott Hinckley said, that he and his sister tried to "figure out the best solution for John."