The lineup of the Senate Banking, Housing and Urban Affairs Committee in the new Congress will be substantially different from the current one, as a batch of conservative Republicans join the panel and several moderate to liberal members depart.

The most notable departure is that of Sen. Richard G. Lugar (R-Ind.), chairman of the housing subcommittee and sponsor of a major housing assistance bill that was passed this year and vetoed by President Reagan. Lugar is also on the Agriculture and Foreign Relations panels, and the Senate Committee on Committees ruled that new senators who wanted to join Banking had the right to bump him.

Lugar is reported to have been very unhappy at having to leave the Banking Committee. "He was pushed kicking and screaming" off the panel, according to one member of his staff. But, as a lobbyist for one major trade association in the housing field put it, "He could have chosen to leave Agriculture or Foreign Relations, but he decided they were more important to him."

Lugar's successor as head of the housing subcommittee will be Sen. John G. Tower (R-Tex.).

Sen. Jake Garn (R-Utah) will continue as chairman of the full committee and Sen. William Proxmire (D-Wis.) as ranking minority member. The committee's size has been increased from 16 to 18 members. Last year there were nine Republicans and seven Democrats; this year the ratio will be 10 to eight.

Conversations this week with legislative affairs staff members at various trade associations active in the housing and construction fields indicated that while most expect smooth relations with the new members, some worry that the new panel may be less receptive to housing assistance programs than in the past.

These sources were reluctant to comment on individual members. "We wouldn't want to antagonize any of them--these are the people we work with," said one. However, the sources seemed pleased with the addition of Sens. Slade Gorton (R-Wash.) and Frank Lautenberg (D-N.J.).

"We're more optimistic than some" about the committee, said another. "When you start to examine where they're from and what their background is, I think we may find it easier to to work with them than it might seem at first blush."

Another trade group worker, commenting on the apparent conservatism of some of the new panel members, also saw a bright spot: "What we want is to keep interest rates down--and if these are fiscal conservatives who are going to do that then we're happy."

Besides Gorton, new Republicans on the panel are Paula Hawkins (Fla.), Mack Mattingly (Ga.), Chic Hecht (Nev.) and Paul S. Trible Jr. (Va.).

Lautenberg is the lone new Democrat.

GOP members of the housing subcommittee, in addition to Chairman Tower, will be Garn, John Heinz (Pa.), Alfonse M. D'Amato (N.Y.), Gorton, Hawkins and Trible.

The Democrats have not yet chosen subcommittee members, according to the panel's staff, but Donald W. Riegle Jr. (Mich.) is expected to be the ranking minority member.

The House has not yet organized its committees. Industry sources say they expect major changes on its banking and housing panels.