President Reagan yesterday named veteran diplomat Philip C. Habib as his new special representative for the Middle East, giving him authority over negotiations both for withdrawal of foreign forces from Lebanon and the Middle East peace initiative.

Habib said he would pursue both objectives simultaneously. "Don't think we can't succeed," he told reporters.

It was Habib who last summer negotiated the withdrawal of Palestinian forces from Beirut, ending the Israeli siege and bringing U.S. forces to the Mideast as part of a multinational unit that left once but returned after the massacre of Palestinians in West Beirut in mid-September.

With Habib's appointment yesterday, Reagan's team for pressing U.S. goals in the troubled region is complete.

Habib will work closely with Deputy Secretary of State Kenneth W. Dam on the Lebanon phase of the negotiations and with Secretary of State George P. Shultz on the overall peace initiative.

Shultz has kept progress on the peace initiative at the top of his priority list and gives the subject close personal attention, according to high-ranking State Department officials.

Habib will work with Ambassador Morris Draper on the Lebanon negotiations and Ambassador Richard Fairbanks on the Reagan peace initiative.

Habib admitted yesterday that "there are problems" with the peace initiative, but added: "The president is committed . . . to the maintenance of the momentum which has been in effect since the first of September," when Reagan announced his peace effort.

Habib said he would stay in the United States through next Friday, when Reagan is to meet with Israeli Prime Minister Menachem Begin. The meeting has been pointed to as a possible showdown over the issue of Israeli settlements on the West Bank.

"The settlements are going to have to be part of the discussions. There's no question," Habib told reporters yesterday, a sentiment underscored by President Reagan in his press conference last night. Habib repeated State Department assertions that expansion of settlements on the West Bank is a "detriment to the peace process."

Reagan met with Habib, Shultz, national security adviser William P. Clark and Assistant Secretary of State Nicholas A. Veliotes yesterday morning in the Oval Office and then announced Habib's appointment.

"I have asked Ambassador Habib to be my representative in the Middle East to take charge there as we continue on the Middle East policy," he said. Habib would concentrate on "continued pursuit of overall peace in the Middle East as well as stabilizing the situation in Lebanon," he said.

Asked if he thought the peace process is stalemated, Reagan said: "Let's just say it works better when he's Habib's there."

Draper has been shuttling for two weeks between Beirut, Jerusalem and Damascus, trying to get momentum going on the troop-withdrawal talks. Top-ranking U.S. officials said 2 1/2 weeks ago that Israel and Lebanon had agreed to direct talks on the subject, but they have not started.

When asked at his press conference if he thought the U.S. Marines serving in the multinational force would be home for Christmas, Reagan dropped the standard U.S. expression of hope that foreign forces would be out of the country by the end of the year, a precondition for the Marines' coming home.

"I wish I could say" how long the Marines will be in Lebanon, Reagan said. "That's one of the reasons Phil Habib is going back over there. The proposed plan . . . requires the new administration in Lebanon to stabilize and take charge of its own borders. It calls for the removal of all foreign forces as quickly as possible. . . . We are trying to push it as fast as we can."

Progress on the broader peace initiative appears more problematical as the Palestine Liberation Organization has yet to decide its position on proposed negotiations and Jordan's King Hussein must decide whether to go to the conference table.