President Reagan's special envoys, Philip C. Habib and Morris Draper, will return to the Middle East next week for a new try at speeding withdrawal of foreign troops from Lebanon, but Habib cautioned yesterday against expecting a fast exit of Israeli, Syrian and Palestine Liberation Organization forces.

Following a White House luncheon where they discussed strategy with Reagan and other senior policy makers, Habib said: "There has been considerable success in narrowing the issues." But he said the United States must try "to accelerate a process"; Reagan, talking to reporters earlier, noted: "We are interested in speeding it up."

The ambassador, whom Reagan has designated as his personal representative to the Middle East, also said it was too early to make a decision on Lebanese President Amin Gemayel's request that the United States increase the size of the Marine contingent in the multinational force in Lebanon.

Administration officials have said repeatedly that they want to reserve a decision until there is a clearer picture of what kind of withdrawal agreement can be worked out and what security and peacekeeping problems then will confront the Lebanese government. Habib said he couldn't "disagree more" with contentions that the Lebanon negotiations are stalled. But he acknowledged that U.S. hopes for a withdrawal by year's end no longer are realistic.

Despite his optimistic assessment of the long-range outcome, U.S. officials privately have indicated concern at the failure so far of efforts to initiate negotiations between the various parties in the Lebanon dispute.

In fact, senior administration officials said earlier this week that Habib and Draper were being recalled for discussion of what tactics and pressures the United States might use to get things moving. Although they refused to discuss specifics, some officials said yesterday that the two envoys would return to the Middle East with what one official called "some new ideas."

However, he added, "Whether they will work is something we can't say yet. It's not entirely in our hands, since it depends on whether the other parties are willing to be reasonable or not."

He did not elaborate. But it is no secret that the administration regards Israel as primarily responsible for the lack of progress. U.S. officials think Israel has made Gemayel's position overly difficult by its insistence on conducting negotiations in Jerusalem, seeking a written accord that would be tantamount to a de facto peace treaty between the two countries and maintaining a big measure of control over southern Lebanon through Lebanese Christian militias equipped by Israel.

U.S. officials acknowledged that a main aim of the effort by Habib and Draper will be to break the impasse over a venue for Israeli-Lebanese talks. As one said, "There's 15 different ways you can cut that if the two sides are willing to be flexible. Now we have to see if they are or whether they want to use the Jerusalem issue as a device for stalling."

Besides the problems with Israel, U.S. negotiators have had difficulty in budging Syria from its demand that there be a total Israeli withdrawal from Lebanon before Syrian forces, and the PLO fighters under Syrian protection, leave.

However, Habib is understood to believe that Syria will show flexibility if it can be assured that it will not be endangered by the placement of Israeli forces or early warning observers in those parts of Lebanon near the Syrian border.