White House efforts to sustain a veto of legislation to protect the textile industry were set back by gloomy new balance-of-trade figures and by an administration agreement to increase South African textile imports, congressional and administration sources said yesterday.

Some Republican leaders of the House have joined Democratic efforts to override President Reagan's veto of the trade legislation, which would give substantial new protection against imports to the U.S. textile and footwear industries.

Administration officials said yesterday they had been caught by surprise by the deterioration of support in the House for sustaining the president's veto.

Rep. Ed Jenkins (D-Ga.), who is leading the House effort to override, said yesterday that supporters of the textile bill picked up about 10 votes because of what is widely perceived in Congress as the administration's latest South Africa blunder.

"I've had people come up to me and say, 'You were right, they don't know what they're doing in these trade negotiations,' " Jenkins said.

He said supporters of the bill are still about 10 votes short of the two-thirds necessary to override the veto in the House showdown next Wednesday. "I think we can do it, but I don't want to be overconfident because I don't have the votes yet," Jenkins said.

"A lot of us who are free traders are increasingly hard pressed to come up with arguments to justify our position," said Rep. Dick Cheney (R-Wyo.).

The textile bill, which passed the House 255 to 161, would cut clothing imports from Hong Kong, South Korea and Taiwan -- the three major exporters to the United States -- by about 30 percent. It would also limit shoe imports to 60 percent of the domestic market, a reduction from the current level of 80 percent.

Supporters of the bill need about 288 votes to override the veto in the House. It would also have to be overridden in the Senate, where Republicans from textile-producing states such as James T. Broyhill of North Carolina are actively working against the administration.

The administration is being strenuously opposed in the House by Minority Whip Trent Lott (R-Miss.), who is part of a solid, bipartisan bloc of lawmakers from the South and New England working to override the veto. But Lott, a key member of the House GOP leadership, offered no apologies.

"For two years, I have been warning them that this day was coming, that they were going to have to toughen up on trade and do something about textiles," he said. "They just thought I would obediently fall into line. I can't do it on this."

Lott, who said close to a majority of House Republicans may oppose Reagan on the textile bill, added that he hoped the textile issue will force the administration to reevaluate its overall trade policy. "If you don't do it now, you're going to have a bigger problem with an overall trade protection bill," he said.

Another House Republican, who asked not to be identified, said the revolt was being fed by other factors, including growing GOP nervousness over the trade issue as the elections approach. He said Reagan had not exercised "effective leadership" in trade policy, which the Democrats are attempting to use as a major campaign issue, and that there is an "undercurrent" of resentment toward Reagan's neutrality in Republican primaries in which incumbent GOP lawmakers are being challenged.

"A lot of members think a good way to send a message is to override the president's veto," he said. "There is a sense that House Republicans are asked repeatedly to cast those tough votes for the administration, but that it is not a two-way street."

The decision to allow a 4 percent increase in textile imports from South Africa, disclosed earlier this week, immediately added impetus to the effort to impose new economic sanctions against the white minority regime in Pretoria. By yesterday, it was also clear that it had compounded the administration's problems across a broad range of trade issues, beginning with the textile measure.

"It is illustrative of what we have been contending all along -- that this administration's trade policy is so loose they don't know what is going on," Jenkins said.

Asked if the South Africa decision was a major blunder, Rep. Lynn M. Martin (R-Ill.) covered her face with her hand.

"I'm sure there is a new definition for dumb, but if there is not that has got to qualify," she said.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) also seized on the new agreement with South Africa yesterday as he addressed a Capitol Hill rally of textile and apparel workers.

"Mr. President," O'Neill said, "your brand of trade policy may play in Pretoria, but it won't play in Peoria and in towns and cities all across this country."