A group of conservative House Republicans urged President Reagan yesterday to withdraw his tax-overhaul proposal, saying that it has been irreparably altered by special-interest lobbying and is now "antigrowth and antifamily."

In a letter to Reagan, the Republicans said that, because of changes in his tax bill made by the House Ways and Means Committee, they can no longer support the measure.

Therefore, they said, "the time has come to abandon the committee bill, abandon the process that is producing the committee bill and rededicate ourselves to genuine pro-growth and pro-family tax reform."

The letter was drafted by Rep. Vin Weber (R-Minn.), head of the House Conservative Opportunity Society, and signed by 38 Republicans who generally are strong allies of Reagan.

However, several of the lawmakers who signed the letter acknowledged yesterday that their support for Reagan's bill, even before the Ways and Means panel began working on it, was at best halfhearted.

But they said committee changes, including lowering Reagan's proposed personal exemption of $2,000 to $1,500 and widening loopholes for some groups, have made it impossible for them to support the measure.

"A number of us had reservations" about Reagan's proposal, Rep. Robert S. Walker (R-Pa.) said. "I was willing to swallow hard and vote for that package." Now, he said, "this is simply a complicated mess that doesn't deserve to live."

There was general discussion by other lawmakers yesterday that the Ways and Means changes simply provided the GOP lawmakers a chance to back away from the bill without appearing to abandon the president.