Former Democratic representative Phil Gramm, who quit Congress last month and switched parties after losing his seat on the House Budget Committee, tonight won reelection to the House as a Republican in a special election.

But Gramm's victory was modest enough to persuade some Texas Democrats that they may be able to defeat him in 1984, if he chooses to run again for his seat in the 6th District.

With all precincts reporting, Gramm had 55 percent in a field of 11 candidates. His stiffest competition came from former Texas state representative Dan Kubiak, who had 39 percent.

Humorist John Henry Faulk, who was blacklisted during the McCarthy era and has gained prominence on the television show "Hee Haw," was third with less than 4 percent.

Gramm said he would be back on the job in Washington Monday morning, working with the budget "to put the American people back to work."

Tonight's counting was marred by slow returns from Gramm's home county, where turnout was heavy, and for several hours, the one-time "Boll Weevil" hovered below 50 percent of the vote needed to avoid a runoff. Boll Weevil is the name given to a group of Democratic congressmen who often have voted with President Reagan.

Gramm lost six of the rural, heavily Democratic counties in the northern end of the district that stretches from Dallas to Houston. But he gained strong support in two of the biggest, and most Republican, counties in the district, and won nearly 66 percent of the vote in his home county near Bryan to cement his victory.

Kubiak told supporters in Bryan that he probably would run against Gramm in 1984.

"I think we gave it all we had," Kubiak said in a phone interview. "It was a stacked election. When you're outspent seven or eight to one, I can't be more pleased than I am right now."

The Democrats were at a distinct disadvantage in the special election, which was held just 38 days after Gramm resigned. The well-known Gramm began the campaign with $200,000 in the bank and had raised about $500,000 more by early last week. He predicted he would spend at least $500,000 in the contest and others estimated he might top $700,000. Kubiak, in turn, spent less than $100,000.

Gramm cosponsored the president's budget package in 1981 and earned the enmity of fellow Democrats for allegedly sharing Democratic strategy with the Republicans and for his outspoken support of the president.

Gramm said his votes in Congress reflected the wishes of his constituents and charged that Democrats, in bouncing him off the House Budget Committee, last month, were disenfranchising the 6th District.

But Kubiak attacked Gramm for "deceit" and "contempt" of his constituents, charging that the economic programs Gramm had endorsed were bankrupting the country.

Although Kubiak was heavily outspent, he was aided by the remnants of the big Democratic victory in Texas last November. The state Democratic party supported his candidacy and a number of newly elected state officials campaigned for him.