Houston's inability to think big may cost it $110 million in federal aid to start building a subway system.

When the House Appropriations Committee takes up the Transportation Department bill today, it is expected to reallocate to other cities $104.5 million of the $110 million its transportation subcommittee earmarked for Houston in response to heavy lobbying by the Texas delegation.

But the delegation hadn't anticipated the behavior of Houston voters. Only 12 percent of them turned out last Saturday, but by a more than 2-to-1 margin, they defeated a revenue bond issue that would have raised up to $2.3 billion for a subway.

The proposal wouldn't have cost the voters anything. It would have authorized the spending of part of the revenues raised through a 1 percent sales tax passed in 1978 to finance Houston mass transit. The sales tax vote has been cited by Houston boosters as evidence of solid local support for mass transit in the automobile-choked city.

"I think we just got shot in the foot by our constituents," said a Texan on Capitol Hill.

What's most interesting about this is how hard Houston fought to make sure there would be federal funds for its subway in fiscal 1984.

Last December, when the Reagan administration pushed a 5-cent-a-gallon gasoline tax increase through the lame-duck session of Congress to pay for highway repairs and transit systems, Sen. Lloyd Bentsen (D-Tex.) agreed to vote for the bill only after Drew Lewis, then secretary of transportation, promised some of the money would be used for new urban rail systems.

The Reagan budget ignored that pledge, but Transportation Secretary Elizabeth Hanford Dole--with a push from such Texans as Sen. John G. Tower (R) and Bentsen--beat back the Office of Management and Budget and freed the money.

When the transportation appropriations subcommittee was through, Houston and Los Angeles each were due to get $110 million. The rest of a $465 million pot was to be divided among Portland, Ore.; Baltimore; Santa Clara County, Calif.; Miami; Detroit; Atlanta and Jacksonville.

After the Houston vote Saturday, the vultures regrouped to divvy up the $110 million. Houston apparently has preserved $5.5 million for planning, but the rest of the money will be spread among the other recipients, which now include St. Louis. The full appropriations committee is expected to ratify that negotiated settlement today.

The subway vote failed, knowledgeable Houston residents claim, because of a perception that Houston's existing bus service is not well run and because of continued sniping at rail systems by an assortment of groups. Both major Houston newspapers, the Chamber of Commerce and the mayor's office enthusiastically supported the subway.