There is almost a note of surprise in Rep. Jack F. Kemp's voice when he tells visitors these days: "Yes, Jack Kemp is going to vote for his first foreign aid bill."

That is actually not quite true; he has voted for one foreign aid bill in his 11 years in the House. But the conservative New York Republican is a certified, out-front critic of foreign aid programs, and the role reversal he is contemplating for this week underscores an unusual dilemma for House Republicans.

Suddenly, they are under the gun to produce votes for the sort of programs many have damned as "giveaways" for years. Suddenly, they are insisting their own president get involved if he wants to salvage his version of foreign aid this year.

In the past, most Republicans have sat out foreign aid votes or voted "no" while Democrats reluctantly passed the bills and suffered opposition gibes about flinging tax dollars overseas. With a diminishing constituency on both sides of the aisle, foreign aid has survived since 1979 under continuing resolutions, not by the conventional appropriation process.

The tables turned this year because of the long-running budget fight between House Democrats and the White House. House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) announced last week that a foreign aid bill would be brought to the floor before the Christmas holidays.

He was angry at President Reagan's persistent complaint that Congress will not send him any appropriations bills and eager to emphasize that his Democrat-controlled House actually has passed all but one, foreign aid, which so many Republicans oppose. Let the Republicans come up with the votes for that one, he said.

With that as prelude, foreign aid will go to the floor in a cross fire that some believe may cripple it. The House bill carries more money for multilateral aid to developing countries than Republicans want. And it has more money for military aid than many Democrats want. The result could be a shambles.

"The Republicans have got to agree not to offer a lot of amendments to cut the development programs," says Rep. Clarence D. Long (D-Md.), chairman of the Appropriations subcommittee on foreign aid and an old-school proponent of economic assistance. "If they do, the liberals will walk right out from under this bill. But if the liberals oppose the military aid, the Republicans will walk out."

It is generally agreed that Republicans must produce between 100 and 125 votes to pass a bill, and no one sees that many in sight yet. Republicans acknowledge that only a major effort by President Reagan and Secretary of State Alexander M. Haig Jr. will pull it off.

"A hundred votes? They're not there but they can be," says Kemp. "It won't pass unless the president gets involved."

The strategy for the hesitant GOP, Kemp adds, is to picture the bill not as traditional foreign aid, full of loans and grants to struggling countries, but as a kind of adjunct to the national defense appropriations both houses have been willing to pass at Reagan's insistence.

" 'Foreign aid' has become such a pejorative phrase," Kemp says, "but if you say it's a 'national security' measure, which I think it really is, then that changes everything."

The justification for that change of definition is a $300 million infusion of military assistance for Egypt, Sudan, Pakistan and other Middle Eastern and African countries that Kemp and his allies managed to get in the bill when it was before the House Appropriations Committee.

In overall military aid, it is $200 million below what the Reagan administration wants--$700 million below if loan guarantees for some arms sales are considered, Kemp points out. Democrats displeased with the growing military component of foreign aid may try to cut out some arms assistance.

But the biggest battles may be fought over the funds that are lent through international organizations to underdeveloped countries with precarious economies. On that issue, some Republicans may find themselves in conflict with the White House.

Although the Reagan administration has in general soured on such multilateral aid, the president has gone along with one of its more controversial variations--a commitment to join other countries in replenishing funds for the International Development Association (IDA), which makes long-term concessional loans for the World Bank. While cutting the domestic budget, Reagan endorsed a full funding of $850 million for IDA in fiscal 1982.

That has proved difficult for his own party to swallow. The Republican-controlled Senate cut it back to $520 million, and House Republicans will make similar efforts to reduce it. Democrats complain that Reagan's nominal support for full IDA funding has not been accompanied by instructions to his troops.

The administration has good cause to help pass the House bill in some form. If it goes down the drain, the alternative would be to accept whatever comes out of the overall 1982 continuing resolution that both houses must fashion by Dec. 15. For foreign aid, that could be much less than the administration wants. The resolution that died under a presidential veto Nov. 23 contained $10 billion for foreign aid, $1.6 billion less than the White House wants.

Because the administration's pet projects for military aid to new Mideast allies were jettisoned in that resolution, both Reagan and Haig stepped into the fray, telephoning key congressmen, including Democrats, for help.

This time, apparently, the administration is planning ahead. Haig has agreed to argue the case for foreign aid before House Republicans this week, and there are reports that Reagan himself will try cajoling his party's members into voting for what they so enjoyed voting against in the past.