Rep. Joseph L. Fisher (D-Va.) says that when he was invited to be a delegate to the Interparliamentary Union's spring meeting here he "swallowed hard . . . and reached the decision to accept the criticism and come."

Fisher, whose congressional district includes Arlington and the northern half of Fairfax County, said it would be "chicken" to be scared off by "criticism that may not be well informed," although he added, "there's a limit on how much flak you can take."

Since his arrival here Monday, after stops in London and Paris, Fisher, like most of the 14 members of congress who are America's official delegates to the IPU, has faithfully attended the twice-daily debating sessions at which legislators from 75 countries are considering such issues as terrorism illiteracy and the neutron bomb.

Fisher hasn't allowed the steady procession of receptions and special events to interfere with his participation in the conference's agenda.

Wednesday Fisher and Rep. J. J.(Jake) Pickle (D-Tex.) missed a reception for the American delegation at the home of Portuguese Prime Minister Mario Soares and were late for a luncheon with U.S. Ambassador Richard Bloomfield because they were drafting a statement on international economic problems.

"If I were in charge, Fisher said, "these meeting would be somewhat more modest and streamlined. But I make no apoligy for being here. In the private sector, and the executive side, it's done without a thought."

Fisher said he realized that "you run the risk of criticism by you (the press) and my constituents" by attending an IPU session, a favorite of junketeering congressmen since 1889.

But at age 64, he said, "I can't wait 10 or 20 years" to pick up the insight he is getting here, which he considers of great value, especially in his assignment here to a subcommittee on international trade policies.

Fisher conceded, however, that "it's a clumsy way to get information, coming all the way over here."

He said that he plans to "pay full freight" for his wife Peggy, even though the funds authorized for the trip include expenses for spouses.

Fisher said he asked Mary McLaughlin, one of nine congressional staffers who made the trip, to "keep as good as record as you can of identifiable expenses" incurred by Mrs. Fisher.

Fisher said it would not be possible to determine the cost of Mrs. Fisher's ride on the Air Force jet that brought the 46-member party here, and therefore he won't have to pay for that.

The plane, part of the fleet that is stationed at Andrews Air Force Base, will leave here this morning and take part of the delegation to London, for a weekend of rest and relaxation.

Then the jet and its 22-member crew will return to Portugal and pick up the remainder of the delegates Sunday morning, rendezvous with the others at London's Heathrow Airport, and head home Sunday afternoon. Except for Rep. David Bowen (D-Miss.), who met the group here, the entire party also spent last weekend in London and Paris.

Among those leaving here early are Sen. John Sparkman (D-Ala.) and his wife, several of the 10 House members, and nearly all of the staffers.

"The conference runs through Saturday," Fisher said, "and since I came for the conference, I'm staying in Lisbon."

Having said that, Fisher took another hard swallow and admitted that he is worried about how his remarks will sit with his colleagues, some of whom are leaving early, skipping meetins or not reimbursing the cost of their wives' trip.

"I don't want to appear to be insufferably moral about this thing," Fisher said. "I don't want to be a stinker."

The $45,000 appropriated by Congress for IPU trips this year is supposed to cover all expenses, but with three meetings in 1978 instead of the usual two. Arthur Kuhl, the assistant secretary of the Senate who serves as executive secretary to the delegation, said it will be necessary to seek additional funds.

The budget squeeze may have prompted the distribution of a note Wednesday night in the control room of operation CODEL (Congressional Delegation) IPU that announced "an assessment of 50 escudos per person" to cover the cost of a party honoring Rep. and Mrs. Robert McClory (R-Ill.) on their ninth wedding anniversary. The assessment worked out to $1.25 a head.

Another note, perhaps repsonding to grumbling about the continental breakfasts that are included in the cost of the rooms at the Hotel Ritz, reported that yesterday's breakfast meeting of the delegation would feature, and the words were underlined, "a full American buffet breakfast."

The 10-story Ritz, whose hilltop location offers a panoramic view of the city's busy harbor and red-tiled roofs, is "perhaps as fine a hotel I've ever been in," Fisher said as he gazed around one of the enormous sitting rooms off the main lobby.

The rooms occupied by the Sparkmans and the three other senators and their wives feature two balconies, open and enclosed, two walk-in closets, marble bathrooms with heated towel racks, a telephone, two showers and a tub, and six light fixtures.

Rooms assigned to the congressmen and staffers are only slightly less [TEXT OMITTED FROM SOURCE]

Fisher said he didn't volunteer for the trip, but rather was picked by Rep. Lee Hamilton (D-Ind.) "after he surveyed the scene carefully." Hamilton was to have been chairman of the House delegation, but cancelled at the last minute.

This is Fisher's second congressional trip. The first one, to Nairobi, Kenya, was part of the U.S. UNESCO Delegation.

"If you think this is a bash," Fisher said, "you should have seen that one in Kenya."

What would he do if he were invited to IPU's upcoming May meeting in Vienna and September session in Bonn, Fisher was asked.

"Vienna would be a dilemma," he answered. "But Bonn is out of the question before the election."