Congress should establish a special fund to finance research on a defense system to protect Europe against attack by medium- and short-range Soviet missiles both nuclear and conventionally armed, Defense Department officials testified yesterday.

Undersecretary Fred C. Ikle told a Senate Armed Services subcommittee that such a fund would help negotiations with West European allies and help "get us in a mode of sharing and matching money."

Sen. Dan Quayle (R-Ind.) said he would offer an amendment to next year's defense authorization bill asking $50 million "for cooperative development" of a defensive sys- tem against tactical ballistic mis- siles.

Ikle and Lt. Gen. James A. Abrahamson, director of the Strategic Defense Initiative, President Reagan's space-based defense system, said testing and development of such systems would not violate the 1972 Antiballistic Missile (ABM) Treaty as long as they could not knock down strategic missiles.

Ikle, however, cited a "bonus effect from SDI work," and Abrahamson noted that "our allies operating inside the ABM treaty can contribute to solutions for both the short-range and long-range ballistic-missile threat."

Citing one terminal defense approach being explored under the SDI program, Abrahamson listed elements, which Europeans could explore, that would cause "no problem" with ABM compliance.

Sen. William S. Cohen (R-Maine) pointed out that Reagan has charged the Soviet Union with "possible violations" of the ABM treaty in tests of a new antiair missile, the SAX12. That system, the administration has said, can hit short-range and "some types of U.S. strategic ballistic missiles."

"Aren't we running the same risk" of violating the ABM treaty in the short-range missile defense program? Cohen asked.

Ikle responded that the United States simply raised questions about the Soviet system, adding: "We have not reached a finding" as to whether it constitutes a violation.

Abrahamson said some projects applicable to a North Atlantic Treaty Organization defense system are being discussed with several countries.

Until now, "Soviet military planners have had a free ride" on their threat to use ballistic missiles in Europe, Abrahamson said, adding: "Even a modest defense against tactical ballistic missiles would have a major effect on the situation."

Lt. Gen. John F. Wall, head of the Army's strategic defense command, said existing systems, such as the Patriot antiair system, would be upgraded as an initial step to meet the Soviet missile threat.