Union Oil Co.'s Retirement Plan Committee yesterday sued Mesa Petroleum Co. Chairman T. Boone Pickens Jr. in federal court in New York on grounds he is using his oil company as an unlicensed investment firm in violation of federal securities laws.

The suit was the latest in a barrage of legal actions Unocal has taken to fend off a hostile takeover threat by Pickens.

On April 8 Pickens' investors group, which already owns 13.4 percent of Unocal's stock, launched a $54-a-share cash tender for an additional 64 million shares that would give the partnership a 51 percent stake in the Los Angeles oil company.

Unocal, the parent of Union Oil, filed suit Thursday in federal court in Shreveport, La., charging that Pickens' takeover attempts breach antitrust laws by restricting competition in the oil industry.

The New York and Louisiana suits asked the courts to stop Pickens from pursuing his unfriendly campaign to seize control of Unocal.

The Retirement Plan Committee for Unocal's pension plan, which purchased 20,000 shares of Mesa Petroleum stock in October 1984, alleges in the U.S. District Court suit in Manhattan that Mesa is operating as an unregistered investment company in violation of the Investment Company Act of 1940.

The suit claims Mesa has adopted bonus and incentive plans to reward Pickens and other key Mesa employes for their investment activities, which the plaintiffs contend is illegal under federal securities regulations. Mesa Petroleum had no immediate comment.