At a hearing highlighted by a film showing an FBI agent haggling over the price of a stolen Rembrant, a Senate Judiciary subcommittee was told yesterday that the bureau's undercover operations have been growing steadily in size and sophistication over the past few years.

"It's a good investment," FBI official Francis X. O'Brien said of the technique that began four years ago with the planning for the bureau's celebrated fencing operation her named "Sting."

He said the FBI currently has nearly 100 criminal undercover operations approved or under way in the United States. Increasingly, he told the subcommittee on administrative practice and procedure, organized and white collar-collar crime operations are becoming the targets.

"The trend today is . . . away from the initial "sting'-type street operation and . . . into more sophisticated types of activity," O'Brien said.

Sens. Edward M. Kennedy (D-Mass.) and James Abqurezk (D-S.D.), who alernated as chairman at the morning session, said they were concerned at the same time about the tough and policy questions underlying the new approach - such as whether and when to wink at or violate state and federal laws.

"We're still groping," Mary C. Lawton, deputy assistant attorney general, testified. "It is new to us. Suddenly we are just beginning to grapple with some of these particular rules."

For instance, she said, the FBI asked, in connection with a lease it was trying to arrange for one operation, whether it had to include the standard nondiscrimination clause required of government agencies.

The answer was no. Otherwise, Lawton said, "You pretty well tip yourself off."

The highlight of the session was the videotape recording made by a hidden camera of an April 11, 1977, transaction in buffalo, N.Y., during a joint FBI-state undercover operation. The film, played at the hearing on a television set, showed an FBI agent, posing as an art expert, and his "source" dickering with a fence over a small Rembrant, entitled "Le Rabbin," the painting was stolen from a French museum in 1971, but the FBI's "art expert" kept expressing doubts abourt its authenticity.

"We think it's a Rembrant," the fence insisted. "I did my own research . . . I know what I'm talking about."

"Do you own your own gallery?" the FBI "expert" asked sceptically.

"No, I own the person that stole that painting off the wall," the fence shot back.

The FBI man finally bought the painting for $22,000 - 10 percent of what all sides evidently agreed was its minimum value. The fence has since been tried and convicted for dealing in stolen property.

John C. Keeney, deputy assistant attorney general, who testified for the Justice Department's criminal division, said the prosecutorial results of the FBi's undercover operations have been "extraordinarily productive," including successes against phony financial schemes by white-collar criminals, waterfront corruption, and frauds in government benefit programs.

According to the panel of four FBI officials who testified, the bureau spent just over $900,000 on its undercover operations last year and recovered more than $109 million worth of property.

O'Brien, who is chief of the FBI's personal and property crime section, said the bureau currently has 71 undercover operations of its own and 21 joint operations approved or under way (not counting foreign intelligence operations). About a dozen of the FBI operations are aimed at white-collar crime, compared to three such operations last year, the subcommittee was told.

Kennedy and Abourezk said this, in turn, raised raised questions about an undercover agent's participation in a legitimate business without notice to the owner and his acess to business records without a warrant. The FBI's John K. Chadwick said the bureau has put an agent to work for a business concern without knowing "full knowledge of a key individual" in that concern only once, and that this was done with the Justice Department's approval.

Lawton said Justice Department officials won't even put their guidelines down on paper for fear that criminal targets will be able to obtain them under the Freedom of Information Act. Kennedy suggested that the department seek changes in the law instead.

"Without criteria," Kennedy told Lawton, "i just don't know how , other than flying by the seat of your pants, you make a judgement." CAPTION: Picture, Agents points out principals in videotape of transaction in Buffalo where stolen Rembrandt was bought for $22,000. By James K. W. Atherton - The Washington Post