The Securities and Exchange Commission yesterday denied it was responsible for the safekeeping of the defense files of First Jersey Securities Inc., which disappeared from a federal building in New York City late in 1980 shortly after an administrative trial of First Jersey was adjourned for negotiations on a possible settlement.

First Jersey, a securities brokerage firm with 23 offices nationwide, is accused by the SEC in a civil complaint of defrauding customers by manipulating stock prices on five occasions between 1975 and 1978, and of other violations of securities regulations and laws during those years. First Jersey has denied those allegations.

The firm obtained a temporary injunction on Sept. 25 from U.S. District Court in Newark, N.J., barring the SEC from resuming the hearings. First Jersey is asking that the hearings be halted until the missing documents are located and returned, with that issue set to be argued in federal court on Oct. 22. The documents, including 22,000 pages of annotated transcripts from 11 months of hearings and the attorneys' notes on interviews and future cross examinations, are necessary for its defense and its ultimate vindication, the firm said.

In its response, the SEC said it took "sharp exception" to First Jersey's contention that only government officials had access to the hearing room and could have removed the files, contained in four cabinets and numerous boxes and bags.

The SEC said that in December 1980, two men appeared at the federal building "on behalf of First Jersey" and removed two of the file cabinets. SEC Investigator Walter K. Walsh said in an affidavit that in mid-December of 1980 he opened the door to the hearing room so that two men could remove the filing cabinets. Neither wore the uniform of General Services Administration employes charged with the maintenance of government buildings, Walsh said. The files were reported missing by First Jersey in late December 1980, about two months after the hearings had been adjourned.

Walsh said that, while the younger of the two men carted two file cabinets outside, he remained in the hearing room with the older man for about a half hour.

"Among other things, he asked me if I had ever seen Bob [First Jersey President Robert E.] Brennan's $2 million estate in Brielle, N.J. He spoke of it as if he had been there," Walsh said.

Walsh also swore that two men appeared to know exactly what to do. He said they complained that "they were back here to remove file cabinets even though they had been here before to bring in file cabinets."

In his affidavit, Walsh said he was told by another SEC official that the two men were from First Jersey, but he does not say that the men identified themselves as representing the firm. Walsh also said that one file drawer opened as it was being removed and he saw it was empty. Other files and boxes remained in the room after the men left, Walsh said.

In other affidavits, the government said the file cabinets in the hearing room were First Jersey's property and had been brought into the building by the securities firm.

One SEC attorney not working on the hearing, Caryn Miller, said she saw unidentified men removing First Jersey file cabinets and places the date at a time when that part of the federal building was being recarpeted -- around the end of December 1981, according to the SEC.

Patrick J. Finley, an SEC attorney working on the case, swore in his affidavit that he warned First Jersey's lawyers "that the hearing room in which they were storing their files was being used for other purposes besides our hearing." He said a key to the hearing room was in a drawer of the nearby desk of a secretary. He said he sometimes unlocked the door and at other times Brennan and his attorneys were already in the hearing room when he arrived. Finley's affidavit does not say the First Jersey representatives knew of the key or used it.

The SEC court filings confirm that in June 1980 the commission authorized a second investigation of First Jersey, concerning trading of stock in Geosearch Inc. in 1980. First Jersey has stated that documents relating to Geosearch were among the missing records.

In response to the SEC filings, Brennan said that two men from a moving company hired by the legal firm representing First Jersey moved two file cabinets into the hearing room in November 1979 and removed them on Dec. 21, 1979, when the hearings adjourned for the holiday season. The files were returned in January 1980, Brennan said, and, although several were added, none of the files was removed after that by First Jersey or its attorneys. "Nobody from First Jersey went to get those files" after the October 1980 adjournment, Brennan said.

The meeting with the workmen cited in Walsh's affidavit must have happened in 1979, not 1980, Brennan said, adding that it was "inconceivable" that the SEC would have permitted two unidentified men to enter the building and remove file cabinets -- as Walsh stated.

He said he and the firm's attorneys "never had access to the hearing room without an SEC officer opening the door."