The New York Stock Exchange today fined the large brokerage house of Paine Webber Jackson & Curtis Inc. $300,000 for the huge paperwork backlog it faced earlier this year during heavy trading in stocks and bonds. The fine was the biggest the New York exchange has ever levied, NYSE officials said.

Paine Webber Jackson & Curtis is a subsidiary of the large financial services company Paine Webber, Inc., which merged last January with the Blyth Eastman Dillon & Co., another big brokerage/financial services company. The paperwork overload dates back to the merger.

Earlier this month the Securities and Exchange Commission cited Paine Webber for reporting transgressions that arose from the huge snarl of paper in Paine Webber's back office. The company reported earlier this month that it lost $6.9 million in its fiscal year ended Sept. 30 because of expenses that arose from the paperwork backlog.

The NYSE alleged, among other things, that Paine Webber made inaccurate reports to the press and the exchange, that it violated its net capital position briefly and that it failed to complete a required quarterly securities report for the SEC twice last year because of the confusion in its operations department.

Paine Webber said that it has resolved most of the problems that came about because of the merger with Blyth and that it will return to profitability in the quarter that ends today.