Rouse Co., a real estate firm and joint developer of Columbia, Md., reported increased revenues and earnings for 1980, with all operating divisions but mortgage banking producing the highest profitability on record.

Revenues of Rouse increased to $119.5 million for $105 million, and operating earnings before noncash charges rose 7 percent to $13.4 million from $12.6 million. Noncash charges include such factors as depreciation of property, deferred income taxes and amortization of deferred costs. Rouse officials say this measure provides the most accurate reading on the company's performance.

Net income also rose to $4.05 million (29 cents a share) from $3.33 million (24 cents) in what President Mathias DeVito called "a good year for the company in a difficult economy."

Retail center earnings rose 11 percent to $17.4 million, while James W. Rouse & Co., the mortgage banking subsidiary, suffered a decline in earnings to $758,000 from $1.8 million because of volatile and record-high interest rates. Five new retail complexes are scheduled to open this year, including White Marsh in the Baltimore suburbs.

Pubco Corp., a Glenn Dale, Md., holding company for printing and publishing businesses, reported 1980 profits of $882,214 (23 cents a share) compared with $758,431 (24 cents, with fewer shares outstanding) a year earlier. Revenues rose to $58.6 million from $53.5 million.

Chairman Charles Lockyer also confirmed a report in The Washington Post that the Merkle Press printing plant will be closed by midyear after heavy losses in 1980 caused by an "inability to increase sales."

Lockyer indicated that Pubco will suffer initial losses because of the Merkle closing but forecast that consolidated earnings from operations "will be substantially improved" in 1982. Founded here in 1920, Merkle currently employs 430 workers.