Executives of the Rouse Co. have met in recent days with representatives of a Canadian investment firm, which earlier this month revealed plans to acquire up to 20 percent of the Maryland retail center and real estate firm, but Rouse President Mathias DeVito declined yesterday to discuss the substance of the talks.

Discussions to date have centered on "whether and under what circumstances" Trizec Corp. Ltd., of Calgary, may exercise on option it has to buy 7 percent of Rouse common shares and then to increase its holdings to 20 percent, DeVito told a large crowd of Rouse stockholders at their annual meeting in Columbia.

Although DeVito said earlier this month that Rouse does not "look favorably" on any party owning as much as 20 percent of the firm, he emphasized yesterday that Trizec does not "seek control or a change in structure or management."

Trizec, described by DeVito yesterday as a "fine Canadian company," is a real estate investment business controlled by members of the Bronfman family, wealthy Canadians who are heirs to the Seagram liquor business.

In a statement filed with the Securities and Exchange Commission, Trizec has stated that it has an option through Aug. 31 to buy 940,000 Rouse shares from a group formerly headed by the late S.H. Scheuer. Trizec also said it may increase this holding to 20 percent by acquiring additional Rouse shares.

One possible problem could be antitrust implications of any large investment in Rouse by Trizec, since the Canadian company has purchased 40 percent of Earnest W. Hahn Inc., a West Coast shopping center operator and Rouse competitior.

Currently, Rouse runs 52 retail complexes -- 34 that it developed and 18 that have been acquired under a three-year-old program of acquiring and managing older malls with future profit potential.

At the company's 25th annual meeting since becoming a public firm, DeVito said more emphasis in future years will be on urban retail centers such as Harborplace in Baltimore, which opened last year and has attracted 18 million persons to date in a major downtown renaissance.

Downtown developments are "harder to do in every way" than suburban malls, but prospects for growth in sales and profitability are "far better" than for suburban sites, he added.

Rouse's newest developments in the Baltimore-Washington market are an expansion of the mall in downtown Columbia, adding 133,000 square feet of space for smaller retailers plus a Sears, Roebuck & Co. unit, due to open on Aug. 6, and the new White Marsh mall northeast of Baltimore.

White Marsh, on schedule for a planned Aug. 12 opening, is expected to start a retail revolution in the regional market.R.H. Macy & Co. of New York is moving into the area with its initial operation, a Bamberger's department store. In addition, Woodward & Lothrop of Washington will be one of the five major tenants, with the most-distant store to date from its 100-year old D.C. base.

Rounding out the unusual competitive environment at White Marsh will be Hutzler's long Baltimore's premiere department store, and two national chains -- Sears and the J.C. Penney Co. The mall will have 350,000 square feet and room for about 200 shops, with 85 percent of the space already leased, according to Rouse Executive Vice President Michael Spear.

Bamberger's will open its second area store closer to Washington in another Rouse mall, in Owings Mill, Md., in 1984 or 1985. Woodies also has signed up for the Owings Mill site, northwest of the city between the Baltimore beltway and Reisterstown, along with Garfinckle's of Washington and Hutzler's.

In the first quarter of 1981, retail center profits before deducting certain noncash charges (depreciation, amortization and deferred income taxes) rose more than 15 percent to $4.4 million, continuing the dominant role of the shopping centers in the company's operations. Overall, earnings before noncash charges rose to $3.23 million from $3.03 million, with all operations except mortgage banking showing gains from last year.

DeVito said Rouse is looking for "a very good year," even though there is concern that continued high interest rates will hurt profitability by boosting interest expenses and start-up losses from projects scheduled to open this year. In addition to Columbia Mall and White Marsh, Rouse this year will complete expansions or renovations at malls in Dayton, Ohio, Milford, Conn., and San Antonio.