An internal working paper circulated at the end of August by the office of the Securities and Exchange Commission's chief economist, Gregg Jarrell, says the practice of paying off corporate raiders with so-called greenmail results in a 4 percent loss in the value of a corporation's stock for other shareholders.

The SEC study refutes a more limited academic study that the Office of Management and Budget used earlier this year when it defended greenmail as economically beneficial.

Greenmail is the payment of a premium price for the shares acquired by an unfriendly investor so that he will sell out rather than continue buying into the company or demanding a voice in management. Legislation has been introduced by Democrats in both houses of Congress to outlaw the practice. A Treasury Department spokesman said yesterday that the Cabinet Council on Economic Affairs was familiar with the SEC study but was maintaining its policy of not taking a definitive public position on the issue. Other sources say the Reagan administration is philosophically opposed to interfering with corporate managers' ability to run their businesses as they deem necessary. INTERNATIONALISM . . .

Late last month the Boston Stock Exchange asked the SEC to let it establish an electronic trading link with the Montreal Stock Exchange. The SEC plans to seek public comment within a week. If approved, the link would be the first between a U.S. stock exchange and a foreign one. (A similar link between the Chicago Mercantile Exchange and the Singapore International Monetary Exchange was approved Aug. 29 by the Commodity Futures Trading Commission. Trading in currency futures began yesterday in Singapore and will begin today in Chicago.)

Canadian investors would be able to trade between 300 and 500 of the most commonly traded U.S. stocks directly within 30 seconds through the link with Boston. Also, 40 Canadian stocks traded in Montreal would be available for trading in Boston. Currently, Canadian orders for U.S. stocks must be placed by Canadian brokers who are members of U.S. exchanges, or vice versa. Electronic communciation would reduce the cost of trading.

Meanwhile, in a speech delivered this week in Vienna, Austria, SEC enforcement chief John M. Fedders told the International Bar Association, "We welcome your participation in U.S. securities markets , but you cannot expect preferential treatment. If you want to trade in our markets, you must agree to play by the rules that apply to all participants in the market."

Fedders has spent time recently explaining to foreign audiences the SEC's "waiver by conduct" approach to policing international capital markets. He appealed to the lawyers for cooperation in policing the markets, while trying to persuade them that the concept would not infringe on other nations' sovereignty.

"Waiver by conduct" means that any person who invests in U.S. markets implicitly gives up the right to hide behind foreign secrecy laws in the event of an SEC investigation into trading abuses. In the past, the agency's efforts to investigate insider trading in U.S. stocks by persons who trade through European banks have been hampered by foreign laws that protect the identity of investors. HOT ISSUES . . .

A report that the SEC issued late last month concluded that the agency has the problem of "hot issues" under control and does not need additional powers at this time. "Hot issues" are initial public offerings of stock, often by small companies without a track record, that soon rise dramatically in price as demand for shares increases and then often drop back just as dramatically. In some cases, investors are subject to high-pressure sales pitches characterized by false information about the issuer and price predictions. Also, some brokers have engaged in fraudulent practices such as market manipulation.

The major markets for bringing out hot issues are Denver and New York. The study was done at the request of Rep. Timothy E. Wirth (D-Colo.). A hearing last year by Sen. Alfonse M. D'Amato (R-N.Y.) resulted in legislation, now on the Senate floor, that would bar commodity dealers from the securities industry whom the CFTC has barred from trading in that business because of abuses. Some barred commodity dealers have been active in the hot issues markets. MUTUAL FUNDS . . .

The SEC yesterday approved a measure that will help mutual funds make investments overseas. From now on, the funds will be able to leave their foreign assets in the overseas banks. Previously, they were required to put their assets in branches of U.S. banks or to get special permission to leave them with local banks. APPOINTMENTS . . .

Robert J. Sack has succeeded L. Glenn Perry as chief accountant in the Enforcement Division. Sack formerly was a partner in Touche Ross & Co., one of the Big Eight accounting firms. Perry has joined another Big Eight firm, Peat, Marwick, Mitchell & Co., as a partner in New York . . . . Richard Chase has been promoted to associate director of the Division of Market Regulation.