The University of Houston may have lost more than $30 million, a sum which is equal to more than one-third of its endowment, through highly speculative trading by one of its officers in virtually unregulated U.S. government securities markets, according to market sources.

The disclosures of the transactions made shortly after Thanksgiving by Samuel A. Harwell, 34, the university's manager of short-term investments since March 1974, was the latest in a number of startling revelations about the dangerously risky trading undertaken in the past two or three years by a number of banks and institutions in these markets.

The scandal at the university has since deepened with the suicide of the member of the board of regents who first questioned Harwell's transactions. The regent, Robert L. Grainger, a stockbroker with Underwood Newhouse in Houston, was not involved in Harwell's dealings, university and market sources insist. But Grainger's death, Harwell's discharge and the revelations about the raids made on the school's assets have shrouded the campus in a gloom that it will not easily shed.

The university and a number of the 25 or more brokerages through which Harwell dealth - including Merrill Lynch, Pierce Fenner & Smith and E. F. Hutton - are the latest institutions to find themselves involved in government regulatory agency inquiries into government securities trading. The markets garnering the most attention are the forward contracts and furtures of mortgage-backed securities guaranteed by the government known as Ginnie Maes. (The name refers to the Government National Mortgage Association, part of the Department of Housing and Urban Development, which guarantees pools of FHA, FMHA and VA mortgages to the investment community.)

The growing concern of such agencies as the Comptroller of the Currency, the Treasury Department, the Securities and Exchange Commission has created a degree of conflict among some agencies and officials over which agency should be given regulatory authority over GNMAs and other governmental-backed securities, such as Treasury Bills. It also has renewed fears among traders, brokers, some exchanges and trade groups such as the GNMA Mortgage-Backed Securities Dealers Association that their sector shortly will be hit with tight registration and trading rules cutting into their profits. Self-Regultory Plan

GNMA dealers, in fact, are trying to head off the Securities and Exchange Commission by proposing a self-regulatory plan similar to that of the National Association of Securities Dealers.

What frightens the dealers most, sources agree, is the specter of market regulation under the supervision of Stanley Sporkin, the SEC's zealous chief of enforcement.

In late summer, Sporkin's staff followed up on the lawsuits and public disclosure of the collapse of some securities firms and Florida banks due to GNMA speculation. Associate enforcement director Wallace Timmeny told the Bureau of National Affairs at that time that the SEC was considering seeking legislation to require registration of government securities dealers and stricter regulation of their activities.

In mid-October, Sporkin outlined preliminary regulatory proposals for GNMA forward contracts in a letter to John Dalton, president of HUD's Government National Mortgage Association.

Sporkin concluded his letter with ". . . we believe that ultimately dealers will have to be regulated and that there will have to be direct regulation over GNMA forward contracts."

The quiet furor that has been building about trading in these markets is based on the machinations that take virtually risk-free investments in government-backed securities and, through a complicated series of transactions in several related markets, transform the investments into highly speculative transactions with the potential for either windfall profits or staggering losses.

Substantial losses of small banks credit unions, savings and loan associations and other institutions which should be restricted to "prudent" or conservative investments with their minimal resources have brought the SEC and national and state banking agencies into the picture. High Pressure

hese small financial institutions have been "preyed upon," one SEC investigator said, by a number of shady firms that retail Ginnie Maes and other government securities as part of boiler-room, high-pressure telephone sales operations.

What brings on the financial troubles here, federal officials say, is that salesmen push the institutions to buy - without a penny down - securities that they can ill afford. The most frequent abuse, they say, is "overselling" - pressuring a small financial institution to commit itself for contracts, which if the bank or credit union had to purchase their full face value, would exceed the amount of the institution's total assets.

The basic security involved in Ginnie Maes is a pool of government-backed home mortgages. Investors may buy shares in the pool in $1 million units, and receive interest payments on the securities which are passed on by the lending institutions from the homeowners. If an investor buys the Ginnie Mae outright he's purchased a sound, virtually risk-free investment, since the GNMA, HUD and the Treasury, in effect promise he'll be paid the interest and principal promptly.

Ginnie Maes, however, can also be bought for delivery at a date in the future in a transaction known as a "forward contract." Ginnie Maes for future delivery are also bought and sold as futures contracts on the Chicago Board of Trade under the organized method of trading government securities as commodity futures. Debt securities increase in value as interest rates decline, so market participants here are "betting" on future moves of interest rates.

It is these two vehicles, both of which can be traded in conjunction with the conventional Ginnie Mae, that introduce the element of speculation to the game. And, for many, it is a costly game to play.

Forward contracts are essentially private contracts between an individual or firm and Wall Street dealers who belong to the GNMA Dealers Association for the future delivery of a Ginnie Mae at a price set at the time on the sale.

Forward contracts, which exist in dozens of other commodities and securities, are not securities and are not regulated by the SEC. But they also are not futures contracts, which are the Commodities Futures Trading Commission.

In a recent highly publicized case the refusal of about 25 banks and credit unions in tiny prairie towns and Southern villages to pay $8 million in losses on Ginnie Mae forward contracts brought about the demise of Winters & Co. of Ft. Lauderdale, Fla., and its subsidiary Winters Government Securities Corp. and an SEC investigation.

Winters was a retail brokerage that bought Ginnie Mae forwards from Wall Street dealers to resell to wealthy speculators and small institutions. While some of its early clients made money as interest rates declined in 1976, the nosedive for debt securities this year resulted in massive losses for firms such as Winters as well as their customers. Winters owned about $3 million to Wall Street GNMA dealers at the time of its collapse, according to Florida court records.

The University of Houston case, however, is no parallel with the scandals arising from high-pressure sales operations.

The story outlined by government and university officials, its legal counsel and outside auditors is that Samuel Harwell was a high-flying speculator who dealth with more than two doezen firms in order to pyramid his transactions into a multimillion dollar operation - on paper.

Harwell was playing with short-term funds, including student fees and receipts of the college bookstore, sources say, which normally would put a crimp in a trader's style. He was not involved in the investment of the university's endowment funds, which are handled by a private investment advisement firm and a committee of the board of regents.

But Harwell was a creative trader and found ways to circumvent the university's natural caution with funds, according to the picture painted by sources.

The university says Harwell pyramided, using securities owned by the college as collateral on a loan from a securities dealer. The proceeds from the loan would be used to buy another security on margin that security would then be used as security on another loan, and the cycle would continue. Pyramid Technique

Wall Street sources who trade Ginnie Maes said the pyramid technique is a leverage tool used by experienced speculators in nearly every market. When employed by a highly capitalized, sophisticated trader attuned the markets it can result in hundreds of millions of dollars of profits.

The key, they said, is the capital backing the moves. "If the market turns against you and you wither have to sell at a loss or capitalize yourself to wait 'til the market recovers, a well-backed speculator will be able to ride it out. Others will just lose and lose and lose," one source said.

Harwell reportedly pyramided a single security into as many as 30 credit purchases; he apparently built more than 10 pyramids.

When he pledged the securities he bought as collateral for loans, the standard deposits required by dealers ("maintenance of margin") would have amounted to 2 to 5 per cent of the value of the security, sources said. Since the securities he dealth in are traded in $1 million units, the deposits on each loan ranged between $20,000 and $50,000, leaving him between $950,000, and $980,000, with which to reenter the market.

The standard loan agreement on securities calls for the securities to be redeemed at a set price. If the market price rises in the interim, the borrower - in this case, the univeristy - would gain.

At the time Harwell was fired, the university's holdings in government securities amounted to about $250 million of which $180 million was in Ginnie Maes of some type.

Harwell's boss, Douglas Maclean, vice chancellor for financial and management services, has been placed on six months administrative leave with full pay and benefits in a "mutual decision," a university spokesman said Thursday.

The spokesman declined to discuss the likelihood that MacLean will rejoin the university's staff, but added, "The facts speak for themselves, I think."

The university has been careful to avoid discussing who was responsible for the trading decisions, and whether Harwell's moves were approved or known by any other university official or adviser.

A task force of auditors from Arthur Young & Co., lawyers for the university's outside law firm, private investment advisers and their inhouse counterparts have been working seven days a week to identify all of Harwell's transactions and to dismantle, albeit delicately, the paper pyramids.

A university spokesman said Thursday it will be "several weeks" before "any bottom line figures" on the transactions can be identified. "We don't know what the loss is, but on the one hand we hope there won't be any. We could even make some money, some of the accountants say."

Wall Street sources familiar with Harwell's transactions, however, stressed privately that the college was losing heavily at the time of Harwell's discharge and it would take a substantial market turnaround for the institution to break even, much less turn a profit. $17 Million Overdraft

Harwell's transactions reportedly resulted in a $17 million overdraft of one of the university's accounts at the First City National Bank, the large Houston-based bank that leads the list of the holdings of First City Bancorp. of Texas, Inc. The chairman of the holding company is J. A. Elkins Jr., the vice chairman of the Univeristy of Houston's board of regents.

Elkins was unavailable for comment yesterday, but a bank spokesman said. "Any overdrafts that may have occured in this bank were properly authorized by responsible financial officials of the university and have since been fully paid. The bank has sustained no loss as a result of any of these transactions and still enjoys a satisfactory customer relationship with the univesity."

An interesting facet to the Harwell incident, related in a story in The Wall Street Journal, is that Harwell apparently was connected with two of the firms with which he had trading accounts on behalf of the university.

SEC officials in Washington and Houston confirmed that the agency "is looking at" the university transactions and Harwell's involvement with two companies, Covington Knox, Inc., and Patrick Sullivan & Associates, both of Houston.

A principal of Covington Knox, Inc., a Houston-based securities firm that brokered some of the university's trades, is identified in the Journal story as having been a partner with Samuel Harwell in another Houston investment company, Harwell-Knox Investments, Inc., between Dec. 1974 and Nova. 1975. Samuel Harwell reportedly was listed as the corporation's registered agent.

The Journal also identified Patrick Sullivan as a half-brother of Harwell's, and the recipient of a $1 million loan by the San Jose, Calif. branch of Japan-California Bank, which was secured by a university bank deposit of more than $5 million as collateral.

No charges have been filed against Harwell or anyone else in the Houseton case, although SEC's Timmeny, said Thursday that the commission is studying probable violations of the anti-fraud provisions of federal securities laws.

Attemps to contact Harwell and his attorneys last week were unsuccessful.

Federal officials stressed that the prevailing questions raised by the University of Houston and Winters cases involve the apparent failure of brokerages and dealers - including some of the largest national firms - to question the authority of employees of financial institutions to trade on such a large scale, and especially, to involve their institutions in speculation; the apparent failure to question or to require documentation about the capitalization of these transactions or to study the instiutions' assets to determine if the account should be permitted to trade on a large-scale.

Wall Street sources indicated that at the time of his discharge, Harwell had entered the Ginnie Mae futures market in Chicago with a hedging account at E. F. Hutton, which ran to $108 million of short positions and $180 million of long positions. A short position requires an individual to deliver at a future date Ginnie Maes of a yield (interest rate) specified in the Chicago Board of Trade contract; a long position would permit one to buy at a future date Ginnie Maes with a set interest rate.

"It certainly raises questions of why a firm would permit a customer to carry such a position," said a source familiar with the regulations of the CFTC. "Customer suitability rules and financial standards would undoubtedly have been violated."

Thomas Re. Hutton's compliance officer, said earlier last week that Harwell had placed "substantial" hedge trades with the firm and that the SEC has made inquiries about them.

He defended Hutton's acceptance of the transactions, however. "Our understanding is that the university is audited by responsible officials and had proper controls. We were only seeing one side of the university's trading."

Robert Fomon, Hutton's president, said, "We aren't concerned about any culpability on our part."

Officials at the SEC and the CFTC indicated this week that both agencies will seek to obtain jurisdiction over the rocky GNMA forward contracts market in early 1978 in order to prevent such scandals from reoccuring.

The SEC's Timmeny said the SEC staff believes forward contracts are securities, not futures contracts. But, he added, "Realistically, it'll be a question for the courts to settle later on."