Housing and Urban Development Secretary Patricia Roberts Harris charged yesterday that Federal National Mortgage Association "has gotten off the track" intended by Congress by concentrating on profits and avoiding inner city or low cost housing investments across the country.

In the strongest criticism of the company by a ranking government official sonce FNMA was transformed from a government agency to the private sector in 1970, the HUD Secretary said mortgage bankers are reluctant to make urban mortgage loans because they believe "Fannie Mae won't buy them."

Nicknamed "Fannie Mae," the Washington company was chartered by Congress and given a mandate to aid the nation's housing market by purchasing home mortgages originated with other companies - thereby freeing up more money for mortgages in periods of tight money.

As the Senate Banking Committee opened two days of hearings on Fannie Mae yesterday, Harris voiced Carter administration support for proposed legislation that would add four additional public directors to the company's board and open up its books to the public under the Freedom of Information Act.

She also criticized her own department under the two previous Republican administrations for failure to exercise statutory authority over the company's operations and for not "clearly informing Fannie Mae" about the government's concern for support of all housing sectors.

"An enlarged public representation on the FNMA board of directors should increase FNMA's sensitivity to its public purpose responsibilities and help it to get back on the track which Congress placed it," Harris told the committee.

Under consideration is legislation sponsored by committee chairman William Proxmire (D-Wis.) and Sen. Alan Cranston (D-Calif.), which would create a board of directors for FNMA consisting of 10 persons elected by stockholders and 9 appointed by the President (compared with 5, currently).

The "high quality" of Carter's recent appointments to FNMA's board are "instructive as to the kind of people we think should be on that board," persons representing public constituencies across the country, Harris said.

"We had hoped" that FNMA itself would "offer a similar variety" of candidates for directorships to stockholders at the company's recent [WORD ILLEGIBLE] meeting. Instead, management offered a slate of "10 white males, good and true the Secretary added.

Carter recently named as FNMA board members the first black director, Washsington realtor John W. Thompson; HUD general counsel Ruth Prokop; former FNMA president Raymond Lapin and University of Delaware urban affairs expert Marvin S. Gilman.

FNMA declined to comment yesterday on the statements by Harris and other individuals or organizations, which similarly condemned past practices by the company and called for enactment of the Proxmire-Cranston bill. Officers of the firm are scheduled to testify today.

In recent weeks, FNMA has hired a Washington law firm to help prepare its case against the proposed legislation, which the firm contends is unconstitutional.

Sen. Edward Brooke said (R. Mass.) yesterday that he agrees with constitutional issues raised by FNMA and expects "to oppose vigorously" the legislation "unless convinced otherwise." It would not be proper to increase the proportional number of government-appointed directors without the approval of stockholders, he declared.

Brooke emphasized his feeling that Fannie Mae "should show greater sensitivity to inner city mortgage needs but he said it is not necessary to add to the board of directors to achieve this goal.

Secretary Harris noted that FNMA's recent attacks on the proposed legislation had sought to emphasize its role as a company in the private sector of the economy. But in an earlier case in New York, when Fannie Mae was opposing state law requirements, the company had emphasized its role as a "federal instrumentality."

"We do not believe that FNMA should have the best of both worlds, changing its status to suit its advantage," he continued.

Harris said FNMA began business with assets previously owned by the government, is given access to public money markets with lower interest rates because of its federal agency status, has a $2.25 billion line of available credit from the Treasury and "is not just 'any corporation,' as the FNMA . . . would suggest."

She also was critical of Fannie Mae's role in the mortgage market. In effect, she charged the company with ignoring its role of selling mortgages at certain periods, possibly because of fears that profits would decrease.

Harris said FNMA should be buying mortgages when money is tight, to provide more housing funds, but should sell mortgates during periods of easy credit and declining interest rates.

Other housing industry experts have argued that because FNMA continued to buy mortgages even when money was available from other sources, the company created an atmosphere in the 1970s real estate "boom-bust" cycle that led to speculative spending for often unnecessary projects other than housing.

From the end of 1968 through last year, FNMA increased its holdings of mortgages from $7.2 billion to $33 billion. Since 1968, sales of mortgages have totaled less than $1 billion; the only periods during which mortgages were sold in any volume were early in 1971 and 1972, and the first quarter of last year.