Record high borrowing costs caused losses of Federal National Mortgage Association to balloon substantially during the third quarter, the Washington company reported yesterday.

Separately, Penril Corp. of Rockville listed higher profits for its fiscal year ended July 31, and two Northern Virginia banking firms--First American and First Virginia Banks--reported higher earnings for the third quarter and first nine months of 1981.

Federal National Mortgage, known popularly as Fannie Mae, is the nation's largest single source of mortgage money through its purchases in the secondary home finance market of lending institutions' mortgages--freeing up more funds for lending.

Faced with a staggering level of interest rates that must be paid to attract investors to its own market instruments and a lower rate of return on the mortgages it owns, Fannie Mae has been running a money-losing business since late last year.

In the quarter ended Sept. 30, Fannie Mae posted a net loss of $79.5 million compared with depressed profits in the same period last year of $5.9 million (10 cents a share). Before taxes or tax credits for losses, the loss in the recent quarter amounted to $152 million compared with earnings of $11 million.

Because of the sharp third-quarter losses, Fannie Mae's net loss for the first nine months rose to $119.6 million compared with profits of $13.5 million (22 cents) in the comparable 1980 quarters. Results for the 1981 period include a $42 million reduction of Fannie Mae's allowances for losses from loans.

Chairman David O. Maxwell, in assessing the record losses, said yesterday that Fannie Mae has added a large volume of higher-yielding assets to its portfolio in recent months while boosting the fees it charges for mortgage purchase commitments. These steps will have beneficial effects in the future "but they were insufficient to offset the corporation's exceptionally high cost of money during the third quarter," he stated.

Commitment fee income jumped to $32 million in the third quarter from $22 million a year earlier, but borrowing costs--which topped 18 percent in July before easing off-- resulted in a loss of $188 million on Fannie Mae's mortgage portfolio. Since March, Fannie Mae has refinanced $372 million of low-yielding mortgages through a program that encourages home owners to refinance properties with FNMA mortgages, and this volume was replaced with $676 million of loans at higher rates; the yield to FNMA on the $304 million of new money averages 17.19 percent, Maxwell said.

The Fannie Mae official also emphasized that the firm had a strong capital position on Sept. 30 of $1.3 billion in shareholders' equity, which he said would permit the business to "weather current interest rate levels" and remain a strong source of housing finance.

Penril Corp. reported record earnings and sales for the year ended July 31. Profits rose to $2.29 million, a gain of 28 percent from $1.78 million a year ago. Earnings per share declined to $1.15 from $1.35 because there now are more shares outstanding. Revenues jumped 20 percent to $40 million.

The Rockville firm, which designs, develops, manufactures and markets a diversified line of high-technology electronic equipment and consumer audio equipment, said fourth-quarter profits rose to $544,000 from $458,000 while per-share earnings declined to 27 cents from 35 cents. Sales rose to $10.6 million from $8.8 million.

First American Bank of McLean, a unit of the Financial General Bankshares holding company here, reported record third-quarter profits of $3 million ($1.70 a share) compared with $2.2 million ($1.25) a year ago, an increase of 36 percent. Nine-month earnings also rose 36 percent to $9.2 million ($5.18) from $6.8 million ($3.80).

President Milton Drewer Jr. credited an increase in net interest income of $2.4 million and an increase of $500,000 in noninterest income for the third-quarter gains. Assets of the bank rose to $945 million from $857 million last year.

First Virginia Banks of Falls Church, a bank holding company, reported third-quarter operating profits of $5.3 million (47 cents a share) compared with $3.7 million (34 cents) a year ago. Counting gains or losses from selling securities, net income was $4.7 million (42 cents) vs. $3.5 million (33 cents).

For the first nine months of 1981, First Virginia operating earnings rose to $14.6 million ($1.31) from $10.7 million ($1.01).