Federal National Mortgage Association profits for 1980 were off 91 percent from the previous year, the nation's largest purchaser of residential mortgages reported yesterday. It was the second-worst year for Fannie Mae since it was transformed from a government agency into a publicly traded corporation a dozen years ago.

Net income totaled $14.2 million (23 cents a share fully diluted) compared with $161.7 million ($2.68) in 1979. (The company's worst year was in 1970 when it earned just $7 million.) Fannie Mae had a loss on its 1980 mortgage and loan portfolio of $41.7 million versus a gain of $241.3 million the previous year.

During 1980 the government-chartered firm, which may draw on the U.S. Treasury for $2.25 billion, experienced its first losing quarter and was forced to halve its divident rate to 16 cents a share. That action caused a record wave of selling of Fannie Mae stock and depressed its price.

The fourth quarter was the third-worst ever, with net income of $671,000 (1 cent), down 89 percent from the previous quarter and down 98 percent from the same period in 1979. Fannie Mae's portfolio loss totaled $11.7 million during the same period in 1979.

Last year's high, volatile interest rates increased Fannie Mae's average borrowing costs until they more than offset the largest yields on its mortgage portfolio. While the return on its portfolio amounted to 9.08 percent in September, the latest figure available, its average debt costs were 9.49 percent. During the year Fannie Mae offered investors purchasing its 2- to 8-year debentures interest rates ranging from 9.5 percent to 15.4 percent. In December its short-term notes were paying 17.8 percent.

First Virginia Banks Inc., a multi-bank holding company, reported net income in 1980 after securities losses was up 10 percent over 1979 levels, from $13.8 million ($1.31 a share) to $15. million ($1.42). Net income for the final quarter of 1980 rose by 32 percent to $4.6 million (43 cents) from $3.5 million (33 cents) during the same period in 1979.

Chairman Thomas K. Malone Jr. attributed the higher fourth-quarter earnings to sharply rising interest on short-term investments and rate-sensitive loans.