Two major automobile insurance firms - State Farm Mutual and Government Employees Insurance Co. - yesterday reported a sharp increase in profitability last year.

American Finance System Inc., a Silver Spring consumer finance holding company, also posted a significant earnings gain.

State Farm, the nation's largest auto insurer, reported 1977 net income of $435 million compared with $200 million the previous year. Profits on underwriting auto policies soared to $319.5 million from $16.5 million in 1976, and the company yesterday declared "dividends" of $126 million to policyholders from earnings in the final six months of the year.

The dividends from the mutual insurance company amount to deductions from premium rates and will be shown on State Farm customers' next bills. Policyholders who don't renew policies will receive checks.

According to the company, based in Bloomington, lll., owners of 15,000 vehicles in D.C. insured by State Farm will receive premium reductions of $682,000 for a 31.2 percent return: owners of 361,000 vehicles in Maryland will receive $5.3 million for a 13.4 percent return of premiums; and owners of 491,000 vehicles in Virginia will receive $5 million for a 12.9 percent return.

State Farm dividends vary from state to state and are paid in states where company profits exceeded the amount required to meet financial needs of the firm. State Farm attributed the growth of profits to reduced claims reported last year in every major coverage area except for uninsured motorists a factor that a spokesman said could reflect sluggish economic growth nationwide.

As of Dec. 31, State Farm insured 18.7 million vehicles, an increase of 1.4 million over 1976. Total sales rose 25 percent to $3.86 billion.

Government Employees Insurance (Geico), of Washington, reported record profits in 1977 of $58.6 million ($2.96 a share) compared with a loss of $26.3 million in 1976. The previous record was in 1972, when Geico earned $32.6 million.

Not only did the Geico statement confirm Chairman John J. Byrne's forecast at last year's annual meeting - that record earnings could follow a turbulent period during which Geico almost collapsed into insolvency - but directors of the firm also increased the quarterly dividend rate for common stock yesterday.

A cash payout of 5 cents a share, up from 3 cents in the prior quarter, will be distributed March 31 to stockholders of record March 10. A regular dividend on preferred stock, of 18.4 cents, will be paid April 1 to owners on March 10.

Geico's statement, based on preliminary and unaudited figures, showed that the annual profits included a $20.7 million tax benefit gain, based on previous losses. Not counting this extraordinary gain or capital gains, Geico's operating profits for the year were $37.8 million compared with a loss of $26.4 million.

In th fourth quarter alone, Geico's total profits were $16.7 million compared with $8 million a year earlier, but per share figures were not available last night.

Byrne said Geico still suffered a loss of $6 million last year in underwriting auto insurance, a factor he said leaves "considerable room" for improvement.

But the year's performance overall indicates that Geico "continued to grow in financial strength through the fourth quarter," building a "solid foundation" for odditional gains this year, Byrne said.

Geico's surplus for policyholders' protection rose to $178.6 million from $136.7 million.

In another development, Geico said its tender offers for shares of three affihates expired at 5 p.m. on Tuesday and will not be extended.

During the offering peroid, Geico was tendered 1.39 shares of Government Employees Life (it offered to buy them at $14 a share): 378,800 shares of Criterion Insurance (it offered to buy at $21.50), and 443,000 shares of Government Employees Financial (it offered to buy at $12.25).

Geico, which had said it would buy up to 300,000 shares each of the three firms, said it would buy all the stock tendered. Thus, when the payments are made in the near future, Geico will become the owner of 55 percent of the life insurance company, 60 percent of Criterion and 61 percent of the finance company, Gefco

American Finance System reported 1977 profits of $6 million ($1.17 a share) compared with $2.8 million (46 cents) the previous year. The annual earnings included extraordinary gains in 1977 of $1.6 million (34 cents) and, in 1976, of $263,000 (6 cents).

Fourth-quarter profits were $749,000 (11 cents) vs. $263,000 (6 cents), not counting an extraordinary gain in the 1976 period of $681,000 (11 cents), which made final net income $944,000 (17 cents).

Total revenues last year were flat at $84.3 million, while outstanding receivable loans were $350 million on Dec 31 compared with $340 million a year earlier. AFS operates 374 consumer finance offices in 23 states and has agreed to a merger into Security Pacific Corp., a Los Angeles bank holding company.

Metropolitan Federal Savings and Loan Association of Bethesda and its wholly-owned subsidiary reported 1977 profits of $1,586,242 ($4.21 a share) compared with net income after taxes of $967,811 ($2.66) in 1976.

Metropolitan also announced a 5 percent stock dividend payable March 15 to shareholders of record Feb. 28.