The Federal Communications Commission, in an effort to loosen its controls over the broadcasting industry, yesterday moved toward allowing the three major television networks to own and operate cable televison systems.

By a 6-0 vote, the commission proposed dropping its 12-year-old rule that has barred the networks from operating cable systems.

At the same time, the commission noted that it was concerned that the elimination of this and other ownership restrictions ultimately could increase concentration in the video market, thereby reducing the diversity of programming available to the public.

As a result, the commission asked the public to help it develop a formula that would set a threshold at which the commission would place curbs on how many video outlets one company could own.

Yesterday's action is expected to be the first of many proposals lifting present broadcasting ownership restrictions. Since assuming office a year ago, FCC Chairman Mark S. Fowler has made it clear he wanted to lift the rule limiting the number of television and radio stations a company may own to seven television stations, seven AM-radio stations and seven FM-radio stations. Other restrictions, including the rule barring local televison stations from owning and operating cable sytems in their own viewing areas, may also be targeted for repeal.

Fowler has argued that with the advent of cable, pay-TV, low-power television stations and direct-broadcast satellite networks, the broadcasting industry has become so diversified with so many different companies in the field that these restrictions no longer are needed. These restrictions, Fowler has said, only hinder competition in the rapidly growing video industry and limit the number of choices available to consumers.

The large number of companies in the cable television market was one reason the commission also agreed yesterday to drop a 12-year-old proposal that sought to limit the number of cable television systems one company could operate. The proposal was made just as the cable television industry was getting off the ground and the commission was concerned that one company could dominate the new industry.

Similarly, the staff argued that the 1970 rule barring networks from owning any cable system was outmoded. At the time the rule was issued, the commission was concerned that networks would try to block cable development by seeking permission to operate cable systems, then failing to run them.

With cable systems now installed in thousands of cities across the United States, that is no longer a concern to the commission. However, several commissioners made it clear they still were very concerned that with the elimination of this and other ownership rules, a handful of companies could control such a large number of stations that the industry would become too concentrated. From that concern grew Commissioner Mimi Weyforth Dawson's proposal to seek public comment on an "alarm system" for multiple holdings.

According to an aide to Dawson, the alarm system probably would be similar to the recent merger guidelines drawn up by the Justice Department in which the antitrust division has set limits, based on a complex mathematical formula, under which it would attack a corporate acquisition. However, the aide said, the threshold under which the FCC would limit ownership of video outlets would probably be lower than the Justice Department's, because the commission would be more concerned about First Amendment diversity issues than Justice is.