The money market mutual fund industry, which has siphoned billions of dollars away from banks, has gone to court to try to prevent banks from getting some of it back by establishing their own funds.

The Investment Company Institute, the trade association for mutual funds, filed suit yesterday in U.S. District Court here against the Federal Deposit Insurance Corp., the government agency that oversees savings banks. The ICI wants the judge to order the FDIC to consider its petition to stop the Boston Five Cents Savings Bank from setting up a money market fund under a subsidiary. Earlier this week the FDIC voted unanimously not to give the petition a hearing.

ICI President David Silver charged that "the FDIC's decision amounts to a complete abdication of the important responsibilities Congress entrusted to it," to determine whether the bank is violating state and federal law. The FDIC had no comment on the suit.

However, attorneys for both sides said the judge would be called upon to decide in effect whether Boston Five's plan requires a change in the general character of its business--which would necessitate prior approval by the FDIC. The FDIC is expected to argue that the plan does not require a change in status and no further action is necessary.

The ICI is trying to force consideration in a public hearing so that it can get to the heart of the issue: the Glass-Steagall Act of 1933, which prohibits commercial banks from engaging in the securities business. Boston Five is believed to be the first savings bank to try to set up its own money market fund, although other banks and thrifts have made arrangements with existing funds.

Lawyers for Boston Five cite a 1981 Supreme Court ruling involving the ICI and the Federal Reserve Board stating that Glass-Steagall does not prohibit a bank subsidiary--as distinguished from the bank itself--from operating a money market fund. The ICI counters that the 1981 decision involved a legitimate, existing affiliate, whereas the proposed Boston Five Mutual Fund would be a sham subsidiary, operated by the same staff in the same place.