The Justice Department asked a federal appeals court yesterday to reject the Federal Communications Commission's landmark decision permitting the expansion of American Telephone & Telegraph Co. into unregulated businesses.

Calling the decision irrational, the Justice Department charged that the FCC does not have the power to implement the so-called Computer II decision, let alone monitor the giant Bell System with accounting rather than regulatory tools. The brief was signed by the assistant attorney general for antitrust, William Baxter.

In its brief, filed with the U.S Court of Appeals here, the Justice Department also announced that it intends to appeal the recent decision of a federal judge in New Jersey that upheld, at AT&T's request, the FCC's authority to permit AT&T's entrance into new fields such as the computer business.

The appeals court action here was brought initially by the Computer & Communications Industry Association, which has been joined in the challenge by a variety of other trade groups and companies. But until yesterday, it was uncertain whether the Justice Department would back the appeal, let alone file a ringing denunciation of the separate-subsidiary concept that is the heart of both the Computer II decision and legislation recently passed by the Senate.

Under both the FCC decision and the Senate bill, AT&T would be free to compete in new fields such as home information and date processing through a separate subsidiary. AT&T has been barred from entering all unregulated businesses since signing an agreement 25 years ago ending a government antitrust suit against it.

The Justice Department said that the FCC has acknowledged its historic inability to develop accounting systems to monitor the financial records of AT&T and the procurement policies of the Bell System companies.

"In view of this heavy reliance on accounting, the commission's decision is remarkable for its blind faith that the necessary accounting tools, heretofore nonexistent, will be developed in time to ensure the success of its Computer II scheme," the Justice Department said.