Peter Jay, a former British ambassador to Washington and media high-flier, was ousted today as head of TV-AM, the troubled company he established to bring American-style commercial morning television to Britain.

Like so much of the enterprise, today's action seemed to be on the American model, where bad ratings lead to bitter boardroom battles. Although TV-AM's leading bankrollers insisted they were behind Jay, they were plainly alarmed by the dismally small audience the program was attracting.

In a statement, Jay said he had resigned "for the sake of the cohesion of the company." His place as chief executive was taken by Jonathan Aitken, a former journalist, who is a Conservative member of parliament, a financier and the holder of 16.7 percent of TV-AM stock.

The program has had serious problems from the beginning. Despite big-name hosts such as David Frost and high production standards, the show drew only about one-fifth of the BBC audience. After a month of scathing reviews, Frost went "on vacation" and the program was revamped.

Still, a segment at 6 a.m. actually failed to record any audience in the ratings and the peak total in the last week surveyed was just 400,000 people, compared with a peak audience of 1.7 million for the BBC early-morning show. Even Jay had conceded those numbers were "unacceptable." Advertisers demanded that prices for commercials be slashed, and the company's revenue for its first year was said to be running at only half its projection.

Morale was plummeting. Even so, as word spread that Jay was about to be dumped, two of the TV-AM stars--Anna Ford and Angela Rippon--accused unnamed company officials of "treachery." Staff members were reported to have voted unanimously to support Jay, and a petition to that effect, with names of about half the production employes, was sent to the decisive board meeting today.

Jay, 46, who served in Washington from 1977 to 1979, is a former editor at the London Times and a successful television pundit. When the franchise for breakfast television was offered by Britain's Independent Broadcasting Authority, Jay mounted a vigorous campaign to get it, raising millions of pounds from British financial institutions and mortgaging his house to become an investor.

Ambitious plans for a high-quality program along the lines of NBC's "Today" show attracted enormous advance publicity. Frost talked about the "sexual chemistry" the stars would generate and Jay gave speeches on the programming philosophy aimed at "explaining" world affairs.

Meanwhile, the BBC, on a more modest scale, started its own early-morning program, which premiered two weeks before the TV-AM version. Its relaxed, almost rambling quality apparently appealed to viewers who also prefer early-morning television without frequent interruptions for commercials.

The inevitable comparisons between the two offerings heightened tensions at TV-AM, which had developed in the months before the shows began. Disputes over management emerged, with the stars, generally speaking, in one camp and the main investors in the other.

As long ago as November there were rumors that Jay's authority would be diluted by appointment of a managing director who would be the operating boss while Jay served as chairman. But Jay apparently forestalled that drive.

Aitken, 40, was joined by Jacob Rothschild of the celebrated banking family in pressing for Jay's ouster, although all are old friends. Terms for Jay's departure were not disclosed although Jay has been given the title of president, apparently a powerless position with no influence or responsibility.

This evening Aitken met with the TV-AM employes in an effort to win them over, but the future shape of the program and its stars is clearly in doubt.

As for the allegations of "treachery," Aitken said: "A lot of people said heated words in a very understandably tense situation. We now have a new situation and new staff and everyone should calm down and get on with the job."