To hear Mayor Frank Rizzo tell it, there are devils loose in our midst, like the legendary Four Horsemen of the Apocalypse. Their names are school busing, affirmative action, public housing and the press.

The mayor, a six-foot-plus slab of intimidating bulk, is warning his constituents that he is the only politician on the scene who can stand between them and the scourge.

"I am the difference," he proclaims as he stumps the predominantly white wards of the city in a heavily contested campaign for political survival. Rizzo wants to amend the city charter to permit him to run for a third term. Philadelphia voters will decide that issue Nov. 7.

Newspaper polls published over the past week show Rizzo's campaign for what a public relations adviser deftly labeled "charter reform" is sagging badly. Opposition to the third-term charter change is running 2 to 1 against Rizzo in a recently published Gallup Poll, a finding echoed in other Philadelphia polls. A direct test of the mayor's popularity by Gallup showed him in disfavor by more than 3 to 1.

Rizzo disparages the newspaper and television polls as hoked up by the "ultraliberal" opposition. He claims his tests of voter sentiment conducted by teams of telephone canvassers show him ahead. "I've never seen anything like the dumping of these phony polls," says Rizzo.

Despite the rising clamor of the final days of the campaign, Philadelphia seems calm and self-contented to the visitor. Joggers plod through Rittenhouse Square, now ablaze with fall's golden foliage. Neighborhoods in center city blossom with new paint and renovated house fronts. At night the downtown is a bright diadem of lights and people.

In the geographical center of the original town, at the junction of Board and Market streets, stands the trophy, City Hall, a 40-story Romanesque pile of columns, pediments and stone dryads. Frank Rizzo sits on the second floor and William Penn's statue stands on the roof.

More than a decade ago, George C. Wallace spoka to the fear and rage of a large segment of the white working class and became a major political influence in the America of the 1960s. Rizzo, though never a public supporter of Wallace, and lacking his national stature, provides a voice today for the same constituency.

He is a combination of Kojak and the Godfather, a dominating presence who speaks the language of the working-class "ethnics" who sing steel, pour concrete, load ships, run restaurants, walk the beats and answer the fire bells and who live in the modest but tidy rowhouse neighborhoods in South and Northeast Philadelphia.

The message he is carrying to the wards bears strongly on the fears of his constituency.Those fears center on race.

"Racism is his fastball," says Paul Tully, a former organizer for Eugene J. McCarthy, Robert F. Kennedy and George McGovern and now working with the Stop Rizzo Coalition, one of four oppointing organizations. "In the World Series when you're in the bottom of the ninth with score tied and two out, you go with the play that got you there."

Rizzo denies the racism charges but he is virtually ignoring the black wards of North and West Philadelphia. Instead, he is patrolling his native South Philly and the sprawling Northeast along Roosevel Boulevard, the upward-mobility corridor along which Philadelphians make their ascent into middle-class status in steady calibrations from rowhouse to detached colonial.

"While I am the mayor of Philadelphia nothing will go into your neighborhood that you don't want," he proclaimed to a crowd in the white working-class neighborhood of Mayfair the other night while dedicating a new high school swimming pool.

"And I can assure you, no matter who the aggressors are who try to do it, they got to run over me (swelling applause) and that's not an easy job (cheers and whistles."

The message carries straight to the gut in the Italian, Irish and Polish working-class just northeast and south of center city that see schools, neighborhoods and jobs threatened by blacks and the "ultraliberal" media.

The same message, with its stark racial overtones, comes through just as clearly to the blacks who comprise about a third of Philadelphia's 1.8 million Populace and who swarmed onto the voter registration rolls this year in unprecedented numbers toward the end of denying Rizzo another term in the Nov. 7 charter referendum.

Last week Coretta Scott King, widow of the Rev. Martin Luther King Jr., came to Philadelphia to accuse Rizzo of using racist rhetoric "to polarize a city along racial lines for his won selfish reasons."

Rizzo equates in one breath the ing press and television - conspiring NAACP, Urban League and Black Panthers as part of a cabal - includ-against him and his constituents.

"I don't buy the paper anymore. I borrow it," he told another audience last week at the Charisma Room, a 49th Ward disco and bingo establishment. "I suggest you do the same. They just scream about the First Amendment and then hide behind it to destroy people, to drop phony polls and to tell untruths." Applause swept

But if Rizzo, the ex-cop from Little Italy, presents himself as the champion of "the guys with the lunch boxes and the oil-stained shirt," his lifestyle had advanced considerably beyond them.

He works in a sumptuous office paneled in teak, carpeted in deep pile, its bathroom resplendant in Italian marble, a six-button phone panel beside the toilet. He lives in an imposing stone house in Chestnut Hill, one of Philadelphia's least pleabian neighborhoods. In a copyrighted series three years ago, the Philadelphia Daily News estimated that the purchase and refurbishment of the house cost "conservatively" $410,000, at a time when Rizzo's salary was $40,000 a year.

When he campaigns Rizzo travels with a retinue of high-ranking police officials and advance men in a squad of official limousines, all smoothly linked together by police radio. If logistical efficiency were the deciding factor in Philadelphia's referendum fight, Rizzo would win hands down.

But as it is, the mayor has succeeded in uniting against his third-term quest an improbably coalition of forces including Main Line lawyers and bankers, black organizations, the editorial pages of the Philadelphia Inquirer, the American Jewish Committee, the Americans for Democratic Action, the Women's International League for Peace and Freedom, the Gay Alliance, the Sholom Aleichem Club and the Revolutionary Student Brigade.

"Frank Rizzo is our best organizer," says Tully of the Stop Rizzo Coalition.

Eight months ago it seemed that Rizzo was through with Philadelphia politics when he announced in an emotional speech that he would not try to amend the charter to run for a third term. He chose as the site Whitman Park, a symbolic community battleground in a white working-class neighborhood of South Philadelphia where residents have fought for more than two decades to block a public housing project. Today Whitman Park is a vacant, littered field.

In the speech Rizzo attacked public housing and preferential quotas for minotirites. It was the beginning of his "ethnic rights" crusade, which quickly took shape as an appeal for whites to band together in what Rizzo called a campaign for "equality for whites."

Despite his Whiteman Park political disclaimer, a campaign was launced in Rizzo's behalf for a third-term charter amendment. By mid-June the mayor let it be known that he was changing his mind. "I tell you there is a force in this city that don't want me around, that don't want Frank Rizzo talking about the issues: busing, quotas, the death penalty," he said.

By Sept. 11 the 57-year-old mayor announced that he would campaign for a revision of the charter that would permit him to run for a third term. Within a week, after a visit by NAACP national director Benjamin Hooks calling for defeat of the charter change, Rizzo unleashed what is now perceived in Philadelphia as his "vote white" campaign.

Rizzo branded Hooks, who had been appointed to the Federal Communications Commission by President Nixon, a "social extremist" along with former Black Panther Bobby Seale.

At a German-American Labor Day rally Rizzo was introduced as the city's "great white hope." And in that early phase of the campaign he appealed openly to that emotion. "I'm going to say to the people of this city, 'Vote white,'" he told campaign audiences in the Northeast.

later, under the public relations tutelage of Sanford Weiner, the California political consultant who helped win the gambling casino referendum in Atlantic City, N.J., Rizzo moderated his language. Weiner helped design the Rizzo charter-revision posters, which dot the main roads and proclaim in soft blue, white and orange hues: "Yes! Charter Reform . . . Protect Your Right to Choose."

At the end of last month leaders of Philadelphia's Catholic, Protestant and Jewish institutions condemned - without mention of Rizzo's name - the injection of racially divisive rhetoric into the charter-revision campaign. One of the signers, William L. Johnston, chairman of the Metropolitan Christian Church, acknowledged that it applied to Rizzo. "There are no indications that others have used such polarizing statements," he said.

Rizzo immediately fired back: "Why don't we turn it around? Why don't they say something about the black radicals who come into town and shoot their mouth off? This is a double standard that I'm fighting."

The pivotal influences in next week's election will be the black voters who went on the registration books this year and the Jewish vote in the Northeast, which Rizzo has been courting assiduously. In his past two elections the mayor has received substantial support from the Jewish middle class concentrated west of Roosevelt Boulevard. But even some of the mayor's political tacticians acknowledge that the strong racial overtones of this campaign have turned off large numbers of Jewish voters.

At a campaign appearance in a Northeast B'nai B'rith lodge last week; rizzo recalled how he supported the investment of $1 million in city funds in Israel bonds several years ago. "There is no stronger supporter of Israel than Frank Rizzo," said Rizzo. "I admire people with courage, who are outnumbered 10,000 to 1. I know more about Israel than most people who are Jewish. I was the only mayor in the United States to support the city purchase of Israel bonds."

In Rizzo's past two elections black voting has been apathetic. This year a heavy turnout is predicted among the 28 predominantly black wards of Philadelphia, which contain about a third of the electorate of 935,000. Philadelphia Daily News columnist Chuck Stone, an influential figure in the black community, has warned that a heavy black turnout is crucial for defeating the charter change.

No one wise in the ways of politics in Frank Rizzo's Philadelphia is prepared to count the mayor out, despite the polls.

Rizzo commands an army of loyal city employes and political allies who serve as the backbone of his organization. He has shown no sign of hurting for money.

"I have IOUs out on the streets all over the city," he reminded a rally of 800 supporters at a $100-a-plate fundraiser in Little Italy last week. (For $100 each donor carried out a plate with a picture of Frank Rizzo on it.)

The City Hall workers, contractors trade unionist and local businessmen who were looking up admiringly at the mayor as they munched small squares of cold pizza knew eactly what he meant.