In a stunning upset that few had predicted, Rep. Harold Washington (D-Ill.) apparently won the Democratic mayoral nomination in unofficial nearly complete returns today showing he held a 15,000-vote margin over Mayor Jane M. Byrne out of a record 1.17 million votes cast.

With 96 percent of the tally in, Washington had 391,000 to Byrne's 376,000. Most of the precincts still to be counted were in the heavily black South side, where Washington was getting up to 87 percent of the vote. As the count continued, confidence ebbed in the Byrne camp.

Al Ronan, director of field operations for the mayor, predicted a Washington victory by about 10,000 votes. But the feisty mayor simply told her campaign workers at a party at the Ambassador West Hotel that she was turning in for the night because "it's too close to call."

Washington did not claim victory, but Cook County State's Attorney Richard M. Daley conceded defeat late last night. Endorsed by the two major city dailies and backed by former vice president Walter F. Mondale, Daley ran third, getting about 31 percent of the vote. A simple plurality takes the nomination and there is no runoff. A Washington victory would be a stunning upset by a black candidate with an underfinanced campaign. It would usher in a new era in Chicago politics, dominated by whites for 150 years.

Washington's showing was built on enormous pluralities in the city's 13 black wards on the South and West sides, while Byrne's strength lay in the North and Northwest wards.

However, in some of the North side wards, such as the 40th, her plurality was only 200 votes instead of the expected 1,500, with much of the difference going to Washington.

The closeness of the results placed even more weight on federal officials who organized an unprecedented Election Day task force to guard against fraud at the polling place and to impound the ballots afterward for use in possible prosecutions.

More than 800 voting irregularities were reported by Chicago Board of Elections officials.

The Democratic primary winner will meet Republican candidate Bernard Epton in the April 12 general election.

Epton, a former state legislator, is rated a poor chance if Byrne is his opponent. But if Washington, the strongest black candidate ever to run citywide, is the Democratic candidate, then Epton's chances are expected to rise on the strength of an anti-black vote.

Byrne and Daley virtually split the white vote, according to the returns counted so far. Meanwhile, blacks, who four years ago turned out large pluralities for Byrne, this time went to Washington.

Washington, who was a state legislator before being elected as an insurgent candidate to Congress from Chicago's First Ward in 1980, stressed the importance of black unity during the closing days of his campaign.

His strong showing is evidence that the 650,000 black voters of what many describe as the most segregated city north of the Mason-Dixon line have coalesced into a voting bloc rivaling the white ethnic populations that have controlled this city's politics almost since Chicago was founded in a swamp beside Lake Michigan in 1837.

Washington's campaign was a shoestring operation from start to finish. He was outspent nearly 15 to 1 by the mayor and Daley, the 40-year-old eldest son of the late Boss Richard J. Daley.

Byrne, 48, used a bankroll of nearly $10 million to refurbish her image from that of a feisty, chaotic Attila the Hen to a "new Jane" of mature, balanced administrative skills.

Byrne bombarded what is called Chicagoland with a sophisticiated television advertising campaign engineered by media consultant David Sawyer. Her ratings, which last fall showed her a poor second behind Daley, soared. By year's end she was the front-runner, overtaking Daley who had planned a quick victory based on his father's name and dissatisfaction with the mayor.

Washington, 60, entered the race only after a voter registration drive last autumn added about 120,000 new black voters to the rolls. He had run for mayor once before, in 1977, and got only 11 percent of the vote. This time, he never received substantial contributions from anyone, and it was not until the final week of the campaign that he matched his opponents' TV campaigns.

He scored most heavily in four well-reported candidates' debates, exhibiting a smooth, commanding oratorical style that was in contrast to the halting Daley delivery, and the mayor's restrained sarcasm.

Washington, the son and grandson of Methodist ministers, once spent a month in jail on a tax evasion conviction and was once suspended from law practice by Illinois bar officials after clients complained that he had failed to represent them. But he dealt with these early in the campaign, saying that he had paid his debt.