A former stock and commodity market operator who once acknowledged violating several federal securities laws - a political aeophyte with no experience in public affairs - has come within striking distance of a seat in the United States Senate here in Montana.

The candidate is Larry William, a 36-year-old conservative Republican who once called himself "a con artist."

A year ago his name meant nothing whatsoever to Montana voters, but according to a recent poll taken for Williams' Democratic opponent, Rep. Max Baucus, the name is well known, and the candidate enjoys virtually the same voter support as Baucus, a popular two-term congressman. (Ten percent of Montana's voters remain undecided, according to the Baucus poll).

Williams has made this remarkable progress with a campaign emphasizing sharp criticism of Baucus (as unusual departure from the gentlemanly traditions of Montana politics) and bold promises to try to cut federal taxes by one-third, decimate foreign aid and double the toll for passage through the Panama Canal, among others.

A few years ago Williams was making promises of another kind as an author, lecturer and adviser to would-be stock and commodity market investors.

In one of his venture, he persuaded hundreds to pay him $1,500 for a weekend of instruction in a system for investing in commodities futures that Williams promised would being 100 percent profit in a year. That promise and other activities landed Williams in trouble with the Securities and Exchange Commission and the Commodity Futures Trading Commission.

In 1972 Williams signed a consentor with the SEC in which he admitted violating numerous laws and regulations in an earier stock market promotion and agreed to a suspension of his registration as an investment adviser. (He was later permitted to re-register).

Early this year the Commodity Futures Trading Commission decided that Williams could register as a commodity trading adviser, although the commission's enforcement division had recommended that his application be denied. The commission found that earlier Williams promotions - though "characterized by deception, misrepresentations of material fact . . . (and) gross puffery" - were not illegal at the time Williams undertook them. They would be illegal today, the commission said.

When 'The Wall Street Journal investigated Williams' operations in 1976, a reporter for the paper asked him - then a modish resident of Southern California who wore beads, open shirts and long hair - if he was a con artist.

"Yes," Williams replied. "I am a con artist. Aren't we all?" (Williams now says he made the remark "in jest.")

The Journal published a long front page article detailing Williams' activities and legal problems and quoting persons who studied his commodity trading system but could not make it work.

Williams' past has played a remarkably small role in the Senate campaign here. Only in the closing days of the campaign have Baucus supporters - clearly nervous at the closeness of the race - begun to circulate accounts of Williams' earlier career.

The state AFL-CIO has mailed to members 50,000 copies of an article on Williams that appeared in Forbes magazine in 1975, including a picture of the candidate dressed and coiffed in the modish style he then favored. The Forbes article makes no mention of Williams' encounters with federal regulatory agencies, but describes some of his exploits in selling advice to investors.

In the last few days the Forbes picture has appeared in most Montana newspapers. It makes a startling contrast with the appearance of Williams the candidate, who has cut his hair and given up the love-beads for dark suits.

But some Baucus supporters are now nervous that this distribution of the Forbes material could backfire and create sympathy for Williams. Baucus has cultivated a "nice guy" image in the tradition of Montana's political heroes, Mike Mansfield and the late Sen. Lee Metcalf (whose seat Baucus and Williams are contesting), and a personal attack on Williams might tarnish him.

Friday's Montana Standard, a morning paper published in heavily Democratic Butte, carried a banner headline: "Baucus-Williams: No More Mr. Nice Guy." Under it were "then and now" photos of Williams.

Many Baucus supporters and campaign workers express frustration that Williams can attack their man with apparent immunity, whereas, any direct response may be regarded as inappropriate by many conservative Montanans.

Williams, a smooth, articulate campaigner, has responded to publication of the picture by saying, "Everybody takes a bad picture sometime - look at your driver's license." He has said that distribution of the picture was a smear that showed "the level to which the Baucus campaign has sunk."

Montana has weak daily newspapers, and one of them has conducted a serious inquiry into Williams' past. A visiting reporter watched a local television newsman interview the Republican candidate here the other day, than asked the newsman how he felt about the campaign.

"Personally, I sure wouln't want that guy to represent me, and I'm a Republican," the newsman said of Williams. Yet his interview had been low-key and friendly, without any probing questions.

That interview took place outside a hearing room where three arbitrators are listening to charges of unfair campaign practices that Baucus and Williams have brought against each other.

Williams' lawyer got Baucus' chief congressional aide to admit that a purportedly complete account of the congressman's voting record was both incomplete and erroneous in some details. Baucus's aide said these were inadvertent mistakes.

Baucus charges that Willliams has made significant factual errors in newspaper ads about out-of-state contributions to the Baucus campaign, and that Williams has no basis for his claim that he is "the only Senate candidate with a professional background in agriculture."

In an interview, Williams said this assertion was based on his career as a commodity trader and adviser. "I know the markets" for agricultural commodities, he said.

Baucus' family owns and operates a 100,000-acre sheep ranch north of Helena.

The arbitrators have promised to adjudicate these claims before election day.

Baucus aides and allies acknowledge privately that the congressman's own campaign - often vague or dull - has helped Williams transform what was supposed to be a runaway into a tight race.

In an interview Baucus said almost wistfully that Williams "has captured a lot more attention that I thought would have bene the case ven two or three weeks ago."

Several prominent Montana politicans from both parties say they can't really imagine Baucus losing. "Max is young (36) and he could come on something like Mansfield did" as a senator, one prominent Republican said, suggesting that Baucus could build up seniority. "I'd be damned surprised if he lost."

But the polling data - provided by Peter D. Hart Research Associates of Washington - indicated that Baucus could lose. He seems particularly vulnerable if turnout is light or if Williams can manage to attract yet more attention in the closing days.

The race has been expensive by Montana standards. In a contest for no more than 300,000 votes, Baucus will spend nearly $600,000, Williams nearly $400,000.