WHEN JULIA Walsh was 34 years old, her husband died. He left her with four boys, aged 11 months to 8 years, and little money. It was one of the few times in her life she was really scared. She took what money she had--$20,000--and invested it all in four promising stocks, Texas Instruments, Hewlett-Packard, Aerojet General and Reynolds Metals.

"It was the only time I was really reckless," she says now. "I had such a little amount of money that I thought if I lost it, it wasn't that much, but if I had a couple of things do well, I might have something really nice."

In 18 months, her investment quadrupled. She bought a $65,000 house in northwest Washington with cash.

Today, at 59, she estimates she is worth nearly $3 million. She still has the house--though she's added some things like a swimming pool--and she still has that shrewd sense of the stock market. One of her business school teachers, George M. Ferris Jr., saw her potential talent for the market when he tapped her for his securities firm, Ferris & Company Inc., in the late '50s, when women stockbrokers were considered little more than amusing.

But she studied the market, learned it and flourished in a world that tends to embrace the conservative, well-connected man and shun everyone else. She stayed at Ferris 22 years, progressing from registered representative to vice chairman. In 1977, she started her own investment firm, Julia M. Walsh & Sons, Inc. Three sons and one stepson are from the family of 12 children she shares with her banker husband, Thomas Walsh, (four are from her first marriage, seven are from his and one is from theirs).

Her successes have brought her wealth and access to goverment and business leaders, and carved for her a place of some influence in the Washington business community. Her colleagues speak highly of her market savvy, charm and wit. "She's very charismatic," says George Ferris Jr.

But she is still at best a fringe member of that club, that network of brokers and investors, grounded in school, family and fraternal ties. "The only frustration is that you never feel part of the male system," Walsh muses without bitterness, leaning back in a chair in her office. "In my age group, at least, you could never be part of that network."

She is tall, sturdy, with striking white hair, strong features, searing blue eyes. She resembles Barbara Bush, the wife of the vice president.

She doesn't belong to the old-line clubs like the Metropolitan, a gathering spot for many businessmen. She doesn't belong to golf clubs; she doesn't play golf. "There's a lot of business that goes on on the golf course," she says. "Also, there's the fact that I didn't go to the 'right' schools." (She went to Kent State University.) "Other people had gone to school with bankers and people like that. I couldn't call up a bank and say, 'Let me talk to my buddy, so-and-so.' "

But even without making it in the old-boy network, she has made it in her business.

Nothing tells so vividly of her success like the color photograph hanging on a wall outside her office. It is a picture of the board members of the American Stock Exchange--business-suited, somber-faced men assembled in rows of chairs in a dark wood paneled room. In a back row, dressed in gray, is Julia Walsh assuming the sober pose of the day. "I should have worn yellow or blue so I'd stick out more," she says.

In the fall of 1972, she became one of the few women to ever sit on the board of the American Stock Exchange. She came home that evening to find her house full of flowers.

"I thought I'd died," she says with a low, raspy chuckle. "There were so many flowers in the house it looked like a coffin should be in the middle. I was so excited. Everyone I knew on Wall Street had sent me flowers."

When she left Ferris in 1977, she was earning the most commissions of any of its 50 executives. But she wanted to work with her own children and bring them up through the business, so she started her own investment firm. She has 35 employes, about 4,000 accounts and a seat on the New York Stock Exchange.

She is, as far as she knows, the only woman in Washington to run her own investment firm and possibly the only woman on the East Coast doing so. She sits on the boards of Pitney Bowes, Esmark Inc., the Shrine of the Immaculate Conception and the East-West Center in Honolulu, among others. She is a regular panelist on the "Wall Street Week" television program. She sits on the board that oversaw the building of the D.C. Convention Center. She also served on Mayor Marion Barry's housing task force during this past transition period. She worked for the Democratic National Finance Committee and was considered by President Carter for secretary of commerce. Not getting the post was the impetus she needed to start her own firm.

She always knew she wanted to have a career.

"I didn't want to spend the rest of my life in Akron," she says of her Ohio home town. "It wouldn't offer the excitement I thought I wanted."

Her traditional Irish-Catholic parents were not pleased. But while her mother was resigned to it, her father, a tire builder, resented it.

She didn't want to teach, and she wasn't artistic. She majored in international business at Kent State. Two years through, she got an opportunity to go to Smith College. "I wanted more than anything else to go to Smith," she says. "Someone I met at an educational conference arranged for me to get room and board to go to Smith. But I didn't have transportation, and I would have had to go both summers to make up liberal arts credits. My father thought it was a bunch of nonsense anyway and wouldn't lend me the money. I just accepted it. We weren't sassy daughters yet," she chuckles.

Later, she was the resentful one. "When I was back in the job market in the '50s, I began to resent my father. I thought, 'Maybe, if I'd gone to Smith, I would have been better connected.' But I think that was just rationalization."

After school, she joined the Foreign Service and went to Germany as a personnel officer. Three years later, she married John Montgomery, an Army lieutenant colonel, and was summarily kicked out of the Foreign Service. "If you got married, the Foreign Service threw you out," she says. "You didn't even think of questioning it."

In the next 10 years of their marriage, they moved constantly, and she had a variety of jobs. In the fall of 1954, Montgomery was assigned to Washington. He encouraged his pregnant wife to go to George Washington business school at night. So did her live-in mother-in-law, who had shocked her own family when she left her Minnesota home to attend college at the University of California at Berkeley. Her father ordered her home one year later.

Walsh zipped through business school. "I ate that stuff up," she says of the financial statements and annual reports they plumbed. "It was like reading a novel." Her mother-in-law babysat. George Ferris Jr., one of her teachers, invited her to work with his firm. "Thirty years ago, it took some courage to hire a woman," she recalls. "In order to reinforce his decision--he knew he'd get resistance--he took a scientific approach." He gave her some type of sales skills test, and she scored well on it.

When Walsh joined Ferris, she was the only woman broker in the office. People were nice, she remembers, but "they took the attitude that I wasn't serious," she says. "It was a lark. I was a flash in the pan.

"The only thing it did which has hung over me my whole life was that I performed less aggressively," she says. "I accommodated the attitude that I wasn't very serious. I tried not to step on anyone's toes."

She looked for sedate clients making safe investments when she wanted aggressive ones whose money she could invest more adventurously. "I didn't do things that weren't ladylike," she says. It was the late '60s, early '70s before she began to follow the riskier but more exciting high-technology stocks that she had believed in ever since they paid for her house.

"If I'd known what I know today, I would have been more aggressive," she says.

In 1957, when Walsh had been in the business only a few years, Ferris stopped brokering and went into full-time management. "I turned over some of my clients to her," he says. "She relates well with people."

"To listen to her close a deal--she's an artist," says her 25-year-old son, Mark Montgomery, an account executive at Julia Walsh & Sons. "It's a cliche' to say she could sell Eskimos snow, but we think she can."

She had quit her job when her husband John was assigned to Kansas. Before she could move there, he was killed on maneuvers. When Ferris & Co. found out, they immediately offered her the job back. "By that time, I'd shown promise," she says.

But she still had problems. Ferris sent her to a week-long educational program after she returned to work. "I'd go to class, and both seats on either side of me would be empty," she says. "I went to a colleague and said, 'I don't think I can take this.' " Her colleague, John Schoenfeld, now a Washington consultant, told her to stay at least another day. He got the men to sit near her and got her into circulation with the other brokers.

In May 1962, she went to a 13-week advanced management program at Harvard Business School. She thought it would enhance her business. She made friends, but it didn't help her business. It was also thick with chauvinism.

"It was a painful experience," she recalls. "It was terrible for some of the men. They very much resented the fact that a woman was there, and they made no bones about it. They thought you were there just to prove a point."

One classmate would sometimes chastise her when she made a remark in class. "He'd say, 'Oh, for God's sake, isn't that just like a woman?' "

The day of her graduation, this classmate approached her at a reception. One of her sons, who was 13, was with her. As he was drinking a glass of punch, the classmate bumped her son's arm and the boy spilled punch down the front of his shirt. The classmate took the boy aside to clean him up. Later that evening, the classmate found Julia Walsh and apologized. "He said he was very embarrassed," Walsh recalls. "He said he thought children of women who were away at work so much would be very emotionally insecure. He said he found my boy very well-adjusted. He said he had bumped him on purpose so he would have a chance to talk to the boy."

Walsh's eyes widen in amazement as she tells the story. "I just couldn't believe it."

Back in Washington, Walsh worked hard to get business. Four nights a week, she taught seminars, she taught at Catholic University, taught at the Y. She spoke to Navy Officers' wives clubs and to Foreign Service officers' wives clubs. Her name popped up in genteel newspaper stories about womens' gatherings in the late '50s and early '60s as often as names of socially prominent hostesses.

"She was out front, lecturing, promoting herself," says Les Silverstone, the manager of a Dean Witter branch on K Street, who has known Walsh for 20-some years. "She was aggressive--just in the way she developed her business. She let people know that she was doing well--in the way she carried herself, the way she dressed."

She wore hats a lot when she was speaking. "I did it to be a little set apart," she says.

"She was well-known. She was one of very few women who'd made it to the upper echelons of the industry," says Delia Emmons, corporate secretary of the American Stock Exchange, about Walsh in her governing board days. "She was vice chairman of a major regional brokerage firm in the early '70s. I can't recall another woman--although there must have been some--like that."

"She's very different from a normal Wall Street stuffy type," says Bonnie Knox, who worked for the American Stock Exchange when Walsh was on the board. "It was very refreshing."

When Julia Walsh & Sons went into business, one of the first issues to settle was--what do the sons call her? 'Mom' was ruled out. Instead, they address her as 'Julia' or 'Mrs. Walsh.' But she often calls each of them, 'Dear.' "We hear 'Dear,' and we don't know who should answer," says Mark Montgomery chuckling.

"I say to John, who's head of research, 'Honey, what do you think of this?' " she says. "He says, 'Now, wait a minute, it's Mr. Honey.' "

Her desk is a mess of papers, folders and sometimes a hair brush. She tends to be late for appointments, unlike her husband Thomas, the vice chairman of the board of National Permanent Federal Savings and Loan. "He's always 10 minutes early for everything," says Mark Montgomery. "She's always a half hour late."

Her office in an old building on 17th Street is carpeted, quiet, conservative-looking. There are no trappings of high-tech, with the exception of a small video display terminal behind her desk on which she can read market standings. On one wall is a watercolor of the White House signed by President Carter. On another wall are two paintings: one depicting Wall Street in 1834 and the other showing Wall Street in 1856. Her sons wander in and out to confer on business or coordinate rides home. (All the children have moved out of the house, though.)

Eighty percent of her business is management of portfolios, she says, not simply phoning in stock orders. Her clients are professionals by and large--doctors, lawyers, business people. They're investing their own money, not conserving inherited wealth. Walsh also handles smaller institutions like charities, schools, several women's groups, the League of Women Voters.

She is still big on high-technology companies and local companies. She follows her own advice: She has a third of her worth invested in a Texas high-technology company not publicly listed yet.

"She talks a lot; she talks very fast," says one woman who is a client. "She's sort of down-to-earth, a 'one-of-the-guys' kind of woman." She went to the firm on the recommendation of a Walsh stepdaughter. "She'd always talked about her mother with great pride, and at the time, we needed someone." When she told a wealthy, conservative uncle, who had been managing her investments, that she was going to Walsh, he balked. "He said, 'Some cockamamie woman. What do you mean you're going to a woman stockbroker?' "

When Julia Montgomery bought her house in Northwest, the seller threw a party for her. One of the guests was Thomas Walsh. A couple of years later, his wife died after a long illness. A few years after that, in 1963, Walsh and Julia Montgomery got married. He and his children moved into her house and the combined group consisted of two 6-year-olds, two 8-year-olds, two 11-year-olds, two 13-year-olds, a 15-year-old, a 16-year-old, a 17-year-old and two maids. "I brought a maid with me; he brought a maid with him," she says. "One would stay until 2 p.m. The other would come in at 1 p.m., overlap an hour and stay through dinner time." They had another baby girl together.

"Our Catholic faith helped us," says Walsh, who attends mass regularly with her husband, and who tithes. "We both went into it thinking 'we're going to make it work.' "

The children were raised knowing about the stock market. They heard about it at the dinner table and some took business courses in school. "I can remember the first Christmas all together, we got stock in something we were interested in--like Disney World," says her 35-year stepson, Thomas Walsh, the president of her company. "If you wanted to buy a bicycle, you had to sell your stock to do it." When Thomas Walsh was 17, he got a part-time job in the stock market--as a runner at Ferris--through his stepmother.

"You could tell when the market was bad," Mark Montgomery says. "She was in a bad mood . . . She'd get in a blue funk."

Even when the market was tough Walsh was able to live comfortably. There were several cars at any given time, and at one point, the Walshes owned a house in Florida and a farm with horses in Virginia. But during the grim years of the early '70s, they cut back. "We really knew things were bad when the farm was gone," chuckles Mark Montgomery.

"My mother is very generous and likes to indulge everyone around her," he says. She's not only generous to her family but also to a variety of organizations. "Her mom is just constantly appalled by her spending habits," he says with a laugh. "She thinks we're all spoiled . . . She's proud of Mom and happy for her, but she just can't see how much she's made."

"I am a terrible spendthrift," says Julia Walsh with a little smile. "But, you know, you can't take it with you."