A SMALL CROWD of spectators ringed the corral, shouting out encouragement as I fought a battle of wills with Gomer, the 12-year-old bay I was riding, on the last day of a Montana dude ranch vacation.

It was Beartooth Ranch's Saturday "Fun Rodeo," a sort of graduation ceremony after a week of horsemanship lessons and trial rides. Sandy, my fiancee, and I were among the 20 competitors, and I was in trouble.

"Hang in there, Jim," I heard someone cry, as a stubborn Gomer shied and danced in tight circles, refusing to go in the direction the rodeo event required. I tightened my hold on the reins and nudged him in the haunches with my boot heels, frightened but determined.

The event, the flag race, isn't one you are likely to find on the rodeo circuit, but for those of us who had almost forgotten a few childhood hours on horseback, it was a challenge. We were to gallop our horses to the end of the arena, reach over and pick up a flag sitting in a barrel, gallop a short distance farther and place the flag in a second barrel and then return to the starting point, racing the clock.

I had managed to snatch up the flag with my right hand, but just at that moment a stiff breeze whipped it alongside Gomer's head, obviously upsetting him. He shot right on by the second barrel and was headed for the starting gate.

By now any chance of placing in the contest was lost but I was set on proving to myself (and my fellow riders) that I had learned something about handling a horse. Holding the flag behind my back to get it out of sight, and thwarting Gomer's every move to bolt with a tight rein, I forced him to turn back to the second barrel, and with a great deal of relief for both of us, dropped in the flag.

That night, when ribbons were handed out in the lodge, Sandy and I learned we had at least placed in a couple of other dude rodeo events, the Kool-Aid and sack races. And Sandy, to her great delight, was chosen by the wranglers as "Rodeo Queen of the Week," probably because of the rapport she had developed with her horse, Hamburger. Not bad for a couple of city folks.

Why did we choose a dude ranch for our vacation? Not because we had any great love for horses and riding. It was simply because it was Out West and out-of-doors and a new experience. The Montana Rockies we chose at random, and Beartooth we picked for its scenery, isolation and informality as described in a guide-book. "Country Vacations U.S.A."

The 160-acre ranch lies in a small valley at more than 5,000 feet near Nye in southwestern Montana, just north of Yellowstone National Park. Running through the ranch is the Stillwater River, which despite its name was a torrent in late June. Peaks, some reaching up to 11,500 feet and still snowcapped that week, surround the ranch.

It is only a dude ranch, rising no cattle or crops, and open only from late spring to fall for up to 40 guests at a time. It adjoins Custer National Forest and the Beartooth Primitive area.

We flew into Billings, Mont., on a Sunday afternoon via Western Airlines from Denver. A ranch stationwagon met us at the airport for the 90-mile drive into the mountains to the ranch.

Our three-room log cabin, set alone in an aspen glade and appropriately named "Trial's End," was spartan but had all the modern facilities. Its redeeming feature was a covered front porch that faced directly on the 1,000-foot Woodbine Falls, cascading down the mountain into the Stillwater.

We spent most of our evenings on the porch, reading watchingthe stars, listening to the river, breathing in the fresh air, sometimes spotting a deer or the twin ponies that had full run of the ranch, Lydia and Pinkham.

Our days were programmed but still informal. Breakfast in the cookhouse from 7 to 8:30 a.m.; saddle up for the morning trail ride at 9 a.m.; lunch at noon and a rest: an afternoon trail ride at 2 p.m.; back in time for a 5:30 p.m. BYOB "happy hour," and dinner at 6:30 p.m.

Our first contact with the horses was on Monday morning, when we were fitted to saddle and stirrups. Stand up in the stirrups, we were told, and make sure there are three inches between the crotch and the saddle. That would keep the saddle from slapping our rear ends when the horse went into a trot.

About 18 of us who were riding that week - from youngsters to a hesitant couple from Rochester, N.Y., in their mid-50s who were more inclined to sample the ranch's fishing - had hoped for docile horses. We were told not to worry, the horses were well-trained, but those huge creatures were not reassuring - and it looks a long way down when you are in the saddle. Sandy and I were wearing riding boots, and hats to keep off the hot sun. We looked the part but didn't feel it.

Our group was led single file to the corral, where we got our first lessons in how to stop and turn our horses. Around the arena we went practicing, building some confidence but still hesitant and probably puzzling our mounts with our inexperience.

Then it was out to the valley and mountain trails an experienced wrangler to lead and one to follow. A couple of miles the first day and sore muscles, steeper trails the next and a 12-mile ride on Friday, some of it at full gallop, whooping and hollering in posse formation across the grasslands, knowing we should have one hand free but keeping it clutched to the saddlehorn to hold on anyway. And then Saturday's rodeo.

On Thursday we rode about four miles up the Stillwater River along a narrow, rocky path and through a long gorge to Sioux Charley Lake for a lunch cookout: hamburgers and fried potatoes over the bonfire for 22 trailriders.

It was one of the first days on the Beartooth for Knobby Brown, a hefty 25-year-old part-Indian from Montana's Blackfoot reservation, looking authentic with his black cowboy hat sporting a long eagle feather. A bareback rider in his youth, he most recently had been herding cows. But on that day's trail ride, he confessed:

"I sure do like trailing dudes: They're a lot easier to handle than cows . . . and you got somebody to talk to." He then let out a whoop that shook dudes and horses alike.

On night, we were all invited to a square dance in the lodge. A caller was on hand with taped music, and so were the wranglers and the cooks and the maids. It was one of the week's big social nights. For two hours we learned to do-si-do and swing our partners, moving in large circles and squares of eight.

On another night, the ranch managers and part-owners, John and Doris Mouat, gathered up a dozen of us for a drive to Nye, the closest "town" - a bar, post office and trading post in one. It was our only trip away from the ranch, and our chance to sample a Western bar with its clientele of dudes, miners, and honest-to-goodness cowpokes.

There were some problems on this vacation.One of the male guests, an air controller from Indiana, got kicked in the face and chest (no serious injury) when he bent in front of his horse to pick up his hat.

Sandy never made friends with the mouse that camped in our living room, ans she had some trouble getting to sleep initially when she heard what she thought were tiny footsteps scurrying across the roof.

I had hoped for some overnight trail rides, but was told it was too early in the season. The high country was still too cold. and the river was too dangerous in June this year for the white-water raft trips promised in the brochure.

A few guests found the cliff-edge portions of some of the trails frightening and skipped those rides. And, of course, there was all that horse manure.

The food was abundant, cooked well but the menu was not especially imaginative - roast turkey, lasagna, stuffed peppers for dinner. Bacon, eggs, pancakes for breakfast. Sandwithces, macaroni and cheese, hot dogs and beans for lunch. The bread and desserts were baked fresh daily. But never a Montana steak, a major disappointment. The meals were served family style, at nicely decorated tables, and the staff that wasn't working joined the guests for meals.

The cost was $189 per adult for seven nights lodging and three meals a day, plus $35 each for a horse for the week and $35 a family for roundtrip transportation from the Billings airport. Roundtrip excursion airfare from Washington to Billings is $252 each. At the ranch, there is almost nothing to spend money on, except soft drinks and setups for "happy hour."