In the last two weeks, Billy Olson twice has broken the indoor pole vaulting record. First, at Toronto, he vaulted 18 feet 8 3/4 inches. He eclipsed that mark last weekend at the Portland Federal Mason-Dixon Games in Louisville, where he went 18-9 1/4.

Don't tell anybody, but Billy Olson has been holding a little something back.

"I haven't had the perfect jump yet," he said yesterday from his home in Abilene, Tex. "I've been going good, to say the least, but I don't think I've gotten everything I can out of the pole yet."

Olson, 23, says he's through fiddling around. He's ready to go for 19 feet. "My parents watched the Toronto jump on TV and they said I was six inches over the bar," said the Abilene Christian University student. "And in Louisville I know I was two, three or four inches over."

The event Olson has been saving up for has arrived. The Millrose Games are Friday at Madison Square Garden in New York, "which I've been planning for all year as the place I'm going to cut loose."

He's bringing one incentive with him. His longtime girlfriend will be along when he heads for New York today, which is unusually early for him to get into town before a meet. He wants plenty of time to practice and plenty of inspiration.

Usually in a meet, all but one or two vaulters are eliminated when the bar gets to 18 feet. Then those remaining get to decide what height to go for.

In the past, when Olson has had a meet wrapped up he's been content to go for a half-inch over the existing mark. But at the Millrose Games, he said, "If we get to 18-4 or so and I have it sewed up, I'm going straight to 19 feet. And I'm going to make it.

"I feel like I have that 19-foot jump in me. I don't want to sound cocky, but if you need a story about a 19-foot pole vault, you'd better be there. It's a great place to jump and I've been psyched about it forever."

Success has been sudden for Olson, who teaches a Bible class in Abilene and who, with his shaggy blond hair, bears a resemblance to squeaky-clean singer John Denver.

Two years ago he was one of the top U.S. vaulters, but nagging minor injuries kept him from soaring to record heights. Then, in a horrible training accident in September 1980, he broke every bone in his left wrist and dislocated his left elbow. It was such a mess doctors told him he'd be lucky to regain use of the wrist at all. He was in a cast for six months. The doctors said he'd never vault again.

"I saw the X-rays," said Olson. "It looked like scrambled eggs."

He had fallen during a rope-vaulting exercise, using a rope hanging from the gym ceiling to soar over a bar. Olson landed on the gym floor from 18 feet up.

But even before the cast came off he was back at work. His coach, Don Hood, told him use of the left hand wasn't critical. He found his worst problem was in carrying the pole on the run, when almost all the weight is borne by the left wrist. In the actual vault most of the stress was on his right side, so he modified his run and carry and this winter things all came together.

In January he set a record of 18-6 1/2 at the Meadowlands in New Jersey. Then came the Toronto and Louisville jumps. Oddly, his emergence is coming where he least expected it: indoors.

He prefers outdoors, where he can use a longer and generally faster runway. His personal indoor record before this year was 17-10, but he's already jumped 18-7 1/2 outdoors, where the world record is three-fourths of an inch over 19 feet.

"That's what's got me so excited," Olson said in his drawl. "I've increased my p.r. (personal record) by almost a foot indoors. If I'd been outdoors, with good conditions, I think I'd already be at 19 feet and over."

The fact that he's from Abilene, where if you shake a tree a pole vaulter is likely to fall out, didn't hurt Olson. He started vaulting seriously in the summer between 10th and 11th grades, when he joined a friend who was working out. "I hadn't even tried it in two years," he said, "but it had gotten easy for me. I jumped 12 feet that day, as high as he did."

That fall his high school track coach asked him to try out and by the end of the season Olson was vaulting 15 feet. The next year he set a state high school record of 15-10.

Then it was on to Abilene Christian, where Hood has created something of a pole vaulting mecca. "In the last five or six years he's had six vaulters over 17-6 and three over 18 feet. There's no other college coach in the country, maybe even the world, who can say that," said Olson.

Hood said in Olson he saw a "skinny kid with tremendous competitive spirit. He was just so skinny, but he blew down the runway as fast as he could. I figured I could work with him."

It wasn't long, Hood said, before he realized that Olson could be "the greatest pole vaulter America's ever had. He can't stand to lose, even in tiddlywinks."

Olson needs nine credits to graduate from Abilene Christian, which he's working on. He lives at home and when you call there and ask for Billy the voice on the other end wants to know, "Big Billy or little Billy?"

Little Billy is the world famous pole vaulter, 6 feet 2 and 170 pounds. Big Billy is his father, 6-2 1/2 and 225.

"He's a health nut," said little Billy. "Solid muscle and he's 50 years old. I mean, he's huge . . ."