A rumor circulating at the annual convention of the National Association of Basketball Coaches has North Carolina Coach Dean Smith returning to his alma mater, the University of Kansas, and Larry Brown, coach of the NBA's New Jersey Nets and a UNC alumnus, replacing him in Chapel Hill.

Few give the rumor validity and Smith, who coached the West team to victory in the NABC All-America all-star game Sunday, checked out of his hotel this morning and was unavailable for comment. But the rumor does point up that the market for coaches is better in college than in the pros.

Ed Martin, Tennessee State coach, said, "In the pros, the players are getting all the money. There are only two kinds of pro coaches--those who have been fired and those who are waiting to get fired."

The good coaches who have taken new jobs, such as Lute Olson at Arizona and George Raveling at Iowa, are getting annual income packages in the $200,000 range.

Raveling has told a Spokane, Wash., newspaper he will accept the Iowa job.

One of the hottest topics at the NABC convention, which ended today, was the number of teams in the Division I tournament. Joe B. Hall of Kentucky is among the many coaches who dislike the 52-team format and first-round byes to 16 teams. But he doesn't want to see a 64-team tournament, as the NABC board of directors has favored.

"The tendency is to expand," Hall said. "But I'd like to see it at 32, or open to everybody."

The final four all-tournament team: Houston's Akeem Abdul Olajuwon (MVP), North Carolina State's Thurl Bailey, Dereck Whittenburg and Sidney Lowe, and Louisville guard Milt Wagner.

Olajuwon, who had 82 points, 59 rebounds and 28 blocked shots for the tournament, received 118 votes for MVP. Whittenburg had 25.

Despite reports that Gary Colson of the University of New Mexico would move to Long Beach State, the California school is continuing interviews . . . No one surfaced here as the likely successor to Ted Owens at Kansas, but West Coast sources say a longshot is Ralph Miller, the successful, longtime coach at Oregon State who is a Kansas alumnus.

Among the spectators here for the final four were David Thompson, Phil Spence and Monte Towe, of North Carolina State's undefeated NCAA champions in 1974 . . . The success of the Houston Cougars has renewed talk of raising the basket from 10 feet to 12 feet. A proponent of this change is Ed Steitz, the editor of the NCAA rules committee, who says the reason Dr. James Naismith placed the basket at 10 feet was it was the only height in the gym at which the basket could be hung.

Tom Jernstedt, NCAA assistant executive director in charge of the basketball tournament, says that under its three-year, $46 million contract with CBS, the NCAA is obligated to bargain exclusively with that network for a 30-day period when negotiations for a new contract begin this fall. The current contract expires after the 1984 tournament.

Those comments came in response to a report in The New York Times that NBC is trying to persuade the NCAA to televise the tournament on two networks.