The Justice Department is investigating allegations of improper conduct involving a senior federal housing official who has earned at least $80,000 speaking and writing for a group that sells books and tapes on how to make money in real estate.

Gordon D. Walker, deputy undersecretary of the Housing and Urban Development Department, gave some of the lectures during official HUD trips that were paid for by the government, according to sources familiar with the investigation. They said investigators have questioned whether Walker has billed HUD for a number of trips on which he did little or no official business and whether the trips were scheduled to accommodate his private appearances at paid seminars.

From 1982 to 1984, the National Institute of Financial Planning paid Walker -- who joined HUD in 1981 -- $80,000 to $200,000 in "royalties," according to Walker's financial disclosure statements. These reports, which list income within broad ranges, have not yet been filed for 1985.

Walker has produced four audio tapes and a book for the firm, which sells them for up to $59.95 apiece. They are promoted along with other cassettes featuring "the biggest names in real-estate investing." The book by Walker, a former Salt Lake City broker and developer, is based on his experience as a land developer. It is called "Develop Your Way to Success."

Walker, who earns $72,300 overseeing HUD's 81 local offices, did not respond to requests for comment.

The allegations about Walker were referred to the Justice Department's Public Integrity Section after an investigation by Paul A. Adams, HUD's inspector general. Adams said he is sending a report on the matter to HUD Secretary Samuel R. Pierce Jr. but declined further comment. A HUD spokesman said Pierce could not comment because he had requested the probe.

Walker, according to the income ranges listed on his financial statements, received $15,000 to $50,000 a year from the institute in 1982 and 1983, but his compensation increased to $50,000 to $100,000 in 1984. Sources said he was paid about $75,000 that year and made more than 30 speeches around the country promoting his books and tapes. He listed all the income as "royalties."

In a 1983 memo, HUD Assistant General Counsel David White told Walker the outside work was "permissible," as long as he was not paid for writings or speeches that are "devoted substantially" to HUD programs.

Walker said in a letter to White that all of his four books and 12 audio cassettes on real estate investing "were produced prior to joining HUD or on my own time." He said he had been asked to promote the books "at a series of book-selling and investor conventions three or four times a year," and that none of the sponsoring firms "does business directly or indirectly at HUD."

At these conventions, Walker said, he does not talk about "any policies of HUD that are not already found in the public press." He said the sponsoring firms pay his travel and lodging expenses, but that he receives no speaking fees, only royalties on the book and tape sales.

Bruce Candrian, seminar director for the institute, said the firm pays speakers for each appearance at its weekly seminars and four conventions a year, which are held across the country and abroad. He said the company recorded Walker's tapes and printed his book several years ago, but that he did not think Walker has spoken at its "Courage to Be Rich" seminars for at least a year.

Walker did work for the institute before joining HUD in late 1981, when he became a special assistant to Pierce. The institute is based in Salt Lake City, where Walker's family development firm, of which he is still a partner, is located.

The institute sells magazines, books and cassettes of what it calls "the greatest accumulation of real estate lecturers in the country." One publication describes Walker as "deputy undersecretary for HUD with responsibility for all HUD offices nationwide."

The group uses direct mail and advertising to sell its products, which include "motivation" tapes by former UCLA basketball coach John Wooden and others. Some of its speakers buy time on late-night cable television shows, where viewers are urged to call a toll-free number for information on such topics as buying foreclosure properties with no money down.

One of the institute's ads is for a $49.95 package of tapes, "Six National Real Estate Experts Show YOU How to Create Wealth." One of these experts is former Baltimore developer Wayne Phillips, author of a book called "How to Get Government Loans," in which Phillips talks of keeping $30,000 of a $120,000 HUD rehabilitation loan that he obtained in Baltimore.

Phillips' Baltimore firm has declared bankruptcy with nearly $200,000 in debts, and both HUD and city officials have barred him from participation in housing programs. Institute officials said they are no longer associated with Phillips, who has his own seminar firm.

HUD officials said Phillips and other institute speakers often put out misleading information about government programs. HUD spokesman Robert Nipp said this forces the agency to field numerous inquiries about low-interest loans and other nonexistent programs.

"They get people all excited, and they don't have the information straight," Nipp said. "Each time they hit an area with a big seminar, we get a flurry of calls."

But Candrian defended the institute's work. "We're a little different from some of these companies that are here today, gone tomorrow, and their material is shoddy," he said . . . . It's simple to make money in real estate, but it's not easy. It takes work, it takes time and you put in your homework. It's not a 'get rich quick.' "