Professional travel agents reacted angrily yesterday to a Civil Aeronautics Board decision paving the way for new competition in the business of selling airline tickets and vowed to challenge it in court and in Congress.

The CAB decision, made Thursday after a three-year proceeding, has the potential to make major changes in a 40-year-old system that now allows only the airlines and their accredited ticket agents to sell airline tickets.

On a 4-to-1 vote, the board decided that airlines should be allowed to appoint anyone they choose -- such as retail stores, banks and outlets such as Ticketron -- to sell tickets on their own routes, effective immediately. The nation's 20,000 accredited travel agents will continue for another two years to have the exclusive right to sell tickets covering connecting travel on more than one airline.

Officials of the American Society of Travel Agents, a trade association representing 10,000 U.S. agents, said yesterday they would use "every legal means" to overturn the board order. "If the CAB decision is allowed to stand, the ensuing chaos would be devastating to the airline industry, agents and, most importantly, to the traveling public," said ASTA Chairman William H. Hunt.

The travel agents' business has grown significantly since airline deregulation. They currently sell more than 65 percent of domestic airline tickets and an even higher percentage of international air tickets. The agents get a commission that averages about 10 percent of the ticket price.

In order to sell tickets, travel agents currently must obtain accreditation from the Air Traffic Conference, a trade group made up of the major airlines, for domestic routes, and from the International Air Transport Association for international travel.

Under agreements that have been approved in the past by the CAB, both groups establish personnel standards, location and financial requirements and other business practices that travel agencies must follow. In this week's decision, the board said that, at the end of 1984, it would lift the immunity from antitrust prosecution those agreements now enjoy.

Despite the ASTA's distaste for the CAB order, Paul M. Ruden, a lawyer for the group, said he doesn't think it will result in a drastic change in the way airline tickets are sold. "I don't see any airline . . . turning blank ticket stock over to grocery stores to be sold along with baloney and canned ham," he said. "We don't see that as the future . . . I see the airlines sticking with the system they know and the system that works."

The ASTA is convinced, though, that the agreements that set industry standards will not survive once antitrust immunity is ended.

The CAB decision allows new competition in the sale of tickets but does not mandate it, and the airlines themselves will determine whether to establish new sales outlets for their tickets. Industry sources suggested yesterday that few airlines would risk the displeasure of travel agents and jeopardize the sale of their tickets by them.

Although travel agents deny it, many airline officials are convinced that agencies have the ability to ticket away from an airline and did so last spring when Braniff International was fighting for its life. A week before Braniff suspended services, its officials complained that travel agency bookings were down between 40 and 50 percent from a year earlier. "

In its decision, the board also upheld the current industry practice of barring business travel departments from becoming accredited travel agents and collecting commissions on the tickets they write. Under the new rules, however, they could seek new arrangements -- and commissions -- from airlines without being accredited.