Efforts to introduce a new generation of home tape recorders into the United States got a boost yesterday when the National Bureau of Standards said a system intended to block copying of music by the machines often doesn't work and can degrade sound quality.

"We feel the system does not achieve the stated purpose," Dr. John Lyons, director of the bureau's National Engineering Laboratory, told a press conference here. "For some listeners, for some selections, there is a discernible difference."

A coalition of manufacturers, retailers and consumer groups hailed the report and predicted it would speed introduction of digital audio tape (DAT) machines in the United States. Manufacturers last year put them on sale in Japan and Western Europe, but have withheld them from the United States pending resolution of the copying dispute.

The report appeared to be a serious setback for major record companies and musicians that have championed the system as necessary to prevent mass piracy and copying of music. Analysts predicted that the report will kill legislation in Congress that would require the anticopying system.

But the Recording Industry Association of America, which represents the major recording companies, said it would not give up. It will sue any company that introduces DAT recorders before the copying issue is settled, association president Jason S. Berman said.

"DAT recorders must not be permitted to rob the American industry of the rights to be paid for the music it creates," he said. The association is part of the Coalition to Save America's Music, which includes many musicians. Yesterday, it released statements from such artists as singer Tina Turner and conductor Leonard Bernstein supporting its position.

The DAT controversy revives the issue of a copyright battle that began in the 1970s, in which the motion picture industry tried to outlaw home copying of television programs. That ended with a Supreme Court decision in 1984 that upheld the right to copy at home for noncommercial purposes.

DAT machines produce distortion-free sound using the same digital technology that is the basis of the popular compact disc.

Audiophiles have praised the machines as revolutionary. But record companies and musicians depict them as a pirate's dream, producer of "perfect clones" of compact dics that would cheat them out of millions of dollars.

The recording industry said it already loses about $1.5 billion a year to home copying. (Recordings made with conventional analog tapes are less of a danger, the industry said, because quality diminishes with each generation of copying.)

With the protection of U.S. "intellectual property" abroad becoming an important trade issue for the Reagan administration, DAT last year attracted major interest on Capitol Hill, with bills being introduced to require the anticopying system, which was developed by CBS Records.

The issue also became wrapped into the tense U.S. trade relationship with Japan, because production of DAT machines is centered there.

Under the CBS system, music on compact discs or prerecorded DAT tapes that a manufacturer wanted to protect would be treated to delete in places a narrow band of frequencies at around 3840 hertz, a pitch lying between the highest B-flat and B on an 88-key piano.

A special chip in the recorder would listen for this "notching." If detected for 15 seconds or more, the chip would shut off the recording function for 25 seconds.

Manufacturers and consumer groups, working together as the Home Recording Rights Coalition, protested. They contended that home copying for noncommercial purposes is legal and should not be blocked.

In addition, they suggested that use of the chip would amount to an intentional rejection of a revolutionary technology, a senseless reversal of progress. Prevention of piracy should be accomplished by enforcement of laws, they said, not crippling of the machines.

The recording companies produced experts who said the difference using the anticopying system could not be detected by the human ear.

Congress, seeking an objective assessment, commissioned the National Bureau of Standards study. After extensive laboratory and listener tests, using a variety of classical and popular recordings, it found that:

The CBS system often does not work. About half the time it failed to detect notched material and allowed it to be recorded. In other, less frequent cases, it shut down the recording of material that had not been notched, apparently because gaps in the frequency range that the chip looks for can occur naturally in certain passages of music.

Some listeners can hear the omission in certain pieces. The omission was most commonly discerned, they said, in synthesizer chords, soprano voices and other high-register passages.

In one case, a piccolo note in a piece by French composer Olivier Messiaen dropped out completely. "You heard the breath going over the piccolo, but the note wasn't there anymore," said NBS official Dan Flynn.

The system can be foiled by people who have basic engineering training, using about $100 worth of components, assuming that they have access to designs for such devices.

The NBS's technicians devised five systems to foil the protection, it said.

Lyons suggested that other systems based on encoding recorded music would run into similar problems. "None of them are going to be foolproof," he said.