In his six years in Congress, Rep. Harold C. Hollenbeck has compiled an unusually liberal and pro-labor voting record that warmed the hearts of New Jersey union leaders and discouraged Democrats from trying to unseat him.

The boyish, 43-year-old lawyer and former state legislator from East Rutherford sided with congressional Democrats more often than with his own party. He regularly received high ratings from the AFL-CIO and the Americans for Democratic Action (ADA).

And, with labor solidly in his corner, he won election to a third term in 1980 with nearly 60 percent of the vote in what is now a predominantly blue-collar district in southern Bergen County.

But since then Hollenbeck's political fortunes have soured a bit, largely because of his brief fling with Reaganomics and a redistricting plan that increased the number of Democrats in his district.

Now Hollenbeck is working hard to stave off an aggressive challenge by Democrat Robert G. Torricelli, 31, a onetime aide to then-Vice President Mondale and then-New Jersey Gov. Brendan Byrne.

The Democratic redistricting plan cut off the northern tip of Hollenbeck's district, in Hudson County, and added the heavily Democratic towns of Hackensack, Teaneck, Fair Lawn and Paramus in Bergen County.

But equally damaging was the defection to Torricelli of several of the largest and most politically active unions, whose members disapproved of Hollenbeck's support of President Reagan's initital budget and tax cuts.

Hollenbeck is a part of a caucus of moderate Republicans, known as "Gypsy Moths," who sought to soften future cuts in domestic spending by cooperating with Reagan in the early round.

Hollenbeck said he broke with the administration late last year after it reneged on a promise to restore funding for certain domestic programs and cut back on defense spending.

"I was never a strong supporter of Reagan," Hollenbeck said. "I cast a couple of votes. It was an answer to a cry in the country for a change in direction."

But Torricelli cites Hollenbeck's early support of Reagan as evidence that Hollenbeck favored a repeal of minimum Social Security benefits, drastic cuts in student loans and a shift of the tax burden to middle-income families.

The powerful state AFL-CIO and more than a dozen key unions like the Teamsters, the airline pilots and the seafarers continue to support Hollenbeck, who serves on the House Public Works and Transportation and Science and Technology committees.

AFL-CIO leaders praise his voting record and his continued support of worker health and safety standards and the Davis-Bacon Act.

But Torricelli picked off the endorsements and financial aid of the 110,000-member New Jersey Education Association, the largest union in the state; the American Federation of State, County, and Municipal Employes, the United Auto Workers, the International Ladies Garment Workers Union, the International Association of Machinists and a handful of other liberal unions.

Hollenbeck said he's not overly concerned about the split in labor and cites a recent poll, taken by his campaign, that shows him holding a comfortable lead. He predicts that the unions remaining in his camp will redouble their efforts and that a large percentage of members of defecting unions will vote for him.

"I probably will gain more support from labor this time -- more money, manpower and phones," he said. "With labor, you're always going to lose some. Labor is not cohesive. They always go their own way."

But Torricelli views the labor split as a highly significant development -- and one that has kept his hard-charging campaign afloat.

"Labor is the only thing that can help a Democrat keep up with the Republican National Committee," he said. "I wouldn't be in this race if it wasn't for labor support."