You'd expect Julian Dixon, a black California congressman, to vote "right" on issues vital to blacks and Hispanics. After all, the two groups make up nearly 70 percent of his constituency. Similarly with Edward Roybal, an Hispanic congressman whose district is 71 percent Hispanic.

But you might be a bit surprised to see the score of George Brown Jr., whose Riverside district is only 8 percent black and 22 percent Hispanic. The three Democrats are among the five members of the 45-member California delegation to rack up perfect scores on a new rating of congressional votes on black and Hispanic issues during 1981. John Rousselot, the San Marino Republican, on the other hand, managed a perfect zero.

The scores are reported by Congressional Education Associates, a Washington-based firm headed by former Senate aide Gordon Alexander.

The ratings are based on recorded votes on 20 key legislative issues, ranging from federal budget proposals and voting rights to food stamps and a move to bar a visit here by a South African Rugby team. Each "right" vote is worth five points; wrong votes or nonvotes count zero.

California had more perfect scores than the entire Black Congressional Caucus, which had only three: Dixon, Parren Mitchell of Maryland and William Clay of Missouri. The lowest ranked Caucus member was Chicago's Gus Savage, at 70. According to David Ruffin, who with Gloria Barajas put the guide together, no Black Caucus member voted "wrong" on any issue; the reduced scores are the results of nonvotes.

Not so with the five Hispanics in the Congress. Roybal was perfect, and New York's Robert Garcia would have been but for two missed votes. But New Mexico's Manuel Lujan, who voted on every key issue, managed only a 45.

Nearer home, no Virginian scored higher than 30, and three scored at zero. Maryland, on the other hand, had three perfect scores in the House, and one (Paul Sarbanes) of only three in the entire Senate. Don Riegle (D-Mich.) and Bill Bradley (D-N.J.) were the others.

The guide, which will be published annually, is designed to help minority voters keep tabs on their representatives, according to Ruffin. "There have been several rating systems--conservative, liberal, and so on--but each one is put together by people based on their own priorities. We think this is the first one based on the priorities of blacks and Hispanics.

"We hope it will be useful, not only as a reference on how elected officials voted on key issues but also in helping to identify voting patterns on issues of mutual interest which can form the basis for congressional coalitions."

The guide, which identifies members by state and congressional district as well as by percentage of black and Hispanic constituencies, makes clear that ethnicity wasn't the only consideration in the key roll-call votes. For instance, Iowa, in all of whose six districts blacks and Hispanics comprise less than 1 percent of the electorate, only one representative scored less than 60. Two (Neal Smith and Berkley Bedell) scored 90 or higher.

On the other hand, Mississippi, whose five districts have black percentages ranging from 20 to 45, no one scored higher than Jamie Whitten's 65. Trent Lott, who is credited with triggering the president's fiasco on tax exemptions for segregated schools, scored zero.

The CEA ledger rates members on issues involving civil rights, the federal budget, government spending priorities, human and community development and foreign affairs.