A new balloting system designed to shorten lines at the polls and speed counting of votes, also appears to encourage voters to participate in more local, less-publicized races, according to a recently completed study by Mongomery County election officials.

The system, in which voters punch their choices onto cards to be tallied later by computer, got a test last year in the primary and general elections in Maryland's 17th Legislative District, an area encompassing Rockville and parts of Wheaton and Potomac.

Next year, the Datavote system will replace the lever-type machines throughout Montgomery County.

In General, Montgomery election officials have found that voters tend to quit registering their choices towards the lower end of long ballots, in the races of less general interest.

But in the Rockville district, the study notes, this "drop off" phenomenon was less marked. "In every contest, in both the general and primary elections, drop-off in the 17th district was lower than drop-off for that contest in the aggregate vote . . .," the report says.

"Why?I don't know," said the county's elections administrator, Marie M. Garber, who coordinated the study. "People will never vote to the end of the ballot. But if we make it easier to read, and provide more voting booths," she noted, more people may be encouraged to vote.

Getting voters to the polls and keeping their attention through a long ballot is of increasing concern to elections officials in Montgomery County, who have watched participation in elections slide in recent years.

Only 54.8 percent of registered Montgomery voters went to the polls in the 1978 general election, Garber noted, as opposed to 60 percent four years before that and 70 percent in 1970. While the drop-off parallels a national trend, Garber attributes part of the apathy, to long lines at the polls. "When the parking lot is full, some people will turn around and go home," Garber said yesterday.

The Rockville study also offers some insights into voter behavior. Noting that voter participation was higher in congressional and gubernatorial races than in local contest, Garber said, "I think it shows very rational behavior on the part of voters. The uncontested candidates, and the state constitutional amendments were not important [to voters.]"

Voter response on ballot proposals was mixed. Those affecting other ies attracted fewer Montgomery. However, in the case of the much publicized TRIM tax-cutting proposal, listed second from last on the ballot, "drop-off was considerably lower than on any other ballot question [and] lower than in local candidate contests," according to the study.

In the cases of local offices, the study found "an unexpectedly large jump in drop-off" between the last partisan race, the race for sheriff, and the following nonpartisan contests for Board of Education seats.

Asked to explain why voters were more interested in the outcome of the sheriff's race than in choosing the Board of Education, which last year engendered a bitter, divisive and well-publicized campaign, Garber said that interparty competition still attracts many voters.

Another factor, she noted, is that of "bullet-voting" - the decision of a voter to vote for only one or two candidates in a large race when they could vote for more. Voters do this to enhance the strength of a candidate by cutting down the vote totals of rival candidates.

Garber expects the Datavote system, manufactured by Diamond International Corp. of San Francisco, to provide a number of benefits when it comes into use in Montgomery's May 1980 primary election.

Because each booth unit is much cheaper than lever-type booths-roughly $180 as opposed to $2,300-there will be more booths: One for every 75 voters instead of one for every 350. This, elections officials hope, will give voters more time to weigh their choices.

An additional benefit, Garber said, is ease of recounting ballots. The computer cards, can be retrieved and recounted if need be.

While Garber contends there are enough safeguards to protect voter privacy and against corruption, not all participants in the Rockville test are so sure. Pili Sheehan, of Potomac, said she and some other Republicans are concerned about the security of the ballots in the boxes. "Someone waiting to use the machine could see over your shoulder," she said.

Such doubts were echoed by Eileen Petrillo, of the Prince George's County Board of Elections. She said Prince George's was "satisfied" with lever-type machines, and had had some problems with a punch card experiment involving absentee ballots.

In Virginia, lever-type machines are mandated by law, although an effort is under way to get a punch card system for absentee ballots.

In the District of Columbia, which gave Datavote a trial in City Council and Board of Education elections last Tuesday, officials have voiced pleasure with the system. "Things went well," said the city elections administrator, Mary Rodgers, yesterday. "But the ultimate decision on whether to purchase it is up to the board. The funds are available."