A narrow plurality of District of Columbia voters favors statehood for the city, but almost as many voters are "not sure" the political change is a good idea, according to a new poll.

The latest Associated Press--WRC-TV poll, conducted April 28-29, also showed that Washingtonians overwhelmingly favor the city's strong handgun-control law, support raising the legal age for beer and wine consumption from 18 to 21 and endorse a nuclear weapons freeze by both the United States and the Soviet Union.

Telephone interviews were conducted with 738 District residents chosen to reflect the demographic makeup of the city. According to the pollsters, the results have a 4 percent margin of error.

With a voter-chosen convention currently drafting a constitution that would point toward statehood, subject to approval by city voters and Congress, the pollsters asked, "Do you favor statehood for the District of Columbia?"

The response: Yes, 40 percent; no, 25 percent; not sure, 35 percent.

On some other questions:

* "Do you favor . . . the . . . current [D.C.] law prohibiting the sale and possession of handguns?" Favor, 69 percent; oppose, 20 percent, not sure, 11 percent.

* "Do you favor or oppose raising the drinking age . . . ?" Favor, 60 percent; oppose, 30 percent, not sure, 10 percent. (Unlike adjoining states, where drinking ages below 21 are a relatively recent and controversial matter, 18-year-olds in the District have been permitted to buy beer and light wine--but not hard liquor--since 1934.)

* "Would you favor or oppose a [joint U.S.-USSR] freeze on the production of nuclear weapons . . . ?" Favor, 71 percent; oppose, 18 percent; not sure, 11 percent.

However, only 22 percent of Washingtonians said the U.S. should move toward nuclear disarmament on its own, and 49 percent said it should move only if the Soviet Union also does so. Sixteen percent thought the U.S. should not curtail nuclear weapons in any case.