A special federal study group on Medicare voted 6 to 3 yesterday against recommending an increase in federal alcohol and tobacco taxes to funnel more money into the failing Medicare trust fund.

Sources said both industries lobbied heavily on the issue. The justification for the proposed taxes was that persons with alcohol- and tobacco-related illnesses use a disproportionate share of Medicare services.

The group, headed by former Indiana Republican governor Otis R. Bowen, a physician, is the Social Security advisory panel appointed every four years to recommend changes in the system.

It is concentrating now on Medicare because the Medicare trust fund is expected to be exhausted near the end of this decade as medical costs increase far faster than the general cost of living.

The panel also:

* Rejected, 6 to 3, a proposal to recommend taxing employes on the value of health-insurance premiums paid by their employers. The proposal was backed by President Reagan but turned down by the House Ways and Means Committee.

* Agreed that low-income persons should not be charged the full premium rate for a proposed plan to enlarge Medicare hospital and doctor benefits. Under the plan, which the panel had recommended, Medicare beneficiaries would be required to pay no more than $200 a year out of pocket for allowable doctor bills.

They also would receive unlimited free hospital days except for the first day of a specified period in a hospital, but would have to pay about $420 a year to Medicare in premiums instead of the current $146.

Under the low-income exemption recommended yesterday, extra premiums for the added coverage would be canceled for beneficiaries with very low incomes and would be fully chargeable only to those with annual incomes of more than $6,000.