The image of the unwed father was always a seedy one. He was the man who skipped town one step ahead of the shotgun. He left behind him a woman "in trouble," and a child who grew up as a swearword.

The only paternity cases ever heard about when I was growing up were the kind that proved who was the father and forced him to 'fess up and pay up. We were far more concerned then with pressing responsibilities on unwed fathers than with ensuring their rights.

But, gradually, things have changed. In 1950, only 3.9 percent of the children in this country were born out of wedlock. In 1975, 14.2 percent were. Most of their fathers still remain anonymous and tragically disconnected. But there are a number of men, and a body of law, gradually developing around the new kind of paternity issue - the rights of the unwed father.

Last week, the U.S. Supreme Court heard the latest and potentially most important of those cases, Caban v. Mohammed. They heard the arguments for and against a father who is challenging a New York statute. It says that an unwed mother can agree to the adoption of her children without the consent of the father.

The case touches on many sensitive legal and emotional points. Should the rights of an unwed father be different from those of a married, or formerly married, father? Should they be different from the rights of an unwed mother? How carefully can we balance the interests of the children and the interests of the adult?

The story behind this litigation is as tangled as many other human relationships. The parents lived together from 1968 to 1973 and together had two children, who bear the father's name. After their separation, the mother married, but the father maintained a strong presence in his children's lives. At one time or another, he had extensive visitation rights or full custody. He always had a major decision-making role.

But about three years ago, the mother and the stepfather petitioned to adopt the children. Adoption would have ended all the legal rights of the natural father, including even his right to visit his children. At that point he fought - he fought the mother and the state, declaring that the law was unconstitutional because it violated his rights to due process and equal protection of the law. Now, when the Supreme Court hands down its decision, it will fill in a major gray area of new law on fathers' rights.

Historically, men like Abdiel Caban had no rights to their illegitimate children. Then, six years ago, in Stanley v. Illinois , the Supreme Court struck down a law that made illegitimate children the wards of the state upon the death of their mother - even if their father was alive and living with them. For the first time, the court acknowledged the status of an unwed father and kept a family intact.

But this year, they ruled against the rights of another unwed father in a Georgia case, Quilloin v. Walcott. They said that this man -who had maintained virtually no contact with his children - could not suddenly come upon the scene to veto the adoption by a new stepfather.

Both of these cases - decided for and against unwed fathers - hinged neatly on the best interests of the children. As Paul Kurtz of the University of Georgia Law School says, "The overriding consideration is the best interest of the child." But he says that in Caban v. Mohammed the question is whether "the best interest of the children is 'best enough" to outweigh the rights of the father."

A lower court had ruled that the two children would fare better in the world if they had the same name as their mother and stepfather and were no longer illegitimate. But is the difficulty of bearing a different name or even the stigma of illegitimacy greater today than the risk of losing contact with a parent? Is is so great that it outweighs the man's right to be a father?

Surely an unwed father shouldn't have an absolute right of veto over adoption. That would create havoc. It seems to me appropriate that, at this time, his rights should be calibrated in terms of his commitment. There's a vast difference between the father who has lived with his children, and the one who has deserted them. Perhaps the rights of both mothers and fathers have to be "deserved."

But in this case, the psychological effects of the adoption on the children are not crystal-clear, and the father has a powerful desire to remain a parent. It seems both unfair and discriminatory to allow an unwed mother the unchecked right to dispense her children - and even to sever their ties with their father.

One of the major issues of this society, after all, is how to restore, maintain or strengthen the relationships between men and their children - in or out of marriage. We can't seek or demand responsibility from fathers while denying them appropriate rights. They, too, need this legal security as insurance for their emotional investment.