My youngest son was only 4 years old when I was sent to prison. He was so attached to me that any time I tried to go anywhere, he would holler and cry for me not to leave. He liked for me to read to him in the morning, and he would sit and listen and stare at the book while I read to him. We watched “The Nutty Professor” so many times together he knew the whole movie by heart. I taught him how to catch and throw a small football in our backyard. I never envisioned I would ever be separated from him.

Three days into my incarceration, he kept telling his mother, “Ask him, Mom, ask him.” I asked her what it was. She told me it was Father-Son day on Friday at school, and he wanted to make sure I would be able to come.

When I heard that, my heart broke into a million pieces. A feeling of desperation come over me that I had never felt in my life. I immediately had to go to my room and lie down and cry. That day I knew that my incarceration would have a devastating effect not only me but all of my family members.

Our nation’s jail and prison population has sharply increased in the past few decades: 2.3 million people in 2010 compared to just 500,000 in 1980. These figures represented the highest incarceration rate in the world. Some 1.2 million inmates—more than half of the incarcerated population—were parents of children under age 18. Two-thirds of these incarcerated parents were serving time for a non-violent crime while one-third were serving time for a violent one.

As a result, there are 2.7 million minor children who have a parent in jail or prison. In other words, one in 28 American children (3.6 percent) has an incarcerated parent. Just 25 years ago, the number was one in 125. A report from the Bureau of Justice Statistics found that half of the mothers (52 percent) and fathers (54 percent) in state prison stated that they were the primary provider for their children before their incarceration.

Prisoners who are released are almost always far worse off than when they were first imprisoned. Most come out with no money, no transportation, no job and no employment or life skills. For my family and many others, the consequences of incarceration were devastating.

I was arrested on July 5, 2000. I’d spent the morning talking with my wife and kids at the breakfast table about where we would go on summer vacation. We decided on Washington, D.C. and New Orleans. I kissed everyone and left on my way to my office.

I was driving up Two Notch Road in Columbia, South Carolina, thinking how fast my kids were growing up, when an unmarked car flashed its blue lights behind me. I pulled over, thinking they were trying to get to a car in front of me, and it followed me into a parking lot.

Before I could even roll my window down to ask what the problem was, a half-dozen police vehicles converged. Cops and detectives tumbled out, guns at the ready in my face. Next thing I knew, I was sitting in the caged back seat of a sheriff’s cruiser, hands behind my husky torso, cuffs biting into my wrist, looking out grimy windows, stunned by what was happening. I had no idea I wouldn’t be free again for more than eight years.

Before my arrest, I owned Professional Management Inc., a sports management company worth more than $125 million. I was known as one of the few super-agents in sports, the first NFL agent to have five No. 1 picks in the same draft. My clients included Vince Carter, Barry Sanders, Sterling Sharpe, Shannon Sharpe and Andre Rison. I served on the boards of charities. I’d been a member of First Northeast Baptist Church board for 17 years. My family was thriving. My children were well-rounded, just kids enjoying life.

On that day in July, everyone I loved had their lives turn upside down. I was arrested for conspiracy to commit fraud and money laundering. I was convicted and, as a non-violent, first-time offender, sentenced to 142 months in prison. I would serve 105. (Later, on appeal, I was cleared of defrauding my clients.)

I had no idea the impact my incarceration would have on me, let alone on my family and the people I love: financial instability, separation, loss of assets, divorce, fear, anger, loss of control, hopelessness and desperation. Upon my arrest, all my finances were frozen and later seized. A few months later, my wife and kids lost our home and moved from South Carolina to Tennessee to live with my wife’s parents.

I was sent into transit from Lexington County, S.C. jail to the United States Penitentiary Atlanta to Detroit’s Wayne County Jail to Milan, Mich.’s federal detention center. This process took more than three months. I only spoke with my wife and kids four or five times a month for just a few minutes at a time. For the first time in my life, I had been taken away from my them and had very little communication with them.

Federal prisoners only get 300 minutes a month of phone time, which comes out to 10 minutes a day. I never understood that rule. How can you even try to comfort your children and have any meaningful conversations with them with only 10 minutes a day? I was my children’s anchor and support base. In a blink of an eye, it was taken away from them. They struggled to find stability and direction.

My daughter had just turned 14 when I was taken away. She had always been a daddy’s girl. Academics came easily to her. The only time my daughter made a B, she came home crying as if the world had ended. She and I decided to go meet with the teacher the next morning to plead for her to do something extra to bring her grade to an A. Thankfully the plea was granted.

During my incarceration, my daughter, normally confident and determined, experienced uncontrollable crying spells and would go into a state of depression. She was no longer a straight-A student and no longer focused on becoming a doctor. She became angry and resentful.

She attended the University of Chattanooga on a full academic scholarship but soon dropped out. She became pregnant and had her first child at age 19. I am certain had I not been incarcerated my daughter would have never dropped out of school, never would have gotten pregnant and she would be a doctor today.

I am blessed to say my daughter is now married and attending college, earning a degree as a pharmacist.

My oldest son, from a previous relationship, had come into my life for the first time in 1999, when I learned he was mine. He was 21 at the time, had a job driving a truck for a major grocery store and was doing well. We were both ecstatic to finally stand on the truth of our relationship. I bought him a new SUV, some clothes, paid his rent for a year and gave him some money — a small token of what I would have and should have been able to do for him. But by the time he turned 22, I was incarcerated.

After I went to prison, he quit his job and started selling drugs, something he had never done. In 2009, he was sentenced to 54 months in federal prison for drug distribution. He was released in 2012. Finding a good job as a convicted felon has been difficult for him.

My middle son, an introvert with a talent for computers and musical instruments, was 16 on the day I went to prison. He had to drop out of public school and get his GED because of the immense national and local negative press concerning me at the time. Everyone knew that I was his father, and he was heckled and unfairly badgered because of my incarceration. I thank God he did not turn to drugs and did the best he could dealing with the loss of his father and the comfortable life he was used to.

My youngest son was just 4. We made a decision to tell him I was gone on a long business trip. Throughout my incarceration, he would continually ask me when I was coming home from my business trip. I would tell him I did not know when I was coming home, to which he once said to me, “Daddy, you have been gone one hundred thousand million days,” How do you tell a 4-year-old that you will not be coming home for years?

My wife had never worked outside of caring for our children and our home. She had a rare case of lupus that attacked her nervous system without warning; the stress after I was taken away only added to her condition.

My wife and I had our bumps in the road in our 20 years of marriage, but we were always able to overcome them. She came to visit me many times throughout my prison term, but after 60 months, she served me with divorce papers. We only got to talk less than 10 minutes a day and visit once every two months because of finances and distance. She felt like her life was at a standstill with me still facing more than three years.

As I sat in the cold holding cell looking through the divorce papers, knowing that my incarceration had ended our decades of marriage, I was flooded with emotion. I thought back to when we first met and the many laughs and happy times we had shared. I worried what our kids would think and how it would affect them.

Studies consistently show that incarceration during marriage is correlated with higher divorce rates. When one spouse has been incarcerated before getting married, the couple isn’t any more likely to split up — but when a spouse is incarcerated during the marriage, the odds of divorce increase. One recent study found that each year of incarceration increases the odds that the inmate’s marriage will end in divorce (before or after the inmate gets out of prison) by an average of 32 percent. Though my wife and I are good friends today, our marriage couldn’t survive prison.

Today’s prisons do not provide any kind of substantial rehabilitation, and the vast majority provide no rehabilitation, employment skills or life skills. Inmates’ families are broken. At the very least our prisons need to provide employable job and life skills, such as money management, technical job skills training, resume and interview classes and credit repair. This would allow a major step towards helping to rebuild the family unit.

Our system of justice is a failed one that has incarcerated millions of non-violent citizens who pose no serious threat to our society, at a cost of billions of dollars. Our system needs to utilize viable and sensible alternatives to incarceration for this group of non-violent citizens such as probation, house arrest, restitution and community service.

For someone like myself who never dreamed of going to prison and does not have a criminal mind, incarceration is the closest thing to death. The emotional pain of losing your family and everything you have is unexplainable. Then upon your release you are branded a convicted felon for life.

This Father’s Day, I thank God for bringing me through the storms of my incarceration and protecting my family. I have learned to appreciate the gift of life far more than I ever did. But it didn’t have to be this way.