James Francis Kelley, 65, a former assistant secretary and comptroller of the Department of Health, Education and Welfare and an authority on government management and budget practices, died of cancer Friday at St. Joseph's Hospital in Asheville, N.C.

Mr. Kelly, who was born in Washington and grew up in Baltimore, spent 36 years in federal service, rising from GS-1 to assistant secretary of HEW. After retiring from the government in 1970, he served for three years as the executive vice president for administrative affairs of Georgetown University. From 1973 to 1979, he was the executive vice chancellor of the State Universities of New York in Albany, N.Y. He then retired to Hendersonville, N.C., where he lived at the time of his death.

Mr. Kelly began his government career in 1934 as a clerk in the old Public Works Administration. In 1938, he transferred to the U.S. Housing Agency. By 1943, when he was commissioned in the Navy for World War II service, he was the principal budget analyst in the budget branch of the Federal Housing Authority.

In 1939, he earned a law degree at the Columbus University School of Law, now part of Catholic University, and was admitted to the D.C. Bar in 1940. He also studied accounting at the Columbus University School of Accountancy.

After the war, he spent two years as regional budget officer in the New York office of the Public Housing Administration. In 1948, he returned to Washington as director of the housing administration's budget branch.

He held that post until 1954, when he was appointed the budget officer of the old Department of Health, Education and Welfare, which was established in 1953. From 1961 to 1966, he was the deputy comptroller of HEW. From 1966 until his retirement, he was assistant secretary-comptroller of the department. He was credited with overhauling HEW's accounting system and developing an improved system for auditing grant-in-aid programs.

Mr. Kelly received the Superior Service and Distinguished Service awards from HEW. He also received the Career Service Award of the National Civil Service League, the Distinguished Leadership Award of the Federal Government Accountants Association, the President's Award from the National Rehabilitation Association and a special citation from the secretary of HEW.

In addition, Mr. Kelly received an honorary degree in political science from the University of the Pacific and an honorary doctorate in science from the New York Medical College. He wrote and lectured widely on public administration.

He was a member of the American Society for Public Administration, the Institute of Medicine of the National Academy of Sciences and the board of directors of the Georgetown University Community Health Program. He was a consultant to the National Institutes of Health and chairman of a task force sponsored by the Association of American Medical Colleges on national health policy.

Mr. Kelly is survived by his wife, Mary Soule Kelly of Hendersonville; four daughters, Mary Carol Kelly of Los Angeles, Joan Renwick of Chapel Hill, N.C., Laurel Cunningham of Reston, and Elizabeth Kelly of Albany, N.Y.; three sons, James F. Jr., of Los Angeles, David, of Winston-Salem, N.C., and John, of Hendersonville; two sisters, Louise Kelly of Silver Spring, and Margaret Quaile of Garden City, N.Y., and five grandchildren.