James Young Pennebaker, 84, who had been a real estate investor and mortage banker in Washington since the 1920s, died Saturday at Georgetown University Hospital after a heart attack.

Mr. Pennebaker, who lived in Arlington, was a horeseman and a member of the Virginia Thoroughbred Association. He raised and trained hunters and steeplechasers. He owned Derrydown, who was successful in steeplechase races in the 1960s and the early 1970s.

Mr. Pennebaker was a native of Washington. He enlisted in the French Army Ambulance Corps in 1916 and served in the U.S. Army after this country entered World War I a year later.

After the war, he worked briefly in the investment business in New York City. He returned to Washington in the early 1920s as a salesman with the real estate firm of Storey & Company. Later in the decade he went into business for himself. He never retired.

Mr. Pennebaker was a member of the Washington Golf and Country Club.

Survivors include his wife, Evelyn Freer Pennebaker of Arlington; a daughter, Jacqueline Clark of Vienna; three sisters, Ann Spransy of Chicago, and Lucinda Converse and Margaret Dawson, both of Stone Harbor, N.J.; two brothers, William B., of Bucks County, Pa., and Charles D. IV, of Media, Pa., and four grandchildren.

The family suggests that expressions of sympathy be in the form of contributions to the American Heart Association.