James Davidson married Amanda Quinn, the daughter of John Jackson Quinn and Julia Rayburn, 11 Apr 1860 Henderson Co KY. Amanda was the daughter of John Jackson Quinn, 1815-1913, and Julia Rayburn, 1815-1893, who are both buried at Corydon Cemetery.

James and Amanda are listed in the 1860 Webster County census with sons William G. and Thomas E., both from James' earlier marriages. A daughter Mary J. was born to them in 1861. By 1870, William G., age 22, and Mary J., age 9, are living with Amanda's parents in Henderson County and their parents are deceased. Mary J. Davidson married J. N. Turner 3 Jul 1884 Henderson County.

Dr. James Davidson is mentioned in several of Cap Spencer's 1913 letters to Thee Davidson. These letters were transcribed by Debbie Mecca and can be read online.