James Madison The oldest of twelve children, James Madison was born on Ma to James and Nelly Madison. James Sr. was a slave owner and planter. In the early s, he constructed a new home in Orange County, Virginia, and the family moved into the plantation estate, Montpelier.