Wharton was born in Philadelphia in 1826, the fifth child of ten in a liberal Hicksite Quaker family. His parents, William Wharton and Deborah Fisher Wharton, were both from prominent early American immigrant families of Quaker descent.  Wharton's father was a typical gentleman and did not hold a regular job because he had several illnesses, but oversaw his estate, served on the Philadelphia School Board, and was active with his wife Deborah in the Hicksite ministry.

Wharton's youth was spent in the family's house near Spruce and 4th Streets in downtown Philadelphia and at the country mansion "Bellevue".  From their country estate, the family often went to the nearby Schuylkill River, visited neighboring estates such as Deborah's grandfather Joshua Fisher's "The Cliffs", and went on weekend horse and carriage excursions to the countryside surrounding Philadelphia, sometimes attending the smaller Quaker Meetings.

Wharton's maternal grandfather, Samuel R. Fisher ran a prosperous mercantile business and shipping packet line between Philadelphia and London

​

Both of his grandmothers were named Hannah and were from Newport, Rhode Island. His grandmother Hannah Rodman was a descendant of Thomas Cornell, the ancestor of Ezra Cornell who founded Cornell University.