Frankel is the only child of horse trainer Robert J. Frankel and Bernadette Burke. Her father was Jewish and her mother was a Roman Catholic of Welsh descent who converted to Judaism when they married. Her father left her mother when Frankel was four years old and when she was five, her mother married another horse trainer, John Parisella. She describes her childhood as difficult. Her biological father stopped supporting the family and her mother, Frankel says, "was always drinking" and often argued violently with her stepfather.