Joseph's paternal grandfather, Milton Levitt, was born Milton Levitz in Ohio; Milton's father, Louis Levitz, was a Russian Jewish immigrant, and Milton's mother, Fraudel "Fannie" Isaacson, was born in 1894, in then-Ottoman Palestine, to a Russian Jewish family (that wave of movement to the area was known as the "First Aliyah"). Joseph's paternal grandmother, Celia Roth, was born in Pennsylvania, to Polish Jewish parents. Joseph's maternal grandfather, Michael Gordon, was born Irving Kunin Gordon in Maryland, to a Jewish family from Lithuania and Russia, while Joseph's maternal grandmother, Elizabeth A. Cohn, was born in Ohio, to a Romanian Jewish father and a Russian Jewish mother. Joseph's grandfather Michael Gordon was a prominent Hollywood film director, who directed the films Cyrano de Bergerac (1950), Pillow Talk (1959), Portrait in Black (1960), and many others. Michael Gordon's career was crippled when he was blacklisted during the Red Scare of the mid-Twentieth Century.
When Marion Cotillard was honored at Hollywood Film Festival in 2012, he presented the tribute to her in French, Cotillard's native language. Levitt and Cotillard were co-stars in Inception (2010) and The Dark Knight Rises (2012).
Good friends with Channing Tatum and David Krumholtz. Also friends with Leonardo DiCaprio, Zooey Deschanel, Anne Hathaway, and Emily Blunt.
Grew up with Beverley Mitchell. They attended the same acting class for many years as aspiring child actors.
Good friends with Mysterious Skin (2004) co-star, Michelle Trachtenberg.