Harris was born in Englewood, New Jersey, and raised in Tenafly, the son of Margaret, a travel agent, and Robert L. Harris, who sang with the Fred Waring chorus and worked at the bookstore of the Art Institute of Chicago. He has an older brother, Robert, and a younger brother, Spencer. Harris was raised in a middle-class Presbyterian family. He graduated from Tenafly High School in 1969, where he played on the football team, serving as the team's captain in his senior year. He was a star athlete in high school and competed in athletics at Columbia University in 1969. Two years later his family moved to New Mexico and he followed after having discovered his interest in acting in various theater plays. He enrolled at the University of Oklahoma to study drama. After several successful roles in the local theater, he moved to Los Angeles, and enrolled at the California Institute of the Arts. Harris's wife is actress Amy Madigan. The couple married on November 21, 1983 while they were filming Places in the Heart in which they played an adulterous couple. They have a daughter, Lilly Dolores Harris, born in 1993. Harris and his wife appeared at a pro-choice rally for the National Abortion and Reproductive Rights Action League in January 21, 2003, held to celebrate the 30th anniversary of the Roe v. Wade case.