Growing up in a postwar Philadelphia housing project, the prospects in life for Walter Williams weren’t great. His father was absent, and Walter, his sister, and their mother lived among the misfit sidewalk characters made famous by his childhood neighbor, Bill Cosby.

He did have two important advantages, however: “My sister and I never really thought of ourselves as poor,” he told an interviewer, “[and] my mother made sure we had library cards by the time we were 6 or so.” Bright, observant, and naturally rebellious, he was also “a jokester and a troublemaker” in school until a teacher who discerned potential in Williams reacted to his classroom antics by declaring that teaching him was like casting pearls before swine.