At the northern limits of Olde Towne are the remnants of a historic, predominantly African American community known by its inhabitants as "The Berg." The area was settled in 1861 by refugees fleeing from enslavement in the Petersburg, VA area and was originally known as Petersburg or Grantsville.[24] In 1915 the neighborhood encompassed several blocks from 1st St. to Bashford Lane and Royal St. to the waterfront railroad line.

Built in 1945, a 260-unit public housing complex covers several blocks in what is now Olde Towne Alexandria. There are two prominent subdivisions in the Berg. The first is the James Bland Homes, built in 1954 and named after an African American musician and songwriter. The second is the Samuel Madden Homes, named after the second African-American pastor of the Alfred Street Baptist Church.[25]

Over the years, the historic roots of the Berg’s name were lost, and many assumed it referred to the monolithic, iceberg-like buildings of this apartment complex. It was mentioned in the movie Remember the Titans, which dramatizes the integration of city public schools in the 1970s.