Was Robert E. Lee’s older brother born at 312 Cooper Street, now the Rutgers—Camden Alumni House?

Possibly yes, but we lack a definitive record of that birth at this location on September 2, 1802.

Most circumstantial evidence supports a story that this house was the birthplace of Sydney Smith Lee, the fourth child of “Light Horse Harry” Lee of American Revolution fame and Ann Hill Carter Lee. However, we have not yet found direct evidence, like a birth certificate or a letter describing the event.

The Lees’ next child, Robert E. Lee, grew up to lead the Confederate forces during the Civil War. The older brother, known as Smith, also fought for the Confederacy. Prior to the Civil War, he also served as commandant of the Philadelphia Naval Yard.

What do we know about the birth of Sydney Smith Lee? 

According to a Lee family history published in 1895, Sydney Smith Lee “was born the 2d of September, 1802, at Camden, New Jersey, where his mother happened to be visiting a friend.” Later biographers have relied on this history. The Lee family Bible records the date of birth, but not the place.

Two other Lee family members described the circumstances of the birth. In a memoir written in the 1860s, an older Lee brother recalled being with his parents on a trip to Philadelphia in 1802 and that his brother born during that trip “was named Smith from the hospitable family, in whose house he was born.” Later, in the 1880s, Sydney Smith Lee’s son Fitzhugh Lee made public statements about his father’s birth. While campaigning to become governor of Virginia, he said that Sydney Smith Lee was born in Camden while his mother was visiting her friends, the Smiths.

How does this connect with the house at 312 Cooper Street?

For nearly fifty years, this was the summer home of iron merchant Edward Smith, his wife Sarah, and their large family. According to deeds and tax records, the house was built by 1802, so it is plausible that the Smiths could have entertained visitors at that time. This could make them the Smith family in Camden that Sydney Smith Lee’s mother was visiting when she gave birth.

Do we have any reason to doubt this?

Maybe. During the 1880s, people in Camden made the connection with 312 Cooper Street after they read news accounts of Fitzhugh Lee’s statements about his father’s birth. By that time, the house was owned by the Camden Republican Club, whose members knew about its earlier ownership. Although many of the members were Union Army veterans, they publicized the connection with a son of the Confederacy and displayed a framed letter from Fitzhugh Lee about his father’s birth.

However, the Smiths also had a home in Philadelphia. Could the birth have taken place there instead? Published accounts consistently identify Camden as the birthplace, but they do not address the existence of another Smith home in Philadelphia. This leaves an open question.  Although we have many references to Camden as Sydney Smith Lee's birthplace, no evidence has been found to exclude the Philadelphia house as the place Mrs. Lee visited in 1802.

How did the Smith family and the Lees of Virginia know each other, if they did?

We do not know for sure. It is plausible that their paths could have crossed in Philadelphia social circles during the late 1790s. At that time, Philadelphia was the nation’s capital, and “Light Horse Harry” Lee served in the U.S. House of Representatives from Virginia. The Smiths, who were related to wealthy landholders in Salem and Cumberland Counties, moved to Philadelphia around the same time. Like Mrs. Lee, Sarah Smith was in her child-bearing years. She was also pregnant around the same time that Sydney Smith Lee was born. When the Lees returned to Philadelphia during a trip in 1802, it can be imagined that both women would have appreciated getting away from hot, congested Philadelphia to spend time at the Smiths’ new country house in Camden.

Is it OK to repeat this story?

Yes, but we can only say that it is an interesting story that has not been thoroughly proven, so far. It is possible that Robert E. Lee’s older brother was born at 312 Cooper Street in Camden, but until better documentation is found, we cannot be entirely sure.

-- By Charlene Mires, posted August 29, 2026.

Sources:

Armes, Ethel. Stratford Hall: The Great House of the Lees. Richmond, Va.: Garrett and Massie, 1936.
Camden and Philadelphia Newspapers (Newspapers.com and GenealogyBank.com); obituaries in the New York Times, the Baltimore Sun, and other newspapers.
Camden County and Gloucester County Deeds.
Family Record (transcription from Lee family Bible), Mabel Brady Garvan Institute Records, Series III, Box 3, Folder 1, Yale University.
Lee, Edmund Jennings. Lee of Virginia, 1642-1892. Philadelphia: Franklin Printing Company, 1895.
“Lee, Henry.” Biographical Directory of the United States Congress. https://bioguide.congress.gov/search/bio/L000195
New Jersey Tax Records, New Jersey State Archives.
Philadelphia City Directories.

Acknowledgments: Bonny Beth Elwell of the Camden County Historical Society and Judy Hynson of the Stratford Hall Historic Preserve provided valuable assistance with this research.

Cite: Charlene Mires, "Sydney Smith Lee," Learning From Cooper Street, https://onlineexhibits.camden.rutgers.edu/s/cooper-street/page/leebirth.