The hexagonal structure was designed by then Brevet-Colonel Robert E. Lee and named for Charles Carroll of Carrollton, a signer of the Declaration of Independence. Construction of the fort began in 1848, under Lee's supervision, by the U.S. Army Corps of Engineers. Fort Carroll was important for the defense of Baltimore — before the fort was created, the only military defensive structure between Baltimore and the Chesapeake Bay was Fort McHenry. In addition, a lighthouse, now abandoned, was built on the ramparts to aid navigation into Baltimore Harbor.
The government abandoned the fort as a military post in 1920, and the island was declared excess property in 1923. However, the War Department took no immediate steps to sell the land. In May 1958, a Baltimore attorney purchased the island for $10,000, but development plans never materialized. The fort now is deserted.
The steel beams attaching the stone quay and the fort (visible in the 1992 pictures) are almost completely rusted away as of 2007, with reportedly only part of one remaining. The fort has become a defacto bird sanctuary.