A rich vein of crystalline marble runs beneath the City of Nelson, and it is upon this bedrock of marble that the town's foundation is built and from which its heritage springs. The history of Georgia's First Marble City and Georgia Marble is inseparable.

Original inhabitants were various Native American Tribes, including the Cherokee who resided in the area until the early 1800s. Settlers first came to where Nelson is now situated in the early years of the nineteenth century and others continued to follow for the next several decades establishing homesteads and farms in the area's valleys and foothills. The town is named for one of the earliest settlers, John Nelson, a landowner, farmer, and gunsmith. However, it was not until the Marietta and North Georgia Railroad extended their rail lines from the south to present-day Nelson in 1883 that any semblance of a town existed.

The railway was attracted to the area to serve the rapidly growing marble industry erupting in and around southern Pickens and northern Cherokee Counties. Eight years following the introduction of the railroad, in 1891, Nelson, Georgia was incorporated, and to this day, the city is one of only three incorporated municipalities within Pickens County. This incorporation followed a period of growth and expansion for what would soon become the City of Nelson.

In 1884, the year following the coming of the railroad, Colonel Sam Tate founded Georgia Marble Company after he had purchased and consolidated many of the area's smaller marble concerns. The founding of Nelson coincided with a period of rapid growth in the United States, and builders in cities across America were thirsty for building material. In an era before the advent of steel girders and concrete, structural stone was the preferred building material, and Georgia Marble "fit the bill to a tee." Because of this, Sam Tate expanded his company and in addition to his well-establish monument and memorial divisions, he formed the Structural Division of Georgia Marble and constructed a structural stone facility in Nelson.

Soon, finished structural marble from the Nelson plant would be used for such famous buildings as the New York Stock Exchange Annex, The Swan Building in New York, The Chicago Water Tower, the Federal Reserve Bank, the National Aeronautical and Aerospace Museum, the Russian Embassy, the east facade of the nation's capitol building, and also, the iconic Lincoln Memorial. Stone from Georgia Marble Company accounts for the large majority of headstones in Arlington National Cemetery, and over sixty percent of all monuments and memorials in Washington DC are made from Georgia Marble.

Because of the company's inordinate success, Georgia Marble Company recruited experienced stonecutters from other nations, and in particular, from Italy's acclaimed stable of marble workers of the Carrara region. A large contingent of Italians, as many as fifty families, migrated to Nelson, as is proven by the many headstones in the Bethesda Baptist Church Cemetery which are inscribed with Italian names. Stoneworkers were also recruited from Scotland. Also, long before any such edicts were mandated from Washington, Colonel Sam Tate was a firm believer in equal employment opportunity, and he offered steady employment to Nelson's substantial African-American population. As a result, original English and Scotch-Irish settlers worked alongside their fellow Italian and Black workers truly making Nelson a multi-cultural community.

For over a century, residents were reassured by the sound of what could be described a "musical backdrop" as they lived and worked in Nelson. The rhythmic sound of gang saws permeated the surrounding atmosphere and was punctuated by the occasional shrill shriek of a steam whistle marking the beginning and end of work shifts. The sounds were "music to the ears" of Nelson residents because it meant prosperity for the community and steady employment for the residents.

The community pays homage to its marble legacy with a museum located at City Hall featuring exhibits of marble specimens and displays of the local history of the marble mining and finishing industry.