The Coolwell First Baptist Church was one of the first African-American churches founded independently of a white congregation. The church was organized in 1862 by the Rev. P. Ferguson. The building was rebuilt in 1973 and added on to in 1995. The location was orginally referred to as a spot with a well, containing cold water, hence the name. The cemetery, in use for over 100 years, contains at least 255 individuals. Roughly half of the gravestones have legible dates, while many are carved fieldstones (lacking inscriptions) or highly eroded metal, marble, or wooden markers.

Many of the individuals buried here once worked at Sweet Briar (both in its plantation days and as a college, founded in 1901). Oral histories reveal that many of the men and women who worked at Sweet Briar, lived in Coolwell and walked several miles on foot to reach work. One of these individuals was Signora Hollins, buried with her family in the cemetery. Later in life, Ms. Hollins recalled playing with Daisy (the only child of Indiana Fletcher and Mr Henry Williams who died at the age of 16 in 1884). Upon her death in 1900, Indiana left money to found a college for women in honor of her deceased daughter.