Jeremiah Brimstone Band was formed in 1999 when Dawid de Villiers—previously of Stellenbosch-based grunge-rock outfit, Mudbelly, and the punk band, King Ilk—abandoned electric for acoustic guitar and approached pianist Andrew Beck with a handful of ballads. The name itself was adopted after the addition of Hendré Retief on percussion and Pedro von Wielligh on bass. From the outset their sound, best described as alternative folk, registered the influence of such acts as Johnny Cash, Leonard Cohen, Nick Cave & the Bad Seeds, Tom Waits and Sixteen Horsepower, augmented by substratal references to the Violent Femmes, Sonic Youth and Velvet Underground.

The group spent the bulk of its time jamming in the cramped confines of Retief's Jonkershoek cottage, occasionally venturing forth to play at small local venues—its most notable appearances at this stage were a Rape Crisis concert in the Stellenbosch town hall in 2001 and a theatre production, at the Kleine Libertas Theatre, entitled His Unexemplary Life (the show incorporated the band's music into a story that also relied on narration by Jacques Coetzee—currently the vocalist for Red Earth & Rust—and a slide show drawn from Retief's photography). With the help of local video artist and occasional member Chris Wait the band had the opportunity to record a demo, Mute Horse Bootleg, at Sunset Recording Studios in 2002.

However, early in 2003, with most of the members facing the problem of gainful employment, the band dissolved. De Villiers—like Retief—found a temporary solution in going to Taiwan, but by 2006 both were back in their home town, Wellington, where Retief met up with Heine du Toit, lead singer and songwriter of the temporarily defunct Afrikaans band, amaBhulu?. The two bands decided to join forces, and as a result Jeremiah Brimstone Band was reborn, this time with Du Toit on bass and slide guitar, and Elmi Badenhorst on accordion and harmonica. The bands started performing in the Boland region in 2007—usually on a double bill featuring both acts—and occasionally appearing at the Barleycorn Music Club in Cape Town. In 2008 they were invited to play at the annual Barleycorn Music Festival. Early in 2009 they were officially joined by Birmingham native, Natalie Mason, on viola, and the band began to explore other venues around the Cape, like Zula Bar, Alma Café, and Stellenbosch’s Dorp Street Theatre.

By May of 2009 the band had recorded some demo material in Du Toit’s home studio, and on the strength thereof ventured to record a full length album over the course of a long weekend in August. The nine-track album—From the Wrong Side of the River—was produced by Du Toit in collaboration with the rest of the band and released on their own label, MUTEHORSEMUSIC. It was launched in November 2009.