↓Skip to main content

Django with SQLite3 made durable with Litestream and Caddy

·4 mins

Lower your database costs to close to nothing #

On the morning of January 1st I stumbled upon this tweet from Tobi, the founder of Shopify where he mentioned running Ruby on rails apps using SQLite3 made durable using Litestream.

This made me really excited, using SQLite is a really nice and inexpensive way to add a database to your apps, but it comes with the drawback on how to backup and restore it, this is where Litestream comes into the picture.

"Despite an exponential increase in computing power, our applications require more machines than ever because of architectural decisions made 25 years ago. You can eliminate much of your complexity and cost by using SQLite & Litestream for your production applications."

I encourage you to read this blog post, written by Ben, the creator of Litestream, which gives you the full details about Litestream and why it was created.

Create a production ready Django server with SQLite, Docker, Litestream and Caddy #

I decided to play around with Litestream so I put together a bare bones Django 4.0 app with SQLite3 as the database, making it durable with Litestream and Caddy. The whole thing is easy to run with docker-compose.

What is Caddy you might wonder? It’s a reverse proxy written in Go with automatic HTTPS, so like NGINX but easier to use, I really enjoy Caddy and its my default choice for reverse proxy now a days. Read more over at their website

Create your local development environment #

Create a virtual env python3 -m venv my_django_app

Then activate it cd my_django_app ; source bin/activate

Next pull this repo git clone https://github.com/fredrikburman/django-litestream-caddy.git

Configure Litestream #

You need to create a replica, ie where your Litestream backups will be stored, you can use for example S3 or any other service, pick and follow a guide here

Once you have created a place to store your replicas you need to create a .env file that contains

DB_FILE=/usr/src/app/database/blogpostdemo.sqlite3  # this is a location inside the Docker container
LITESTREAM_ACCESS_KEY_ID=<you access key id from the previous step>
LITESTREAM_SECRET_ACCESS_KEY="<you secret access key from the previous step>"
REPLICA_URL=<replica url from the previous step>

The file should be located in the django-litestream-caddy directory

Verify that Litestream is doing its thing #

Now go ahead and launch the services docker-compose up -d, open your browser and navigate to http://localhost and you should see something like this.

Django it worked

Verify that Litestream was started properly, you should see something similar to this.

$ docker logs web
.....
litestream v0.3.7
initialized db: /usr/src/app/database/blogpostdemo.sqlite3
replicating to: name="s3" type="s3" bucket="********" path="blogpostdemo.sqlite3" region="" endpoint="" sync-interval=1s
/usr/src/app/database/blogpostdemo.sqlite3: init: cannot determine last wal position, clearing generation; primary wal header: EOF
/usr/src/app/database/blogpostdemo.sqlite3: sync: new generation "983aaff5a14f0ec0", no generation exists
/usr/src/app/database/blogpostdemo.sqlite3(s3): snapshot written 983aaff5a14f0ec0/00000000
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
January 03, 2022 - 07:57:19
Django version 4.0, using settings 'web.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

Next head over to the service where you are storing your replicas and check that files have been written there. Im using Amazon S3 and I can see that a folder has been created and that there are files within that folder structure written at the time my serivce was launched.

S3 content

Thats it, now you are running Django with SQLite, all database changes are synced to your replica url.

The cool thing is that if your Docker container is destroyed (ie your database file is lost) then Litestream will automatically pull the latest version from your replica url the next time you start the service. This is all done in the web/entrypoint.sh file.

Can I add this to my existing Django project? #

Of course you can!

  • copy the Dockerfile into your Django app directory
  • copy the caddy directory and place it one level up from your Django app directory
  • copy the .env and docker-compose.yml files to the same level as the caddy directory.

So it should look something like this.

├── .env
├── caddy
│   ├── Caddyfile
│   └── site
├── docker-compose.yml
└── your_existing_django_app_directory
    ├── Dockerfile

Summary #

It was really fun to play around with Litestream and Im in the process of refactoring all of my sideprojects to use it instead of PostgreSQL.

Im also working on a post where I will describe how to create a CI/CD flow with GitHub actions that deploy the service to a DigitalOcean droplet.

Resources #

↑