Happy Birthday! 5 Years of Docker at Overleaf
John · March 21, 2018This week is Docker’s 5th birthday, so we thought we’d mark the occasion with a rundown of how we’ve been using this great technology at Overleaf. Overleaf also celebrated its 5th birthday at the start of this year, and we’ve been using Docker since its very early days.
Docker is an application that helps us safely run large numbers of programs on the same computer. Each program runs in its own ‘container’, isolated from all other programs. It works a bit like your smartphone does — you can install and run hundreds of apps, and they can’t interfere with each other or the core features of your phone.
Every time we compile your Overleaf project into a PDF, your project gets its very own Docker container. In every container there is a full installation of TeX Live, with over 4000 latex packages, thousands of fonts, and all of the supporting tools, such as pdflatex and bibtex. We then run latexmk in the container to automatically compile your project. The next time you compile, you get a new container, which helps ensure that you get consistent, repeatable results.
What makes containers remarkable compared to other technologies is how quickly all of this happens. The main alternative to running programs in containers is to run them in ‘virtual machines’, which run not just a single program in a container but a whole operating system. Just like booting up your computer, starting a virtual machine can take a while. With Docker, we can stand up a new container and tear it down all within 100ms — hundreds or even thousands of times faster than a virtual machine.
Under the hood, Docker is a user-friendly (for developers, that is) wrapper for some powerful and relatively new features of the Linux operating system kernel, mainly control groups, namespaces and the OverlayFS filesystem. Many of these underlying technical features were around in some form for years before, but Docker brought them together in a more approachable way and made them accessible to a much wider audience. I like to think that Overleaf is helping to do the same with LaTeX.
Another major application of Docker is to manage the environments that application developers use to build their software, mainly to make sure that all of the developers on a team have consistent development environment. Most companies that use Docker start by using it for their development environments and then gradually move to running services with Docker on their production servers. At Overleaf we did this a bit backward! We were running Docker in production for years before we got around to Dockerizing our development process in 2016.
One outcome of that effort was a blog post about the lessons we learned from trying to get some of our node.js applications running under Docker. That post proved very popular — it was translated into Japanese, and I was even invited to join the Docker Captains program, through which I’ve learned and shared a lot more Docker knowledge.
Since ShareLaTeX joined Overleaf last summer, Docker has taken on an even more important role at Overleaf. Whereas Overleaf has historically used a monolithic architecture with a single large, well-structured Ruby on Rails application and a small number of extracted services, ShareLaTeX uses a microservices architecture with dozens of small, loosely coupled services. Managing all these services natively proved difficult for the development team, especially after we doubled its size last summer. Now we have all the services running under Docker and supported by better tooling for managing them, which brought a major improvement to developer quality of life (though we still have further improvements we'd like to make).
As part of our integration of Overleaf and ShareLaTeX to produce Overleaf v2, we are also working toward running our microservices under Docker in production. This will mean running not only many programs on the same computer, but many, many programs on many computers! We are currently trialling the Kubernetes container orchestration system, which manages the allocation of containers to a cluster of servers.
I think we at Overleaf are very lucky to have started up alongside Docker just 5 short years ago, because we’ve found it useful in so many parts of our stack. It's been essential to Overleaf's growth into a community of over two million authors around the world. Many happy returns!
If Overleaf sounds like the kind of place you’d like to work, we’d love to hear from you. You can find out about our current vacancies on our jobs page.
Image credit: Docker Inc.