A robust distributed locking algorithm based on Google Cloud Storage

Many workloads nowadays involve many systems that operate concurrently. This ranges from microservice fleets to workflow orchestration to CI/CD pipelines. Sometimes it's important to coordinate these systems so that concurrent operations don't step on each other. One way to do that is by using distributed locks that work across multiple systems.

Distributed locks used to require complex algorithms or complex-to-operate infrastructure, making them expensive both in terms of costs as well as in upkeep. With the emergence of fully managed and serverless cloud systems, this reality has changed.

In this post I'll look into a distributed locking algorithm based on Google Cloud. I'll discuss several existing implementations and suggest algorithmic improvements in terms of performance and robustness.

Read more »

Docker and the host filesystem owner matching problem

Containers are no longer only used on servers. They are increasingly used on the desktop: as CLI apps or as development environments. I call this the "container-as-OS-app" use case. Within this use case, containerized apps often generate files that are not owned by your local machine's user account. Sometimes they can't access files on the host machine at all. This is the host filesystem owner matching problem.

  • This is bad for security. Containers shouldn't run as root in the first place!
  • This is a potential productivity killer. It's annoying having to deal with wrong file permissions!

Solutions are available, but they have major caveats. As a result it's easy to implement a solution that only works for some, but not everyone. "It works on my machine" is kind of embarrassing when you distribute a development environment to a coworker, who then runs into issues.

This post describes what causes the host filesystem owner matching problem, and analyzes various solutions and their caveats.

Read more »

Traveling Ruby 20210206: maintenance update featuring Ruby 2.4

Traveling Ruby allows you to easily ship Ruby apps to end users. It lets you create self-contained Ruby app packages that run on multiple versions of Windows, Linux and macOS.

Today I’ve released version 20210206. This release supports Ruby 2.4, bumps all the gem versions, bumps the minimum supported macOS and Linux versions, and fixes some bugs.

It has been a long time since the last release. So this post also adresses an elephant in the room: is Traveling Ruby back?

Read more »

An alternative to macOS's DYLD_LIBRARY_PATH

In my last blog post about Traveling Ruby's future, I said that it's hard to democratize the development of Traveling Ruby because of System Integrity Protection (SIP). Traveling Ruby's build process relies on DYLD_LIBRARY_PATH, which is blocked by SIP. This means that:

  • Contributors that build Traveling Ruby on their own laptops, must disable SIP.
  • Traveling Ruby cannot be built on many CI hosting services, such as Azure DevOps and Github Actions, because it's not possible to disable SIP there.

After some research and experimentation, I've found an alternative to DYLD_LIBRARY_PATH, meaning that it's no longer necessary to disable SIP. This significantly changes the ability to democratize Traveling Ruby's development.

Read more »

The future of Traveling Ruby

A couple of years ago, I had a dream: to make it dead-easy to distribute Ruby CLI apps to end users, without requiring those users to install Ruby or muck about with gems and Bundler. And thus Traveling Ruby was born.

Traveling Ruby hasn't seen updates for quite a while now. Recently I tried making a new bugfix release, but I found it to be more challenging than I had hoped. In this article I reflect on those challenges, as well as on the future of Traveling Ruby.

Read more »

How Debian packaging works

Debian packaging can be quite mysterious and hard to figure out. In this guide I'll provide a simple introduction into the Debian packaging process and its most important concepts.

This is not a full guide into all aspects of packaging. Instead, I'll cover just enough of the basics to help you develop a mental model of what Debian packaging is about, and to be able to produce useful results.

I'll cover:

  • What a Debian package is.
  • The anatomy of a package.
  • How to inspect a package.
  • How to create a package.
  • What APT repositories are.
  • How to create an APT repository.

Read more »

Why piping to 'cat' can stuck a CI job 🙀

What's the difference between command and command | cat? There shouldn't be any, right? The first prints the output directly, and the latter prints output via cat, but they should have the same effect. Not so: the latter can cause the command to get stuck indefinitely. This has given a particular CI pipeline of mine quite some headache.

To learn why this happens, and how we can mitigate this problem, we need to dive into the arcane magic that is Unix process management. Join me on this journey.

Read more »

Why Fullstaq Ruby?

A while ago, the people from Fullstaq and I started the Fullstaq Ruby project: a Ruby distribution that's optimized for server use cases. Compared to normal MRI Ruby, Fullstaq Ruby uses 50% less memory, is faster, and is easier to install and security-patch because of RPM and DEB packages.

Since I announced Fullstaq Ruby on EuRuKo 2019, I have received many questions about Fullstaq Ruby's vision, purpose and nature:

  • Is Fullstaq Ruby a commercial product, or are there such plans?
  • How will Fullstaq Ruby stay maintained?
  • Who is in control of Fullstaq Ruby?
  • Why are the changes in Fullstaq Ruby not in upstream Ruby? What is the current, and envisioned, relationship between the Ruby core developers and Fullstaq Ruby?

In other words, people are wondering: "how do I know this is, and stays, a real thing that I can count on?"

These are legit questions! As the author of Passenger and Ruby Enterprise Edition, I've experienced first-hand what the challanges are of building a healthy open source project. In this post, I will describe my vision on this matter.

Read more »
Page 1 of 5