pump.io

Social server with an ActivityStreams API

This project is maintained by pump.io contributors

Blog

Show only 2016 2017 2018

Zero-downtime restarts have landed

I'm thrilled to announce that zero-downtime restarts, which I've been hacking on for the past week or two, have just landed in pump.io master!

Zero-downtime restarts require at least two cluster workers and MongoDB as a Databank driver (we'll eventually relax the latter requirement as we continue to test the feature). Here's how it works:

  1. An administrator sends SIGUSR2 to the master pump.io process (note that SIGUSR1 is reserved by Node.js)
  2. The master process builds a queue of worker processes that need to be restarted
  3. The master process picks a random worker from the queue and sends it a signal asking it to gracefully shut down
  4. The worker process shuts down its HTTP server, which causes it to stop accepting new connections - it will do the same for the bounce server, if applicable
  5. The worker shuts down its database connection once the HTTP server is completely shut down, meaning that it's done servicing in-flight requests
  6. The worker closes its connection with the master process and Node.js automatically terminates due to there being no listeners on the event loop
  7. The master recognizes the death of the worker process, replaces it, waits for the new worker to signal that it's listening for connections, and repeats from step 3 until the queue is empty

This works because only one worker is shut down at a time, allowing the other workers to continue servicing requests while the one worker is restarted. We wait until the new worker actually signals it's ready to process requests before beginning the process for another worker.

Such a feature requires careful error handling, so there are a lot of built-in checks to prevent administrators from shooting themselves in the foot:

  • If there's a restart already in progress, SIGUSR2 is ignored
  • If there's only 1 cluster worker, the restart request is refused (because there would be downtime and you should just restart the master)
  • The master process will load a magic number from the new code and compare it with the old magic number loaded when the master process started - if they don't match, SIGUSR2 will be refused. This number will be incremented for things that would make zero-downtime restarts cause problems, for example:

    • The logic in the master process itself changing
    • Cross-process logic changing, such that a new worker communicating with old workers would cause problems
    • Database changes
  • If a worker process doesn't shut itself down within 30 seconds, it will be killed
  • If a zero-downtime restart fails for any reason, the master process will refuse SIGUSR2 and will not respawn any more cluster workers, even if they crash - this is because something must have gone seriously wrong, either with the master, the workers, or the new code, and it's better to just restart everything. Currently this condition occurs when:

    • A new worker died directly after being spawned (e.g. from invalid JSON in pump.io.json)
    • A new worker signaled that it couldn't bind to the appropriate ports

While these checks do a lot to catch problems, they're not a silver bullet, and we strongly recommend that administrators watch their logs as they trigger restarts. However, this is still a huge win for the admin experience - the most exciting part of this for me is that it's the first step we need to take towards having fully automatic updates, which has been a dream of mine for a long while now.

Admins running from git master can start experimenting with this feature today, and it will be released during the next release cycle - i.e. with the 5.1 beta and stable, not the current 5.0 beta. Since this is highly experimental, we want this to have as much time for testing as possible. You can also check out the official documentation on this feature.

I hope people enjoy this! And as always, feel free to report any bugs.


pump.io 5.0 beta released

I'm excited to announce that pump.io 5.0.0 is now officially in beta!

This is another big release and makes a wide variety of improvements. Here are some highlights from the changelog:

  • More complete documentation
  • Small improvements to the administrator experience
  • A better web UI, including some user experience polishing as well as an upgrade to more performant and better-licensed libraries
  • A fix for crashes related to "login with remote account" (although this one was backported in 4.1.1)
  • Significant security improvements in the systemd service shipped with the package
  • Lots of internal refactoring and simplification made possible by dropping Node 0.10/0.12 support

Many of these changes - particularly the systemd changes and the fact that (as previously announced) Node 0.10 and 0.12 are no longer supported - will require administrator intervention. Be sure to read our upgrade guide for details on how to deal with these changes.

All of these features add up to make pump.io 5.0 beta the most stable and secure release yet. As always, it will go through our beta period for about a month before being released as a fully stable version. If you try it out, the community would love to hear about it - and be sure to report any bugs you encounter!


pump.io 4.1 is out

pump.io 4.1 is out!

As usual, nothing much has changed since the beta release. This release includes:

Remember that with this release, Node 0.10 and Node 0.12 support is officially obsolete. This change paves the way for a lot of really important improvements, and in fact, we've already introduced a lot of changes that reduce technical debt. Note that since the beta announcement, Debian Stretch has been released, which ships Node 4 in main. You can read more about our Node.js support policy on our wiki.

Enjoy the new release, and remember to report any bugs!


Out now: pump.io 4.1 beta

Perhaps some of you were wondering where pump.io's 4.1 release was, since our release schedule says it should've been released at the beginning of this month?

Well, wonder no longer. This release was unfortunately delayed because of some big stuff in my personal life that got in the way of free software stuff (sorry!), but as of yesteray, pump.io 4.1 is officially in beta! Huzzah!

This was a relatively minor release, primarily improving some aspects of the web UI. Notably, the pump.io 4.1 beta includes support for Subresource Integrity, a web standard which will improve security for users on nodes with "noCDN": false in their pump.io configurations. A more complete list of changes is available in the change log.

As always, we advise caution when upgrading to beta releases. Please report any problems or bugs you encounter.

pump.io 4.1 beta is a drop-in replacement for pump.io 4.0.

First time contributors

I would like to specially thank the following people, who contributed to the pump.io project for the first time during this release cycle:

  • Camilo QS, who contributed a number of improvements to the web UI
  • @bio-boris, who implemented Subresource Integrity for the web UI
  • Ryan Riddle, who made the web UI proactively warn if the user was trying to sign up with a blacklisted/reserved username (like api or robots.txt)

Node.js 0.10 and 0.12 support ending

Finally, the stable release pump.io 4.1 will mark the end of pump.io's support for Node.js 0.10 and 0.12. These versions of Node are extremely old and are preventing us from making serious improvements to the codebase - you can see a list of these in issue #1234. Note that in particular we cannot simultaneously support Node 7 and Node 0.10 due to our browser unit testing library, Zombie.

With the exception of Red Hat Enterprise Linux and CentOS, there are no major Linux distributions not shipping Node 4+ in some form. Most of them ship it out-of-the-box; the only one that does not is Debian. For our admins on Debian, Node 4 is available in jessie-backports and will become the officially supported version with the release of Debian Stretch (which is imminent). Plus, for all of these platforms, NodeSource offers binary packages for all major Node versions that can be installed and managed through the system package manager. We'll also soon be making a Docker image available, which will allow people to run pump.io on any Docker-capable host, regardless of what Node version is shipped with their distribution.

You can read more about our Node.js version support policy on our wiki.

Here's to another pump.io beta!


pump.io XSS security releases available

Well, apparently I forgot to make a blog post announcing that pump.io 4.0.0 stable is out. Surprise! Besides the version number it's the same as 4.0 beta 5 anyway.

What is not the same as 4.0 beta 5, however, is the security releases I've just published. A semver-major release went out within the past 6 months, so per our security support policy, we've released security patches for the past three stable releases:

  1. pump.io 4.0.0 has been updated to pump.io 4.0.1
  2. pump.io 3.0.2 has been updated to pump.io 3.0.3
  3. pump.io 2.1.1 has been updated to pump.io 2.1.2

So what exactly required these releases? Well, the library we use to prevent cross-site scripting, DOMPurify, released some security patches recently. While we could conceivably just tell pump.io users to rerun npm install -g to get the updated library, it seemed safer to issue patch releases that bumped the minimum version for DOMPurify and have people upgrade to those. This gives a 100% guarantee that pump.io users will be protected with the absolute latest DOMPurify version.

As with any security release, we encourage admins to upgrade ASAP. If you've already installed the 4.0 release via npm, that's great! That's our recommended configuration, and you'll be able to upgrade by issuing:

$ npm install -g pump.io@4

If you're still on the 3.x release series, you really should upgrade to 4.x, which contains significant work impacting security which was too big to go into a patch release (specifically, the upgrade to Express 4.x). But in the meantime, you can fix your XSS problems by running:

$ npm install -g pump.io@3

And something similar if you're on 2.1.x:

$ npm install -g pump.io@2

Though in this case it's even more urgent that you upgrade to 4.x.

Note that the above comands also assume you have an npm-based install, which we strongly recommend. If you have a source-based install, you will need to upgrade however you usually do - this will depend on how exactly you have pump.io set up.

As always, if you need help, get in touch with the community.