pump.io

I'm really excited to announce that pump.io 5.1 is officially in beta!

This release makes a wide array of improvements. One of the features I'm most excited about is zero-downtime restarts, which will allow administrators to gracefully roll over to new configurations and codebases with absolutely no user-visible impact! Aside from that, here's a list of highlights:

• The daemon now generates startup log warnings on bad configurations, including insecure secret values and internal parameters
• An official Dockerfile is now included with the release
• The logged-out mobile homepage's menu icon is no longer incorrectly styled as black
• SockJS connections no longer fail due to authorization problems

As usual, this release also includes some small updates to dependencies. Plus, we started tracking how much code our test suite covers! We've also significantly cleaned up the documentation and moved almost all of the in-repository documentation to ReadTheDocs, where it's a lot more organized and accessible. If you want more than just these highlights, you can check out the changelog.

Note that while the repository includes a Dockerfile now, we have not yet begun publishing on Docker Hub due to some security logistics that need to be figured out - when we do, it'll be announced in all the usual places (including this blog).

I'm so excited for people to try out this release. 5.1 beta 0 is a drop-in replacement for 5.0, so if you're on 5.0 and want to live (more) on the bleeding edge, you can upgrade with npm install -g pump.io@5.1 if you have an npm-based install. If you have a source-based install, you should merge and/or switch to the v5.1.0-beta.0 tag. And as always, if you encounter any problems you can reach out to the community or file bugs you find.

Recently some denial-of-service vulnerabilities were discovered in various modules that we indirectly depend on. I've bumped Express and send to pull in patched versions, and I've updated our fork of connect-auth to require a patched version of Connect, too. The remaining vulnerabilities I've confirmed don't affect us.

Because of these version bumps, I've just put out security releases which all administrators are encouraged to upgrade to as soon as possible. A semver-major release (5.0.0) was released within the past 6 months so per our security support policy this means there are three new releases:

1. pump.io 5.0.2 replaces 5.0.0 and is available now on npm
2. pump.io 4.1.3 replaces 4.1.2 and is available now on npm
3. pump.io 4.0.2 will replace 4.0.1 and is currently undergoing automated testing (it'll be on npm shortly) Update: pump.io 4.0.2 is now on npm

As these are security releases we encourage admins to upgrade as soon as possible. If you're on 5.0.0 installed via npm - our recommended configuration - you can upgrade by issuing:

$ npm install -g pump.io@5

If you're on 4.1.3, you can upgrade by issuing:

$ npm install -g pump.io@4

And when 4.0.2 is out, if you're on 4.0.1 you can upgrade by issuing:

$ npm install -g pump.io@4.0

Note though that 4.1.3 is a drop-in replacement for 4.0.2, so you should consider just upgrading to that instead. Or even better, upgrade to 5.x!

If you don't have an npm-based install, you'll have to upgrade however you normally do. How to do this will depend on your particular setup.

As always, if you need help, you should get in touch with the community. I'd also like to specifically thank Jason Self, who generously deployed a 24-hour private beta of these fixes on Datamost. One of the version bumps was ever-so-slightly risky, and being able to test things in production before rolling out patches for the entire network was invaluable. I wouldn't be as confident as I am in these releases without his help. So thanks, Jason - I really appreciate it!

I'm super excited to announce that as of a couple days ago, pump.io 5.0 has now been declared stable and released!

This release cycle had an extra beta since I found a bug with the new display of shares. The fix ended up being trivial, though, and other than that there's been no changes from the beta announcement:

• Documentation has been expanded
• Small improvements to the administrator experience have landed
• The web UI has gotten, among other things, some user experience polishing as well as upgrades to more performant and better-licensed libraries
• "Login with remote account" no longer crashes (although this one was backported in 4.1.1)
• The systemd service shipped with the package has significant security improvements
• Lots of internal refactoring and simplification made possible by dropping Node 0.10/0.12 support

As I said in the beta announcement, some of these changes - particularly the systemd changes and the fact that (as previously announced) Node 0.10 and 0.12 are no longer supported - will require administrator intervention. Be sure to read our upgrade guide for details on how to deal with these.

pump.io 5.0 is the most stable and secure release yet, so as always, I'd encourage all administrators to upgrade as soon as possible. And if you get stuck, the community is always here to help.

Cheers!

I'm thrilled to announce that zero-downtime restarts, which I've been hacking on for the past week or two, have just landed in pump.io master!

Zero-downtime restarts require at least two cluster workers and MongoDB as a Databank driver (we'll eventually relax the latter requirement as we continue to test the feature). Here's how it works:

1. An administrator sends SIGUSR2 to the master pump.io process (note that SIGUSR1 is reserved by Node.js)
2. The master process builds a queue of worker processes that need to be restarted
3. The master process picks a random worker from the queue and sends it a signal asking it to gracefully shut down
4. The worker process shuts down its HTTP server, which causes it to stop accepting new connections - it will do the same for the bounce server, if applicable
5. The worker shuts down its database connection once the HTTP server is completely shut down, meaning that it's done servicing in-flight requests
6. The worker closes its connection with the master process and Node.js automatically terminates due to there being no listeners on the event loop
7. The master recognizes the death of the worker process, replaces it, waits for the new worker to signal that it's listening for connections, and repeats from step 3 until the queue is empty

This works because only one worker is shut down at a time, allowing the other workers to continue servicing requests while the one worker is restarted. We wait until the new worker actually signals it's ready to process requests before beginning the process for another worker.

Such a feature requires careful error handling, so there are a lot of built-in checks to prevent administrators from shooting themselves in the foot:

• If there's a restart already in progress, SIGUSR2 is ignored
• If there's only 1 cluster worker, the restart request is refused (because there would be downtime and you should just restart the master)

• The master process will load a magic number from the new code and compare it with the old magic number loaded when the master process started - if they don't match, SIGUSR2 will be refused. This number will be incremented for things that would make zero-downtime restarts cause problems, for example:

  • The logic in the master process itself changing
  • Cross-process logic changing, such that a new worker communicating with old workers would cause problems
  • Database changes
• If a worker process doesn't shut itself down within 30 seconds, it will be killed

• If a zero-downtime restart fails for any reason, the master process will refuse SIGUSR2 and will not respawn any more cluster workers, even if they crash - this is because something must have gone seriously wrong, either with the master, the workers, or the new code, and it's better to just restart everything. Currently this condition occurs when:

  • A new worker died directly after being spawned (e.g. from invalid JSON in pump.io.json)
  • A new worker signaled that it couldn't bind to the appropriate ports

While these checks do a lot to catch problems, they're not a silver bullet, and we strongly recommend that administrators watch their logs as they trigger restarts. However, this is still a huge win for the admin experience - the most exciting part of this for me is that it's the first step we need to take towards having fully automatic updates, which has been a dream of mine for a long while now.

Admins running from git master can start experimenting with this feature today, and it will be released during the next release cycle - i.e. with the 5.1 beta and stable, not the current 5.0 beta. Since this is highly experimental, we want this to have as much time for testing as possible. You can also check out the official documentation on this feature.

I hope people enjoy this! And as always, feel free to report any bugs.

I'm excited to announce that pump.io 5.0.0 is now officially in beta!

This is another big release and makes a wide variety of improvements. Here are some highlights from the changelog:

• More complete documentation
• Small improvements to the administrator experience
• A better web UI, including some user experience polishing as well as an upgrade to more performant and better-licensed libraries
• A fix for crashes related to "login with remote account" (although this one was backported in 4.1.1)
• Significant security improvements in the systemd service shipped with the package
• Lots of internal refactoring and simplification made possible by dropping Node 0.10/0.12 support

Many of these changes - particularly the systemd changes and the fact that (as previously announced) Node 0.10 and 0.12 are no longer supported - will require administrator intervention. Be sure to read our upgrade guide for details on how to deal with these changes.

All of these features add up to make pump.io 5.0 beta the most stable and secure release yet. As always, it will go through our beta period for about a month before being released as a fully stable version. If you try it out, the community would love to hear about it - and be sure to report any bugs you encounter!