pump.io

Social server with an ActivityStreams API

This project is maintained by pump.io contributors

Blog

Show only 2016 2017

pump.io 4.1 is out

pump.io 4.1 is out!

As usual, nothing much has changed since the beta release. This release includes:

Remember that with this release, Node 0.10 and Node 0.12 support is officially obsolete. This change paves the way for a lot of really important improvements, and in fact, we've already introduced a lot of changes that reduce technical debt. Note that since the beta announcement, Debian Stretch has been released, which ships Node 4 in main. You can read more about our Node.js support policy on our wiki.

Enjoy the new release, and remember to report any bugs!


Out now: pump.io 4.1 beta

Perhaps some of you were wondering where pump.io's 4.1 release was, since our release schedule says it should've been released at the beginning of this month?

Well, wonder no longer. This release was unfortunately delayed because of some big stuff in my personal life that got in the way of free software stuff (sorry!), but as of yesteray, pump.io 4.1 is officially in beta! Huzzah!

This was a relatively minor release, primarily improving some aspects of the web UI. Notably, the pump.io 4.1 beta includes support for Subresource Integrity, a web standard which will improve security for users on nodes with "noCDN": false in their pump.io configurations. A more complete list of changes is available in the change log.

As always, we advise caution when upgrading to beta releases. Please report any problems or bugs you encounter.

pump.io 4.1 beta is a drop-in replacement for pump.io 4.0.

First time contributors

I would like to specially thank the following people, who contributed to the pump.io project for the first time during this release cycle:

  • Camilo QS, who contributed a number of improvements to the web UI
  • @bio-boris, who implemented Subresource Integrity for the web UI
  • Ryan Riddle, who made the web UI proactively warn if the user was trying to sign up with a blacklisted/reserved username (like api or robots.txt)

Node.js 0.10 and 0.12 support ending

Finally, the stable release pump.io 4.1 will mark the end of pump.io's support for Node.js 0.10 and 0.12. These versions of Node are extremely old and are preventing us from making serious improvements to the codebase - you can see a list of these in issue #1234. Note that in particular we cannot simultaneously support Node 7 and Node 0.10 due to our browser unit testing library, Zombie.

With the exception of Red Hat Enterprise Linux and CentOS, there are no major Linux distributions not shipping Node 4+ in some form. Most of them ship it out-of-the-box; the only one that does not is Debian. For our admins on Debian, Node 4 is available in jessie-backports and will become the officially supported version with the release of Debian Stretch (which is imminent). Plus, for all of these platforms, NodeSource offers binary packages for all major Node versions that can be installed and managed through the system package manager. We'll also soon be making a Docker image available, which will allow people to run pump.io on any Docker-capable host, regardless of what Node version is shipped with their distribution.

You can read more about our Node.js version support policy on our wiki.

Here's to another pump.io beta!


pump.io XSS security releases available

Well, apparently I forgot to make a blog post announcing that pump.io 4.0.0 stable is out. Surprise! Besides the version number it's the same as 4.0 beta 5 anyway.

What is not the same as 4.0 beta 5, however, is the security releases I've just published. A semver-major release went out within the past 6 months, so per our security support policy, we've released security patches for the past three stable releases:

  1. pump.io 4.0.0 has been updated to pump.io 4.0.1
  2. pump.io 3.0.2 has been updated to pump.io 3.0.3
  3. pump.io 2.1.1 has been updated to pump.io 2.1.2

So what exactly required these releases? Well, the library we use to prevent cross-site scripting, DOMPurify, released some security patches recently. While we could conceivably just tell pump.io users to rerun npm install -g to get the updated library, it seemed safer to issue patch releases that bumped the minimum version for DOMPurify and have people upgrade to those. This gives a 100% guarantee that pump.io users will be protected with the absolute latest DOMPurify version.

As with any security release, we encourage admins to upgrade ASAP. If you've already installed the 4.0 release via npm, that's great! That's our recommended configuration, and you'll be able to upgrade by issuing:

$ npm install -g pump.io@4

If you're still on the 3.x release series, you really should upgrade to 4.x, which contains significant work impacting security which was too big to go into a patch release (specifically, the upgrade to Express 4.x). But in the meantime, you can fix your XSS problems by running:

$ npm install -g pump.io@3

And something similar if you're on 2.1.x:

$ npm install -g pump.io@2

Though in this case it's even more urgent that you upgrade to 4.x.

Note that the above comands also assume you have an npm-based install, which we strongly recommend. If you have a source-based install, you will need to upgrade however you usually do - this will depend on how exactly you have pump.io set up.

As always, if you need help, get in touch with the community.


pump.io 4.0 in beta

pump.io 4.0.0 is officially in beta! Whooo!

Highlights

This is a positively huge release, and I'm so excited to share it with the community. Some highlights:

  • Express 4.x - I wrote about the significance of this change here, but suffice to say that this significantly improves security, performance, and future maintainability
  • Performance and correctness improvements to the web UI's JavaScript
  • Better administrative experience, including the ability to specify configuration via environment variables
  • Better interoperability with the IndieWeb

Upgrading

The upgrade to Express 4.x and the improvements to configuration loading have the potential to break some existing pump.io installations, although 95% of installs should be completely unaffected. If you want to help test this beta, please set aside extra time as necessary to perform this upgrade - full documentation can be found on ReadTheDocs.

As always, this release will follow our normal release cycle, which means that the stable 4.0.0 release will go out in about a month.

Test days

Due to the complexity of this upgrade, we've decided to have some test days during the beta where we upgrade prominent nodes for a day, then downgrade them again. This will help expose problems earlier and make the upgrade smoother for everyone. So far Jason Self, who runs Datamost, has volunteered for this - if you're interested in joining him, please get in touch!

Happy hacking!


Dianara v1.3.7 released

A new version of Dianara, my desktop client for the Pump social network, has just been released. This will be the last of the 1.3.x series, and the last one to support Qt 4.

The image viewer

It’s another small release. The biggest change is the updated image viewer, which now supports zooming with the mouse/touchpad, and dragging the image around, as full-blown image viewers usually support ;)

If you want to build it with Qt 5.x, note that your system will need a Qt 5 build of the QOAuth library, which not all GNU/Linux distributions provide yet. Mageia 6 does, and so do Archlinux, openSUSE (in a separate repository) and Debian (in Experimental). If you have that, but experience problems building it this way, your QOAuth might be missing a feature file (.prf), like used to be the case in Debian.

Changelog

The changes since v1.3.6 are:

  • The image viewer now supports dragging the image around with the mouse, zooming with the wheel, and rotating animated images correctly.
  • Fixed case-insensitive sorting of contacts in auto-completion lists, when typing @ while creating a post. This means that “Person A” will appear next to “person b”, not after “Z-contact”.
  • Your server’s version will be shown in the log.

Express 4.x in pump.io core

So I thought I'd take a moment to announce that the upgrade from Express 2.x to Express 4.x is finally complete! I fixed up the last couple test failures last Wednesday, and the branch got merged on Thursday.

A long time coming

Believe it or not, the work to do this upgrade started almost an entire year ago. Express 2.x has been outdated and unmaintained for a long time now, so upgrading has been a high priority. However, it wasn't as simple as adjusting a version number - there were a staggering number of changes that needed to be made due to Express deprecating, removing, and changing things around. One of the most significant problems was the fact that the old template system that we used, utml, was not compatible with Express 3.x and above. That meant that we had to rewrite every single template into a modern language - an effort that resulted in over a thousand lines changed!

However, the time for Express 4.x has finally arrived. With that and some other trivial version bumps, I'm proud to announce that pump.io is fully up-to-date in terms of dependencies with only three, non-critical exceptions. Whooooo!

Immediate benefits

There are a lot of reasons this is immediately awesome:

  1. Express 4.x fixes significant performance problems that existed in Express 3.x
  2. Relatedly, Express 4.x fixes some security problems present in 3.x
  3. The fact that our dependencies are finally up-to-date means that we can (and do!) now make use of Greenkeeper and the Node Security Platform to automatically track dependencies to make sure they're up-to-date and not introducing security vulnerabilities

That last one is particularly significant. Greenkeeper and NSP will continuously monitor the project's dependencies and automate away a lot of the pain that's associated with keeping pump.io up-to-date. Everyone will get a more secure and stable codebase because of this setup.

Looking forward

The Express 4.x upgrade is a big change, and it's definitely possible that stuff has broken. We want to make sure that breakage doesn't make it into production. This change went into pump.io 4.0, which will go through our normal release cycle. That means it'll be in beta for a month before being released. As a part of that, Jason Self - who's kind enough to administer Datamost - has agreed to have a test day where Datamost upgrades to the beta for a day, then downgrade it again. This test day will give us much wider exposure than we would've gotten otherwise, which will be incredibly valuable feedback in the effort to identify and fix regressions. We haven't set a date yet, but if you'd like to join Jason in helping us find bugs, please get in touch with the community. We'd love your help.

Beyond the immediate release, though, there's still things to look forward to. Express 4.x gives us a better way to structure routing code, and a refactor to use this structure is planned. There's a lot of room for improvement. But really, the most important benefit is this: technical debt is a far less pressing issue than before. That means that we can shift focus and spend more time fixing user-facing bugs, adding useful features, and generally improving the experience for our users. I couldn't be more excited.


Pump.io 2.0.4 is available

Greetings!

After a beta period of just over a week, pump.io 2.0.4 is now available on npm and GitHub. Whoohoo!

(This was originally going to be 2.0.0, but we had to do a couple patch releases due to some outdated documentation and several critical bugs. 2.0.4 is mostly the same thing as 2.0.0.)

Changes

Note that this release includes security improvements - namely, a newer Express version and a better TLS configuration - and therefore admins are encouraged to upgrade ASAP.

For the full list of changes, see the change log.

Breaking changes

(As I said in the beta announcement:)

Pump.io 2.0.4 is a drop-in replacement for 1.0.0 unless you have any plugins configured or you modify the templates.

Plugins are likely to be affected by the upgrade to Express 3.x. The easiest way to migrate is probably to just run pump.io, test out the relevant parts of the app, and see where your plugin crashes. You might also want to look at the Express 3 change log.

If you modified the templates, you'll be affected by the templates' rewrite from utml into Jade. Migration should be relatively painless but has to be done manually. Your best bet will be to save a copy of the diff you created, undo your changes, upgrade, then use the diff you saved to reintroduce your changes. You'll have to run npm run build after making changes to Jade files.

Non-breaking changes

This release is actually relatively minor in terms of non-breaking changes; however, we do have some nice new improvements:

  • A pump(1) manpage is now included
  • Any internal web UI link with a data-bypass attribute is now ignored by the routing logic (useful for e.g. custom pages added by the admin)
  • YouTube links in posts are now shown as embeds by the web UI (#1158)
  • TLS connections now use Mozilla's "intermediate" cipher suite and forces server cipher suite preferences (#1061)
  • Various minor fixes and improvements

Upgrading

Upgrading is dead-simple. If you used our recommended install method, and installed from npm, you can upgrade with:

sudo npm install -g pump.io@2

If you installed from source, you can upgrade with:

git fetch
# If you modified templates, save the diff at this step
git checkout .
git checkout v2.0.4
npm install
# Restore your template changes
npm run lint:jade # Optional but recommended if you changed templates
npm run build

Both of these methods will work whether you're running 0.3.0, 1.0.0, or 2.0.0 beta. Make sure to restart pump.io after performing the upgrade.

Getting help

If you have any issues with the upgrade, get in touch with the community. You can also email me at alex@strugee.net.


Pump.io 2.0.0 beta is here

Greetings!

As pump.io is gearing up for our 2.0.0 release, I just published a beta to npm. We'd love it if any interested admins could upgrade their nodes and report any bugs you may run in to. (All bug reports are helpful, but just so we're clear, we're likely to only fix regressions from 1.0.0.)

Changes

Pump.io 2.0.0 beta is a drop-in replacement for 1.0.0 unless you have any plugins configured or you modify the templates.

Plugins are likely to be affected by the upgrade to Express 3.x. The easiest way to migrate is probably to just run pump.io, test out the relevant parts of the app, and see where your plugin crashes. You might also want to look at the Express 3 change log.

If you modified the templates, you'll be affected by the templates' rewrite from utml into Jade. Migration should be relatively painless but has to be done manually. Your best bet will be to save a copy of the diff you created, undo your changes, upgrade, then use the diff you saved to reintroduce your changes. You'll have to run npm run build after making changes to Jade files.

For the list of non-breaking changes, see the change log.

Upgrading

Upgrading is very easy. If you used our recommended install method, and installed from npm, you can upgrade to the beta with:

sudo npm install -g pump.io@beta

If you installed from source, you can upgrade with:

git fetch
# If you modified templates, save the diff at this step
git checkout .
git checkout v2.0.0-beta.1
npm install
# Restore your template changes
npm run lint:jade # Optional but recommended if you changed templates
npm run build

Make sure to restart pump.io after performing the upgrade.

Getting help

If you have any issues upgrading to or running the beta, please don't hesitate to get in touch with the community. You can also email me at alex@strugee.net.


Pump.io 1.0.0 is now available!

Pump.io 1.0.0 is officially available! Whoooo!

I just wanted to write up an announcement real quick to celebrate. Here's a sample what's gone into this release:

  • Node 4.x support
  • Lots of security improvements including a better cross-site scripting scrubber and security-related headers that help protect the web UI (most notably, the web UI now declares a Content Security Policy)
  • Minor improvements to the API to make it (slightly) smarter
  • LibreJS support
  • Numerous dependency upgrades, most notably Connect
  • And of course, tons of minor bugfixes and improvements

For more details, see the brand-new change log.

And of course since we're now past 0.x.x releases, we're now officially making a commitment to the community to make only API-compatible changes going forward (or at least, until 2.0.0!).

As this release does improve security and fixes a lot of bugs, node administrators are encouraged to upgrade as soon as possible. If you have a global, npm-based install, you can upgrade with:

sudo npm install -g pump.io

And with a source-based install:

git pull
git checkout v1.0.0
npm install --production

If you're upgrading from 0.3.0, everything should Just Work(tm). Don't forget to restart your daemon!

One final note - the rumors are true. While we're not doing so yet, we are, in fact, planning to deprecate running under Node.js 0.10 and 0.12 very soon. Also, if you upgrade to Node.js 4.x early, the new, better XSS scrubber will be enabled - however, be aware that pump.io is far less tested under Node.js 4.x and you are likely to run into more bugs than you would under 0.10 or 0.12. This is an unfortunate situation, but sadly there's really nothing to be done about it. :(

Special thanks to Menno Vossen, Laura Arjona, Evan Prodromou, Jan Kusanagi and all the other volunteers who did so many different things to make this release happen. It truly wouldn't have happened without you.

Enjoy the release!

With <3,

AJ


Pump.io: call for testers, call for feedback

So in my last post, I mentioned that I'd left a couple things for a second blog post. This is that post, and instead of being about all the cool stuff going on in the pumpiverse, it's about stuff that you - yes, you! - can do! It's super easy too.

Call for testers

So as I mentioned, the utml-to-jade branch is basically finished (see PR #1170). Since switching templating languages is a huge, huge change, by definition touching every single part of the Web UI, we want to make sure it's well-tested. This is especially critical given the fact that the Web UI unfortunately has very little test coverage.

That's where pump.io system administrators come in. If you're a sysadmin and you're willing to test this change on your node, we'd very much appreciate it. There's very little risk, since I think I've squashed all the regressions that happened, but you should be willing to report bugs if you do run into them. In particular, you should look out for:

  1. Links that seem to encompass too much text
  2. Missing spaces - e.g. Alex Jordanat [date] instead of Alex Jordan at [date]
  3. HTML code showing up on the page - e.g. Test note<br /> instead of Test note followed by a line break

Note that the utml-to-jade branch incorporates all changes in the master branch, so you may want to check out the advice in Running from Git master.

Sound interesting? Want to take part in the development of pump.io? Installing is super easy:

$ [sudo] npm install -g e14n/pump.io#utml-to-jade

This will work even if you already have a (non-source) install of pump.io - just make sure to restart the server afterwards.

Note that this command has some semi-terrible logic to build Jade templates on install (this is a workaround for a deficiency in npm). If you get a scary warning message from npm, please file an issue, making sure to include the full log.

Call for design feedback

The other big thing that's happening is the pump.io 1.0.0 tshirt we're designing! I've spent quite a bit of time working on a variety of candidate designs which can be viewed in this ownCloud share. Obviously we want the coolest tshirt possible, so we're looking for any design feedback that people have. Anyone with some spare time can glance through the designs, and we'd be thrilled to get everyone's opinions.

If this sounds interesting, I'd welcome you to check out the drafts. As always, get in touch with the community through our chatroom or if you'd prefer, you can email me directly at alex@strugee.net.

Thanks so much! :)


New stuff in pump.io

So I promised a (long) while ago that I'd blog about all the stuff going on in pump.io. And there is a lot going on. Where to even begin?

LFNW talk

I think the first thing I should mention is the talk I gave at LinuxFest Northwest this year. It went really, really well (even though I finished half the slides the night before), and people seemed to be really engaged, especially during questions. It starts off by covering the history behind pump.io: that includes the relevant protocols, like OStatus and ActivityStreams, but also the motivation behind abandoning StatusNet (aka current GNU Social) in favor of a brand-new network. Next I move on to the way that pump.io itself works, namely, its function as a generic ActivityStreams distribution engine. To put this another way, I explain why this quote from the README:

I post something and my followers see it. That's the rough idea behind the pump.

is a pretty accurate way of describing what pump.io actually does. (I quote that quite a few times in the slides themselves.) I end with a discussion of the recent developments in the community, which are of course wonderful, and a short call-to-action for people to contribute - either directly to the software, or by joining the network and spreading pump.io.

Oh, and by the way - the talk was recorded! So you can watch it on YouTube or, even better, on my personal MediaGoblin. Fitting, since (as I mention in the talk) MediaGoblin federation will soon be released, and it's based on (and fully interoperable with) the pump.io protocol!

Conservancy application

Pump.io is applying to the Software Freedom Conservancy! This is super fantastic for us for a number of reasons. One of the biggest advantages is the fact that inside Conservancy (assuming our application is accepted), we'll be able to take donations much, much easier. This is primarily important because nodes in the pump.io network are volunteer-run, but they still require funds to operate. We're thinking about models wherein people will be able to donate to "pump.io", and then some (most?) of those funds will be used to pay for the network. (In particular, they'll be used to pay for the existing E14N nodes that Evan currently runs, which will become extremely useful once we transition those nodes to community owners). Conservancy also provides useful miscellaneous services, like owning our logo and making sure that if we encounter license violations, the license is properly enforced. But perhaps most importantly, becoming part of Conservancy cements pump.io even more in the community - both the group of people working on the software & network, but also the larger free software world.

ReadTheDocs

We've started a dedicated space for project documentation, hosted on ReadTheDocs. We're plannning to move a bunch of content from the GitHub wiki into this project, and hopefully it'll become a thorough and central place for pump.io documentation - both for users and for deployers.

Triage

We've spent quite a bit of time going through open issues and prioritizing them. A lot of issues have a release target now, and it feels really nice to feel like our issue tracker is a bit more organized.

Special thanks to Laura Arjona for driving this work.

Various minor code improvements

There have been a bunch of small bugfixes and improvements that have gone into the master branch - some of them user-facing, and some of them making the development experience better. Notable changes include:

  • Migrating from Connect 1.x to Connect 2.x (this is just a start - Connect 2.x is still deprecated, but it gets us closer than we were to relying on a fully non-deprecated stack)
  • JSHint is now automatically run against bin/ and routes/ when npm test is run. This makes it super obvious when there are regressions in code quality, especially in Pull Requests (since Travis CI will fail if JSHint doesn't succeed.)
  • JSCS is now used to enforce code style. It's automatically run against the entire codebase (whoo!) when you run npm test, and it's awesome for the same reason - much of the style-related feedback that would've previously ended up in a Pull Request can now be dealt with directly on a local development machine, reducing PR review time for both the reviewer and the contributor.
  • LibreJS is now supported
  • Tests now pass! Whooooooooo! (Thanks to Menno Vossen for sending the enormous Pull Request that made this happen.)

Those are just the bigger ones, of course - there are a bunch of even smaller problems that got squashed as well. I'd also like to point out that quite a few of these were long-standing PRs which finally made it into core, which is awesome for everyone.

Express 3.x migration

I've been putting in a lot of work to migrate pump.io to Express 3.x. It's a huge amount of work, but when complete, it will bring us very, very close to being able to migrate onto Express 4.x, which is modern and fully-supported by upstream. Basically what I've been doing is just running the app, seeing where it crashes, going to the exception site, and fixing the problem. Rinse, repeat. You can check out this work on the express-3.x branch - currently, this branch can successfully start up the app, but will crash pretty soon after you try to do almost anything else.

This work, unfortunately, is on hold while another important project is completed: converting all the templates from utml to Jade.

utml to Jade transition

This is basically what it sounds like. Previously, the templates in pump.io were based on utml, which is essentially a thin wrapper around Underscore.js's _.template() function. However, utml doesn't work with Express 3.x (and it's not really worth making it work), plus it's not the prettiest to work with. Jade is an extremely popular templating language in Node-land nowadays, so a couple months ago I spent somewhere between 14 and 18 hours going through and rewriting all the utml in Jade, which was absolutely brutal - but necessary. Then, of course, I had to fix the client-side templating logic to handle Jade instead of Underscore templates, which took quite a while, along with the fact that I made a very large number of minor (largely cosmetic) errors in my conversions.

As I said above, this was kind of awful work (especially the beginning), but it's necessary and great, as it paves the way for Express 3.x and massively improves the contributor experience.

You can check out the gory details of this work in PR #1170, and the original reasoning behind why we're doing this in issue #1167. This work is actually done, but I'm going to write a separate blog post about it, calling for testers.

Upcoming 1.0.0 release

Last but certainly not least, we're gearing up for our 1.0.0 release! From a codebase standpoint, this is really just a small bugfix release (although it will make a lot of things less broken and - if I recall correctly - fix the actual installation process), but more importantly, it means that we're now committing to semantic versioning, which is a win for everyone (but especially administrators). The main thing that needs to be fixed before this goes out the door is the behavior of the XSS scrubber, which was accidentally made a little too aggressive. This is being tracked in issue #1169.

As a bonus, I'm also designing a t-shirt that (if there's sufficient interest) we may print as a celebration of this release - but more on this in my next post.