By: Nolan Lawson
Published:06 June 2016
"Hold the pickles, hold the lettuce; special orders don’t upset us."
– Burger King jingle
When you’re building a large open-source project, the challenge often becomes less about "how do we add new features?" and more about "how do we let people opt-out of the features they don’t want?".
npm install ed, especially if you’re only using a small part of it.
Large dependencies can mean a loading spinner in your user’s browser, as well as an npm spinner in your own console.
PouchDB – not so pocket-sized anymore
PouchDB is an ambitious project, and as such it’s grown a lot over the years. It started out as a simple way to sync CouchDB to IndexedDB (originally only supporting Firefox Nightly !), whereas now it’s used in a variety of contexts:
- In Node.js using LevelDB
- In hybrid apps using native SQLite
- As its own server
- As an AJAX interface to CouchDB
- As a layer over WebSQL, LocalStorage, or even in-memory
- What if I don’t want map/reduce?
- What if I’m only using the IndexedDB adapter?
- What if I’m only using PouchDB to talk to CouchDB?
If you’re not using any of the dependencies, it’s a shame to include them in the bundle that gets shipped to your users. We feel ya.
Also, because PouchDB has some large native dependencies (such as LevelDB), the first
npm install can take a very long time, and it can even fail on certain operating systems (Windows, looking at you). This is especially bothersome when you consider that those dependencies are only required if you’re using PouchDB in Node.js. (And even then, you could just use the in-memory adapter instead.)
So unfortunately, folks who are only using PouchDB in the browser still have to pay the extra cost of installing LevelDB. And ditto for SQLite, even though it’s only used for the optional
node-websql adapter! This is a less-than-ideal scenario.
Breaking up is hard to do
With PouchDB 5.4.0, though, we’ve made a huge architectural change to support users who want a slimmer or more customized PouchDB experience. Starting with this release, many of the core modules that were previously internal to PouchDB can now be
npm install ed and configured separately.
As it turns out, this monorepo strategy is perfect for PouchDB. Logically, we have many small interdependent components (such as the replication module, the map/reduce module, or the IndexedDB adapter), but we prefer to test them all in one large test suite, because a change in one module can have cascading effects in another module. Breaking everything up into many tiny repos, which would be tested, versioned, and developed independently, is just not practical for us.
Using Lerna, though, we can keep all of PouchDB’s codebase in one repo, and still release it as separate packages. There are some tricky bits – such as how to manage source code versus bundled code, and how to measure code coverage – but in those cases Rollup remains a great tool to allow us to de-bundle and re-bundle at will. In particular, Rollup allows us to build a "debug" version of PouchDB with many more endpoints exposed, which we then test as a single
index.js for measuring code coverage. Then, we are still free to ship a "production" version of PouchDB with those endpoints disabled.
With this release, we are also committing to the
jsnext:main standard as promoted by Rollup , meaning that all of our packages (including
pouchdb itself) now ship with both ES modules and CommonJS, and you are free to use whichever one you want. In general, the ES modules will tend to give you smaller bundle sizes (assuming you are using Rollup, SystemJS, or Webpack 2 as your bundler), whereas the CommonJS bundle is more appropriate for backwards compatibility. (Neither release makes use of ES6 features aside from ES modules, so you do not need a transpiler like Babel or Bublé ).
Cool, how does it work?
PouchDB now ships with a few presets, which are a good demonstration of how the new plugin feature works. Here’s
pouchdb-browser (as CommonJS), which is the "browser" version of PouchDB:
var PouchDB = require('pouchdb-core'); PouchDB.plugin(require('pouchdb-adapter-idb')) .plugin(require('pouchdb-adapter-websql')) .plugin(require('pouchdb-adapter-http')) .plugin(require('pouchdb-mapreduce')) .plugin(require('pouchdb-replication')); module.exports = PouchDB;
As you can see,
pouchdb-browser is just an extension of
pouchdb-core , with some key modules added in: the IndexedDB adapter, the WebSQL adapter, the HTTP adapter, map/reduce, and replication. If you’re not using any of these features, you can just delete that plugin and enjoy the file savings.
Additionally, you’ll notice that
pouchdb-browser does not depend on LevelDB at all. Instead, that lives in
pouchdb-node , a.k.a. the "Node" version of PouchDB:
var PouchDB = require('pouchdb-core'); PouchDB.plugin(require('pouchdb-adapter-leveldb')) .plugin(require('pouchdb-adapter-http')) .plugin(require('pouchdb-mapreduce')) .plugin(require('pouchdb-replication')); module.exports = PouchDB;
The only difference here is that
pouchdb-browser uses IndexedDB and WebSQL, whereas
pouchdb-node uses LevelDB. Hence, LevelDB is not installed if you do
npm install pouchdb-browser .
Also, if you’d like to use a custom adapter, those are now shipped separately. So if you want to use PouchDB in a purely in-memory mode, you can do:
var PouchDB = require('pouchdb-core') .plugin(require('pouchdb-adapter-memory'));
Or if you’re only using PouchDB to talk to CouchDB:
var PouchDB = require('pouchdb-core') .plugin(require('pouchdb-adapter-http'));
This applies to all of the plugins we previously shipped in the”extras” API. Any of these can now be mixed and matched as desired:
Note that the plugin order of the non-HTTP adapters matters. So for instance, if you wanted to fall back from IndexedDB to WebSQL to LocalStorage to in-memory, you would do:
var PouchDB = require('pouchdb-core') .plugin(require('pouchdb-adapter-idb')) .plugin(require('pouchdb-adapter-websql')) .plugin(require('pouchdb-adapter-localstorage')) .plugin(require('pouchdb-adapter-memory'));
How many bytes can it save?
Here is a breakdown of the bundle size of various possible PouchDB configurations, using Browserify, bundle-collapser , and Uglify.
|Preset||Browserify + Uglify||Browserify + Uglify + Gzip|
||140.06 kB||45.15 kB|
||148.3 kB||47.54 kB|
||64.29 kB||21.54 kB|
|PouchDB, IndexedDB only||83.47 kB||27.79 kB|
|PouchDB, WebSQL only||85.15 kB||28.16 kB|
|PouchDB, replication only||76.28 kB||25.26 kB|
|PouchDB, in-memory only||350.74 kB||100.81 kB|
pouchdb-browser is slightly bigger than
pouchdb , because
pouchdb is heavily optimized with Rollup before publishing, whereas
pouchdb-browser is not. Also, some optional plugins (like
pouchdb-adapter-memory ) are still quite large. In the future, we may consider using Rollup more aggressively, or trying to reduce the size of
pouchdb-core . Or we might just wait for the ecosystem to switch to tools that do automatic tree-shaking for ES modules, such as Rollup or Webpack 2. (If you use such a tool, and it supports
jsnext:main , then you should already be able to get smaller bundle sizes.)
In the meantime, though, a major benefit of PouchDB unbundling is still in reduced
npm install times, as well as avoiding compatibility issues with native dependencies like
Yes indeed, as suggested by the 5.4.0 release version, this is a fully backwards-compatible change . The
pouchdb module will still act exactly as it did before, including the
extras/ APIs. Essentially, a
npm install pouchdb will install the kitchen sink. We believe this is the best approach, because it provides a batteries-included experience for beginners, without having to burden them with understanding a vast plugin API.
extras/ API is deprecated, and will be removed in a later version. Furthermore, the
PouchDB.ajax , and
PouchDB.Errors APIs, which have been undocumented and unrecommended for years, are now removed. Plugin authors should update their code to use the new sub-packages.
That said, there is an important note about the new sub-packages: all of them are pegged to PouchDB’s version number, meaning that when PouchDB cuts a new release, each of the sub-packages will also be released with the same version number. Furthermore, not all of these sub-packages follow semantic versioning , because conceptually they are internal modules, which are only exposed on npm as an implementation detail. The APIs for these packages may change at any time, so you should use exact versioning if you decide to use them directly. These packages will be clearly marked as non-semver in their READMEs.
In the new system, the packages that serve as "plugins" (as well as
pouchdb itself) are considered user-facing and semver-compliant. In particular, these packages follow semver:
Other modules, such as
pouchdb-checkpointer , and
pouchdb-promise , may be freely used, but they make no guarantees as to semver. As we did with the
extras/ API, all of these are used at your own risk.
We considered using scoped packages (e.g.
@pouchdb/foo ) as opposed to prefixed packages (e.g.
pouchdb-foo ) for the new monorepo architecture. In fact, we even asked the community to vote on the issue , and as of today the vote stands at 27 for scopes and 24 for prefixes.
However, the core team ultimately decided that it’s too early to hop onto the scoping bandwagon. There are still plenty of thorny issues with scopes, such as the fact that npm organizations are not free for open-source yet, meaning we would have to register a single
pouchdb account and share it among all the contributors. (See the voting issue for discussion about other issues with scopes.) We look forward to these issues being resolved as scoping matures, at which point we may consider moving over to scoped packages.
For the foreseeable future, though, PouchDB’s sub-packages will sport names like
pouchdb-ajax , and
pouchdb-replication . Even if it’s a bit old-fashioned, we hope that PouchDB’s users will experience fewer surprises and confusion with this more traditional naming convention. And considering the popularity of
babel-* , and
react-* packages, it’s safe to say PouchDB is in good company.
Since there are dozens of combinations for building your own PouchDB, we encourage the community to mix-and-match, and to publish any presets that they find useful. As of 5.4.0, there are three presets that PouchDB will officially support as first-party packages:
pouchdb-browser– the "browser" version of PouchDB
pouchdb-node– the "Node" version of PouchDB
pouchdb-http– PouchDB as an HTTP interface to CouchDB (sans map/reduce)
In the future, we also plan to move some packages that have previously existed in third-party repos (including
pouchdb-express-router , and
pouchdb-find ) into the main PouchDB repository, so that they can be tested, versioned, and released at the same time as PouchDB.
So please, try out the new customizable PouchDB, and let us know what you think! And if there are any bugs, there’s only one repo you need to worry about, so please direct your bugs to the PouchDB issues page . Happy unbundling!