Deploying private NPM modules to Zeit

Share on Facebook0Share on Google+0Tweet about this on TwitterShare on LinkedIn0

Update Zeit has just released the official way to use private NPM modules,
see the announcement.

I have been playing with Zeit.co – a super quick immutable deployment
environment targeted at Node projects. If I have a server in a local folder,
it could be deployed to new server almost instantly. It was a perfect
solution for deploying a small web hook,
or a chat server.

Yet there were two limitations:

  • lack of environment variables support; this was simple to
    work around
    using a separate private repo and a tool like as-a
  • the deployment tool does not install any private NPM dependencies due
    to lack of NPM authentication

This post shows how I worked around the second limitation by bundling
the entire server code using browserify, preparing a single
source file to be deployed. At the end there will be just a single
JavaScript deployed, with zero NPM dependencies.

Example application

As an example I took a chat application server implemented
on top of Feathers framework. The original example already had
some modifications to make it work on Zeit

  1. No Socket.io on Zeit – I have left the REST api only
  2. Local data can be written into /tmp folder only.
  3. (Optional) the server can determine its own host url using environment
    variable NOW_URL

I have published the chat app on NPM under name
feathers-chat-app-gleb, but to make the example
more realistic, I have included a dummy private NPM dependency.
The dependency @bahmutov/private-foo comes from public GitHub repo
private-foo but the module has
been published as a private scoped module on NPM registry. The chat application
just prints the value exported by the private module

src/app.js
1
2
3
4
5
const privateFoo = require('@bahmutov/private-foo');
console.log('I include', privateFoo);
// npm start
// I include private foo
// Feathers application started on localhost:3030

How can we deploy this application to Zeit using the zeit now tool?

Authentication using NPM_TOKEN

First, let us try deploying the application as is.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ now -f
> Deploying "/Users/irinakous/git/feathers-chat-app"
> Using Node.js 6.2.1 (requested: `6`)
> Ready! https://feathers-chat-app-gleb-fhzggnzoij.now.sh (copied to clipboard) [2s]
...
NPM install errors (too fast too see)
...
> Building
> ▲ npm install
> Installing package subarg@^1.0.0
> Installing package through2@^2.0.0
> Installing package xtend@^4.0.0
> Installing package bn.js@^4.0.0
> Installing package inherits@^2.0.1
> Installing package minimalistic-assert@^1.0.0
> Installing package duplexer2@~0.1.0
> Installing package readable-stream@^2.0.2
> Installing package acorn@^1.0.3
> Installing package defined@^1.0.0
> ▲ npm start
> module.js:442
> throw err;
> ^
> Error: Cannot find module 'feathers'
> at Function.Module._resolveFilename (module.js:440:15)
> at Function.Module._load (module.js:388:25)
> at Module.require (module.js:468:17)
> at require (internal/module.js:20:19)
> at Object.<anonymous> (/home/nowuser/src/src/app.js:4:21)

So the deployment fails, but the install log passes way too fast to see the
problem. To read the exception message, let us record the terminal output using
the awesome asciinema utility.

1
2
3
4
5
6
$ asciinema rec
$ now -f
$ exit
~ Asciicast recording finished.
~ Press <Enter> to upload, <Ctrl-C> to cancel.
https://asciinema.org/a/acp3tvh8mui1o7y6goox42t29

Opening the movie url, we can observe the detailed output, and even embed it
below

Now we can see the install error clearly

1
2
3
4
5
message: 'Response code 404 (Not Found)',
host: 'registry.npmjs.org',
method: 'GET',
path: '[email protected]%2Fprivate-foo',
statusCode: 404

The install fails, as it should;
the registry returns 404 when someone is requesting our
private module. Can we copy the personal NPM auth token into the local
.npmrc file to allow Zeit deploy to install the private module? If we keep
the repo private, this could be acceptable solution. We can even git ignore
the .npmrc file to avoid checking it into the repo (and even use
a tool like ban-sensitive-files to prevent this
from happening).

Yet, running the now tool shows that the local .npmrc file is ignored
and the private scoped module is still NOT installed. Hmm.

Shrink packing

If we cannot install private dependencies from the NPM registry, we could try
including the downloaded dependencies directly in the git repo. The best way
to keep code dependencies together with source is shrinkpack.

First, use npm shrinkwrap command to “fix” the exact versions of all
dependencies, including the private ones. We are excluding the dev dependencies
though.

1
2
3
$ npm prune
$ npm shrinkwrap
wrote npm-shrinkwrap.json

The written file npm-shrinkwrap.json has every
dependency resolution. Here is the start of the file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"name": "feathers-chat-app-gleb",
"version": "1.0.0",
"dependencies": {
"@bahmutov/private-foo": {
"version": "1.0.0",
"from": "@bahmutov/private-foo@*"
}
,

"accepts": {
"version": "1.3.3",
"from": "accepts@>=1.3.3< 1.4.0",
"resolved": "http://registry.npmjs.org/accepts/-/accepts-1.3.3.tgz"
}

}

}

Next, we will run the command to pack every resolved dependency back into
the .tar.gz file (almost like it was downloaded from the NPM registry).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ npm install -g shrinkpack
$ shrinkpack
i 231 dependencies in npm-shrinkwrap.json
i 0 need removing from ./node_shrinkwrap
i 231 need adding to ./node_shrinkwrap
i 209 are in your npm cache
i 22 need downloading
i 1 have a missing "resolved" property
? @bahmutov/private-foo@1.0.0 has no "dist.tarball" in
[email protected]/private-foo/package.json
? @bahmutov/private-foo@1.0.0 contacting registry...
set missing "resolved" property for @bahmutov/private-foo@1.0.0 to
https:[email protected]/private-foo/-/private-foo-1.0.0.tgz
...
shrinkpack +231 -0221 00:07

All the .tar.gz files were placed into node_shrinkwrap folder

1
2
3
4
5
6
$ ls -l node_shrinkwrap/
total 9880
-rw-r--r-- 1 576 Jul 1 08:52 @bahmutov-private-foo-1.0.0.tgz
-rw-r--r-- 1 5102 Jul 1 08:52 accepts-1.3.3.tgz
-rw-r--r-- 1 136743 Jul 1 08:52 acorn-1.2.2.tgz
...

The tool also updated the npm-shrinkwrap.json file, setting the paths
to new local files

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"name": "feathers-chat-app-gleb",
"version": "1.0.0",
"dependencies": {
"@bahmutov/private-foo": {
"version": "1.0.0",
"from": "@bahmutov/private-foo@*",
"resolved": ".[email protected]"
}
,

"accepts": {
"version": "1.3.3",
"from": "accepts@>=1.3.3< 1.4.0",
"resolved": "./node_shrinkwrap/accepts-1.3.3.tgz"
}

}

}

We can check in the node_shrinkwrap folder into our Git repository, and
it will make npm install instant from now on, because the registry will
NOT be used – all the module resolutions are present locally after cloning.

Trying now deployment command, and … it does not work. The now command
only uploads the JavaScript files, NOT the .tar.gz files. Even when I
forced it to upload by moving the node_shrinkwrap folder into the src
folder, the installation command did not resolve the local .tar.gz files.
Maybe the now command does not respect the npm-shrinkwrap.json, or
maybe there is some other reason. We need to find another way.

Bundling the server code

Our goal is to run the application. In the “normal” Node mode, each piece
of JavaScript can be loaded from a separate file using require(<filename>)
call. The <filename> is resolved relative to the path, or from the
node_modules folder. If we cannot have some files loaded from the
node_modules because the install failed, we can try “bundling” the source
code into a single source file – just like we would for running an app
inside a browser!

I installed browserify and used a simple
build.js to pass options that build a bundle targeted for
Node environment (instead of the browser).

build.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var fs = require('fs')
var browserify = require('browserify')
var insertGlobals = require('insert-module-globals')
browserify('./src/index.js', {
builtins: false,
commondir: false,
insertGlobalVars: {
__filename: insertGlobals.vars.__filename,
__dirname: insertGlobals.vars.__dirname,
process: function() {
return;
},
},
browserField: false,
})
.bundle()
.pipe(fs.createWriteStream('./src/out.js'))

Let us build the bundle and see if we can run the project without
node_modules folder.

1
2
3
4
5
6
7
8
9
10
11
12
$ node build.js
$ ls -l src/out.js
-rw-r--r-- 1 1575733 Jul 1 09:09 src/out.js
$ node src/out.js
I include private foo
Feathers application started on localhost:3030
^C
$ mv node_modules tmp
$ node src/out.js
I include private foo
Feathers application started on localhost:3030
^C

I also had to move folders config and public into src for deployment
to include them.

We have all the code needed inside a single file src/out.js. Let run use
this bundle when running now. Oops, still there is a problem

1
2
3
4
5
6
7
8
> Error: Cannot find module '/git/feathers-chat-app/src/config/default'
> at Function.Module._resolveFilename (module.js:440:15)
> at Function.Module._load (module.js:388:25)
> at Module.require (module.js:468:17)
> at require (internal/module.js:20:19)
> at s (/home/nowuser/src/src/out.js:1:176)
> at /home/nowuser/src/src/out.js:1:367
> at EventEmitter.<anonymous> (/home/nowuser/src/src/out.js:20848:26)

The feathers-configuration module loads config
JSON files by looking through the files in the config folder. This fails,
because the browserify does not include these – there is no way it can know
which files to include! So we need to hack around the config and just
load the files we need for this case ourselves. Change the src/app.js
to load the config/default.json ourselves and set the properties.

app.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// app.configure(configuration(__dirname));
const defaults = require('./config/default')
Object.keys(defaults).forEach((key) => {
// TODO adjust the paths, merge environment settings, etc
var value = defaults[key]
function isPath(s) {
return /^\./.test(s)
}
if (isPath(value)) {
console.log('resolving', value)
value = path.resolve(__dirname, value)
console.log('resolved', value)
}
app.set(key, value);
});

In the future I will refactor the feathers-configuration module to
allow sending multiple config objects without loading them on the fly
from the file system.

Next, I removed favicon middleware, since the .ico file is not bundled
into JavaScript

app.js
1
2
3
app
// .use(favicon( path.join(app.get('public'), 'favicon.ico') ))
.use('/', serveStatic( app.get('public') ))

At this point, the REST API is working on Zeit, which we can see
if we try to grab the list of users, for example.

1
2
3
4
5
6
7
8
9
10
$ http https://feathers-chat-app-gleb-kmljntdwmf.now.sh/users
HTTP/1.1 401 Unauthorized
X-Powered-By: Express
{
"className": "not-authenticated",
"code": 401,
"errors": {},
"message": "Authentication token missing.",
"name": "NotAuthenticated"
}

But if we try to open the deployed site in the browser we get 404!
Why do we not get served the HTML file public/index.html?

Bundling the mock read-only file system

The deployed application does not include any non-javascript files
from the public folder. We have the missing static pages there

1
2
3
4
5
6
7
8
$ ls -l src/public/
total 56
-rw-r--r-- 1 3295 Jun 19 10:26 app.js
-rw-r--r-- 1 3443 Jun 19 10:26 chat.html
-rw-r--r-- 1 5533 Jun 18 23:24 favicon.ico
-rw-r--r-- 1 1410 Jun 18 23:29 index.html
-rw-r--r-- 1 1357 Jun 18 23:32 login.html
-rw-r--r-- 1 1365 Jun 18 23:33 signup.html

Somehow we need to

  1. include the text contents of these files in the built bundle
  2. serve the bundled text for each file, as the server tries to read
    it from its “file system”

I love mocking environments using browserify, see blog posts
Angular in WebWorker and
Express in ServiceWorker.
Our current problem looks like a fun little challenge.

The static folder in Feathers is served using the plain Express
serve-static package,
which uses send module to open
a file stream to a file and pipe it back into the resource. If only these
modules had a file system that pointed at contents from the bundle!

Luckily there is module mock-fs that
does exactly this

1
2
3
4
var mock = require('mock-fs');
mock({
'public/index.html': 'hi there'
});

During the build step we can collect all HTML files to be bundled into a single
file, then require this mock object when we bundle the code. We are going
to turn on the mocking, and the server will have no choice but return the
bundled responses. I have described the steps on a tiny example in separate
repo browserify-server.
First, the build collects all public HTML and JS files and puts the text
into a single JSON object.

build.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
const glob = require('glob')
const files = glob.sync('src/public/*.js').concat(glob.sync('src/public/*.html'))
const mockOptions = {}
files.forEach((fullName) => {
mockOptions[fullName] = read(fullName, 'utf8')
})
const publicFiles = './public-bundle.json'
write(publicFiles,
JSON.stringify(mockOptions, null, 2) + '\n', 'utf8')
console.log('saved public files contents to', publicFiles)
// browserify commands ...
browserify('./start.js', {
...
})

We also change the entry file for browserify to start.js, which right
away requires the original index.js file, and mocks the file system
using the file created.

start.js
1
2
3
4
require('./src/index')
const mockOptions = require('./public-bundle')
const mock = require('mock-' + 'fs')
mock(mockOptions)

To avoid getting into recursive bundling, I split the mock-fs module name
in the require('mock-' + 'fs') expression; this is a common trick to
stop browserify from messing with things it should leave alone.

We can find our HTML inside the bundle

1
2
$ grep "<title>" src/out.js
"src/public/chat.html": "<html>\n<head> ...

We can even run the bundled code with the folder src/public removed

1
2
$ mv src/public src/tmp
$ npm start

Deploying the bundle to zeit and … in production the module mock-fs
is not found! Seems Zeit does not let you mock the file system.

Ok, time to find one more work around.

Writing the temp file system

Zeit does allow you to write data into /tmp folder. This is where we write
the messages database for example. We have the public folder bundled into
our single application JavaScript file; we can just dump the files into the
/tmp folder at the startup step. The server app then can serve
the static files normally.

The build file still collects all .js and .html files in the public
folder, and dumps their contents into a single .json file

build.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
function collectPublicFiles () {
const glob = require('glob')
const files = glob.sync('public/*.js').concat(glob.sync('public/*.html'))
console.log('public files\n' + files.join('\n'))
const read = require('fs').readFileSync
const mockOptions = {}
files.forEach((fullName) => {
mockOptions[fullName] = read(fullName, 'utf8')
})
return mockOptions
}
function mockPublicFiles () {
const mockOptions = collectPublicFiles()
// console.log(mockOptions)
const publicFiles = './public-bundle.json'
write(publicFiles,
JSON.stringify(mockOptions, null, 2) + '\n', 'utf8')
console.log('saved public files contents to', publicFiles)
}
mockPublicFiles()
browserify('./start.js', {...})

Then start.js file dumps the collected contents into /tmp folder

start.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
function writePublicFiles () {
const mockOptions = require('./public-bundle')
const fs = require('fs')
const outputFolder = '/tmp'
const dirname = require('path').dirname
const inTemp = require('path').join.bind(null, outputFolder)
const mkdirp = require('mkdirp')
const write = require('fs').writeFileSync
Object.keys(mockOptions).forEach((name) => {
const full = inTemp(name)
console.log('writing file', full)
const folder = dirname(full)
mkdirp.sync(folder)
write(full, mockOptions[name], 'utf8')
})
}
writePublicFiles()
require('./src/index')

Let us deploy and see if we finally got a happy server

1
2
3
4
5
6
7
8
9
10
11
12
13
$ now
> Ready! https://feathers-chat-app-gleb-yhqzmzizjz.now.sh (copied to clipboard) [2s]
> Upload [====================] 100% 0.0s
> Sync complete (1.51MB) [8s]
> ▲ npm run now-start
> writing file /tmp/public/app.js
> writing file /tmp/public/chat.html
> writing file /tmp/public/index.html
> writing file /tmp/public/login.html
> writing file /tmp/public/signup.html
> I include private foo
> Feathers application started on localhost:3030
> Deployment complete!

From the terminal let us fetch the index page

1
2
3
4
5
6
7
8
9
10
$ http https://feathers-chat-app-gleb-yhqzmzizjz.now.sh
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<title>Feathers Chat</title>
...
</head>
</html>

These are not tears of sadness, these are tears of happiness

You can check the app yourself at the above url
https://feathers-chat-app-gleb-yhqzmzizjz.now.sh.

As the final change, I added bundling and writing the config JSON files
the same way to reuse the original feathers-configuration code.

Installing nothing

To speed up the deploys, we can skip the dependency deployment on Zeit.
We have already bundled everything into the single src/out.js file; there
is no need to run npm install when deploying. We can write a simple
script to empty the dependencies and devDependencies in package.json

no-deps.js
1
2
3
4
5
6
'use strict'
const pkg = require('./package.json')
pkg.dependencies = pkg.devDependencies = []
const write = require('fs').writeFileSync
write('./package.json', JSON.stringify(pkg, null, 2) + '\n', 'utf8')
console.log('removed dependencies from package.json')

We will run the above script before the deployment, and will restore the
package.json back to its full state after the deploy.

packate.json
1
2
3
4
5
6
7
8
9
{
"scripts": {
"now-start": "node src/out.js",
"bundle": "node build.js",
"predeploy": "npm run bundle && node no-deps.js",
"deploy": "now",
"postdeploy": "git checkout package.json"
}

}

Let us try this. Without removing the superfluous node_modules the bundling
and deployment takes 60 seconds.

1
2
3
4
5
6
7
8
9
$ time npm run plain-deploy
> Building
> ▲ npm install
> Installing package subarg@^1.0.0
> Installing package through2@^2.0.0
> Installing package xtend@^4.0.0
...
> Deployment complete!
real 0m59.948s

With just the bundling and moving just the single unchanged file it
takes 25 seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ time npm run deploy
> feathers-chat-app-gleb@1.0.0 predeploy /git/feathers-chat-app
> npm run bundle && node no-deps.js
> feathers-chat-app-gleb@1.0.0 bundle /git/feathers-chat-app
> node build.js
public files
public/app.js
public/chat.html
public/index.html
public/login.html
public/signup.html
saved public files contents to ./public-bundle.json
removed dependencies from package.json
> feathers-chat-app-gleb@1.0.0 deploy /git/feathers-chat-app
> now
> Deploying "/git/feathers-chat-app"
> Using Node.js undefined (requested: `6`)
> Ready! https://feathers-chat-app-gleb-rjynpwiawt.now.sh (copied to clipboard) [2s]
> Upload [====================] 100% 0.0s
> Sync complete (837B) [989ms]
> Initializing…
> Building
> ▲ npm install
> Error {
Error: ENOENT: no such file or directory, scandir '/home/nowuser/src/node_modules'
> I include private foo
> Feathers application started on localhost:3030
> Deployment complete!
> feathers-chat-app-gleb@1.0.0 postdeploy /git/feathers-chat-app
> git checkout package.json
real 0m24.659s

Fast and simple.

Conclusion

I wish Zeit had support for private modules, yet lack of this feature was a
great opportunity for hacking together a work around. Notice that I had
pursued multiple solutions that lead to a dead end. Yet every solution that did
not work taught me something new. One just has to be patient and keep
coming up with new ideas, chipping away at the problem.

The final solution is a weird one; I am sure the original problem has to do
with browserified bundle not being able to properly load the regular files
due to some path mangling.
I will not pursue this further, instead I hope to see the Zeit team
implement installation of private NPM modules using proper authentication.

You can find the code at
bahmutov/feathers-chat-app
and see the running application at
https://feathers-chat-app-gleb-rjynpwiawt.now.sh

If you like hacking the code like me, read
How to become a better hacker blog post.

Share on Facebook0Share on Google+0Tweet about this on TwitterShare on LinkedIn0