NPM Package-lock and semver caret versions.

by Corey Snyder

Maintaining dependencies of our front-end apps has been a struggle. The problems we’re facing are unique to Aver’s situation in that we maintain a micro-services-front-end. You can read more about that here. TLDR; Our web product is made up of 25+ different micro-frontends. We are have tried all of the industry-standard tools like package-lock of NPM, yarn, shrinkwrap, etc appropriately, but they still have their shortcomings in a micro-services world and we’re feeling the pain.

Our Current Setup

In our current setup we pin all 3rd party library versions (react: 16.8.6), and we use semver caret versions for our own dependencies (@aver/ui-scaffolding: ^2.3.3). The idea behind this:

  1. We are not ok with our 3rd party libraries getting updated whenever their developers release new code. We want to vet that new code and make sure it's bug free and tested before we pull it into our apps. Most of the time, the risk of making changes is not worth the reward of whatever they add. We only upgrade if there's a feature/bugfix we need, or if there's a known security vulnerability. Note: We make a point that our npm audit has zero Major, Minor, Trivial issues 100% of the time.
  2. Alternatively, because we use a micro-services-front-end consisting of 25+ *-web front-end repos, pinning the versions of 1st party libraries would mean each change would require 25+ package.json changes, branches, PR reviews, and deploys. As our # of front-ends grows, so does the pain of rolling out updates. We want frequent changes to our own libraries, which we code-review and vet all changes on, to be able to be rolled out to the many front-end apps easily without code reviews or a bunch of process and blockers.

We originally switched to trying to use ^versions for our first-party libraries not necessarily because it was better, but because it was less of a burden on developers.

The Latest Issue

So my thought was that if we do things this way and we check in our package-lock file, that at deploy time, it'll run npm ci which will install the dependencies from the dependency tree that exists in the package-lock file which will pick up our 1st party library updates and not those of 3rd party dependencies.

SPOILER: This is not the case.

Our current issue is that even when we attempt to use semver ^versioning of our internal libraries, they are not being updated when re-deploying our front-end apps.

What I've learned:

Package.json > Package-lock.json

As of NPM@5.0.1 - package.json will overrule the package-lock if package.json has been updated. What does this mean? If you make changes to your package.json file and run npm i it will result in changes being made to your lock file. So if you are running npm i as part of your deploy process, it's going to negate your lock file. Instead you should use npm ci which installs dependencies based on the lock file alone.

You cannot reliably use caret versions with package-lock.json

If you have caret versions of your assets in your package.json such as ui-scaffolding: ^1.0.1, there is no guarantee that it will be updated to the latest non-major version when deployed. Essentially whatever version is placed into the lock file at the time that you last ran npm i locally is what is going to be used when your run npm ci or npm i in your deploy process. So say it the latest version 1.1.0 at the time you run npm i. That's what's going to get put in the lock file and that is what is going to get installed at deploy time. Even if version 1.2.0 is released and you redeploy your app, it's going to still use 1.1.0 instead because that's what's "locked" in the lock file.

Furthermore, if version ui-scaffolding@1.2 gets released, and you re-run npm i locally to update it, version 1.2.0 will not get pulled in! This is because npm will look to see what's installed and it'll be 1.1.0 and that satisfies what is in the lock file ^1.0.1. So it'll make no changes to the lock file.

Conversely, If you didn't commit the lock file OR your node_modules directory, and did a deployment, when npm i was executed it would go grab the latest version because there's nothing already in place which satisfied the package.json file and so it'll go grab the latest version which does.

If you want to change the lock file so it FOR SURE pulls in 1.2.0 you have a few options.

Option 1: npm i --save ui-scaffolding@^1.2.0 Update your package.json to require the later version of ui-scaffolding. When it runs it'll detect that you don't have a version installed which satisfies the package.json file and so it's going to fetch the latest file which does satisfy it, and update the package-lock to describe the version it used.

Option 2: (rm -rf node_modules || rm package-lock.json) && npm i. If the node modules  or the package-lock file isn't present, it's going to detect that you don't have a version installed which satisfies the package.json file and so it's going to fetch the latest file which does satisfy it, and update the package-lock to describe the version it used. The lock file is updated because:

package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json.

So how can we have our cake and eat it too?

That's what I've got to figure out. How can we lock down our dependency tree for all 3rd party libraries while still being able to roll out changes to our 1st party libraries painless?

My first thought (But not the right answer):

I could possibly use npm-update as part of a pre-install script like: "preinstall": "npm update". What this is going to do is, before it runs npm ci, it will run npm update which will update all versions which are not pinned to an exact version number.  

When I first tried this I got an error in my deployment npm WARN lifecycle MyApp@1.0.0~preinstall: cannot run in wd MyApp@1.0.0 npm update (wd=/code). I think this is telling me that the user in the docker container that is trying to execute npm updatedoesn't have the proper permissions to safely perform this action so  have to add this to my package.json to allow it: "config": {"unsafe-perm":true}

On second thought

I no longer believe our current idea will reliably work without some the unsafe-perm hackiness. I just don't believe that is the best option for us going forward. Instead, I propose that we write a simple dev tool that we can use to roll-out the library changes to the web repos and subsequently the environments. Something that is more straightforward and intentional with how we deploy 1st-party library code.

Initial thoughts:

  • It might be nice if it were to live as a code-build tool so that developers don’t have to run/install it on their machines
  • It would be great if it didn’t create a branch/PR but rather merge directly to master. The code being merged has already been PR Reviewed and tested at this point. No point in creating 30 PRs to bug a developer to OK a 1-line change that they aren’t really doing any verification on.
  • We would want to be able to select which front-end Apps receive which version of which 1st party library. [List of *-web Repos to be affected] [1st-party-library] [version]

I'm going to be forming a team including representation from devops to try and see if this is a viable option.