The Tedium of Managing Code

There’s a place motivation goes to die for web developers. It’s when something we have to do is simultaneously very, very hard and very, very uninteresting. You know, the corner of Hard and Boring Streets in downtown Must-Ship-to-Clientsville. In my personal map of life, that’s the intersection where you’ll find anything that has to do with client-side JavaScript packaging and dependency management.

Article Continues Below

I have some slides about this in recent talks. They always cause a stir. On one occasion, I had to pause for people to finish cheering. Not at me, but at the notion. There was, I think, relief and amusement at the shared recognition that bundling up and managing our client JavaScript is challenging and, at least for some of us, not our favorite thing to endure.

What are we trying to do here?#section1

A first clue to the wide-flung nature of this beast is that I don’t even have a concise term for what I’m talking about. The summarized goal is that we need to get our various bits of JavaScript put together in a way that a browser can use it. This actually entails several things, as we’ll see, but it can feel like one general objective.

We have left behind the rickety old days, when we wrote and downloaded scripts, tossed them in a directory and stuck them in <script> tags in our web pages. We’re making For-Real Things with JavaScript nowadays, and that comes with the baggage of needing to package, manage, and maintain our code, third-party code, and dependencies.

Starting basic, our first need is to take our application code, package it into one (remember: staying basic here) file, and output it somewhere. Then we can reference the output package in a <script> tag.

This may sound like an exercise in concatentation. But to get distracted by concatenation is to miss the actual thrust of what we need to do: make our code modular and handle dependencies.

You can depend on this#section2

We write code, and as we write it, bits (I’m not going to say modules just yet because that comes in a minute—hang on) of our code need other bits of code from other places. Those needed things—dependencies—might be within our own codebase or external to it.

A primary task is not just to smoosh all of our code together in a package, but to resolve and load the dependencies it needs as part of that process.

That means we need to be able to reference the dependencies we need in a way the packaging tool understands, and the tool needs to know how to find them. Not only that, our code and the code of our dependencies needs to be modularized in the right shape or our tool will rage quit.

Keeping it contained#section3

That is, our code and its dependencies need to use the appropriate module syntax. This is fine when everyone is in the same universe and gets along well, as in the case of pairing npm modules with the popular browserify tool.

browserify can seem so simple and magical. require npm modules that you need in your code just like in node, then bundle it up and, whammo, it spits out a script that works in the browser. So far, so awesome.

But code is modularized and written in different ways—AMD, UMD, CommonJS—or not at all. Some of it might be ES6 (JS 2015 to the cool kids), which we need to transpile.

“Just shim it” and other things easier said than done#section4

There are methods for subduing or translating modules that are in the wrong shape for your packaging tool of choice—packaging tools can be extended or configured to shim wayward modules. But the overhead of managing for many different flavors of modules can be a tedious addition to an increasingly cumbersome process.

Meanwhile, we have additional things to deal with. We also have to manage the source of code and dependencies. Not everything we need for the web comes from npm. Browser-targeted JavaScript has many sources: bower, CDNs, application code from your own repository, third-party code that isn’t managed at all. Fun.

Another common scenario is including a core dependency from a CDN—a classic example is jQuery—within a <script> tag. We need to tell our tool that that dependency is already accounted for, and not to worry about it. And if we can get the config syntax right (grrrr, this one bites me every time), provide that jQuery dependency to our own modules as $ instead of jQuery. Et cetera.

Yes, it’s all very possible#section5

At this point some, maybe many of you are squinting and thinking Come on, it’s not that hard. And, in the grand scheme of things, it’s not. I guess. It’s doable.

But here’s the punchline. Or, at least, the point that makes me want to lie down on the floor for a while. Every single tool or system or combination of tools does each of the things I’ve talked about in a slightly different way. browserify, require.js, webpack, others. Whether that’s the expectation of AMD module syntax or a standalone config file or different command-line options, each solution has its own learning curve that proves remarkably unfun for me when I’d rather be, you know, implementing features. It sabotages my focus.

And then we add more#section6

Any single aspect, like shimming for non-conformant module syntax, can be trivial in isolation, but typically I am at least one layer removed from the packaging by way of a build workflow abstraction. I’m usually looking at my packaging config through a murky lens of gulp or grunt. And often there are other bits at play, like a watch task to spawn packaging builds when relevant code has changed.

It’s a telling sign that the browserify task in my most recent gulp workflows is the only one I don’t fully have a handle on—it’s sourced from a boilerplate example. At one point I went through the code, line by line, and added my own comments, as a learning exercise. For five minutes, I had the whole system glowing and complete in my head. Now I look at the code and it is, once again, soup.

But, ES6!#section7

ES6 (JS 2015) is a significant update to the JavaScript language and has its own, built-in module syntax. And that is great! Especially if we could now go blow up all of the existing code in the world and start over.

Just this morning I was pondering the readme on babel-loader, an ES6 module transpiler and loader. We’ve got a project using webpack and we want to write our own stuff for it in ES6. But now, here I am again. How do I configure webpack correctly vis-a-vis babel-loader? How can I be certain that I can import non-ES6 dependencies into my freshly-minted ES6 modules?

The reality is that even when ES6 support becomes more widespread, there are going to be multiple, co-existing module syntaxes and package managers and unmanaged third-party code. The complexity is a sign that JavaScript has really come of age as the programming language of the web, but mastering this stuff takes some effort. Excuse me while I go off to debug the Uncaught ReferenceError: ufSkpO1xuIl19 is not defined exception that browserify just barked at me.

9 Reader Comments

  1. This article just proved for me that I can no longer call myself a real web developer.. but I feel your pain

  2. I love this column, Lyza. You’ve expertly summarized the frustration inherent in our increasingly varied and oft-changing workflows.

    I’m curious if you also see this as a side-effect of increased modularization of build tools? As modules become more focused and specialized (which I’m not at all convinced is a net negative, but bear with me), the scope of problems they can solve on their own is narrowed, requiring additional modules that combine and repackage common concerns (babel-loader, cssnext), complimentary modules for different scenarios (watchify versus browserify) or modules that extend one another (gulp + gulp-if). It’s no longer enough to understand the build tasks we’ve written ourselves or borrowed from boilerplates; we must also spelunk through numerous interdependencies and middlemen to truly grasp what’s going on.

  3. Thanks for this article, Lyza. The last few years have introduced many components to our workflow that make it increasingly complex just to manage elements like dependencies. Pointing it out makes us realize how much has changed so quickly.

  4. A very good article Lyza.

    I started my IT carreer 44 years ago, there were no PC’s then and certainly no internet and I learned Cobol, RPG and Assembler.
    When the PC’s came and the internet was opened for the public, I did my own things on the PC and worked in my job on a terminal as we had mainframes and no PC’s in the companies. The rest of the history can be found anywhere on the web.

    My first web pages I coded in a simple editor like blocnote.
    My first documents (like doc and docx in Word) I wrote in WordPerfect using the horizontal split-screen to manage my layout.
    Later I used “HotDog” to create my web pages, great tool then.

    Word can be used very simple today because everything you yourself needed to do before, is done for you automaticly (+-) now.
    Also we have seen a great evolution in HTML and CSS.

    My point is that we need to focus on HTML and CSS and not on the multiple (how nice and helpful it can be) front-end and back-end tools and frameworks because who can manage them and who can maintain frameworks etc. The forest of good but un-manageable tools (modules ea) keeps growing and this will force us into a central managed direction.

    This all for the sake of our profession as web-site builders who love their work.

  5. The variety of tools available is quite overwhelming. But what can be done to bring it under control?

  6. @davidshq

    That’s a good question. I don’t have a solution now.
    It is possible to organize this in the same way as how the process and contol upon HTML and CSS has been organized. The WHAT question will be more difficult.
    Browser developers (what more? Providers?) should make clear what they accept (less can be more) and what not.

    We need to organize a discussion forum first. Let’s think together.

    Just remember that website development is unnecessary expensive allready and will be more expensive in the future because more and more specialists are needed in the development process. This means bad bussiness for everyone.

  7. @davidshq I tend to just learn a few new tools that provide me with a simpler way to do something that I already do (or should be doing). If the tool looks too complicated to learn in a couple of days for marginal benefits I leave it. Eventually they’ll get simpler or die off.

    The other thing to bear in mind is a lot of these tools do the same thing. Pick one and stick with it. If after a couple of months it’s not working for you, move on. You don’t need to know multiple ways to do package management for example – just one.

    I think a lot of people get wrapped up in thinking they need to know everything. Backbone, React, Angular, Grunt, Gulp, Require, Bower, Sass, Less… the list is endless but you probably only need around 5 core tools. One or two JS frameworks. One task manager. One package manager and one preprocessor.

  8. Yes, JavaScript on the client-side is more complex than programming for Node.js in the server-side. However, there’s a nice package manager out there for the client-side called JSPM (http://jspm.io/), which is built on top of SystemJS. It doesn’t solve all of the problems you’ve mentioned. However, it solves at least some of them.

  9. Thanks for this article, Lyza.we need to focus on HTML and CSS and not on the multiple (how nice and helpful it can be) front-end and back-end tools and frameworks because who can manage them and who can maintain frameworks etc. Harga Sepeda Gunung

Got something to say?

We have turned off comments, but you can see what folks had to say before we did so.

More from ALA