Node Overview

Ryan Dahl started the Node runtime in 2009 after he was inspired by the performance of the V8 JavaScript engine in the Google Chrome web browser. V8 uses an event-driven model, which makes it efficient at handling concurrent connections and requests. Ryan wanted to bring this same high-performance, event-driven architecture to server-side applications. The event-driven model is the first and most important concept you need to understand about the Node runtime (and the V8 engine as well). I’ll explain it briefly in this section, but we’ll have a chance to talk about it a lot more in Chapter 4.

In its core, Node enables developers to use the JavaScript language on any machine without needing a web browser. Node is usually defined as “JavaScript on backend servers”. Before Node, that was not a common or easy thing. JavaScript was mainly a frontend thing.

However, this definition isn’t really an accurate one. Node offers a lot more than executing JavaScript on servers. In fact, the execution of JavaScript is not done by Node at all. It’s done with a Virtual Machine (VM) like V8 or Chakra. Node is just the coordinator. It’s the one who instructs a VM like V8 to execute your JavaScript.

Node is better defined as a server environment that wraps V8 and provides small modules that can facilitate building software applications with JavaScript

When you write JavaScript code and execute it with Node, Node will pass your JavaScript to V8, V8 will execute that JavaScript and tell Node what the result is, and Node will make the result available to you. In addition to that, Node has a few handy built-in modules that provide easy-to-use asynchronous APIs. Let’s talk about that, and a few other reasons why developers are picking Node over many other options when it comes to creating services for their backends.

Arguments Against Node

Node’s approach to handling code in an asynchronous and non-blocking manner is a unique model of thinking and reasoning about code. If you’ve never done it before, it will feel weird. You need time to get your head wrapped around this model and get used to it.

Node has a relatively small standard library. This means that developers need to rely on third-party modules to perform most big tasks. There is a large amount of third-party modules available for Node. You need to do some research to pick the most appropriate and efficient ones. Many of these modules are small, which means you’ll need to use multiple modules in a single project. It’s not uncommon for a Node project to use hundreds of third-party modules. While this can enhance maintainability and scalability, it also requires more management and oversight. As modules are regularly updated or abandoned, it becomes necessary to closely monitor and update all modules used within a project, replacing deprecated options and ensuring that your code is not vulnerable to any of the security threats these modules might introduce.

Additionally, Node is optimized for I/O and high-level programming tasks but it may not be the best choice for CPU-bound tasks, such as image and video processing, which require a lot of computational power. Because Node is single-threaded, meaning that it can only use one core of a CPU at a time, performing tasks that require a lot of CPU processing power might lead to performance bottlenecks. JavaScript itself is not the best language for high-performance computation, as it is less performant than languages like C++ or Rust.

Node also has a high rate of release and version updates, this can create the need for constant maintenance and updates of the codebase, which can be a disadvantage for long-term projects.

Finally, the language you use in Node, JavaScript, has one big valid argument against it. It is a dynamically typed language, which means objects don’t have explicitly declared types and they can change during runtime. This is fine for small projects but for bigger ones, the lack of strong typing can lead to errors that are difficult to detect and debug and it generally makes the code harder to reason with and to maintain.

Executing Node Scripts

If you have Node installed on your computer, you should have the commands node and npm available in a terminal. If you have these commands, make sure the Node version is a recent one (20.x or higher). You can verify by opening a terminal and running the command node -v.

If you don’t have these commands at all, you’ll need to download and install Node. You can download the latest version from the official Node website (https://nodejs.org/). The installation process is straightforward and should only take a few minutes.

For Mac users, Node can also be installed using the Homebrew package manager with the command brew install node.

Another option to install Node is using Node Version Manager (NVM). NVM allows you to run and switch between multiple versions of Node, it works on Mac and Linux, and there’s an NVM-windows option as well.

To get started, open a terminal and issue the node command on its own without any arguments:

$ node

This will start a Node REPL session. REPL stands for Read, Eval, Print, Loop. It’s a convenient way to quickly test simple JavaScript and Node code. You can type any JavaScript code in the REPL. For example, type Math.random() and then, press Enter:

Node will read your line, evaluate it, print the result, and loop over these 3 things until you exit the session (which you can do with a CTRL+D).

Note how the “Print” step happened automatically. We didn’t need to add any instructions to print the result. Node will just print the result of each line you type. This is not the case when you execute code in a Node script. Let’s do that next.

Create a new directory for the exercises of this book, and then cd into it:

$ mkdir efficient-node
$ cd efficient-node

Open up your editor for this directory, then create a file named index.js. Put the same Math.random() line into it:

Math.random();

Now to execute that file, in the terminal, type the command:

node index.js

You’ll notice that the command will basically do nothing. That’s because we have not outputted anything from that file. To output something, you can use the console object, which is similar to the one available in browsers:

console.log(
  Math.random()
);

Executing index.js now will output a random number

Note how in this simple example we’re using both JavaScript (Math object), and an object from the Node API (console). Let’s look at a more interesting example next.

const http = require('http');

const server = http.createServer((req, res) => {
  res.end('Hello World\n');
});

server.listen(3000, () => {
  console.log('Server is running...');
});

$ node server.js

$ npm install lodash

$ ls node_modules

{
  "name": "efficient-node",
  "version": "1.0.0",
  "description": "A comprehensive guide to learning the Node.js runtime from scratch",
  "license": "MIT"
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "lodash": "^4.17.21"
  },
}

$ npm init

$ npm install -D eslint

$ npm install express

import express from 'express';

const app = express();

app.get('/', (req, res) => {
  res.send('Hello Express');
});

export default app;

import app from './server.mjs';

app.listen(3000, () => {
  console.log('Server listening on http://localhost:3000');
});

$ node index.mjs

  "type": "module"

Summary

Node is a powerful framework for building network applications. Its event-driven, non-blocking I/O model, single-threaded event loop, and built-in module system make it easy for developers to create efficient and scalable applications.

Node wraps a VM like V8 to enable developers to execute JavaScript code in a simple way.

Node built-in modules provide easy-to-use asynchronous APIs. Node’s module system allows developers to organize their code into reusable modules. These modules can be imported and used in other parts of the application.

Node has a large and active community that has created many popular modules that can be easily integrated into Node projects. These modules can be found and downloaded from the npm registry.

The node Command

In Chapter 1, we used the node command briefly to explore Node’s REPL mode and then to execute Node scripts. The node command has many options and its behavior can be customized. It also supports arguments and environment variables to further customize what it does and pass data from the operating system environment to Node’s process environment. Let’s take a look:

In the terminal, type

$ node -h | less

This will output the “help” documentation for the command (on page at a time because we piped the output on the less command). I find it useful to always get myself familiar with the help pages for the commands I use often.

Usage: node [options] [ script.js ] [arguments]
       node inspect [options] [ script.js | host:port ] [arguments]

Options:
  -                           script read from stdin (default if no
                              file name is provided, interactive mode
                              if a tty)
  --                          indicate the end of node options
  --abort-on-uncaught-exception
                              aborting instead of exiting causes a
                              core file to be generated for analysis
  --build-snapshot            Generate a snapshot blob when the
                              process exits. Currently only supported
                              in the node_mksnapshot binary.
  -c, --check                 syntax check script without executing
  --completion-bash           print source-able bash completion
                              script
  -C, --conditions=...        additional user conditions for
                              conditional exports and imports
  --cpu-prof                  Start the V8 CPU profiler on start up,
:

The first 2 lines specify how to use the node command. Anything in square brackets is optional, which means, according to the first line, that we can use the node command on its own without any options, scripts, or arguments. That’s what we did to start a REPL session. To execute a script, we used the node script.js syntax (“script” can be any name there).

What’s new here is that there are options and arguments that we can use with the command. Let’s talk about these.

In the help page, right after the usage lines, there is a list of all the options that you can use with the command. Most of these options are advanced, but knowing of their existence is a helpful reference. You should scan through this list just to get a quick idea of all the types of things that you can do, but let me highlight a few of the options that I think you should be aware of.

The -c option (or --check) lets you check the syntax of a Node script without running that script. An example use of that option is to automate a syntax check before sharing code with others.

The -e and -p options (or --eval and --print) can both be used for executing code directly from the command line. I like the -p one more because it executes and prints (just like in the REPL mode). To use these options, you pass a string of Node code in quotes. For example:

$ node -p "Math.random()"

This is handy, as you can use it to create your own powerful commands (and alias them if you want). For example, say you need a command to generate a unique random string (to be used as a password maybe). You can leverage Node’s crypto module in a short -p one liner:

$ node -p "crypto.randomBytes(16).toString('hex')"

Pretty cool, isn’t it!

How about a command to count the words in any file?! This one will help us understand how to use arguments with the node command:

$ node -p "fs.readFileSync(process.argv[1]).toString().split(' ').length" ~/.bashrc

Don’t panic. There’s a lot going on with this one. It leverages the powers of both Node and JavaScript. Go ahead and try it first. You can replace ~/.bashrc with a path to any file on your system.

Let’s decipher this one a bit:

The readFileSync takes a file path as an argument and synchronously returns a binary representation of that file’s data. That’s why I chained a .toString call to it, to get the file’s actual content (in UTF-8). Furthermore, instead of hardcoding the file path in the command, I put the path as the first argument to the node command itself and used process.argv[1] to read the value of that argument (see explanation of that next). This allows the word-counting one-liner to be generic. We can alias it without the path argument and then use the alias with a path argument.

Then once I have the content of the file, I use JavaScript’s split method, which is available on any string, to split the content using spaces, giving me an array of words. Then I just counted those with a .length call to estimate the number of words.

The -r option (or --require) allows you to require a module before executing the main script. This is useful if you need to load a specific module before running your code or if you want to set up certain configurations or variables.

For example, let’s say you have a Node project that requires the use of a module called dotenv, which loads environment variables from a file. Normally, you would need to include something like require('dotenv').config() at the beginning of your main file to use the dotenv module. However, with the -r option, you can load the module automatically without having to add it to each file:

$ node -r dotenv/config index.js

The --watch option allows you to watch a file (and its dependencies) for changes. It automatically restarts Node when a change is detected. This is very useful in development environments. You can test it with any of the files we wrote so far. For example, to run the Express.js example in watch mode, you can run:

$ node --watch index.mjs

This will start the server in watch mode. Make a change to the server.mjs file (change the “Hello Express” string, for example) and notice how the node command will automatically restart.

The --test option makes Node look for and execute code that’s written for testing. Node uses a simple naming convention for that. For example, it’ll look for any files named with a .test.js suffix, or files whose names begin with test-.

There are a lot more options, but most of them are for advanced use. It’s good to be aware of them so that in the future, you can look up if there’s one particular option that might make a task you’re doing simpler.

Since Node is a wrapper around V8, and V8 itself has CLI options, the node` command accepts many V8 options as well. The list of all the V8 options you can use with the node command can be printed with:

$ node --v8-options | less

This is an even bigger list! You can set JavaScript harmony flags (to turn on/off experimental features), you can set tracing flags, customize the engine memory management, and many other customizations. As with the node command options, it’s good to know that all these options exist.

Toward the end of the node -h output, you can see a list of environment variables, like NODE_DEBUG, NODE_PATH, and many more. Environment variables are another way to customize the behavior of Node or make custom data available to the Node process (similar to command arguments)

Every time you run the node command, you start an operating system process. In Linux, the command ps can be used to list all running process, if you run it while a Node process is running (like the Express.js example), one of the listed process will be Node (and you can see its process ID, and stop it from the terminal if you need to). Here’s a command to output process details and filter the output for processes that have the word “node” in them:

$ ps -ef | grep "node"

Node’s global process object represents a bridge between the Node environment and the operating system environment. We can use it to exchange information between Node and the operating system. In fact, when you console.log a message, under the hood, the code is basically using the process object to write a string to the operating system stdout (standard output) data stream.

Environment variables are one way to pass information from the operating system environment (used to execute the node command), to the Node environment, and we can read their values using the env property of the process object.

Here’s an example to demonstrate that:

$ NAME="Reader" node -p "'Hello ' + process.env.NAME"

This will output “Hello Reader”. It sets an environment variable NAME then reads its value with process.env.NAME. You can even set multiple environment variables if you need, either directly from the command line like this example, or using the Linux export command prior to executing the node command:

$ export GREETING="Hello"; export NAME="Reader"; \
  node -p "process.env.GREETING + ' ' + process.env.NAME"

You can use environment variables to make your code customizable on different machines or environments. For example, the Express.js example in Chapter 1 hard-coded the port to be 3000. However, on a different machine, 3000 might not be available, or you might need to run the server on a different port in a production environment. To accomplish that, you can modify the code to use process.env.PORT ?? 3000 instead of just 3000 (in the listen method) and then run the node command with a custom port when you need to:

$ PORT=4000 node index.mjs

Note that if you don’t specify a port, the default port would be 3000 because I used the ?? (nullish) operator to specify a value when process.env.port does not have one. This is a common practice.

The list of environment variables shown toward the end of node -h output are Node’s built-in environment variables. These are variables that Node will look for and use if they have values. Here are a few examples:

• NODE_DEBUG can be used to tell Node to output more debugging information when it uses certain libraries. We give it a comma-separated list of modules to debug, for example, with NODE_DEBUG=fs,http, Node will start outputting debugging messages when the code uses either the fs or http modules. Many packages support this environment variable.

• NODE_OPTIONS is an alternative way to specify the options Node supports instead of passing them to the command line each time.

• NODE_PATH can be used to simplify import statements by using absolute paths instead of relative ones. We’ll see an example of that later in the chapter.

Node’s REPL Mode

In Node’s REPL mode, as we learned in Chapter 1, you can type any JavaScript code, and Node will execute it and automatically print its result. This is a convenient way to quickly test short JavaScript expressions (and it works for bigger code too). However, there are a few other helpful things you can do in REPL mode beyond the quick tests.

In REPL mode, you usually type an expression (for example: 0.1 + 0.2), and hit Enter to see its result. You can also type statements that are not expressions (for example: let v = 21;) and when you hit Enter, the variable v` will be defined, and the REPL mode will print undefined since that statement does not evaluate to anything. If you need to clear the screen, you can do so with CTRL+L.

If you try to define a function, you can write the first line and hit Enter, and the REPL mode will detect that your line is not complete, and it will go into a multiline mode so that you can complete it. Try and define a small function to test that.

The REPL multiline mode is limited but there’s an integrated basic editor available within REPL as well. While in REPL mode, type .editor to start the basic editor mode, then you can type as many lines of code as you need, you can define multiple functions, or paste code from the clipboard, then, when you are done, hit CTRL+D to the have Node execute all the code you typed in the editor.

The .editor command is one of many REPL commands which you can see by typing the .help command:

> .help
.break    Sometimes you get stuck, this gets you out
.clear    Alias for .break
.editor   Enter editor mode
.exit     Exit the REPL
.help     Print this help message
.load     Load JS from a file into the REPL session
.save     Save all evaluated commands in this REPL session to a file

Press Ctrl+C to abort current expression, Ctrl+D to exit the REPL

The .break command (or its .clear alias) lets you get out of some weird cases in REPL sessions. For example, when you paste some code in Node’s multiline mode and you are not sure how many curly braces you need to get to an executable state. You can completely discard your pasted code by using a .break command. This saves you from killing the whole session to get yourself out of situations like these.

The .exit command exits the REPL or you can simply press Ctrl+D.

The .save command enables you to save all the code you typed in one REPL session into a file. The .load command enables you to load JavaScript code from a file and make it all available within the REPL session. Both of these commands take a file name as an argument.

One of my favorite things about Node’s REPL mode is how I can inspect basically everything that’s available natively in Node without needing to require them. All the built-in modules (like fs, http, etc) are defined globally and you can use the TAB key to inspect their APIs.

Just like in a terminal or editor, hitting the TAB key once in a REPL session will attempt to auto-complete anything you partially type. Try typing cr and hit TAB to see it get auto-completed to crypto. Hitting the TAB key twice can be used to see a list of all the possible things you can type from whatever partially-typed text you have. For example, type a and hit TAB twice to see all the available global objects that begin with a.

This is great if you need to type less and avoid typing mistakes, but it gets better. You can use the TAB key to inspect the methods and properties available on any object. For example, type Array. and hit TAB twice to see all the methods and properties that you can use with the JavaScript Array class. This works with Node modules as well. Try it with fs. or http..

It even works with objects that you create. For example, create an empty array using let myArr = [];, then type myArr. and hit TAB twice to see all the methods available on an array instance.

Best of all, TAB discoverability works on the global level, if you hit TAB twice on an empty line, you get a list of everything that is globally available. This is a big list, but it’s a useful one, it has all the globals in the JavaScript language itself (like Array, Number`, and Math), and it has all the globals from Node (like process and setTimeout), and it also lists all the core modules that are available natively in Node (like fs and http).

AbortController                   AbortSignal                       AggregateError
Array                             ArrayBuffer                       Atomics
BigInt                            BigInt64Array                     BigUint64Array
Blob                              Boolean                           BroadcastChannel
Buffer                            ByteLengthQueuingStrategy         CompressionStream

...

How Modules Work

A module is simply a reusable block of code. Something you can include and use in any application, as many times as you need. In Node, a module can be a single file or a group of files with a main one. There’s always a main file in a Node module, and that’s the file that we “require” (or “import”). Modules have public APIs. When we require a module, we usually get back an object that represents the module’s API.

To understand one important aspect about Node modules, let’s create a new module, name it main.js and put the following line in it:

console.log(arguments);

What do you think executing this file will output?

If you don’t know that all Node modules are wrapped in special functions, you’d say undefined. But the output of that line will reveal 5 argument values!

Node wraps every module implicitly with a function. When you execute a module, Node calls that function, and - as the output reveals - passes 5 arguments to that function as well.

You can actually see the wrapping function detail if you print the value of require('module').wrapper (You can do that in a REPL session).

(function(exports, require, module, __filename, __dirname) {
  // Module code actually lives in here
});

When you use the exports/require/module/__filename/__dirname keywords in a module, you’re not using a “global” variable, you’re just using the implicit wrapper function’s arguments.

The __filename value has the name of the file. The __dirname value has the path to the directory where the file is hosted.

The exports, require, and module arguments are Node’s way to manage a module’s API. To understand them, let’s create another module in the same directory as main.js, let’s name this one config.js. Usually, you’d put any configuration logic in a separate module like that.

Since config.js is yet another module that will be wrapped by Node, it’ll have the 5 arguments as well. Let’s console.log the exports argument in config.js and execute the file with node config.js:

console.log(exports)

As you can see, the value for exports is simply an empty object, and we can change that object and add properties to it, just like we can change any JavaScript variable.

There are 2 ways to execute a module in Node. So far we used the first and main way, which is to specify the file path for the node command. The other way to execute a module is through the require argument (which is a function), one module can require another module using that function, for example, the main.js module, can require the config.js module:

require('./config.js');

We invoke the require function with the path to the module we’re interested in. The path can be a relative one when it starts with a ., or an absolute one (for example: /Users/samer/efficient-node/main.js).

When we execute main.js now using the node command, we’ll see the console.log line from config.js.

Now we can say that the main module “depends” on the config module, or that the config module is a dependency for the main module. This is where the term “dependency management” comes from. We are managing the dependencies of a module here and bringing one module’s API to use in another module.

Let’s define the API for the config module. Let’s define a static property and a function property:

exports.PORT = process.env.PORT ?? 8080;
exports.SERVER_URL = (host = process.env.HOST ?? "localhost") =>
  `http://${host}:${exports.PORT}`;

The exports argument in CommonJS modules is an alias to module.exports which is initialized as an empty object. The official API for the module is the module.exports value. As long as that value is an object, we can use the exports alias to define the API. In some cases, you might need the top-level API object to be a function or a class, or anything else that’s not a simple aliased object. In these cases, you’ll need module.exports to define the API (we’ll see an example of that soon).

When we use the require function in main.js to get the API for config.js, we’re basically invoking the wrapping function for config and getting back the value of module.exports. It’s a bit more complicated than that, but that is a good simplification to remember.

Let’s capture that value and print it:

const config = require('./config.js');

console.log(config);

When you execute main.js now, you’ll see the 2 properties we defined in config.js (PORT and SERVER_URL).

Note how I used process.env variables to make the configurations customizable on different environments. I also made SERVER_URL a function that receives a host argument, which is customizable through the environment as well. Making a configuration value a function allows it to be customizable at run time.

To understand another concept about how Node modules work, let’s repeat the require line in main.js multiple times:

require('./config.js');
require('./config.js');
require('./config.js');

Given these 3 require lines, when we execute main.js, how many times will the console.log in config.js be outputted?

The answer is not 3 times. It’ll only be outputted once.

Modules in Node are cached after the first call. A module is executed the first time you require it, then when you require it again, Node loads it up from a cache.

If you look at front-end applications, like React for example, all component files require the React module, and that’s okay, because only the first require will do the work, the rest will use the cache.

But what if I do want the console.log message to show up multiple times every time we require config.js?

You can actually clear the modules cache, but generally, that’s not a good practice. However, you can make the top export of config.js a function instead of an object, put all the code there inside the function, and call the function every time you need the code to be executed. The cache, in that case, will cache the definition of the function. The Non-object APIs sidebar has an example of that.

Node Modules Lookup

When you require a module in Node, Node uses the following procedure to determine how and where to look for the required module:

If the module does not start with a . (denoting a relative path) or a / (denoting an absolute path), Node will first check if the module is a core one (like fs or http). If it is, it’ll load it directly.

If the module is not a core one, Node will look for it under node_modules folders starting from the directory where the requiring module is, and going up in the folders hierarchy. For example, if the requiring module is in /User/samer/efficient-node/src, Node will first look under src for a node_modules folder, if it does not find one, it’ll look next under efficient-node, and so on all the way to the root path.

You can use this lookup nature to localize modules dependencies by having multiple node_modules folders in your project, but that generally increases the complexity of the project.

You can also use this lookup nature to have multiple projects share a node_modules folder by placing that folder in a parent directory common to all projects, or even have a global node_module folder for all projects on your machine. While this might be useful in some cases, having a single node_modules folder per project is the standard and recommended practice.

If the required module starts with a . or /, Node will look for it in the relative or absolute directory specified by the path.

If you set the NODE_PATH environment variable before executing a script. Node will first look for required modules in the paths specified by NODE_PATH (which can be a single path, or multiple paths separated by a comma). This can be useful to use short absolute paths instead of confusing relative ones. For example, with NODE_PATH set to src, you can require a module under src using require('module') even when the requiring module is multiple levels deep under src, instead of doing something like require('../../../module').

Timer Functions

You can delay the execution of a code block, or make it repeat regularly using timer functions in Node like setTimeout or setInterval. These functions behave very similarly to how they do in browser environments.

A timer function receives a function as an argument. Here’s an example:

const printGreeting = () => console.log('Hello');

setTimeout(printGreeting, 4_000);

This code uses the setTimeout timer function to delay the printing of “Hello” by 4 seconds (the second argument to setTimeout is the delay period in milliseconds).

The printGreeting function (which is passed as the first argument to setTimeout) is the function whose execution will be delayed. This is usually referred to in Node as a callback function.

If we run this script with the node command, Node will pause for 4 seconds and then it’ll print the greeting and exit after that.

To repeat the execution of a block of code, you can use the setInterval timer function. If we replace setTimeout with setInterval in the last example, Node will print the “Hello” message every 4 seconds, forever.

All timer functions can be canceled once they are defined. When you call a timer function, you get back a unique timer ID. You can use that timer ID to cancel the scheduled timer. We can use clearTimeout(timerId) to stop timers started by setTimeout, and clearInterval(timerId) to stop timers started by setInterval.

For example, in this code:

const timerId = setTimeout(
  () => console.log('Hello'),
  0,
);

clearTimeout(timerId);

Even though we started a timer to print a message after 0 milliseconds, that message will not be printed at all because we canceled the timers right after it was defined.

0-milliseconds delayed code is a way to “schedule” code to be immediately invoked when all the synchronous code defined after it is done executing. This is an example of why Node is “non-blocking”. You can basically define code to be executed in a way that does not block the code after it. Here’s an example to understand that:

setTimeout(
  () => {
    for(let i=0; i <= 1_000_000_000; i++) {
      // ...
    }
  },
  0,
);

console.log('Hello');

In this example, although we defined a loop that ticks 1 billion times, that code will not block the printing of the Hello message. The printing will happen first, then the big loop will be executed.

I’m using a big loop here as a simplification of something that’ll take a long time to execute, but in practice, you should never use a big loop like that synchronously in Node, because Node is single-thread, any loop like that will actually block the code after it. For example:

setTimeout(
  () => console.log('Hello!'),
  0,
);

for(let i=0; i <= 1_000_000_000; i++) {
  // ...
};

Here, even though the printing of “Hello” is scheduled to be executed immediately, it will not. Node will have to wait on the for loop to finish first, and then, a few seconds later (when I tested this on my machine), it’ll execute the delayed function.

This is a general observation about timer functions, their delays are not guaranteed to be exact, but rather a minimum amount. Delaying a function by 10 milliseconds, means that the execution of the function will happen after a minimum of 10 milliseconds, but possibly longer depending on the code that comes after it!

Why exactly does a for loop block the code that was scheduled before it? It’s time to dive into the details of Node’s concurrency model.

Concurrency in Node

We learned that Node uses a single-threaded event loop for its non-blocking nature. To understand how that is achieved, we need to learn about a stack and a queue! The stack is known as “The Call Stack”, and the queue is known as “The Event Queue”.

The call stack is part of V8 (not Node), and it’s how V8 manages function calls. A stack is a last-in/first-out data structure. Every time we call a function in our code, a reference to that function is placed on the call stack. When you nest function calls (when functions call other functions), the function references are stacked in the call stack. Then V8 will pop one function at a time (from the top of the stack) to complete the initial call.

Any JavaScript code you write in Node has to be placed in the call stack for V8 to execute it. The call stack is single-threaded, which means when there are functions in the call stack, everything else (including event-driven callbacks) will have to wait until the call stack is available again.

This is exactly why the for loop in the previous example blocked the execution of a function that was scheduled to be immediately executed. We simply made the call stack busy with that loop and you should never do that. Any code that needs to run for a long time should be done with either asynchronous tools, or in its own worker thread (more on that later).

When an asynchronous function like setTimeout is placed on the call stack and it’s time for it to be popped, Node will take control of it, freeing the call stack to pop the next stacked function if any. Asynchronous functions usually have a callback function that needs to be invoked once the asynchronous function is done.

Callback functions can be generalized under the “event” terminology. We define an event, and a function to be executed after that event. For the timer case, the event was “time has passed”, but other events can include user input, changes in system state, or messages from other parts of the program.

This is why there is an “Event Queue” in this structure, Node queues the event functions that are ready to be executed in a queue. When the timer is ready, Node will queue its callback function into the event queue. Multiple event functions can be queued to later be processed in order (a queue is a first-in/first-out structure).

This is where the event loop comes into action. The event loop is a simple infinite loop, continuously ticking to monitor both the call stack and the event queue. When the call stack is free, and there are queued functions in the event queue, the event loop takes the top function in the queue, and places it on the call stack for V8 to execute it in our program. The event loop keeps doing that until there are no functions left in the event queue, in which case, the Node process will exit.

Summary

Node CLI has many powerful options that we can control. We can also pass arguments to it, set environment variables before running it, and both of these options allow us to pass data from the operating system environment to a running Node process. Node’s process object is the bridge.

Node’s REPL mode is a good way to explore everything you can use in Node, and take a quick look at the API of anything, including core modules, installed modules, and even objects you instantiate.

CommonJS Modules in Node are implicitly wrapped in a function and are passed 5 arguments. We use the require function (which is one of the 5 arguments), to make modules depend on each other and get access to their APIs. Node manages a cache for all required modules. To discover where a required module is, Node follows a predefined set of rules depending on the path of the module. A path can be a relative one, an absolute one, or just a name. For the latter case, Node looks for the module in node_modules folders.

Node’s event loop handles asynchronous tasks using the call stack and event queue. The call stack is a data structure managed by V8 that tracks function calls. Any JavaScript code in Node must be placed in the call stack for V8 to execute it. The event queue is used to handle asynchronous tasks such as timers or I/O operations. When an asynchronous function is ready, its callback function is registered to the event queue. The event loop monitors the call stack and event queue, and when the call stack is free, it pops the first function off the event queue and adds it to the call stack for execution.

Why Do We Need a Package Manager

If packages are just folders, why exactly do we need a “package manager” for them?

Keeping track of these package folders becomes challenging when there are many of them, and when these packages depend on other packages. This is especially true for a team of developers working on the same Node project. Package management tools provide a simplified and systemic approach to handling the common tasks around packages. They provide simple commands to install, update, and remove packages, and to ensure that a project has exactly what it needs to function correctly, and similarly on all machines that are running it.

More importantly, package management tools can manage any conflicts among all the dependencies in the project, which are usually referred to as “the dependency tree”. It’s a tree because a project has a main list of dependencies, and these main dependencies have their own dependencies, and so on. The term “transitive dependency” is often used to refer to all the dependencies in a project that are beyond the first level of the dependency tree.

The term “npm” is mainly used to refer to the CLI (npm command) that ships with Node and provides tools to manage Node packages. There is also an npm website (npmjs.com) which hosts the public registry of many open-source npm packages. The npm registry is like a big warehouse full of JavaScript packages, offering many options for common features and functionalities that you might need to add to your projects. For example, if you need your project to handle web requests, web sockets, or connect to a database, you do not need to build these features from scratch or deal with low-level code. You can download and use ready-made and often battle-tested generic solutions from a package registry, and then build your custom needs on top of them.

But, are these ready-made solutions to be trusted? You need to be the judge of that, but many of these packages have already established the trust and respect of the JavaScript and Node communities. All npm packages are open-source, so you can do your own research. There have been bad actors in the space before, so pick your packages carefully and keep an eye on their updates. Even a trusted package might be hacked but looking at the source code changes, and the activities around the code changes (like GitHub issues, pull requests, etc) helps mitigate the risk.

Adopting a systemic approach to managing package dependencies is essential for a team project. With npm, because all package dependencies are configured in the project’s package.json file (which is shared among all developers), it becomes easy to set up a new environment, or update an older one. All developers on the team use similar versions for the project packages, and when conflicts happen, they can be detected early.

Packages usually get updated often to fix bugs, add new features, and improve things overall. With a package manager, you’re in control of how to handle these updates. You can specify which exact versions of packages the project needs, to ensure compatibility and prevent conflicts. You can also automate installing important security patches.

The npm project started with a small set of Node scripts to manage common tasks around folders that contain code for Node and it has since evolved into a fully featured package manager that is useful for all JavaScript code. Not just Node. If you browse the packages that are hosted on https://npmjs.com[npmjs.com], you’ll find packages that are for Node and packages that are libraries and frameworks meant to be used in a browser or a mobile application. If you dig deep enough you’ll even see examples of apps for robots, routers, and countless other places where JavaScript can be executed.

Node packages come in all shapes and sizes. Some represent big frameworks, some represent smaller libraries of a certain utility, and many others provide small isolated utility functions. A typical Node project will have hundreds of npm packages managed under the dependency tree.

The Fundamentals of npm

The npm CLI is the main tool you need to learn. It’s a powerful one that supports many commands. To see usage instructions and the list of all the available commands, you can run npm --help.

~ $ npm --help

npm <command>

Usage:

npm install        install all the dependencies in your project
npm install <foo>  add the <foo> dependency to your project
npm test           run this project's tests
npm run <foo>      run the script named <foo>
npm <command> -h   quick help on <command>
npm -l             display usage info for all commands
npm help <term>    search for help on <term>
npm help npm       more involved overview

All commands:

    access, adduser, audit, bin, bugs, cache, ci, completion,
    config, dedupe, deprecate, diff, dist-tag, docs, doctor,
    edit, exec, explain, explore, find-dupes, fund, get, help,
    hook, init, install, install-ci-test, install-test, link,
    ll, login, logout, ls, org, outdated, owner, pack, ping,
    pkg, prefix, profile, prune, publish, rebuild, repo,
    restart, root, run-script, search, set, set-script,
    shrinkwrap, star, stars, start, stop, team, test, token,
    uninstall, unpublish, unstar, update, version, view, whoami

Don’t be overwhelmed by the amount of commands you see here. You don’t really need many of them. The commands you’ll use often are install, and update. You’ll also probably use run commands like start and test which we’ll learn about later in the chapter.

Most other commands you’ll use infrequently. Here are a few highlights:

• npm init: Initializes a new npm package in a project folder. We used this one in Chapter 1. It asks a few questions about the project, like the name, version, description, and more. It also tries to detect some information about the project and include them as default answers to the interactive questions. It’ll then use the answers to create a package.json file.

• npm search <search terms>: Searches the npm registry for packages based on the provided search query. For example: Try npm search lodash.

• npm list: Displays a tree-like view of installed packages and their dependencies. A common alias here is npm ls.

• npm publish: Publishes your package to the npm registry, making it available for others to install. We’ll see an example of that later in the chapter.

• npm link: Creates a symbolic link between a package in your local file system and a package installed under node_modules (or globally). This allows you to develop and test packages locally without the need for publishing or reinstalling.

• npm cache clean: Clears the npm cache, which can help resolve certain installation issues or outdated package versions.

You can get further details and instructions on any npm command using npm <command> -h. Here’s an example help summary for the npm install command:

~ $ npm install -h
Install a package

Usage:
npm install [<@scope>/]<pkg>
npm install [<@scope>/]<pkg>@<tag>
npm install [<@scope>/]<pkg>@<version>
npm install [<@scope>/]<pkg>@<version range>
npm install <alias>@npm:<name>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>

Options:
[-S|--save|--no-save|--save-prod|--save-dev|--save-optional|..
...

aliases: add, i, in, ins, inst, ...

As you can see, we can use the install command in many ways and with many options. It also has many aliases, like i for example (so you can use npm i express to install the express package).

You don’t need to remember all of the usage ways and options, but a quick scan for later reference is certainly helpful. This is actually the summarized version of the install help page. You can see the full help page using npm help install.

Here are a few challenges for you to figure out from the help text of npm install:

• Install a package that’s hosted under a scope. An npm scope is a way to group related packages under a specific namespace or organization. An example scope is @babel. An example package under that scope is core.

• Install a package directly from GitHub. Try to install lodash from GitHub. To verify, look at the dependencies section of the project’s package.json file. lodash should have a github label.

• Install a package globally to make it available to any Node project on the machine. This option is commonly used for command-line tools. For example, you can install the yarn package globally using npm, and that will make the yarn command available everywhere.

  "lodash": "^4.17.21"

$ npm install lodash@3.9.1

$ npm i express@3

$ npm outdated -a

$ npm ls

efficient-node@1.0.0 /Users/samer/efficient-node
├── accepts@1.3.8 extraneous
├── array-flatten@1.1.1 extraneous
├── body-parser@1.20.1 extraneous
...

$ npm prune

removed 58 packages, and audited 2 packages in 1s

found 0 vulnerabilities

Creating and Publishing Packages

Let’s create and then publish a simple npm package that provides a function named printInFrame. That function takes a string argument and outputs that string within a frame made of * characters.

Let’s name the package “print-in-frame”.

Here’s an example of how we’d use it:

import printInFrame from "print-in-frame";

printInFrame("Hello World");

This should output:

***************
* Hello World *
***************

First, make a new folder to host this package code. The name of the folder usually matches the name of the package (although that’s not a requirement):

$ mkdir print-in-frame & cd print-in-frame

Next step is to make this empty folder into an npm package. We do that by adding a package.json file. We can use npm init for that.

$ npm init

Answer the questions and confirm. You can use the default answers. After the file is created, manually add the "type": "module" to instruct Node that this project will exclusively use ES modules.

Open up your code editor on this folder. Then create an index.js file in the root of the project, and define an empty printInFrame function in there, and make it the default export:

const printInFrame = (text) => {
  // ...
};

export default printInFrame;

To write an implementation for printInFrame, let’s start with a test. Node has a few built-in tools to write and run tests.

Create an index.test.js file in the root of the project, and start it by importing the Node test and assert objects, and the printInFrame function that we need to test:

import test from "node:test";
import assert from "node:assert/strict";

import printInFrame from "./index.js";

The “node:test” module provides a way to organize your tests and describe them. The “node:assert” module provides assertion methods to implement the logic of the tests.

Here’s what I came up with to implement the “Hello World” test:

// ...

const output = printInFrame("Hello World");

const expectedOutput = `
***************
* Hello World *
***************
`.trim();

test("printInFrame", (t) => {
  assert.equal(output, expectedOutput);
});

To run tests with Node, you can use the --test argument:

$ node --test

This will run all tests by scanning through all files to locate the ones named using certain patterns. I like to use the file.test.js pattern. You can also add the --watch command to make Node rerun the tests everytime you change the code.

The one test we have here should obviously fail. To implement the printInFrame function, we basically needs to read the length of the text and use that to print a set of * characters before and after. This can be done in many ways. Here’s what I did:

import times from "lodash.times";

const printInFrame = (text) => {
  const frameWidth = text.length + 4; // 2 stars + 2 spaces

  let textToPrint = "";

  times(frameWidth, () => (textToPrint = textToPrint + "*"));

  textToPrint = textToPrint + "\n" + "* " + text + " *" + "\n";

  times(frameWidth, () => (textToPrint = textToPrint + "*"));

  console.log(textToPrint);

  return textToPrint;
};

export default printInFrame;

I made the function depend on “lodash.times” which provides a function that can repeat a block of code any number of times. I used that to prepare the frame header and footer lines.

You need to npm install lodash.times. After that, running the test again should make it pass.

To use the “print-in-frame” package in a Node project, we need to install it. We can actually install it directly from the file system:

$ npm install ../print-in-frame

While this works okay, when you share your code with others, you’ll have to share the “print-in-frame” folder as well. To keep them separate, we’ll need to use an npm registry and publish the package there.

If you want to publish your package on npmjs.com, you need to have an account there. Then you can use the npm login command to authenticate your local npm client with your account. It’ll ask you for your username and password.

Since the package name is unique at the npm registry, to avoid conflict, add a prefix to your package name. I changed the name property in package.json to “samer-print-in-frame”. While you’re there, add a description to the package as well. It’s optional, but it makes the package easier to discover.

When you’re ready, run the npm publish command. If everything works, your package will be available at npmjs.com (use the UI search there to find it). You can also use the npm search command to find it.

With the package published, in your main Node project, install it with: npm install PREFIX-print-in-frame, replacing “PREFIX” with the prefix that you used.

Now look at the output of npm ls. You should see 2 new dependencies: print-in-frame, and lodash.times.

Now to make updates to your package and test them in a project before you publish a new version, you can use the npm link command to temporarily make a project use a local package rather than the one installed through the registry. In the print-in-frame folder, run npm link, then in the main project folder, run npm link PREFIX-print-in-frame.

Now you can make changes to your local package folder and test them in your main project. Once you’re done, you can increment the version property in the package.json file under your package, and run npm publish again.

  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },

  "scripts": {
    ...
    "list-unused-packages": "npm ls | grep 'extraneous'"
  },

  "scripts": {
    ...
    "pretest": "npm prune && npm install"
  },

  "scripts": {
    ...
    "lint": "eslint"
  },

$ npm run lint --help

npm view eslint versions

npx create-react-app your-app-name-here

npx @vue/cli --help

Summary

A package manager like npm is an important part of working on a Node project. It introduces a simple and standard way to deal with project external dependencies and keep them updated, consistent, and conflict-free.

npm packages are hosted on a public registry and the npm command is configured to work with that registry. npm commands like install, update, search, and more work with that registry.

The package.json and package-lock.json files are automatically modified by npm every time there is a change to the project dependency tree. These files store what versions of packages are installed and what range of versions to use when updating packages.

In addition to the npm command, there’s also an npx command that can be used to execute local or remote command-line tools.