ExpressJS Advanced Topics

Middleware

• In ExpressJS we already mentioned about the Middleware. So, basically the Middleware function is basically a function that takes a request object, and either returns a response to the client, or passes control to another Middleware function.
• So in express, every route handler function we have is technically a Middleware function, because it takes a request option, and in this case it returns a response to the client, so it terminates the request response cycle.

• Express includes a few built in Middleware functions, but we can also create custom Middleware functions that we can put at the front of our request processing pipeline. So every request that we get on the server, will go through our Middleware function, but this custom Middleware function, we can perform cross cutting concerns. For example, we can do logging, authentication, authorization, and so on.

Create A Custom Middleware

• Before beginning with the example, we will clone the existing example express-demo in ExpressJS and name it as express-demo-middleware.
• To use a Middleware function into Express we need to use the method use. For example.

app.use(express.json());
...

• In Express, the use() method is a core function that is used to apply middleware to the application's request-response cycle.

  • Add Middleware Globally: We can use app.use() to add middleware that will be executed for every request made to the application. This is often used for tasks like logging, authentication, parsing request bodies, setting headers, and more.

  • Add Middleware to Specific Routes: We can use app.use() with a specific path to apply middleware only to requests that match that path. For example, app.use('/api', middlewareFunction) would apply the middleware only to routes that start with /api.

  • Chain Multiple Middleware Functions: We can chain multiple middleware functions together using multiple app.use() calls, which will execute in the order they are added.

const express = require('express');
const app = express();

// Global Middleware
app.use((req, res, next) => {
    console.log('Middleware 1 - Global');
    next(); // Pass control to the next middleware
});

// Middleware for a specific route
app.use('/api', (req, res, next) => {
    console.log('Middleware 2 - /api');
    next();
});

// Another global middleware
app.use((req, res, next) => {
    console.log('Middleware 3 - Global');
    next();
});

app.listen(3000, () => {
    console.log('Server started on port 3000...');
});

• Creating a Middleware function in Express involves defining a JavaScript function that takes three arguments: req (request), res (response), and next. The next argument is a function that, when called, passes control to the next middleware in the stack. If next is not called, the request-response cycle will not proceed beyond this middleware.

• Okay, let's start with simple global Middleware functions. Let's create 2 files: logger.js and authentication.js with the content as below in our express-demo-middleware.

function log(req, res, next) {
  console.log("Logging...");
  next();
}

module.exports = log;

function authenticate(req, res, next) {
  console.log("Authenticating...");
  next();
}

module.exports = authenticate;

• As we can see, two methods above contains 3 arguments req, res and next and we will call next() at the end of the function to make sure we pass the control to the other Middleware.
• Next, we will import those Middleware functions and use them as in the code below.

const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');

app.use(express.json());

app.use(logger);
app.use(authentication);

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Now, let's start our node application and try to call any api, then we can see the Middleware functions are triggered.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js 
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
listening on port 5000...
Logging...
Authenticating...

Built-In Middleware

• In express we have a few built in Middleware functions. Below is the table contains all the built-in Middleware functions of Express.

Middleware Function	Description
express.json()	Parses incoming JSON payloads and makes the parsed data available on req.body.
express.urlencoded()	Parses incoming URL-encoded payloads and makes the parsed data available on req.body.
express.text()	Parses incoming request payloads as plain text and makes the data available on req.body.
express.raw()	Parses incoming request payloads as buffers and makes the data available on req.body.
express.static()	Serves static files (e.g., HTML, CSS, images) from a specified directory.
express.Router()	Creates a modular route handler using the Router class.
express.cookieParser()	Parses Cookie header and populates req.cookies with an object of cookie key-value pairs.
express.session()	Adds session support, allowing you to store user data across requests.
express.csrf()	Implements Cross-Site Request Forgery (CSRF) protection by generating and validating tokens.
express.logger()	Logs information about incoming requests, including method, URL, response status, and response time.
express.compress()	Compresses response data before sending it to the client to reduce transfer size.
express.timeout()	Sets a maximum request processing time, terminating requests that exceed the specified duration.
express.errorHandler()	Handles errors and sends appropriate error responses.
express.cookieSession()	Provides session support by storing session data in cookies.
express.bodyParser()	Parses incoming request bodies (deprecated in favor of express.json() and express.urlencoded()).
express.methodOverride()	Overrides HTTP methods (e.g., POST to PUT/DELETE) using the _method query parameter or X-HTTP-Method-Override header.
express.basicAuth()	Implements basic authentication for routes.
express.query()	Parses the query string and populates req.query with the parsed query parameters.
express.responseTime()	Adds an X-Response-Time header to responses indicating the response time in milliseconds.
express.favicon()	Serves a favicon (icon associated with a website) from a specified file.
express.directory()	Serves a directory listing of files in a specified directory.
express.limit()	Limits the size of incoming request bodies.
express.vhost()	Supports virtual hosting (multiple domains on a single server) by directing requests to different middleware based on the host value.
express.jsonp()	Responds to JSONP requests (cross-domain requests using the <script> tag) by wrapping the JSON response in a callback function.
express.rawBody()	Provides raw request bodies without parsing.
express.sslify()	Redirects HTTP requests to HTTPS for secure communication.
express.xmlBodyParser()	Parses incoming XML request payloads and makes the parsed data available on req.body.
express.cors()	Adds Cross-Origin Resource Sharing (CORS) headers to responses, allowing controlled sharing of resources across domains.

• Now, let's take an example for using built-in middlewares, we will use express.urlencoded() and express.static() methods as below.

const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');

app.use(express.json());

app.use(logger);
app.use(authentication);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));


const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• With app.use(express.urlencoded({extended: true})); we will parses incoming request payloads as plain text and makes the data available on req.body.
• Next with app.use(express.static('public')); we will serve a static files (e.g., HTML, CSS, images) from a specified directory public.
• Let's create a folder public which contains a simple index.html file as below.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
    This is a static html file!
</body>
</html>

• Then let's start the node application and use postman to test the express.urlencoded() built-in middleware as below.

• Then if we use browser and open localhost:5000 we can see the static html file is loaded as below.

Third-Party Middleware

• Third-party middleware in Express refers to middleware functions that are not built-in to the Express framework itself but are instead developed and maintained by the broader developer community. These middleware modules are created to handle specific tasks or functionalities that might be common in web applications, such as authentication, logging, data validation, security enhancements, and more.
• We can find all of third-party middleware at this official link of Express.
• Let's take an example by using some third-party middleware like Helmet and Morgan which are used for securing Express apps by setting HTTP response headers and logging HTTP Requests.
• Firstly, let's install the Helmet and Morgan third-party middlewares as below.

npm install helmet
npm install morgan

• Then we use them in the index.js file as below.

const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');
const helmet = require('helmet');
const morgan = require('morgan');

app.use(express.json());

app.use(logger);
app.use(authentication);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());
app.use(morgan('tiny'));


const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Now, let's start node application and call any api we will get the results as below.

• As you can see, with Helmet middleware we can see there are many security headers added into the response.
• Then if we look into the node application log, we can see the api that we call had logged.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js 
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
listening on port 5000...
Logging...
Authenticating...
GET /api/courses 200 82 - 1.230 ms

Environments

• In a more complex, or enterprise like application, we need to know what environment our code is running on. Is this a development environment, or a production environment?Perhaps we may want to enable or disable certain features based on the current environment.
• We learned about this process object in Global Object, this global object gives us the way to access to the current process. This process object has a property called env, which gives us the environment variables.
• Now we have a standard environment variable called NODE_ENV, and this environment variable returns the environment for this node application. If it's not set, here we're going to get undefined.

process.env.NODE_ENV  //return undifined if NODE_ENV is not set

• Now we have another way to get the current environment, and that is for the app object. So this app object has a method called get that we use to get various settings about this application.

app.get('env');

• This method internally uses this environment variable to detect the current environment. However, if this environment variable is not set, this will return development by default.

• Okay let's add the log below into the index.js file.

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');
const helmet = require('helmet');
const morgan = require('morgan');

app.use(express.json());

app.use(logger);
app.use(authentication);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());
app.use(morgan('tiny'));

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Now, when we start the application then we can see in the node application log 2 results undefined and development as below.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js 
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: undefined
app: development
listening on port 5000...

• Then if we want to enable morgan only on the development machine. So we can write code like this.

const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');
const helmet = require('helmet');
const morgan = require('morgan');

app.use(express.json());

app.use(logger);
app.use(authentication);

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());

if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  console.log('Morgan enabled...');
}

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Now, let's start the node application then you can see the Morgan is enabled as below.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: undefined
app: development
Morgan enabled...
listening on port 3000...

• Then now, let's set env as test then we can see the Morgan is not enabled as below.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ export NODE_ENV=test
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: test
app: test
listening on port 3000...

Configuration

• Okay, now we will continue to learn how to store configuration settings for our node application and override them in each environment.
• There are many node packages for managing configuration and the most popular one is config.
• The config provides a way to define configuration parameters for different environments (development, production, testing, etc.) and makes it easy to access these settings within our code.

• The config package allows use to:

  • Define configuration files in various formats (JSON, YAML, etc.) for different environments.
  • Access configuration values based on a hierarchical structure that includes environment-specific overrides.
  • Easily switch between different configurations based on the environment in which your application is running.
  • Provide default values for configuration options.
  • Keep sensitive data, such as API keys or database credentials, separate from your code.

• Now, let's install config for using.

npm install config

• After that, let's create a folder config for storing application configurations following environment.
• In side the folder config, let's create 3 json files: default.json, development.json and production.json as below.

{
    "name": "My Express App"
}

{
    "name": "My Express App - Development",
    "mail": {
        "host": "dev-mail-server"
    }
}

{
    "name": "My Express App - Production",
    "mail": {
        "host": "prod-mail-server"
    }
}

• Then in the index.js , let's import the config package, then to get the configuration in json files we just simply use config.get('<fieldName>') as in the example below.

const config = require('config');
const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');
const helmet = require('helmet');
const morgan = require('morgan');

app.use(express.json());

app.use(logger);
app.use(authentication);

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());

// Configuration
console.log('Application Name: ' + config.get('name'));
console.log('Mail Server: ' + config.get('mail.host'));

if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  console.log('Morgan enabled...');
}

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• As we can see, we will log out values that we get from json file using config. Let's start our node application with development env then we can see the result as below.

export NODE_ENV=development

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ export NODE_ENV=development
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: development
app: development
Application Name: My Express App - Development
Mail Server: dev-mail-server
Morgan enabled...
listening on port 5000...

• Then let's try with production env.

export NODE_ENV=production

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ export NODE_ENV=production
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: production
app: production
Application Name: My Express App - Production
Mail Server: prod-mail-server
listening on port 5000...

• Okay, please note that in the json configuration files, we should not put sensitive value such as username/password or credentials because when we're checking our source code, to a repository, that password or that secret is visible to anyone who has access to that source control repository. So to deal with it we let's store these secrets in environment variables.
• For example, we will export an app_password environment variable and we will use config module to read it.

export app_password=123456

• Then let's create a json file custom-environment-variables.json with the value is the environment name that we have just exported above.

{
    "mail": {
        "password": "app_password"
    }
}

• Next, let's go back to the index.js and add a using the config module to get the environment variable through custom-environment-variable.json as below.

const config = require('config');
const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');
const helmet = require('helmet');
const morgan = require('morgan');

app.use(express.json());

app.use(logger);
app.use(authentication);

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());

// Configuration
console.log('Application Name: ' + config.get('name'));
console.log('Mail Server: ' + config.get('mail.host'));
// console.log('Mail Password: ' + config.get('mail.password'));

if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  console.log('Morgan enabled...');
}

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.send("Hello World!!!");
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Now, let's reload our node application again then we can see the app_password that we set into the environment.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-middleware$ nodemon index.js 
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: production
app: production
Application Name: My Express App - Production
Mail Server: prod-mail-server
Mail Password: 123456
listening on port 5000...

Debugging

• When we work with nodejs, we usually use console.log for debug logging but this approach has some problems because not every time we will need to log out and usually we will remove it when we have done some code implementation and when we need it again we have to add the console.log manually. It will take time for us.
• So there is a better way is using debug node package.

• The debug package is a third-party npm package for debugging Node.js applications. It provides a flexible and convenient way to add debug logs to your codebase, allowing you to toggle these logs on and off based on environment variables. The debug package is particularly useful for gaining insights into the behavior of your application during development and troubleshooting issues.

  • Namespaced Debugging: The debug package allows you to create debug instances with different namespaces. This makes it easy to categorize and control different parts of your application's debugging output.
  • Environment-Based Control: Debugging logs can be enabled or disabled based on environment variables. This means you can keep the debug logs in your code and control their visibility without having to modify the code itself.
  • Color-Coded Output: The debug package provides color-coded output for better readability of debugging messages.
  • Configurable Output Streams: You can configure where the debugging logs are sent. By default, they are sent to the console, but you can redirect them to other streams, such as files.

• Now, firstly let's create another project name express-demo-debug with the package.json as below.

{
  "name": "express-demo-debug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2",
    "morgan": "^1.10.0",
    "nodemon": "^3.0.1"
  }
}

• Then, let's create a simple index.js file as below.

const express = require("express");
const app = express();
const morgan = require('morgan');


if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  console.log('Morgan enabled...');
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Then, let's install the debug node package using the command below.

npm install debug

• Okay, let's create debug instances with namespaces. To create an instance with a namespace we will use the syntax as below.

const namespace = require('debug')('<namespace>');

• In the example below we will create 2 namespaces app:startup and app:db and use them for logging information.

const startupDebugger = require('debug')('app:startup');
const dbDebugger = require('debug')('app:db');
const express = require("express");
const app = express();
const morgan = require('morgan');


if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  startupDebugger('Morgan enabled...');
}

// DB work...
dbDebugger('Connected to the database...');

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Next, we will set environment variable to decide which namespace we will use for debugging in our node application. For example we will use the namespace app:startup.
• So let's export an environment variable DEBUG with value is the namespace app:startup

export DEBUG=app:startup

• Then let's start our node application then you can see the result as below.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ export DEBUG=app:startup
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ nodemon index.js 
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
  app:startup Morgan enabled... +0ms
listening on port 3000...

• As you can see, only the log with namespace app:startup is logged out.
• Let's try to set the DEBUG environment with value app:db and start the node application again.

export DEBUG=app:db

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ export DEBUG=app:db
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
  app:db Connected to the database... +0ms
listening on port 3000...

• Again, only the log with namespace app:db is logged out.

• Okay then, if we want to apply multiple namespace we can set the value for DEBUG environment as below.

export DEBUG=app:startup,app:db

• We will use the , between namespaces.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ export DEBUG=app:startup,app:db
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
  app:startup Morgan enabled... +0ms
  app:db Connected to the database... +0ms
listening on port 3000...

• In case, we want to apply all the namespaces we can set like this.

export DEBUG=app:*

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ export DEBUG=app:*
duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
  app:startup Morgan enabled... +0ms
  app:db Connected to the database... +0ms
listening on port 3000...

• Now there's also a faster way to set the level of debugging we want to see, so we don't have to explicitly set the environment variable, using the export command. We can set the environment variable at the time of writing our application using the command below.

DEBUG=app:db nodemon index.js

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-debug$ DEBUG=app:db nodemon index.js
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
  app:db Connected to the database... +0ms
listening on port 3000...

Templating Engines

• A templating engine in Node.js is a tool that helps generate dynamic HTML content by combining templates and data. It allows us to create reusable templates with placeholders that are replaced with actual data when the template is rendered. This is particularly useful for generating web pages, emails, or any other content where you need to display dynamic data.
• There are various templating engines available for express applications. The most popular ones are Pug, which used to be called Jade. We also have Mustache, and EGS.
• Okay now, let's take an example with Pug.
• Firstly, let's clone project express-demo-middleware and name it as express-demo-pug.
• Next, let's install Pug as in the command below.

npm install pug

• Next, let's create a folder views with a pub file index.pug as below.

html
    head 
        title= title 
    body 
        h1= message

• The title element's content is dynamically set using the Pug variable title. The = syntax denotes interpolation, meaning the value of title will be inserted into the HTML.

• Inside the <body>, there is an h1 element. Similar to before, the h1 element's content is dynamically set using the Pug variable message.

• Then let's configure express to use pug as in the code below.

app.set('view engine', 'pug');
app.set('views', './views'); //default

• Then let's update the default api / which will return an html file with the response as below.

//...
res.render('index', {title: 'My Express App', message: 'Hello'});
//...

• As you can see, the first argument contains index value, it is the name of pug file in folder views. The second argument is the object that contain key-value pairs in which the key set in index.pug.

const config = require('config');
const Joi = require("joi");
const express = require("express");
const app = express();
const logger = require('./logger');
const authentication = require('./authentication');
const helmet = require('helmet');
const morgan = require('morgan');

app.set('view engine', 'pug');
app.set('views', './views'); //default

app.use(express.json());
app.use(logger);
app.use(authentication);

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());

// Configuration
console.log('Application Name: ' + config.get('name'));
console.log('Mail Server: ' + config.get('mail.host'));
console.log('Mail Password: ' + config.get('mail.password'));

if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  console.log('Morgan enabled...');
}

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

app.get("/", (req, res) => {
  res.render('index', {title: 'My Express App', message: 'Hello'});
});

app.get("/api/courses", (req, res) => {
  res.send(courses);
});

app.get("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

app.post("/api/courses", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

app.put("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

app.delete("/api/courses/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

// app.get('/api/courses/:year/:month', (req, res) => {
//     // res.send(req.params);
//     res.send(req.query);
// });

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• Now, let's start our node application and go to browser, we will see the result as below.

Structuring Express Applications

• In a real world application we don't want to write all that code inside index.js. So in this lecture, we will learn how to structure our application.
• Firstly, let's clone the project express-demo-pug to new project express-demo-structure.
• Then we will restructure the apis, let's create a folder routes and create a js file base on the api. For example in the index.js we have /api/courses apis. So we will create courses.js file in routes folder as below.

const Joi = require("joi");
const express = require("express");
const router = express.Router();

const courses = [
  { id: 1, name: "course 1" },
  { id: 2, name: "course 2" },
  { id: 3, name: "course 3" },
];

router.get("/", (req, res) => {
  res.send(courses);
});

router.get("/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");
  res.send(course);
});

router.post("/", (req, res) => {
  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  const course = {
    id: courses.length + 1,
    name: req.body.name,
  };
  courses.push(course);
  res.send(course);
});

router.put("/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const { error } = validateCourse(req.body);
  if (error) return res.status(400).send(error.details);

  course.name = req.body.name;
  res.send(course);
});

router.delete("/:id", (req, res) => {
  const course = courses.find((ci) => ci.id === parseInt(req.params.id));
  if (!course)
    return res.status(404).send("The course with the given Id was not found");

  const index = courses.indexOf(course);
  courses.splice(index, 1);

  res.send(course);
});

function validateCourse(course) {
  const schema = Joi.object({
    name: Joi.string().min(3).required(),
  });
  return schema.validate(course);
}

module.exports = router;

• As you can see, we will use express.Router() to create a router object and we don't need to add /api/courses for every api again, we will do it in index.js. At the end of this file we will export the router object.
• We will do the same for home api / with file home.js as below.

const express = require('express');
const router = express.Router();

router.get("/", (req, res) => {
    res.render('index', {title: 'My Express App', message: 'Hello'});
  });

module.exports = router;  

• Next, as we remember that in the index.js we have used a middlewares name logger and authentication from logger.js and authentication.js files respectively. So let's create a folder middleware and move these js files there.
• Finally in the index.js, we will remove all apis, we just need to import routes and middlewares and use them.

const config = require('config');
const express = require("express");
const app = express();
const logger = require('./middleware/logger');
const authentication = require('./middleware/authentication');
const helmet = require('helmet');
const morgan = require('morgan');
const courses = require('./routes/courses');
const home = require('./routes/home');

app.set('view engine', 'pug');
app.set('views', './views'); //default

app.use(express.json());
app.use(logger);
app.use(authentication);

console.log(`NODE_ENV: ${process.env.NODE_ENV}`);
console.log(`app: ${app.get('env')}`);

app.use(express.urlencoded({extended: true}));
app.use(express.static('public'));
app.use(helmet());
app.use('/api/courses', courses);
app.use('/', home);


// Configuration
console.log('Application Name: ' + config.get('name'));
console.log('Mail Server: ' + config.get('mail.host'));
console.log('Mail Password: ' + config.get('mail.password'));

if (app.get('env') === 'development') {
  app.use(morgan('tiny'));
  console.log('Morgan enabled...');
}

const port = process.env.PORT || 3000;
app.listen(port, () => console.log(`listening on port ${port}...`));

• As you can see, for the imported routes. We will define the path for specific route that we imported.

//...
app.use('/api/courses', courses);
app.use('/', home);
//...

• Let's start our node application again then we can see it can run normally.

duc@duc-MS-7E01:~/study/nodejs-backend/udemy/express-demo-structure$ nodemon index.js 
[nodemon] 3.0.1
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node index.js`
NODE_ENV: undefined
app: development
Application Name: My Express App - Development
Mail Server: dev-mail-server
Mail Password: 1234
Morgan enabled...
listening on port 3000...

• Okay, that's all we will learn about connecting to MongoDb and setup authentication in next lectures.

See Also

• ExpressJS

References

• Node.js: The Complete Guide to Build RESTful APIs
• Full Source Code