Fluentd in docker with Node.js

Run Fluentd in docker container and let node.js push log messages to it

This is is going to be very short blog for those who know about Node.js, express.js web apps, Little bit about docker containers but don’t know how to use fluentd inside docekr container and push logs to fluentd from Node js web app. I am assuming yoyu are using ubuntu 16.4 or greater. Please refer to respective documents for your linux.

Let’s get started….

refer below commands to run fluentd in docker so that it can listen for log messages coming from node.js on HTTP protocol. Open terminal window (i will refer it as window 1 so that it is easy to tell you which window i am referring to while you progress) Make sure you are at root path.

/$ pwd
/
/$ cd tmp/
/tmp$ touch fluentd.conf

Now open the file in gedit to edit it

/tmp$ gedit fluentd.conf

and add the configuration as shown below. This example accepts records from http, and outputs to stdout.

<source>
  @type http
  port 9880
  bind 0.0.0.0
</source>
<match **>
  @type stdout
</match>

Now verify the configuration with cat command

/tmp$ cat fluentd.conf 
<source>
  @type http
  port 9880
  bind 0.0.0.0
</source>
<match **>
  @type stdout
</match>

Now start the docker container of fluentd with below command. Note that it is ready to listen for incoming logs on http.

/$ sudo docker run -p 9880:9880 -v $(pwd)/tmp:/fluentd/etc -e FLUENTD_CONF=fluentd.conf fluent/fluentd:v1.6-debian-1
[sudo] password for user: 
2021-01-06 11:16:29 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2021-01-06 11:16:30 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type http
    port 9880
    bind "0.0.0.0"
  </source>
  <match **>
    @type stdout
  </match>
</ROOT>
2021-01-06 11:16:30 +0000 [info]: starting fluentd-1.6.3 pid=7 ruby="2.6.3"
2021-01-06 11:16:30 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluentd.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2021-01-06 11:16:30 +0000 [info]: gem 'fluentd' version '1.6.3'
2021-01-06 11:16:30 +0000 [info]: adding match pattern="**" type="stdout"
2021-01-06 11:16:30 +0000 [info]: adding source type="http"
2021-01-06 11:16:30 +0000 [info]: #0 starting fluentd worker pid=14 ppid=7 worker=0
2021-01-06 11:16:30 +0000 [info]: #0 fluentd worker is now running worker=0
2021-01-06 11:16:30.495744139 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}

Once the fluentd container is ready open another terminal window (i will call it window 2) and create node app by executing commands as follows. It is not necessary to create Node app directory in home/user/test-code path. It could be anywhere where you need it to be. I have added in this path because i prefer it that way.

/$ pwd
/
/$ cd home/user/
~$ mkdir test-code/
~$ cd test-code/
~/test-code$ pwd
/home/user/test-code
~/test-code$ mkdir node-fluent-app
~/test-code$ cd node-fluent-app/
~/test-code/node-fluent-app$ npm init -y
Wrote to /home/user/test-code/node-fluent-app/package.json:
{
  "name": "node-fluent-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Now start creating node app as follows. first by installing express and axios npm packages.

~/test-code/node-fluent-app$ npm i --save express axios
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN node-fluent-app@1.0.0 No description
npm WARN node-fluent-app@1.0.0 No repository field.
+ express@4.17.1
+ axios@0.21.1
added 52 packages from 41 contributors and audited 52 packages in 11.458s
found 0 vulnerabilities
~/test-code/node-fluent-app$ ls
node_modules  package.json  package-lock.json

Now create index.js file which will be your Node server.

~/test-code/node-fluent-app$ touch index.js

Now edit index.js in your code editor. The code is as follows.

const axios = require('axios')
const express = require('express')
const app = express()
app.get('/',(req,res)=>{
axios.post('http://localhost:9880/sample.test', {
    requestPath:req.path
})
res.send(` added req.rawHeaders to fluentd docker container ${req.path} Just check the terminal window`)
})
app.listen(3000,()=>{
    console.log('listening on 3000')
})

Now goto terminal window (window 2) and run the Node app as shown below.

~/test-code/node-fluent-app$ node index.js 
listening on 3000

Now goto browser window and visit http://localhost:3000

After you visit above url, node app will push log message to fluentd running in docker container. If you goto the terminal window (window 1) where you started docker container, you will see the logs (at the end) as follows. This is the request path that I wanted to log. But you can add logs according to your need.

2021-01-06 14:22:14.947613413 +0000 sample.test: {"requestPath":"/"}

Reference : Fluentd-In-Docker