Spring Boot & ELK Stack with Log4j2 Socket Appender

Overview

In this article we are getting spring boot integration with logstash and ELK Stack via log4j2.

ELK Stack

Logstash : is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination. It is most often used as a data pipeline for Elasticsearch, an open-source analytics and search engine.

Elasticsearch : is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

Kibana : is a data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support.

Dockerizing ELK Stack

First of all create a directory with name volumes, than go to inside that directory. Now we should create directories with names elasticsearch, kibana and logstash. Let’s create a new file with name elasticsearch.yml in ./volumes/elasticsearch directory. The file content ;

cluster.name: "docker-cluster-deneme"
network.host: 0.0.0.0
discovery.type: single-node

Now we need kibana.yml file at ./volumes/kibana directory, file content ;

server.name: kibana
server.host: "0"
elasticsearch.hosts: ["http://elasticsearch:9200"]

Finally we are gonna create logstash.conf file at ./volumes/logstash directory

input {
   tcp {
        port => 9999
        codec => line
    }
}
output {
    elasticsearch {
        hosts => "elasticsearch:9200"
    }
}

Now we are ready to create docker-compose.yml file in root directory.For dockerizing ELK Stack you can use this simple docker-compose

version: '3'

services:
  elasticsearch:
    image: elasticsearch:7.10.1
    container_name: elasticsearch
    volumes:
      - ./volumes/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
  logstash:
    image: logstash:7.10.1
    container_name: logstash
    command: -f /etc/logstash/conf.d/
    volumes:
      - ./volumes/logstash/:/etc/logstash/conf.d/
    ports:
      - "9999:9999"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch
  kibana:
    image: kibana:7.10.1
    container_name: kibana
    volumes:
      - ./volumes/kibana/:/usr/share/kibana/config/
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

Now we are need an application to send logs ELK Stack. Let’s create a new spring boot application.

In pom.xml file we need some dependencies.

<dependencies>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter</artifactId>
      <exclusions>
         <exclusion>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
         </exclusion>
      </exclusions>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
   </dependency>
   <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <optional>true</optional>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-log4j2</artifactId>
   </dependency>
</dependencies>

After adding dependencies now create a simple hello world controller.

@RestController
@Log4j2
public class HelloWorldController {

    @GetMapping
    public void helloWorld() {
        log.info("Hello world");
    }
}

That’s all we need for testing ELK Stack with spring boot. In this article we are sending logs to logstash over log4j2. Now we need to configure log4j2.xml under resources package.

Log4j2.xml ;

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Properties>
        <Property name="logPath">logs/elk-example/</Property>
        <Property name="rollingFileName">elk-example_app</Property>
        <Property name="defaultPattern">[%highlight{%-5level}] %d{DEFAULT} %c{1}.%M() - %msg%n%throwable{short.lineNumber}</Property>
        <Property name="logStashPattern">[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n]</Property>
    </Properties>
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout pattern="${defaultPattern}" />
        </Console>
        <RollingFile name="rollingFile" fileName="${logPath}/${rollingFileName}.log" filePattern="${logPath}/${rollingFileName}_%d{yyyy-MM-dd}.log">
            <PatternLayout pattern="${defaultPattern}" />
            <Policies>
                <OnStartupTriggeringPolicy />
                <TimeBasedTriggeringPolicy interval="1" modulate="true" />
            </Policies>
        </RollingFile>
        <Socket name="socket" host="${sys:logstash.host.name:-localhost}" port="${sys:logstash.port.number:-9999}" reconnectionDelayMillis="5000">
            <PatternLayout pattern="${defaultPattern}" />
        </Socket>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="console" />
            <AppenderRef ref="rollingFile"/>
            <AppenderRef ref="socket"/>
        </Root>
    </Loggers>
</Configuration>

In log4j2.xml file socket appender allows send logs to logstash. We are getting logstash host name and port number as environment variable.

Dockerizing Spring Boot Hello World Application

For dockerizing spring boot application create Dockerfile in root directory of project.

FROM maven:3.6.3-adoptopenjdk-11 as maven-builder

WORKDIR /tmp

COPY . ./

ENV MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=128m"

RUN mvn clean package -DskipTests=true

FROM adoptopenjdk/openjdk11:latest

ENV JAR_FILE=target/*.jar

COPY --from=maven-builder /tmp/$JAR_FILE /opt/app/
RUN mv /opt/app/*.jar /opt/app/app.jar

WORKDIR /opt/app

RUN chgrp -R 0 /opt/app && \
    chmod -R g=u /opt/app

ENV PORT 8080

EXPOSE 8080

ENTRYPOINT ["java","-Dlogstash.host.name=logstash","-Dlogstash.port.number=9999","-jar","app.jar"]

This dockerfile is a multi stage dockerfile. After docker build command application creates a new jar file and we serve that jar file.

Now we can add spring application service to docker-compose.yml file.

version: '3'

services:
  elasticsearch:
    image: elasticsearch:7.10.1
    container_name: elasticsearch
    volumes:
      - ./volumes/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
  logstash:
    image: logstash:7.10.1
    container_name: logstash
    command: -f /etc/logstash/conf.d/
    volumes:
      - ./volumes/logstash/:/etc/logstash/conf.d/
    ports:
      - "9999:9999"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch
  kibana:
    image: kibana:7.10.1
    container_name: kibana
    volumes:
      - ./volumes/kibana/:/usr/share/kibana/config/
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
  app:
    build: .
    ports:
      - "8080:8080"
    links:
      - logstash

Building Containers

If you already installed docker just run $ docker-compose build command.

Running Containers

Containers builded successfully then you can start containers with $ docker-compose up command.

http://localhost:5601 => You can visualize logs at ELK Stack.Open the kibana and click discover menu. First of all you should add index pattern. Then open discover panel again.

Hit the application rest endpoint with $ curl http://localhost:8080.Now you can see Hello world message in Kibana discover panel.

You can find full example at Github.

https://github.com/farukkkaradeniz/spring-boot-elk-via-docker