Skip to content

Docker-Based Installation

This short guide describes how to run, for testing and exploration, the latest version of n6 – using the Docker and Docker Compose tools.

The goal of this guide is to give you an example of how to easily run the n6 system, so that you can learn – by experimenting with the stuff – what the basic building blocks of n6 are and how they interact with each other and with the outside world.

Disclaimer: what these materials are and what they are not

This installation guide, as well as the stuff you can find in the etc/ and docker/ directories of the n6 source code repository, concern setting up an n6 instance just for testing, exploration and experimentation, i.e., not for production (at least, not without careful security-focused adjustments).

In other words, these materials are not intended to be used as a recipe for a secure production setup – in particular, when it comes to (but not limited to) such subjects as X.509 certificates (note that those in the etc/ssl/* directories of the source code repository are purely example ones – they should never be used for anything related to production systems!), authentication and authorization settings (which in these materials are, generally, either skipped or reduced to what is necessary just to run the stuff), or file access permissions.

It should be obvious that an experienced system administrator or security expert should prepare and/or carefully review and adjust any configuration/installation/deployment of services that are to be made production ones, in particular if those services are to be made public.

Requirements

Note

This guide is Linux-dedicated. Despite the use of Docker, we do not guarantee that the described operations will work on other systems (such as Windows or Mac OS).

Important

Internet access is required during the entire installation process.

Building the environment

Note

Make sure you are in the top-level directory of a cloned n6 source code repository.

To build our demonstrational n6 environment we use Docker Compose which binds all the services needed to run the n6 infrastructure.

docker compose build

The result of the process are ready-to-use docker images.

Tip

Sometimes docker images might not be built correctly due to external factors. In case of any errors, try building the images again.

Note

The Docker stack requires all images to be built correctly. In case of an irrecoverable error, please do not hesitate to create an issue on our GitHub site.

After successfully building the images, run the following command:

docker images | grep n6

The command output should be similar to this:

n6_mysql            latest              a34ee42c8e58        20 minutes ago      551MB
n6_rabbit           latest              841d42d17010        20 minutes ago      250MB
n6_web              latest              1f219d032515        21 minutes ago      2.39GB
n6_worker           latest              ec6c16c8bee5        22 minutes ago      1.67GB
n6_base             latest              1263daaf01e0        22 minutes ago      1.42GB
n6_mailhog          latest              8df1ccd08649        22 minutes ago      392MB

Basic features of the prepared Docker stack

n6-specific Docker images to be run:

  • n6_worker – Python environment running the n6 pipeline stuff (collectors, parsers etc.)
  • n6_mysql – MariaDB database instance, running the n6’s Event DB and Auth DB
  • n6_rabbit – RabbitMQ message broker
  • n6_web – n6 services: REST API, Portal API+GUI, Admin Panel

Another image we also make use of:

  • n6_mongo:4.2 – MongoDB database (NoSQL) running the n6’s Archive DB
  • n6_base – Dependencies running the n6’s

By default, the stack exposes the following ports:

  • 80 – redirects to 443 (to use HTTPS)
  • 443 – n6 Portal GUI + n6 Portal API (/api)
  • 3001 – n6 Portal GUI parameterization configurator
  • 4443 – n6 REST API
  • 4444 – n6 Admin Panel
  • 1025 – Mailhog SMTP Server
  • 8025 – Mailhog Web GUI
  • 15671 – RabbitMQ Management

Note

Make sure none of these ports are already in use for localhost. If a port is occupied by another service, please change it in the docker-compose.yml file (and, in the case of the GUI parameterization configurator, also in N6Portal/react_app/config/run_app_server.js).

Launching the system

To start the n6 environment, execute:

docker compose up

Now, give Docker a few minutes to initialize.

Tip

You can use the -d flag to run the application in the background (detached mode).

Port 3001 is used by the Portal GUI parameterization configurator. It is not necessary for n6 to work; to use the configurator, you need to set it up manually:

docker compose exec web bash

…and then:

cd /home/dataman/n6/N6Portal/react_app && yarn run config

First startup

At first run you have to create the Auth DB database tables and its schema. To create the tables use the n6create_and_initialize_auth_db script (use the -D flag to first drop any existing Auth DB tables, and the -y flag to suppress any confirmation prompts):

docker compose run --rm worker n6create_and_initialize_auth_db -D -y

The command output should be similar to this:

* The 'n6create_and_initialize_auth_db' script started.
* Dropping the auth database if it exists...
...
* Creating the new auth database...
* Creating the new auth database tables...
* Inserting new `criteria_category` records...
  * CriteriaCategory "amplifier"
  * CriteriaCategory "bots"
  * CriteriaCategory "backdoor"
...
* Invoking appropriate Alembic tools to stamp the auth database as being at the `head` Alembic revision...
...
* The 'n6create_and_initialize_auth_db' script exits gracefully.

Here you go! n6 is ready to use.

But, please, read on (to learn, in particular, how to get into n6…).

Populating the Auth DB with example data

Let us add some example data to the Auth DB, in particular, creating an example user and its organization. You will be prompted to enter the user’s password. Remember it. You will need it to log in to the n6 Portal.

docker compose run --rm worker n6populate_auth_db -F -i -t -s -p example.com login@example.com

To see the results, restart (or reload) the apache2 service:

docker compose exec web apache2ctl restart

As mentioned in the disclaimer

This is not a production setup, so it may produce error:

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.32.5. Set the 'ServerName' directive globally to suppress this message

For local development or testing of n6 you can simply ignore it.

Then you can try the n6 Admin Panel’s interface: https://localhost:4444/org. There should be new records in the following Auth DB tables: org, user, source, subsource.

Tip

In case you ever forgot password typed while populating auth_db. You can always change it in admin panel -> user.

Multi-Factor Authentication Setup

In your browser, type URL https://localhost/, where the password authentication pages will appear. Insert credentials to log in. Login: login@example.com and password configured while creating account. Then follow the directions in Multi-Factor Authentication Setup.

Checking availability of services and monitoring the system

RabbitMQ Management (with RabbitMQ-generated GUI):

n6 Admin Panel:

n6 Portal API user authentication status:

n6 Portal GUI:

  • URL: https://localhost/

  • Credentials to log in:

    • username: login@example.com,
    • organization: example.com,
    • password: the one entered during the execution of the n6populate_auth_db script.

Mailhog Web GUI:

Additional tools

MongoDB Compass / Studio 3T Free - client GUI for MongoDB:

  • Connection:

    • name: any, e.g.: n6-open
    • hostname: localhost
    • port: 27017

  • Authentication:

    • database name: n6 or admin
    • username: admin
    • password: password
    • auth mechanism: MONGODB-CR or Default in MongoDB Compass

Working with Docker environment

Start worker container in the interactive mode:

docker compose exec worker bash

Within the container, look at the directory structure:

ls -l

The command output should be similar to this:

drwxr-xr-x 3 dataman dataman     4096 Jan 27 10:18 certs
-rwxr-xr-x 1 dataman dataman       80 Jan 27 10:19 entrypoint.sh
drwxr-xr-x 1 dataman dataman     4096 Jan 27 10:18 env
drwxr-xr-x 1 dataman dataman     4096 Jan 27 10:22 logs
drwxr-xr-x 1 dataman dataman     4096 Jan 27 10:19 n6
-rw-r--r-- 1 dataman dataman 39026234 Jan 27 10:19 node_modules.tar.gz
drwxr-xr-x 1 dataman dataman     4096 Jan 27 10:22 supervisord
drwxr-xr-x 2 dataman dataman     4096 Jan 27 10:17 tmp

Some files and directories are worth mentioning – namely:

  • the entrypoint.sh script wraps the given command, adding a necessary Python environment. It is used on every run/execution of the worker image. For example, we can run a n6 collector script in two ways:

  • docker compose exec worker ./entrypoint.sh n6collector_certplshield or

  • docker compose exec worker bash and then ./entrypoint.sh n6collector_certplshield

Note

You could also go with docker compose run worker ..., but keep in mind that it will create new worker container. While using run command, you don’t need to use ./entrypoint.sh before n6collector_certplshield.

  • the logs directory contains log files created by n6 components. Every collector, parser or other n6 component will write its logs here.

Note

The logs directory will be lost on every container stop. You can use the Docker’s volumes feature to make the directory persistent.

  • the supervisord directory contains Supervisor-related configuration files; the subdirectory programs contains a list of n6 components that will be run by the supervisord process. You are free to mount the programs directory as a volume into the container and add more parsers.

  • the n6 directory contains the cloned repository. The n6 infrastructure has been installed with the -a dev option. This means that you can mount the whole locally cloned n6 directory as a volume into the container. As a result, every change in locally stored n6 code will be immediately applied inside the container on every docker compose run <n6parser/n6collector/n6component> (without the need to reinstall n6 each time).

Supervisor

Note

This setup requires running docker compose up.

Run supervisorctl to examine the status of all n6 components:

docker compose exec worker supervisorctl -c supervisord/supervisord.conf

The command output should include lines similar to the following:

n6aggregator                         RUNNING   pid 34, uptime 0:05:55
n6archiveraw                         RUNNING   pid 35, uptime 0:05:55
n6comparator                         RUNNING   pid 36, uptime 0:05:55
n6enrich:n6enrich_00                 RUNNING   pid 37, uptime 0:05:55
n6filter:n6filter_00                 RUNNING   pid 38, uptime 0:05:55
n6log_std                            RUNNING   pid 42, uptime 0:05:53
n6parser_abusechfeodotracker202110   RUNNING   pid 40, uptime 0:05:55
n6recorder                           RUNNING   pid 41, uptime 0:05:55

Running the container initializes and starts processes configured to be run by Supervisor.

Components being run by Supervisor work as daemon processes. Running a parser triggers creation of a per-data-source RabbitMQ queue.

Collecting data

In the interactive mode, after executing source ./entrypoint.sh, you can type, in bash, n6 + the TAB key to see all available n6 executable scripts. To see some data flowing through components via message broker, run one of the collectors, e.g.:n6collector_certplshield.

docker compose exec worker bash

source ./entrypoint.sh

n6collector_certplshield

Now you can log in to the n6 portal and:

  • go to All Incidents page.
  • click the events tab.
  • press search button (you can also set start date on datepicker).

and Events will load. If you used different collectors you can also add filters to search events. To see more detailed information on how n6 store the data, you might want to connect to MariaDB database via terminal or GUI client.

Using REST API

n6 provide you with REST API, but to use it you need to set your API key.

Setting API Key

To set your API key follow these steps:

  • Log into n6 Portal.
  • Click the user icon located in the top right corner.
  • Go to user settings.
  • Under multi-factor authentication you will see API key section.
  • Click generate key.
  • Now you can click on the generated key to copy it to clickboard.

Then you can make a request to the REST API, for example to obtain the collected data (if any) for the current user, do:

docker compose exec worker bash

curl -k 'https://web:4443/search/events.json?time.min=2015-01-01T00:00:00' -H 'Authorization: Bearer YOUR_API_KEY'

And you should see some json data.

Note

Initially, as in the case of Portal, there is no data available via REST API. To fetch and view some data, you need to run a collector – for example, by executing n6collector_certplshield.

Shutdown and cleanup

Stop and remove all containers, network bridge and Docker images:

docker compose down --rmi all -v