Configuring n6 Components¶

Warning

This guide suggests to base your n6 configuration on the configuration prototype files shipped with the n6 source code (namely: those you can find in n6/etc/...). Then, for the purposes of this guide, only several settings in a few files need to be adjusted (see below…), and the rest can be left intact.

Please keep in mind, however, that many elements acceptable for the purposes of this guide would not be acceptable in the case of a production system – in particular:

any insecure passwords/secrets (like those set to the silly password text, and generally all passwords/secrets you can find in the configuration prototype files shipped with the n6 source code; but also any invented or generated by you if they were too weak…)
any uses of the example SSL (TLS) certificates/keys (aka X.509 certificates/keys) shipped with the n6 source code (more generally: beware of using any potentially insecure/compromised certificates and keys; possible oversights may also include too wide file access permissions when it comes to an SSL (TLS) private key file)
any network communication not secured with SSL (TLS) at all
any cases of unnecessarily wide permissions (e.g., the use of the `root`@`localhost` MariaDB acoount)

Where Are We?¶

Again, before any operations, ensure the current working directory is the home directory of dataman (which is supposed to be the user in whose shell you execute all commands):

cd ~

Also, make sure the Python virtual environment in which n6 has been installed is active:

source ./env_py3k/bin/activate

Configuring n6 Pipeline¶

Configuration files for the n6 Pipeline components are to be placed in the .n6 subdirectory of the dataman user’s home directory.

Copying Configuration Prototypes¶

First, copy the configuration prototype files shipped with the n6 source code:

cp n6/etc/n6/*.conf .n6/

Now, the output from the command:

ls -l /home/dataman/.n6/

…should be similar to the following:

-rw-r--r-- 1 dataman dataman  1868 Jun 20 13:41 00_global.conf
-rw-r--r-- 1 dataman dataman  1152 Jun 20 13:41 00_pipeline.conf
...
-rw-r--r-- 1 dataman dataman   625 Jun 20 13:41 05_enrich.conf
-rw-r--r-- 1 dataman dataman   710 Jun 20 13:41 07_aggregator.conf
-rw-r--r-- 1 dataman dataman   396 Jun 20 13:41 07_comparator.conf
-rw-r--r-- 1 dataman dataman  1887 Jun 20 13:41 09_auth_db.conf
...
-rw-r--r-- 1 dataman dataman  1106 Jun 20 13:41 21_recorder.conf
-rw-r--r-- 1 dataman dataman   204 Jun 20 13:41 23_filter.conf
...
-rw-r--r-- 1 dataman dataman  1311 Jun 20 13:41 60_abuse_ch.conf
-rw-r--r-- 1 dataman dataman  3539 Jun 20 13:41 60_amqp.conf
-rw-r--r-- 1 dataman dataman   328 Jun 20 13:41 60_cert_pl.conf
-rw-r--r-- 1 dataman dataman   462 Jun 20 13:41 60_cesnet_cz.conf
-rw-r--r-- 1 dataman dataman   313 Jun 20 13:41 60_dan_tv.conf
...
-rw-r--r-- 1 dataman dataman 15320 Jun 20 13:41 admin_panel.conf
-rw-r--r-- 1 dataman dataman  4033 Jun 20 13:41 logging.conf

Info

All those configuration files are in the popular INI-like format.

Generally, each n6 pipeline component tries to load – from the /etc/n6/ and ~/.n6/ directories (in this order; although, concerning this guide, only the latter is present) – all configuration files whose names match the pattern <two decimal digits>_<whatever>.conf. In the case of each of those two directories (if it exists), the files are loaded in the order resulting from sorting the filenames alphabetically.

Note that configuration sections and options may be repeated. In such cases, the option value encountered later “wins” (so the loading order matters).

Each component picks only the configuration section(s) relevant to it – ignoring all others.

Note

What is important for each component are names of configuration sections and options – not names of configuration files (apart from the required <two decimal digits>_<whatever>.conf format and the loading order discussed in the Info box above).

Nevertheless, the names of the configuration files from n6/etc/ do suggest which components their contents apply to. In particular, these files are relevant to these n6 pipeline components:

00_global.conf and 00_pipeline.conf – to most components
05_enrich.conf – to the n6enrich component
07_aggregator.conf – to the n6aggregator component
07_comparator.conf – to the n6comparator component
09_auth_db.conf – to any components connecting to Auth DB
21_recorder.conf – to the n6recorder component
23_filter.conf – to the n6filter component
any 60_*.conf files – to respective collectors and parsers, i.e., particular n6collector_* and n6parser_* components supposed to deal with respective external data sources

See also:

Architecture and Data Flow Overview

There are also two special configuration files:

logging.conf – loaded by a separate (logging-dedicated) machinery, relevant to all n6 pipeline components (and also to the Admin Panel web component, discussed later in this chapter…)
admin_panel.conf – ignored by n6 pipeline components (as it is relevant only to the Admin Panel web component)

Adjusting Configuration Files¶

For the purposes of this guide, just a few n6-pipeline-related configuration options in a few files need to be adjusted…

RabbitMQ Connectivity¶

Edit the ~/.n6/00_global.conf configuration file to ensure that the host, password_auth, username and password options in the [rabbitmq] section are set as follows:

[rabbitmq]

host = localhost
# ...
password_auth = true
username = guest
password = guest

(Make sure that these options are not commented out with a leading ; character)

Keep the remaining options in that file intact.

Event DB Connectivity (`n6recorder`)¶

Edit the ~/.n6/21_recorder.conf configuration file to ensure that the uri option in the [recorder] section is set as follows:

[recorder]

uri = mysql://root:password@localhost/n6

(And keep the remaining options in that file intact.)

Auth DB Connectivity¶

Edit the ~/.n6/09_auth_db.conf configuration file to ensure that the url option in the [auth_db] section is set as follows:

[auth_db]

url = mysql://root:password@localhost/auth_db

Note

For historical reasons, the relevant option name is url in the case of Auth DB (here), even though it is uri in the case of Event DB (see above).

Also, in the same [auth_db] section, adjust the three SSL-related options by setting them to none:

[auth_db]
# ...
ssl_cacert = none
ssl_cert = none
ssl_key = none

(And keep the remaining options in that file intact.)

Logging Settings¶

Edit the special configuration file ~/.n6/logging.conf to adjust the general n6 pipeline’s configuration of logging, by modifying the content of the [logger_root] section – to make it look like this:

[logger_root]
level = INFO
handlers = file

The rest of the settings in ~/.n6/logging.conf should be OK for the purposes of this guide. Of course, you can customize some of them if you like (the possibilities are wide…).

Optionally: Other Settings…¶

You may also want to adjust some other configuration options, relevant to various n6 pipeline components – customizable by editing various ~/.n6/*.conf configuration files… See the Note box near the end of the Copying Configuration Prototypes section earlier in this chapter.

Also, see the helpful comments you can find in many of those configuration files.

Initializing n6’s Databases¶

As mentioned earlier, n6 uses the following two SQL databases:

Event DB – stores all network incidents information collected by the n6 system, in the n6-pipline-components-processed form of event records (aka normalized events). Given that amounts of stored events may be huge and write throughput may be high, this database makes use of the RocksDB engine.
Auth DB – stores information needed to authenticate users of n6 Portal and clients of the n6’s public APIs, and to authorize those users/clients to access different subsets of data stored in Event DB. Apart from that, some auxiliary information (not necessarily related to authentication and authorization) is also stored in Auth DB. This database makes use of the standard InnoDB engine.

Event DB¶

Create and initialize Event DB (as a new MariaDB database called n6):

cat \
    n6/etc/mysql/initdb/1_create_tables.sql \
    n6/etc/mysql/initdb/2_create_indexes.sql \
  | sudo mysql -u root

Warning

The command first drops the target database (if it already existed).

The above command may take several seconds to execute. It is not expected to print any output.

Auth DB¶

Create and initialize Auth DB (as a new MariaDB database called auth_db):

n6create_and_initialize_auth_db --drop-db-if-exists --yes

Warning

The command first drops the target database (if it already existed).

Near the end of the command’s output the following line should appear:

* The 'n6create_and_initialize_auth_db' script exits gracefully.

Now, populate Auth DB with some example data…

n6populate_auth_db \
  --access-to-inside \
  --access-to-threats \
  --access-to-search \
  --set-password \
  example.com \
  login@example.com

This command first asks you to enter (interactively!) a password for the newly created user (see below…), and then adds the following data to Auth DB:

a bunch of declarations of data sources with appropriate general access subsources (i.e., definitions of data feeds to which client organizations can have access)
a new client organization –
- whose identifier (org_id) is example.com
- which has access to:
  - the inside access zone (n6 REST API’s resource report/inside), with the aforementioned subsources enabled
  - the threats access zone (n6 REST API’s resource report/threats), with the aforementioned subsources enabled
  - the search access zone (n6 REST API’s resource search/events), with the aforementioned subsources enabled
a new n6 user –
- whose identifier (login) is login@example.com
- whose password is what you just entered interactively (remember it, as it will be needed later…)
- which belongs to the aforementioned example.com organization (with all its access rights)

Info

The identifier of an organization (org_id) is supposed to be a DNS domain of that organization.

The identifier of a user (login aka user_id) is supposed to be an e-mail address of that user.

Near the end of the above n6populate_auth_db command’s output the following line should appear:

* The 'n6populate_auth_db' script exits gracefully.

Tip

You could execute the above command specifying also other command-line options…

In particular, the --full-access option would set the full_access marker to True on the newly created organization. If an organization has full_access=True, all users belonging to it are privileged – i.e., authorized to see all event attributes without limitations and any anonymization, and also to access events with the restriction attribute set to "internal" (such events can never be accessed by non-privileged users).

Try n6populate_auth_db --help to learn more about the available command-line options.

Configuring n6’s Web Components¶

Note

In this tutorial, for the sake of simplicity, some of the n6 web components’ configuration prototype files (shipped with the n6 source code) are (ab)used as actual configuration files (i.e., you will edit them in-place, without copying).

n6 Portal¶

Building the n6 Portal’s Frontend (GUI)¶

cd ~/n6/N6Portal/react_app

yarn

npm_config_yes=true npx yarn-audit-fix

yarn build

cd ~

Configuring Event DB and Auth DB Connectivity¶

Edit the n6/etc/web/conf/portal.ini configuration file to modify the sqlalchemy.url and auth_db.url options, so that they have the following values:

# ...
# event db configuration
# ...
sqlalchemy.url = mysql://root:password@localhost/n6

# ...
# auth db configuration
# ...
auth_db.url = mysql://root:password@localhost/auth_db

Disabling (or Configuring) Mail Notices¶

Info

E-mail messages are sent to users of n6 when something related to them or their organization happens, e.g., a password reset is requested, an organization configuration update request is submitted by any of the organization’s users, etc.

This feature is called mail notices. In production, it is definitely necessary (without it the service would be hardly usable), but for the purposes of this guide you can disable it.

To disable the mail notices feature in n6 Portal, edit the n6/etc/web/conf/portal.ini configuration file to modify the mail_notices_api.active option, so that it is set to false:

# ...
# mail notices configuration + jinja rendering configuration
# ...
mail_notices_api.active = false

Tip

On the other hand, you may want to keep this feature enabled – to test it using a fake SMTP (e-mail) server. If you decide to do so, then just:

edit the same n6/etc/web/conf/portal.ini configuration file to modify the mail_sending_api.smtp_host option by setting it to localhost and ensure that the mail_sending_api.smtp_port option is set to 1025 (of course, keeping the aforementioned mail_notices_api.active option set to true);
install the MailDump fake SMTP server by executing the command: pip install maildump, and then run it in the background by executing the command: maildump -p maildump.pidfile (it will start listening for SMTP connections on port 1025; see also: maildump --help).

From now on, you can use a web browser to visit http://localhost:1080 to interactively explore any e-mail notices sent by n6 Portal.

Logging Settings¶

Edit the content of the sections placed near the end of the same n6/etc/web/conf/portal.ini configuration file: [loggers], [handlers], [formatters] and further ones – to customize the n6 Portal backend’s configuration of logging (the possibilities are wide…).

In particular, it is worth modifying the content of the [logger_root] section to make it look like this:

[logger_root]
level = INFO
handlers = file, console

The rest of the n6 Portal backend’s logging settings should be OK for the purposes of this guide.

Optionally: Other Settings…¶

You may also want to customize some other configuration options in various parts of the same n6/etc/web/conf/portal.ini file. See the numerous helpful comments you can find in that file…

Prepare, Enable, Start!¶

Copy these two Apache site files shipped with the n6 source code:

sudo cp \
    ~dataman/n6/etc/apache2/sites-available/000-default.conf \
    /etc/apache2/sites-available/

sudo cp -a \
    ~dataman/n6/etc/apache2/sites-available/n6-portal.conf \
    /etc/apache2/sites-available/

Then edit the new /etc/apache2/sites-available/n6-portal.conf file to replace the existing WSGIDaemonProcess directive, consisting of several lines, with its new version, consisting of the following three lines:

    WSGIDaemonProcess n6-portal \
        python-path=/home/dataman/env_py3k/lib/python3.11/site-packages \
        python-eggs=/home/dataman/.cache/n6/n6portal/python3k-eggs

Do not forget to set the appropriate file access permissions:

sudo chmod 644 /etc/apache2/sites-available/000-default.conf \
  && sudo chown root:root /etc/apache2/sites-available/000-default.conf

sudo chmod 644 /etc/apache2/sites-available/n6-portal.conf \
  && sudo chown root:root /etc/apache2/sites-available/n6-portal.conf

Also, create the directory for the python-eggs cache:

cd ~ \
  && mkdir -m 710 .cache/n6/n6portal \
  && chown dataman:www-data .cache/n6/n6portal \
  && mkdir -m 770 .cache/n6/n6portal/python3k-eggs \
  && chown dataman:www-data .cache/n6/n6portal/python3k-eggs

Now, enable the newly configured Apache site:

sudo a2ensite n6-portal

The command’s output should include:

Enabling site n6-portal.

Finally, start serving your own n6 Portal!

sudo service apache2 restart

n6 Portal should now be available. Try to visit it using a web browser at https://localhost/. You should see the n6 Portal’s log-in page.

Note

Because of the use of the insecure example certificate shipped with the n6 source code, any modern web browser is expected to warn you that the connection is not secure. Before the browser agrees to display the site, you may need to confirm that you accept the risk. Also, for your convenience, you may want to add in your browser a permanent security exception for this site.

See also:

n6 Portal – Some Finer Points

n6 REST API¶

Configuring Event DB and Auth DB Connectivity¶

Edit the n6/etc/web/conf/restapi.ini configuration file to modify the sqlalchemy.url and auth_db.url options, so that they have the following values:

# ...
# event db configuration
# ...
sqlalchemy.url = mysql://root:password@localhost/n6

# ...
# auth db configuration
# ...
auth_db.url = mysql://root:password@localhost/auth_db

Logging Settings¶

Edit the content of the sections placed near the end of the same n6/etc/web/conf/restapi.ini configuration file: [loggers], [handlers], [formatters] and further ones – to customize the n6 REST API’s configuration of logging (the possibilities are wide…).

In particular, it is worth modifying the content of the [logger_root] section to make it look like this:

[logger_root]
level = INFO
handlers = file, console

The rest of the n6 REST API’s logging settings should be OK for the purposes of this guide.

Optionally: Other Settings…¶

You may also want to customize some other configuration options in various parts of the same n6/etc/web/conf/restapi.ini file. See the numerous helpful comments you can find in that file…

Prepare, Enable, Start!¶

Copy the relevant Apache site file shipped with the n6 source code:

sudo cp -a \
    ~dataman/n6/etc/apache2/sites-available/n6-restapi.conf \
    /etc/apache2/sites-available/

Then edit the new /etc/apache2/sites-available/n6-restapi.conf file to replace the existing WSGIDaemonProcess directive, consisting of several lines, with its new version, consisting of the following three lines:

   WSGIDaemonProcess n6-restapi \
       python-path=/home/dataman/env_py3k/lib/python3.11/site-packages \
       python-eggs=/home/dataman/.cache/n6/n6restapi/python3k-eggs

Do not forget to set the appropriate file access permissions:

sudo chmod 644 /etc/apache2/sites-available/n6-restapi.conf \
  && sudo chown root:root /etc/apache2/sites-available/n6-restapi.conf

Also, create the directory for the python-eggs cache:

cd ~ \
  && mkdir -m 710 .cache/n6/n6restapi \
  && chown dataman:www-data .cache/n6/n6restapi \
  && mkdir -m 770 .cache/n6/n6restapi/python3k-eggs \
  && chown dataman:www-data .cache/n6/n6restapi/python3k-eggs

Now, enable the newly configured Apache site:

sudo a2ensite n6-restapi

The command’s output should include:

Enabling site n6-restapi.

Finally, start serving your own n6 REST API!

sudo service apache2 restart

n6 REST API should now be available. Try to visit one of its endpoints using a web browser: https://localhost:4443/report/inside.json. You should see an 401 Unauthorized error page (HTTP status 401); that’s OK for now!

Note

Because of the use of the insecure example certificate shipped with the n6 source code, any modern web browser is expected to warn you that the connection is not secure. Before the browser agrees to display anything, you may need to confirm that you accept the risk.

n6 Admin Panel¶

Auth DB Connectivity¶

Edit the ~/.n6/admin_panel.conf configuration file (one of those you created when preparing the n6 pipeline configuration) to ensure that the url option in the [auth_db] section is set as follows:

[auth_db]

url = mysql://root:password@localhost/auth_db

Also, in the same [auth_db] section, adjust the three SSL-related options by setting them to none:

[auth_db]
# ...
ssl_cacert = none
ssl_cert = none
ssl_key = none

RabbitMQ Connectivity¶

Edit the same ~/.n6/admin_panel.conf configuration file to ensure that the host, password_auth, username and password options in the [rabbitmq] section are set as follows:

[rabbitmq]

host = localhost
# ...
password_auth = true
username = guest
password = guest

(Make sure that these options are not commented out with a leading ; character)

Disabling (or Configuring) Mail Notices¶

Info

E-mail messages are sent to users of n6 when something related to them or their organization happens, e.g., an organization configuration update request is accepted or rejected by an administrator, etc.

This feature is called mail notices. In production, it is definitely necessary, but for the purposes of this guide you can disable it.

To disable the mail notices feature in n6 Admin Panel, edit the ~/.n6/admin_panel.conf configuration file to modify the active option in the [mail_notices_api] section, so that it is set to false:

[mail_notices_api]
# ...
active = false

Tip

On the other hand, you may want to keep this feature enabled – to test it using a fake SMTP (e-mail) server. If you decide to do so, then just:

edit the same ~/.n6/admin_panel.conf configuration file, specifically its section [mail_sending_api], to modify the smtp_host option in by setting it to localhost and ensure that the smtp_port option is set to 1025 (of course, keeping the aforementioned active option in the [mail_notices_api] section set to true);
if you haven’t already done so when configuring n6 Portal, install the MailDump fake SMTP server by executing the command: pip install maildump, and then run it in the background by executing the command: maildump -p maildump.pidfile (it will start listening for SMTP connections on port 1025; see also: maildump --help).

From now on, you can use a web browser to visit http://localhost:1080 to interactively explore any e-mail notices sent by n6 Admin Panel.

Logging Settings¶

Unlike other web components, n6 Admin Panel shares its logging settings with the n6 pipeline components – see the relevant section earlier in this chapter…

Optionally: Other Settings…¶

You may also want to customize some other configuration options in various parts of the ~/.n6/admin_panel.conf file. See the helpful comments you can find in it…

Prepare, Enable, Start!¶

Copy the relevant Apache site file shipped with the n6 source code:

sudo cp -a \
    ~dataman/n6/etc/apache2/sites-available/n6-adminpanel.conf \
    /etc/apache2/sites-available/

Then edit the new /etc/apache2/sites-available/n6-adminpanel.conf file to replace the existing WSGIDaemonProcess directive, consisting of several lines, with its new version, consisting of the following six lines:

    WSGIDaemonProcess n6-adminpanel \
        user=dataman \
        home=/home/dataman/ \
        threads=2 \
        python-path=/home/dataman/env_py3k/lib/python3.11/site-packages \
        python-eggs=/home/dataman/.cache/n6/n6adminpanel/python3k-eggs

Do not forget to set the appropriate file access permissions:

sudo chmod 644 /etc/apache2/sites-available/n6-adminpanel.conf \
  && sudo chown root:root /etc/apache2/sites-available/n6-adminpanel.conf

Also, create the directory for the python-eggs cache:

cd ~ && mkdir -p .cache/n6/n6adminpanel/python3k-eggs

Now, enable the newly configured Apache site:

sudo a2ensite n6-adminpanel

The command’s output should include:

Enabling site n6-adminpanel.

Finally, start serving the Admin Panel app:

sudo service apache2 restart

n6 Admin Panel should now be available. Try to visit it using a web browser at https://localhost:4444/. You should see the n6 Admin Panel home page.

Note

Because of the use of the insecure example certificate shipped with the n6 source code, any modern web browser is expected to warn you that the connection is not secure. Before the browser agrees to display the site, you may need to confirm that you accept the risk. Also, for your convenience, you may want to add in your browser a permanent security exception for this site.

Warning

Unlike other web components of n6, the n6 Admin Panel application by itself does not have any authentication mechanism! (So, in particular, never make it public, unless you protect it with some external authentication layer…)

Summary of Available HTTP/HTTPS Services¶

Web components of n6:

https://localhost/ (port 443) – n6 Portal
- http://localhost/ (port 80) – redirects to 443 (to use HTTPS)
https://localhost:4443/ – n6 REST API
https://localhost:4444/ – n6 Admin Panel

Extra tools:

https://localhost:15671/ – web interface of RabbitMQ
http://localhost:1080/ – web interface of MailDump (if you installed it…)

Configuring n6 Components¶

Where Are We?¶

Configuring n6 Pipeline¶

Copying Configuration Prototypes¶

Adjusting Configuration Files¶

RabbitMQ Connectivity¶

Event DB Connectivity (n6recorder)¶

Auth DB Connectivity¶

Logging Settings¶

Optionally: Other Settings…¶

Initializing n6’s Databases¶

Event DB¶

Auth DB¶

Configuring n6’s Web Components¶

n6 Portal¶

Building the n6 Portal’s Frontend (GUI)¶

Configuring Event DB and Auth DB Connectivity¶

Disabling (or Configuring) Mail Notices¶

Logging Settings¶

Optionally: Other Settings…¶

Prepare, Enable, Start!¶

n6 REST API¶

Configuring Event DB and Auth DB Connectivity¶

Logging Settings¶

Optionally: Other Settings…¶

Prepare, Enable, Start!¶

n6 Admin Panel¶

Auth DB Connectivity¶

RabbitMQ Connectivity¶

Disabling (or Configuring) Mail Notices¶

Logging Settings¶

Optionally: Other Settings…¶

Prepare, Enable, Start!¶

Summary of Available HTTP/HTTPS Services¶

Event DB Connectivity (`n6recorder`)¶