Skip to content

Supervisor

Supervisor is a system for controlling processes’ state under UNIX, useful to manage the n6 pipeline (N6DataPipeline) components and (N6DataSources) sources.

(env_py3k)$ pip install supervisor

Configuration

Create directories for Supervisor’s config and log files. Create a configuration file in ~/supervisor/supervisord.conf. You can find example configuration in n6 repository, in /home/dataman/n6/etc/supervisord/supervisord.conf.

(env_py3k)$ mkdir -p ~/supervisord/{log,programs}
(env_py3k)$ cp ~/n6/etc/supervisord/supervisord.conf ~/supervisord

or for Python 2 N6Core components:

(env)$ mkdir -p ~/supervisord/{log,programs_py2k}
(env)$ cp ~/n6/etc/supervisord/supervisord_py2k.conf ~/supervisord

Generate config files for n6 parsers and copy them to the supervisord directory:

(env_py3k)$ cd ~/n6/etc/supervisord/
(env_py3k)$ python get_parsers_conf.py
(env_py3k)$ cp programs/*.conf ~/supervisord/programs

If you want to use configuration files for Python 2:

(env)$ cd ~/n6/etc/supervisord/
(env)$ python get_parsers_conf_py2k.py
(env)$ cp programs/*.conf ~/supervisord/programs_py2k

If you install n6 in a different Virtualenv than recommended (i.e., other than env_py3k), then you have to adjust a proper path in all of the files in n6/etc/supervisord/programs in the PATH environment variable and in the command option:

command=/home/dataman/<ENV_NAME>/bin/{program_command}
environment=PATH="/home/dataman/<ENV_NAME>/bin/"

Running the Supervisor’s process

To run the supervisord process:

(env_py3k)$ supervisord -c ~/supervisord/supervisord.conf

If you want to run the supervisord process for components in Python 2:

(env_py3k)$ supervisord -c ~/supervisord/supervisord_py2k.conf

Controlling the processes

You can manage processes managed by the supervisord with the supervisorctl command.

Inside the supervisorctl prompt type status to list all processes and their status:

supervisor> status
n6aggregator                            RUNNING   pid 3922, uptime 0:00:15
n6archiveraw                            RUNNING   pid 3942, uptime 0:00:15
n6comparator                            RUNNING   pid 3957, uptime 0:00:15
n6enrich:n6enrich_00                    STARTING  
n6filter                                RUNNING   pid 3927, uptime 0:00:15
n6parser_abusechfeodotracker202110.conf RUNNING   pid 3929, uptime 0:00:15
n6recorder                              RUNNING   pid 3930, uptime 0:00:15

You can use commands like start or stop to control a single or all of the processes.

Managing message broker queues, created by running components

Check the RabbitMQ’s management GUI for new queues, which should have been created by each component that consumes incoming messages.

To start some data flow, you should run some collectors. You can do it manually, or schedule Cron jobs to run them periodically.

Back to top