Implementing a New Data Source¶
This guide needs an update regarding the recent migration of n6 from
Python 2.7 to 3.9, especially that the n6‘s data sources and data
pipeline stuff now works under Python 3.9 and resides in
N6DataPipeline (where some names and other interface details
differ from the corresponding, Python-2.7-only, legacy stuff kept in
The aim of this guide is to describe how to implement new n6 components necessary for collecting and parsing data from some external security data source.
This guide consists of two parts:
Collectors. The first part focuses on collectors which are n6‘s data entry points. The job of a collector is to obtain data from some external data source (for example, from a particular web site) and send those data for further processing.
Parsers. The other part focuses on parsers which are the next stage of the n6‘s data processing pipeline. The job of a parser is to parse and validate external data obtained by the corresponding collector, transform those data to a form digestible by n6 (i.e., normalize the data), and then send those (already normalized) data for further processing.