Dependencies Net Agent

The Dependencies Net Agent is a Java-based daemon that continuously tracks network connections of a system (e.g. GNU/Linux or Apache 2.0), represents these tracked connections in RDF (according to a given conversion), and continuously streams the RDF results out.

The agent can be attached to the following sources to track connections of a system:

1. SystemTap (supports GNU/Linux)
2. net-tools netstat (any UNIX-compatible operating system supporting net-tools netstat)
3. input file in CSV format (enables to connect any data source that produces the relevant networking information in CSV format, for example the access.log of Apache, but with an adapted log format)

The RDF is streamed out in ntriples or a simple json format. The output can be sent to the following targets:

1. stdout
2. file
3. RabbitMQ (quick approach to merge streams of various systems in one place and allow others to subscribe to the merged stream, for example a reasoner)

The conversion to RDF can be customized using a query in the TARQL language (TARQL is meant for CSV-to-RDF conversions). The agent is delivered with a query that transforms a simple CSV input to an instance of the TIMBUS DIO ontology.

Usage Requirements

Build

1. Apache Ant

Execution

1. If used as source, either SystemTap or NetStat

Usage

Build

1

ant jar

Execution

Details on how to configure the steps of the streaming pipeline in the agent can be found here:

1

java -jar net-agent.jar --help

Some examples to transform the connections of a GNU/Linux system or Apache 2.0 into an ontology (here: the MusicClassification example) follow:

Example, Source: SystemTap, Target: stdout

1

java -jar net-agent.jar --st --sts ../systemtap/net.stp --tq ../queries/net_to_timbus_dio.tarql

Example, Source: SystemTap, Target: RabbitMQ

The RabbitMQ is running on 127.0.0.1 here. By default (configurable, see agent help, as referenced above) the output stream is sent to a Rabbit MQ exchange called "logs".

1

java -jar net-agent.jar --st --sts ../systemtap/net.stp --tq ../queries/net_to_timbus_dio.tarql --mq --mqh 127.0.0.1

Example, Source: netstat, Target: stdout

1

java -jar net-agent.jar --ns --tq ../queries/net_to_timbus_dio.tarql

Example, Source: netstat, Target: RabbitMQ

The RabbitMQ is running on 127.0.0.1 here. By default (configurable, see agent help, as referenced above) the output stream is sent to a Rabbit MQ exchange called "logs".

1

java -jar net-agent.jar --ns --sts ../systemtap/net.stp --tq ../queries/net_to_timbus_dio.tarql --mq --mqh 127.0.0.1

Example, Source: Apache access.log with adapted log format, Target: stdout

Here, the access.log of Apache is assumed to contain a compatible CSV format and that it is located at: /var/log/apache2/timbus.log.

The following Apache config snippet would do the trick:

1
2

LogFormat "%V,%a,%{remote}p,%A,%{local}p,%v,%{User-agent}i,%P,%m" timbuslog
CustomLog /var/log/apache2/timbus.log timbuslog

Now, the agent could be launched as follows:

1

java -jar net-agent.jar --input /var/log/apache2/timbus.log --tq ../queries/net_to_timbus_dio.tarql

Example, Source: Apache access.log with adapted log format, Target: RabbitMQ

Here, the access.log of Apache is assumed to contain a compatible CSV format and that it is located at: /var/log/apache2/timbus.log.

The following Apache config snippet would do the trick:

1
2

LogFormat "%V,%a,%{remote}p,%A,%{local}p,%v,%{User-agent}i,%P,%m" timbuslog
CustomLog /var/log/apache2/timbus.log timbuslog

The RabbitMQ is running on 127.0.0.1 here. By default (configurable, see agent help, as referenced above) the output stream is sent to a Rabbit MQ exchange called "logs".

Now, the agent could be launched as follows:

1

java -jar net-agent.jar --input /var/log/apache2/timbus.log --tq ../queries/net_to_timbus_dio.tarql --mq --mqh 127.0.0.1

Author

Martin Alexander Neumann mneumann@teco.edu
Till Riedel riedel@teco.edu

Changelog

Initial release 2014-06-02

License

see LICENSE file