TPM - Data Storage Interaction

Dan Bendas (VTT)

Updated: 11.11.2016

Premises

The Third Parties Mediator retrieves data from external services and makes them available to cloud components via the Data Storage layer. This document describes the process with sufficient detail level to support the work of software developers.

C2NET project applies lean development principles, where software archiecture is not enforced but is discovered through an ongoing process lead by customer's business needs. The main use case for TPM comes from Pilot 2 - Dermo Cosmetics.

The document is work in progress and is updated to reflect the result of discussions between VTT, Uninova, and other involved parties.

Scenario

A) Story

PFDC wants to use weather forecasts to predict demand for certain cosmetic products. C2NET system will be configured so that:

  • weather forecast data is fetched periodically (e.g. daily) from an external service,
  • it's converted in a machine-friendly format
  • is stored inside the cloud
  • is analysed for abnormal patters (by LL-CEP mechanism)
  • if certain patterns are found (e.g. warmer than average temperatures in summer months), a notification is sent to the responsible person (production planner of PFDC) via UCP interface and mobile application and email.

B) Setup

Note: DKMS is able to store any persistent data associated with a Hub/Sensor/DataSource (Resource) through the use of open-ended key-value properties (RM is operating under this assumption). If TPM requires some persistence this can be an option. Won't be used in the next milestone (to maintain simplicity), but it's an elegant way to store and share setting and should be reconsidered once main functionaly is implemented.

  • User uploads a script to TPM (via UI) which describes the data source, data extraction methods (e.g. html scrapping procedures), and data output format
  • Function not available yet. Instead, a hard-coded set of scripts is included in the TPM component
  • User defines a timer in TPM's UI which defines which script to call, at what time interval, with what parameters
  • Function available in TPM, but not in UI {{will be implemented in TPM' UCP module}}
  • User defines a resource in RM, which connects a "stream" of data from TPM to a named storage location and also specifies the data format
  • ?? {{How? Status needed. Link to documentation if ready.}}
  • User defines a LL-CEP Rule Set which analyses incoming data and forwards the notification request to the relevant components
  • ?? {{How? Status needed. Link to documentation if ready.}}

C) Operation

  1. TPM self-activates (by timer), retrieves data from source
  2. TPM pushes data in "output format" via its tpmPush channel to RMBackend through an RM API reply channel (unnamed for now). The message protocol is being formalized. {{to be finalised}}
  3. Data format is JSON, a schema proposal is below.
  4. RMBackend stores the received data as sensor data ((timestamp-datasource) : value). Since there are no STable mappings for data from 3rd party services (such as weather forects) 1, it can't be stored in STables.
  5. RMBackend triggers LL-CEP. {{must be addressed}}
  6. Triggering the LLCEP is not yet fully specified, especially in cases that fit outside STables(pure sensor data).
  7. If needed, LL-CEP creates a notification request and forwards it to the relevant components

1: as discussed in Valencia I meeting

Ideas

Weather forecast data format


The output of Weather Forecast script can be in form of an array of tuples or dictionaries, each with the content listed below (example data in parantheses). Thus the script can be treated like any sensor, with the particularity that it sends bursts of ~30 data points at a time.
{{Q: is this reasonable?}}

  • Timestamp (2016-11-11 10:00:00)
  • Location ("Paris,FR")
  • Source ("Accuweather")
  • TargetDate (2017-02-04)
  • ForecastHi (12)
  • ForecastLo (7)
  • AverageHi (10)
  • AverageLo (5)