ikats/ingestion

!https://ikats.github.io/img/Logo-ikats-icon.png IKATS Ingestion

!https://img.shields.io/docker/automated/ikats/ingestion.svg !https://img.shields.io/docker/build/ikats/ingestion.svg !https://img.shields.io/microbadger/image-size/ikats/ingestion.svg

This module provides features in order to ingest data in IKATS.
The application is deployed on a Tommee server and runs independantly of IKATS datamodel. It can be reached by two means:

1. an IKATS operator: https://ikats.github.io/doc/operators/importTs.html
2. or via HTTP API (see below).

HTTP API

• The ingestion tool's HTTP API manages some ingestion sessions as jobs.
• Currently, the tool can only run one ingestion session at the same time.
• Practical Use cases are presented as user guidelines.

The API is available from the URL base: http://host:port/ikats-ingestion/api where host is the IP of the application server host.

Services available from HTTP API are:

• Create and start ingestion session
• Restart a session
• Get the list of the sessions handled
• Get the session by ID
• Get statistics about one session with ID

Create and start an ingestion session

Request:

• API resource: /sessions
• HTTP Url: http://host:port/ikats-ingestion/api/sessions
• HTTP Verb: POST
• HTTP Body: provides the DATA: with the JSON descri***g the session.

Nominal response:

• header code HTTP 201
• header location property, useful to grep the session <ID> from the value.
  Location example: http://host:port/ikats-ingestion/api/sessions/<ID>

Error response:

• header code HTTP: error code 500|400|...

• body: HTML content for additional error information.

  See also how to submit HTTP requests

Ingestion session description

The JSON data to be sent shall contain the following properties:

• dataset The name of the future dataset into IKATS database

• description A description of that dataset for the end user

• rootPath The root path of the dataset on the import server where files are located

  • Could be absolute, in that case, represent the path on the server
  • If relative, a configuration property will be used as prefix to the path (default: /IKATSDATA)

• pathPattern Regex pattern rules for defining tags and metric of dataset:
  

  • The path is described with a regex
  • The root of the absolute path is the rootPath, and is not included in the pattern
  • The metric and tags should be matched into regex named groups. Metric and tags will be saved as Meta-data useful for querying timeseries in IKATS.
  • The pattern must define one metric with: (?<metric>.*), it defines mandatory information (for OpenTSDB )
  • Each tag is defined with a regex group defined with: (?<tagname>.*)
    Examples: patterns encoded in JSON string (\ needs to be doubled \\):

    \\/(?<equipement>\\w*)\\/(?<metric>.*?)_(?<validity>bad|good)\\.csv

• funcIdPattern Pattern configuring how is generated the Functional Identifier. This substitution pattern defines sections ${metric} or ${<tagname>} refering to pathPattern groups, and replaced by respective group values matched by pathPattern. It follows Apache Commons Lang StrSubstitutor variable format, with any <tagname> or metric as variables names.
  Examples:

  ${equipement}_${metric}_${validity}

• importer Fully Qualified Name of the java importer used to transfer the Time-Serie data to the IKATS dedicated database.

  • This is a plugin definiton adapted to a database. The default plugin is in example below: applicable to OpenTSDB database.
  • serializer Set the Fully Qualified Name of the input serializer.
    • This is a plugin definition adapted to a specific file format. The default plugin is in example below

Example of the JSON document:

json
  {
    "dataset": "Dataset_name",
    "description": "Dataset exemple",
    "rootPath": "data_dataset_example",
    "pathPattern": "\\/(?<equipement>\\w*)\\/(?<metric>.*?)_(?<validity>bad|good)\\.csv",
    "funcIdPattern": "${equipement}_${metric}_${validity}",
    "serializer": "fr.cs.ikats.datamanager.client.opentsdb.importer.CommonDataJsonIzer",
    "importer": "fr.cs.ikats.ingestion.process.opentsdb.OpenTsdbImportTaskFactory"
  }

Restart a session

• API resource: /sessions/{id}/restart
• HTTP URL: http://host:port/ikats-ingestion/api/sessions/{id}/restart
  • Where {id} is the id of the session.
• HTTP Verb: PUT

See also how to submit HTTP requests

Get sessions list

Warning: Do not use it for a large dataset, the current output is the full data of the sessions !

• API resource: /sessions
• HTTP URL: http://host:port/ikats-ingestion/api/sessions
• HTTP Verb: GET

See also how to submit HTTP requests

Get ingestion session

Request:

• API resource: /sessions/<ID>
• HTTP Url: http://host:port/ikats-ingestion/api/sessions/x
• HTTP Verb: GET

Response: the session JSON content

See also how to submit HTTP requests

Get statistics about an ingestion session

Request:

• API resource: /sessions/<ID>/stats
• HTTP Url: http://host:port/ikats-ingestion/api/sessions/x/stats
• HTTP Verb: GET

Response: the session statistics JSON content

See also how to submit HTTP requests

Use case: launch new ingestion session

1. Make sure that the application server is started or start it
2. Make sure that the ingestion tool services are started or start them from the Tomee admin page.
3. Prepare the imported folder:

• IKATS convention: prepared under /var/lib/ikats/IKATSDATA/ filer, must be visible from server host

4. Prepare your new session JSON content in your favourite text editor (JSON descri***g the session)
5. Submit the service Create and start ingestion session

Use Case: start another ingestion while one is running

You can only run one ingestion at once! There are 2 workarounds:

• Solution 1: starting another application server on another node
  • required: install the application server on a new node of the same cluster: not straightforward !!
  • And follow the use case launch new ingest session from the new node.
• Solution 2: stopping the running session, launching the second one, and finally restarting the stopped session.
  1. Stop the services from the Tomee admin page or more drastically stop and start the Tomee server.
  2. follow the use case launch new ingest session
  • Submit the stats service to follow the new session by its ID.
  3. Once the session is finished, restart the stopped session, with stopped ID.

Ingestion tool services

Manage the ingestion tool services

Firstly make sure that Tomee server is installed and started.

The services are provided by the application ikats-ingestion running on the Tomee server.

You can manage the services from the Tomee admin page, at line ikats-ingestion as described in Check the server status

• Démarrage: is inactive when services are started
• Arrêt: is inactive when services are not started

How to submit HTTP requests

From linux bash

Use the command http installed by httpie

1. For a body with JSON content: edit the JSON content in a file req.json

2. Launch the http command and read the response header and body

Example:
http POST http://host:port/ikats-ingestion/api/sessions < req.json

HTTP/1.1 201
Content-Length: 0
Date: Fri, 12 May 2018 07:02:50 GMT
Location: http://host:port/ikats-ingestion/api/sessions/6
Server: Apache TomEE

From firefox

1. Browse your <HTTP URL>: for a specific body or a verb different from GET, this does not yet work, and you have to go to step 2
2. Open Development tools and select the Net frame
3. From the browser: refresh the page
4. Select the Net tab of the development tool: view this request and edit it: * Select the Header frame * and click on Modify and resend: from that new panel, you can customize your HTTP request
   • edit the HTTP Verb: PUT|POST|DELETE|GET
   • add the json content in request:
     1. complete the Request Request Header with property content-type Content-Type: application/json
     2. edit the Request HTTP body: <JSON content>
   • change the HTTP URL if needed.
   • Submit * Check the Response Header from the submitted requests: visible from the Header.
     For example when required to retrieve the session ID: read the header response location property.

Check the server status

Check that the server is started: browse the http://host:port: TomEE welcome page should be available.

For further informations, select Server status from the welcome page, and enter the well-known admin login/password to reach admin page:

• Follow the link Lister les applications

• The ingesting tool services should be visible at line having the path /ikats-ingestion

Start and stop the server

Start server

To start the application server you have to run from the server host

bash
cd /home/ikats/ingestion # will be the directory where the ingestion database file will be stored
cd apache-tomee-plume-7.0.3
bin/./catalina.sh jpda start # to tell the server to start with JPDA remote capabilities activated

Stop server

To stop the server from the server host use:

bash
/home/ikats/ingestion/apache-tomee-plume-7.0.3/bin/./shutdown.sh