tdengine/tdengine

TDengine Cloud | Learn more about TSDB

What is TDengine？

TDengine is an open-source, high-performance, cloud-native time-series database optimized for the Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. TDengine differentiates itself from other time-series databases with the following advantages:

• High Performance: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while outperforming other time-series databases for data ingestion, querying and data compression.

• Simplified Solution: Through built-in caching, stream processing, and data subscription features, TDengine provides a simplified solution for time-series data processing. It reduces system design complexity and operation costs significantly.

• Cloud Native: Through native distributed design, sharding and partitioning, separation of compute and storage, RAFT, support for Kubernetes deployment and full observability, TDengine is a cloud-native Time-Series Database and can be deployed on public, private or hybrid clouds.

• Ease of Use: For administrators, TDengine significantly reduces the effort to deploy and maintain. For developers, it provides a simple interface, a simplified solution, and seamless integrations for third-party tools. For data users, it gives easy data access.

• Easy Data Analytics: Through super tables, storage and compute separation, data partitioning by time interval, pre-computation, and other means, TDengine makes it easy to explore, format, and get access to data in a highly efficient way.

• Open Source: TDengine’s core modules, including the cluster feature, are all available under open-source licenses. It has gathered 19.9k stars on GitHub. There is an active developer community and over 139k running instances worldwide.

For a full list of TDengine competitive advantages, please check here. The easiest way to experience TDengine is through TDengine Cloud.

How to use Docker to run TDengine

Note: Start from 3.3.7.0, the Docker image for TDengine is renamed to tdengine/tsdb.

running TDengine server inside Docker

bash
$ docker run -d \
   -p 6030-6060:6030-6060 \
   -p 6030-6060:6030-6060/udp \
   tdengine/tdengine

This command starts a docker container with TDengine server running and maps the container's ports from 6030 to 6060 to the host's ports from 6030 to 6060. If the host is already running TDengine server and occupying the same port(s), you need to map the container's port to a different unused port segment. (Please see TDengine 2.0 Port Description for details). In order to support TDengine clients accessing TDengine server services, both TCP and UDP ports need to be open.

Further, you can also use the docker run command to start the docker container running TDengine server, use the --name command line parameter to name the container tdengine, use --hostname to specify the hostname as tdengine-server, and use -v to mount the local directory (-v) to synchronize the data inside the host and the container to prevent data loss after the container is deleted.

$ docker run -d \
   --name tdengine \
   --hostname="tdengine-server" \ 
   -v ~/work/taos/log:/var/log/taos \
   -v ~/work/taos/data:/var/lib/taos \
   -p 6030-6060:6030-6060 \
   -p 6030-6060:6030-6060/udp \
   tdengine/tdengine

• --name tdengine: set the container name, we can access the corresponding container by container name
• --hostnamename=tdengine-server: set the hostname of the Linux system inside the container, we can map the hostname and IP to solve the problem that the container IP may change.
• -v: Set the host file directory to be mapped to the inner container directory to avoid data loss after the container is deleted.

Use the docker ps command to verify that the container is running correctly

bash
$ docker ps
CONTAINER ID   IMAGE               COMMAND   CREATED          STATUS          ···
c452519b0f9b   tdengine/tdengine   "taosd"   14 minutes ago   Up 14 minutes   ···

• docker ps: list all containers in running state.
• CONTAINER ID: container ID.
• IMAGE: the image used.
• COMMAND: the command to run when starting the container.
• CREATED: container creation time.
• STATUS: container status. UP means running.

Enter the docker container to do development via the docker exec COMMAND

bash
$ docker exec -it tdengine /bin/bash
root@tdengine-server:~/TDengine-server-2.0.20.13# 

• docker exec: Enter the container by the docker exec command, if exited, the container will not stop.
• -i: use interactive mode.
• -t: specify a terminal.
• tdengine: container name, needs to be changed according to the value returned by the docker ps command.
• /bin/bash: load the container and run bash to interact with it.

After entering the container, execute the taos shell client program.

bash
root@tdengine-server:~/TDengine-server-2.0.20.13# taos

Welcome to the TDengine shell from Linux, Client Version:2.0.20.13
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.

taos> 

The TDengine shell successfully connects to the server and prints out a welcome message and version information. If it fails, an error message is printed.

In the TDengine shell, you can create/delete databases, tables, super tables, etc., and perform insert and query operations via SQL commands. For details, please refer to the TAOS SQL documentation.

Accessing TDengine server inside Docker container from the host side

After starting the TDengine Docker container with the correct port mapping with the -p command line parameter, you can access the TDengine running inside the Docker container from the host side using the taos shell command.

$ taos

Welcome to the TDengine shell from Linux, Client Version:2.0.22.3
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.

taos>

You can also access the TDengine server inside the Docker container using the curl command from the host side through the RESTful port.

$ curl -u root:taosdata -d 'show databases' 127.0.0.1:6041/rest/sql
{"status":"succ","head":["name","created_time","ntables","vgroups","replica","quorum","days","keep0,keep1,keep(D)","cache(MB)","blocks","minrows","maxrows","wallevel","fsync","comp","cachelast","precision","update","status"],"column_meta":[["name",8,32],["created_time",9,8],["ntables",4,4],["vgroups",4,4],["replica",3,2],["quorum",3,2],["days",3,2],["keep0,keep1,keep(D)",8,24],["cache(MB)",4,4],["blocks",4,4],["minrows",4,4],["maxrows",4,4],["wallevel",2,1],["fsync",4,4],["comp",2,1],["cachelast",2,1],["precision",8,3],["update",2,1],["status",8,10]],"data":[["test","2021-08-18 06:01:11.021",10000,4,1,1,10,"3650,3650,3650",16,6,100,4096,1,3000,2,0,"ms",0,"ready"],["log","2021-08-18 05:51:51.065",4,1,1,1,10,"30,30,30",1,3,100,4096,1,3000,2,0,"us",0,"ready"]],"rows":2}

This command accesses the TDengine server through the RESTful interface, which connects to port 6041 on the local machine, so the connection is successful.

TDengine RESTful interface details can be found in the official documentation.

Running TDengine server and taosAdapter with a Docker container

Docker containers of TDegnine version 2.4.0.0 and later include a component named taosAdapter, which supports data writing and querying capabilities to the TDengine server through the RESTful interface and provides the data ingestion interfaces compatible with InfluxDB/OpenTSDB. Allows seamless migration of InfluxDB/OpenTSDB applications to access TDengine.

Note: If taosAdapter is running inside the container, you need to add mapping to other additional ports as needed, please refer to https://github.com/taosdata/taosadapter/blob/develop/README.md for the default port number and modification methods for the specific purpose.

Running TDengine version 2.4.0.0 image with docker.

$ docker run -d \
   --name tdengine-taosa  \
   -p 6030-6060:6030-6060 \ 
   -p 6030-6060:6030-6060/udp \
   tdengine/tdengine:2.4.0.0

Verify that the RESTful interface taosAdapter provides working using the curl command.

$ curl -H 'Authorization: Basic cm9vdDp0YW9zZGF0YQ==' -d 'show databases;' 127.0.0.1:6041/rest/sql

{"status":"succ","head":["name","created_time","ntables","vgroups","replica","quorum","days","keep","cache(MB)","blocks","minrows","maxrows","wallevel","fsync","comp","cachelast","precision","update","status"],"column_meta":[["name",8,32],["created_time",9,8],["ntables",4,4],["vgroups",4,4],["replica",3,2],["quorum",3,2],["days",3,2],["keep",8,24],["cache(MB)",4,4],["blocks",4,4],["minrows",4,4],["maxrows",4,4],["wallevel",2,1],["fsync",4,4],["comp",2,1],["cachelast",2,1],["precision",8,3],["update",2,1],["status",8,10]],"data":[["log","2021-12-28 09:18:55.765",10,1,1,1,10,"30",1,3,100,4096,1,3000,2,0,"us",0,"ready"]],"rows":1}

taosAdapter supports multiple data collection agents (e.g. Telegraf, StatsD, collectd, etc.), here only demonstrates how StasD is simulated to write data, and the command is executed from the host side as follows.

$ echo "foo:1|c" | nc -u -w0 127.0.0.1 6044

Then you can use the taos shell to query the taosAdapter automatically created database statsd and the contents of the super table foo.

taos> show databases;
              name              |      created_time       |   ntables   |   vgroups   | replica | quorum |  days  |           keep           |  cache(MB)  |   blocks    |   minrows   |   maxrows   | wallevel |    fsync    | comp | cachelast | precision | update |   status   |
====================================================================================================================================================================================================================================================================================
 log                            | 2021-12-28 09:18:55.765 |          12 |           1 |       1 |      1 |     10 | 30                       |           1 |           3 |         100 |        4096 |        1 |        3000 |    2 |         0 | us        |      0 | ready      |
 statsd                         | 2021-12-28 09:21:48.841 |           1 |           1 |       1 |      1 |     10 | 3650                     |          16 |           6 |         100 |        4096 |        1 |        3000 |    2 |         0 | ns        |      2 | ready      |
Query OK, 2 row(s) in set (0.002112s)

taos> use statsd;
Database changed.

taos> show stables;
              name              |      created_time       | columns |  tags  |   tables    |
============================================================================================
 foo                            | 2021-12-28 09:21:48.894 |       2 |      1 |           1 |
Query OK, 1 row(s) in set (0.001160s)

taos> select * from foo;
              ts               |         value         |         metric_type          |
=======================================================================================
 2021-12-28 09:21:48.840820836 |                     1 | counter                      |
Query OK, 1 row(s) in set (0.001639s)

taos>

You can see that the simulation data has been written to TDengine.

Application example: write data to TDengine server in Docker container using taosdemo on the host

1, execute taosdemo in the host command-line interface to write data to the TDengine server in the Docker container

bash
$ taosdemo

taosdemo is simulating data generated by power equipments monitoring...

host:                       127.0.0.1:6030
user:                       root
password:                   taosdata
configDir:                  
resultFile:                 ./output.txt
thread num of insert data:  10
thread num of create table: 10
top insert interval:        0
number of records per req:  30000
max sql length:             1048576
database count:             1
database[0]:
  database[0] name:      test
  drop:                  yes
  replica:               1
  precision:             ms
  super table count:     1
  super table[0]:
      stbName:           meters
      autoCreateTable:   no
      childTblExists:    no
      childTblCount:     10000
      childTblPrefix:    d
      dataSource:        rand
      iface:             taosc
      insertRows:        10000
      interlaceRows:     0
      disorderRange:     1000
      disorderRatio:     0
      maxSqlLen:         1048576
      timeStampStep:     1
      startTimestamp:    2017-07-14 10:40:00.000
      sampleFormat:      
      sampleFile:        
      tagsFile:          
      columnCount:       3
column[0]:FLOAT column[1]:INT column[2]:FLOAT 
      tagCount:            2
        tag[0]:INT tag[1]:BINARY(16) 

         Press enter key to continue or Ctrl-C to stop

After entering, this command will automatically create a super table meters under the database test, there are 10,000 tables under this super table, the table name is "d0" to "d9999", and each table has 10,000 records, and each record has four fields (ts, current, voltage, phase), the timestamp is from "2017-07-14 10:40:00 000" to "2017-07-14 10:40:09 999", each table has a tag location and groupId, groupId is set from 1 to 10 and location is set to "beijing" or "shanghai".

It takes about a few minutes to execute this command and ends up inserting a total of 100 million records.

3, Go to the TDengine terminal and view the data generated by taosBenchmark.

• Go to the terminal interface.

bash
$ root@c452519b0f9b:~/TDengine-server-2.0.20.13# taos

Welcome to the TDengine shell from Linux, Client Version:2.0.20.13
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.

taos> 

• View the database.

bash
$ taos> show databases;
  name        |      created_time       |   ntables   |   vgroups   |    ···
  test        | 2021-08-18 06:01:11.021 |       10000 |           6 |    ···
  log         | 2021-08-18 05:51:51.065 |           4 |           1 |	   ···

• View Super Tables.

bash
$ taos> use test;
Database changed.

$ taos> show stables;
              name              |      created_time       | columns |  tags  |   tables    |
============================================================================================
 meters                         | 2021-08-18 06:01:11.116 |       4 |      2 |       10000 |
Query OK, 1 row(s) in set (0.003259s)

• View the table and limit the output to 10 entries.

bash
$ taos> select * from test.t0 limit 10;

DB error: Table does not exist (0.002857s)
taos> select * from test.d0 limit 10;
           ts            |       current        |   voltage   |        phase         |
======================================================================================
 2017-07-14 10:40:00.000 |             10.12072 |         223 |              0.34167 |
 2017-07-14 10:40:00.001 |             10.16103 |         224 |              0.34445 |
 2017-07-14 10:40:00.002 |             10.00204 |         220 |              0.33334 |
 2017-07-14 10:40:00.003 |             10.00030 |         220 |              0.33333 |
 2017-07-14 10:40:00.004 |              9.84029 |         216 |              0.32222 |
 2017-07-14 10:40:00.005 |              9.88028 |         217 |              0.32500 |
 2017-07-14 10:40:00.006 |              9.88110 |         217 |              0.32500 |
 2017-07-14 10:40:00.007 |             10.08137 |         222 |              0.33889 |
 2017-07-14 10:40:00.008 |             10.12063 |         223 |              0.34167 |
 2017-07-14 10:40:00.009 |             10.16086 |         224 |              0.34445 |
Query OK, 10 row(s) in set (0.016791s)

• View the tag values for the d0 table.

bash
$ taos> select groupid, location from test.d0;
   groupid   |     location     |
=================================
           0 | shanghai         |
Query OK, 1 row(s) in set (0.003490s)

Stop the TDengine service that is running in Docker

bash
$ docker stop tdengine
tdengine

• docker stop: Stop the specified running docker image with docker stop.
• tdengine: The name of the container.