Misplaced Pages

Apache IoTDB

Article snapshot taken from Wikipedia with creative commons attribution-sharealike license. Give it a read and then ask your questions in the chat. We can research this topic together.
Open Source Database Project
This article may be too technical for most readers to understand. Please help improve it to make it understandable to non-experts, without removing the technical details. (April 2023) (Learn how and when to remove this message)
Apache IoTDB
Developer(s)Apache Software Foundation
Stable release1.1.0 / 3 April 2023
Repositorygithub.com/apache/iotdb
Written inJava
PlatformCross-platform
Type
LicenseApache License 2.0
Websiteiotdb.apache.org

Apache IoTDB is a column-oriented open-source, time-series database (TSDB) management system written in Java. It has both edge and cloud versions, provides an optimized columnar file format for efficient time-series data storage, and TSDB with high ingestion rate, low latency queries and data analysis support. It is specially optimized for time-series oriented operations like aggregations query, downsampling and sub-sequence similarity search. The name IoTDB comes from Internet of Things (IoT) Database, which means it was designed as an IoT-native TSDB that resolves the pain points of the typical IoT scenarios, including massive data generation, high frequency sampling, out-of-order data, specific analytics requirements, high costs of storage and operation & maintenance, low computational power of IoT devices.

History

Apache IoTDB is a project initiated by Prof. Jianmin Wang's team in the School of Software at Tsinghua University. In 2011, the team chose to use open source NoSQL technology instead of Oracle for a project with mass machine data management, and noticed the insufficiency of NoSQL in the industrial internet of things (IIoT) scenarios. The team started to develop a data management system and formally proposed TsFile, an optimized columnar compact file storage format for time series data, in March 2016. The source code was then opened on GitHub.

In June 2016, based on TsFile, the team began to develop IoTDB, an IIoT database supporting real-time reading & writing and analysis.

In November 2018, the project IoTDB entered incubator at the Apache Software Foundation (ASF).

On September 16, 2020, the ASF officially issued a resolution to promote Apache IoTDB to the global Top-Level Project (TLP) following a public discussion vote by the community and a show of hands vote by the board.

Architecture

The Architecture of Apache IoTDB

The complete storage system of Apache IoTDB follows a client-server architecture, including IoTDB engine (server) and several components as IoTDB suite (client). IoTDB suite can provide a series of functions in the real situation such as data collection, data writing, data storage, data query, data visualization and data analysis. This allows data collected by the sensor to constantly persist in server, where the data can then be used for native query or shipped to other open-source platforms for data analysis. In particular, IoTDB provides a mode called "Edge-Cloud Cooperation", which can synchronize data collected at every user-configured interval from one IoTDB instance to another using Sync Tool.

Users can use JDBC to write time series data to local/remote IoTDB. This time series data may represent system state data (such as server load and CPU memory, etc.), message queue data, time series data from applications, or other time series data in the database. The data can be directly written to TsFile locally or on Hadoop Distributed File System (HDFS).

TsFile is a column storage file format developed for accessing, compressing and storing time series data in Apache IoTDB. Its structure is based on LSM-Tree, which reduces the computational resources and optimizes the performance of Apache IoTDB.

TsFile could be written to the HDFS, thereby implementing data processing tasks such as abnormality detection and machine learning on the Hadoop or Spark data processing platform.

For the data written to HDFS or local TsFile, users can use TsFile-Hadoop-Connector or TsFile-Spark-Connector to allow Hadoop or Spark to process data. The results of the analysis can be written back to TsFile in the same way. Also, IoTDB and TsFile provide client tools to meet the various needs of users in writing and viewing data in SQL form, script form and graphical form.

Features

Flexible and cross-platform deployment

IoTDB is designed to fit three deployment scenarios: 1) file-based storage or embedded time-series database on edge appliance like Raspberry Pi, 2) standalone TSDB on Industrial PC and 3) distributed TSDB or Hadoop cluster with TsFile. IoTDB provides users a one-click installation tool on the cloud, once-decompressed-used terminal tool and the bridging tool between cloud platforms and terminal tools (Data Synchronization Tool).

Low storage cost

IoTDB can reach a high compression ratio of disk storage, which means IoTDB can store the same amount of data with less hardware disk cost.

Efficient directory structure

IoTDB supports efficient organization of complex time-series data structures from intelligent networking devices, organization of time-series data from devices of the same type, fuzzy searching strategy for massive and complex directory of time-series data.

High-throughput read and write

IoTDB supports millions of low-power devices' strong connection data access, high-speed data read and write for intelligent networking devices and mixed devices mentioned above. Currently, IoTDB supports the ingestion rate of up to 30 million data points per second on a single node.

Rich query semantics

IoTDB supports time alignment for timeseries data across devices and sensors, computation in timeseries field (frequency domain transformation) and rich aggregation function support in time dimension.

Easy to get started

IoTDB supports SQL-Like language, JDBC standard API and import/export tools which are easy to use.

Intense integration with open source ecosystem

IoTDB supports Hadoop, Spark, etc. analysis ecosystems and Grafana visualization tool.

Licensing

The Apache 2.0 License is a permissive free software license written by the Apache Software Foundation. It allows end users to modify parts of the original code as long as it contains the appropriate documentation that Apache requires within the redistributed code.

References

  1. ^ Sally (23 September 2020). "The Apache Software Foundation Announces Apache® IoTDB™ as A Top-Level Project". The Apache Software Foundation Blog. Retrieved 18 November 2022.
  2. ^ Wang, Chen; Huang, Xiangdong; Qiao, Jialin; Jiang, Tian; Rui, Lei; Zhang, Jinrui; Kang, Rong; Feinauer, Julian; McGrail, Keven A.; Wang, Peng; Yuan, Jun; Wang, Jianmin; Sun, Jiaguang (August 2020). "Apache IoTDB" (PDF). Proceedings of the VLDB Endowment. 13 (12): 2901–2904. doi:10.14778/3415478.3415504. S2CID 221352039.
  3. ^ Hou, Haonan (14 March 2022). "TsFile Format". ASF Confluence. Retrieved 18 November 2022.
  4. "Apache IoTDB Project Incubation Status". Apache Incubator. Retrieved 18 November 2022.
  5. online, heise (23 September 2020). "Apache Software Foundation erhebt IoTDB zum Top-Level-Projekt". Developer (in German). Retrieved 2022-12-13.
  6. ^ "IoTDB User Guide: System Architecture". Apache IoTDB. Retrieved 18 November 2022.
  7. "Apache IoTDB". Database of Databases. 27 June 2022. Retrieved 18 November 2022.
  8. Xiao, Jinzhao; Huang, Yuxiang; Hu, Changyu; Song, Shaoxu; Huang, Xiangdong; Wang, Jianmin (2022-09-07). "Time series data encoding for efficient storage: a comparative analysis in Apache IoTDB". Proceedings of the VLDB Endowment. 15 (10): 2148–2160. doi:10.14778/3547305.3547319. ISSN 2150-8097. S2CID 252135944.
  9. Huang, Xiangdong; Wang, Jianmin; Wong, Raymond; Zhang, Jinrui; Wang, Chen (2016-10-24). "PISA". Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. CIKM '16. New York, NY, USA: Association for Computing Machinery. pp. 979–988. doi:10.1145/2983323.2983775. ISBN 978-1-4503-4073-1. S2CID 12456810.
  10. Kang, Rong; Wang, Chen; Wang, Peng; Ding, Yuting; Wang, Jianmin (2018). "Matching Consecutive Subpatterns over Streaming Time Series". In Cai, Yi; Ishikawa, Yoshiharu; Xu, Jianliang (eds.). Web and Big Data. Lecture Notes in Computer Science. Vol. 10988. Cham: Springer International Publishing. pp. 90–105. arXiv:1805.06757. doi:10.1007/978-3-319-96893-3_8. ISBN 978-3-319-96893-3. S2CID 21687305.
  11. Wu, Jiaye; Wang, Peng; Pan, Ningting; Wang, Chen; Wang, Wei; Wang, Jianmin (2019). "KV-Match: A Subsequence Matching Approach Supporting Normalization and Time Warping". 2019 IEEE 35th International Conference on Data Engineering (ICDE). pp. 866–877. arXiv:1710.00560. doi:10.1109/ICDE.2019.00082. ISBN 978-1-5386-7474-1. S2CID 46926461.
  12. Mao, Dongfang; Li, Tianan; Huang, Xiangdong; Yuan, Jun; Xu, Yi; Wang, Jianmin (27 April 2020). "The design of Apache IoTDB distributed framework". National Database Conference. 50 (5): 621–636. doi:10.1360/SSI-2019-0189. S2CID 219053248.
  13. Qiao, Jialin; Huang, Xiangdong; Wang, Jianmin; Wong, Raymond K. (2020-01-01). "Dual-PISA: An index for aggregation operations on time series data". Information Systems. 87: 101427. doi:10.1016/j.is.2019.101427. ISSN 0306-4379. S2CID 201127537.
  14. ^ "IoTDB User Guide: Features". Iotdb Website. Retrieved 18 November 2022.
  15. vogler. "Automation Gateway with Apache IoTDB… | RocWorks". Retrieved 2022-12-13.
  16. "Apache IoTDB Dashboard v0.13.1". Grafana Labs. Retrieved 2022-12-13.
  17. "Apache License, version 2.0". The Apache Software Foundation. Retrieved 18 November 2022.
Categories: