mirror of https://github.com/apache/iotdb
Add design documents (#757)
* add design doc and instructions for inserting figures Co-authored-by: Zesong Sun <szs19@mails.tsinghua.edu.cn>
This commit is contained in:
Jialin Qiao
GitHub
parent
c8f7afcadb
commit
23950758a5
|
|
@ -175,7 +175,7 @@ matrix:
|
|||
# - mvn -B apache-rat:check
|
||||
# - mvn -B clean package -pl server,grafana,client,example,:kafka-example,:rocketmq-example -am integration-test
|
||||
- os: linux
|
||||
# if: fork = false #only fork=true (i.e., the committer has permission to write the repo)
|
||||
if: fork = false #only fork=true (i.e., the committer has permission to write the repo)
|
||||
name: sonar-analysis
|
||||
dist: xenial
|
||||
jdk: openjdk8
|
||||
|
|
|
|||
|
|
@ -95,6 +95,9 @@ Then,
|
|||
* Step 1: create a new folder named "7-RoadMap", and add document named `1-B.md` in folder "7-RoadMap";
|
||||
* Step 2: modify `0-Content.md` file by adding "# Chapter 7: RoadMap" in the end, and adding `* 1-B.md` in the list of this new chapter.
|
||||
|
||||
If you need to insert **figures** into documents, you can firstly update the figures in [this issue](https://github.com/thulab/iotdb/issues/543) for storing pictures in IoTDB website or other MD files.
|
||||
Drag a picture and then quote the figure's URL link.
|
||||
|
||||
### Contributing Bug Reports
|
||||
|
||||
If you encounter a problem, try to search the mailing list and JIRA to check whether other people have faced the same situation. If it is not reported before, please report an issue.
|
||||
|
|
@ -45,4 +45,7 @@ Then,
|
|||
* Step 1: create a new folder named "7-RoadMap", and add document named `1-B.md` in folder "7-RoadMap";
|
||||
* Step 2: modify `0-Content.md` file by adding "# Chapter 7: RoadMap" in the end, and adding `* 1-B.md` in the list of this new chapter.
|
||||
|
||||
If you need to insert **figures** into documents, you can firstly update the figures in [this issue](https://github.com/thulab/iotdb/issues/543) for storing pictures in IoTDB website or other MD files.
|
||||
Drag a picture and then quote the figure's URL link.
|
||||
|
||||
> If you want to contribute more (for example, reviewing Changes, reporting bugs, or even being commiters), please refer to [this page](/#/Development/Contributing).
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 应用概览
|
||||
|
||||
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625222-ddd88680-467e-11ea-9098-e808ed4979c5.png">
|
||||
|
||||
物联网时序数据库 Apache IoTDB 的架构图如上所示，覆盖了对时序数据的采集、存储、查询、分析以及可视化等全生命周期的数据管理功能，其中灰色部分为 IoTDB 组件。
|
||||
|
||||
## IoTDB 架构介绍
|
||||
|
||||
IoTDB 采用客户端-服务器架构，如下图所示。
|
||||
|
||||
<img style="width:100%; max-width:400px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625221-ddd88680-467e-11ea-9cf3-70367e5886f4.png">
|
||||
|
||||
其中服务器端主要包括查询引擎，用来处理用户的所有请求，并分发到对应的管理组件，包括数据写入层、数据查询、元数据管理、权限管理等模块。
|
||||
|
||||
* [数据文件](../1-TsFile/1-TsFile.md)
|
||||
* [查询引擎](../2-QueryEngine/1-QueryEngine.md)
|
||||
* [元数据管理](../3-SchemaManager/1-MManager.md)
|
||||
* [存储引擎](../4-StorageEngine/1-StorageEngine.md)
|
||||
* [数据查询](../5-DataQuery/1-DataQuery.md)
|
||||
* [权限管理](../6-Administration/1-Administration.md)
|
||||
|
||||
## 连接器
|
||||
|
||||
IoTDB 与大数据系统进行了对接。
|
||||
|
||||
* [Hadoop-TsFile](../7-Connector/1-Hadoop-TsFile.md)
|
||||
* [Hive-TsFile](../7-Connector/2-Hive-TsFile.md)
|
||||
* [Spark-TsFile](../7-Connector/3-Spark-TsFile.md)
|
||||
* [Spark-IoTDB](../7-Connector/4-Spark-IoTDB.md)
|
||||
* [Grafana](../7-Connector/5-Grafana.md)
|
||||
|
||||
## 工具
|
||||
|
||||
* [同步工具](../8-Tools/1-Sync.md)
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 数据文件
|
||||
|
||||
TsFile 是 IoTDB 的底层数据文件，专门为时间序列数据设计的列式文件格式。
|
||||
|
||||
|
||||
## 相关文档
|
||||
|
||||
* [文件格式](2-Format.md)
|
||||
* [写流程](3-Write.md)
|
||||
* [读流程](4-Read.md)
|
||||
|
|
@ -7,9 +7,9 @@
|
|||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
|
|
@ -19,7 +19,8 @@
|
|||
|
||||
-->
|
||||
|
||||
# 第8章: 系统设计
|
||||
# TsFile 文件格式
|
||||
|
||||
|
||||
## 1. TsFile 设计
|
||||
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# TsFile 写流程
|
||||
|
||||
* org.apache.iotdb.tsfile.write.*
|
||||
|
||||
TsFile 的写入流程如下图所示：
|
||||
|
||||
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625238-efba2980-467e-11ea-927e-a7021f8153af.png">
|
||||
|
||||
其中，每个设备对应一个 ChunkGroupWriter，每个传感器对应一个 ChunkWriter。
|
||||
|
||||
文件的写入主要分为三种操作，在图上用 1、2、3 标注
|
||||
|
||||
* 1、写内存换冲区
|
||||
* 2、持久化 ChunkGroup
|
||||
* 3、关闭文件
|
||||
|
||||
## 1、写内存缓冲区
|
||||
|
||||
TsFile 文件层的写入接口有两种
|
||||
|
||||
* TsFileWriter.write(TSRecord record)
|
||||
|
||||
写入一个设备一个时间戳多个测点。
|
||||
|
||||
* TsFileWriter.write(RowBatch rowBatch)
|
||||
|
||||
写入一个设备多个时间戳多个测点。
|
||||
|
||||
当调用 write 接口时，这个设备的数据会交给对应的 ChunkGroupWriter，其中的每个测点会交给对应的 ChunkWriter 进行写入。ChunkWriter 完成编码和打包（生成 Page）。
|
||||
|
||||
|
||||
## 2、持久化 ChunkGroup
|
||||
|
||||
* TsFileWriter.flushAllChunkGroups()
|
||||
|
||||
当内存中的数据达到一定阈值，会触发持久化操作。每次持久化会把当前内存中所有设备的数据全部持久化到磁盘的 TsFile 文件中。每个设备对应一个 ChunkGroup，每个测点对应一个 Chunk。
|
||||
|
||||
持久化完成后会在内存中缓存对应的元数据信息，以供查询和生成文件尾部 metadata。
|
||||
|
||||
## 3、关闭文件
|
||||
|
||||
* TsFileWriter.close()
|
||||
|
||||
根据内存中缓存的元数据，生成 TsFileMetadata 追加到文件尾部，最后关闭文件。
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# TsFile 读流程
|
||||
|
||||
* org.apache.iotdb.tsfile.read.*
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 查询引擎
|
||||
|
||||
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625242-f648a100-467e-11ea-921c-b954a3ecae7a.png">
|
||||
|
||||
## 设计思想
|
||||
|
||||
查询引擎负责所有用户命令的解析、生成计划、交给对应的执行器、返回结果集。
|
||||
|
||||
## 相关类
|
||||
|
||||
* org.apache.iotdb.db.service.TSServiceImpl
|
||||
|
||||
IoTDB 服务器端 RPC 实现，与客户端进行直接交互。
|
||||
|
||||
* org.apache.iotdb.db.qp.Planner
|
||||
|
||||
解析 SQL，生成逻辑计划，逻辑优化，生成物理计划。
|
||||
|
||||
* org.apache.iotdb.db.qp.executor.PlanExecutor
|
||||
|
||||
分发物理计划给对应的执行器，主要包括以下四个具体的执行器。
|
||||
|
||||
* MManager: 元数据操作
|
||||
* StorageEngine: 数据写入
|
||||
* QueryRouter: 数据查询
|
||||
* LocalFileAuthorizer: 权限操作
|
||||
|
||||
* org.apache.iotdb.db.query.dataset.*
|
||||
|
||||
分批构造结果集返回给客户端，包含部分查询逻辑。
|
||||
|
||||
## 查询流程
|
||||
|
||||
* SQL 解析
|
||||
* 生成逻辑计划
|
||||
* 生成物理计划
|
||||
* 构造结果集生成器
|
||||
* 分批返回结果集
|
||||
|
||||
## 相关文档
|
||||
|
||||
* [查询计划生成器](2-Planner.md)
|
||||
* [计划执行器](3-PlanExecutor.md)
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 执行计划生成器 Planner
|
||||
|
||||
* org.apache.iotdb.db.qp.Planner
|
||||
|
||||
将 SQL 解析出的语法树转化成逻辑计划，逻辑优化，物理计划。
|
||||
|
||||
## SQL 解析
|
||||
|
||||
SQL 解析采用 Antlr4
|
||||
|
||||
* server/src/main/antlr4/org/apache/iotdb/db/qp/strategy/SqlBase.g4
|
||||
|
||||
mvn clean compile 之后生成代码位置：server/target/generated-sources/antlr4
|
||||
|
||||
## 逻辑计划生成器
|
||||
|
||||
* org.apache.iotdb.db.qp.strategy.LogicalGenerator
|
||||
|
||||
## 逻辑计划优化器
|
||||
|
||||
目前有三种逻辑计划优化器
|
||||
|
||||
* org.apache.iotdb.db.qp.strategy.optimizer.ConcatPathOptimizer
|
||||
|
||||
路径优化器，将 SQL 中的查询路径进行拼接，与 MManager 进行交互去掉通配符，进行路径检查。
|
||||
|
||||
* org.apache.iotdb.db.qp.strategy.optimizer.RemoveNotOptimizer
|
||||
|
||||
谓词去非优化器，将谓词逻辑中的非操作符去掉。
|
||||
|
||||
* org.apache.iotdb.db.qp.strategy.optimizer.DnfFilterOptimizer
|
||||
|
||||
将谓词转化为析取范式。
|
||||
|
||||
* org.apache.iotdb.db.qp.strategy.optimizer.MergeSingleFilterOptimizer
|
||||
|
||||
将相同路径的谓词逻辑合并。
|
||||
|
||||
## 物理计划生成器
|
||||
|
||||
* org.apache.iotdb.db.qp.strategy.PhysicalGenerator
|
||||
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 计划执行器
|
||||
|
||||
* org.apache.iotdb.db.qp.executor.PlanExecutor
|
||||
|
||||
执行物理计划。
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 元数据管理
|
||||
|
||||
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625246-fc3e8200-467e-11ea-8815-67b9c4ab716e.png">
|
||||
|
||||
IoTDB 的元数据管理采用目录树的形式，倒数第二层为设备层，最后一层为传感器层。
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 存储引擎
|
||||
|
||||
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625255-03fe2680-467f-11ea-91ae-64407ef1125c.png">
|
||||
|
||||
## 设计思想
|
||||
|
||||
存储引擎基于 LSM 设计。数据首先写入内存缓冲区 memtable 中，再刷到磁盘。内存中为每个设备维护当前持久化的（包括已经落盘的和正在持久化的）最大时间戳，根据这个时间戳将数据区分为顺序数据和乱序数据，不同种类的数据通过不同的 memtable 和 TsFile 管理。
|
||||
|
||||
每个数据文件 TsFile 在内存中对应一个文件索引信息 TsFileResource，供查询使用。
|
||||
|
||||
此外，存储引擎还包括异步持久化和文件合并机制。
|
||||
|
||||
## 写入流程
|
||||
|
||||
### 相关代码
|
||||
|
||||
* org.apache.iotdb.db.engine.StorageEngine
|
||||
|
||||
负责一个 IoTDB 实例的写入和访问，管理所有的 StorageGroupProsessor。
|
||||
|
||||
* org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor
|
||||
|
||||
负责一个存储组一个时间分区内的数据写入和访问。管理所有分区的TsFileProcessor。
|
||||
|
||||
* org.apache.iotdb.db.engine.storagegroup.TsFileProcessor
|
||||
|
||||
负责一个 TsFile 文件的数据写入和访问。
|
||||
|
||||
|
||||
### 单行数据（一个设备一个时间戳多个值）写入
|
||||
|
||||
* 对应的接口
|
||||
* JDBC 的 execute 和 executeBatch 接口
|
||||
* Session 的 insert 和 insertInBatch
|
||||
|
||||
* 总入口: public void insert(InsertPlan insertPlan)
|
||||
* 找到对应的 StorageGroupProsessor
|
||||
* 根据写入数据的时间以及当前设备落盘的最后时间戳，找到对应的 TsFileProcessor
|
||||
* 写入 TsFileProcessor 对应的 memtable 中
|
||||
* 记录写前日志
|
||||
* 根据 memtable 大小，来判断是否触发异步持久化 memtable 操作
|
||||
* 根据当前磁盘 TsFile 的大小，判断是否触发文件关闭操作
|
||||
|
||||
### 批量数据（一个设备多个时间戳多个值）写入
|
||||
|
||||
* 对应的接口
|
||||
* Session 的 insertBatch
|
||||
|
||||
* 总入口: public Integer[] insertBatch(BatchInsertPlan batchInsertPlan)
|
||||
* 找到对应的 StorageGroupProsessor
|
||||
* 根据这批数据的时间以及当前设备落盘的最后时间戳，将这批数据分成小批，分别对应到一个 TsFileProcessor 中
|
||||
* 分别将每小批写入 TsFileProcessor 对应的 memtable 中
|
||||
* 记录写前日志
|
||||
* 根据 memtable 大小，来判断是否触发异步持久化 memtable 操作
|
||||
* 根据当前磁盘 TsFile 的大小，判断是否触发文件关闭操作
|
||||
|
||||
|
||||
## 数据访问
|
||||
|
||||
* 总入口（StorageEngine）: public QueryDataSource query(SingleSeriesExpression seriesExpression, QueryContext context,
|
||||
QueryFileManager filePathsManager)
|
||||
|
||||
* 找到所有包含这个时间序列的顺序和乱序的 TsFileResource 进行返回，供查询引擎使用。
|
||||
|
||||
## 相关文档
|
||||
|
||||
* [写前日志 (WAL)](2-WAL.md)
|
||||
|
||||
* [memtable 持久化](3-FlushManager.md)
|
||||
|
||||
* [文件合并机制](4-MergeManager.md)
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 写前日志
|
||||
|
||||
## 相关代码
|
||||
|
||||
* org.apache.iotdb.db.writelog.*
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# Flush Memtable
|
||||
|
||||
## 设计思想
|
||||
|
||||
内存缓冲区 memtable 达到一定阈值后，会交给 FlushManager 进行异步的持久化，不阻塞正常写入。持久化的过程采用流水线的方式。
|
||||
|
||||
## 相关代码
|
||||
|
||||
* org.apache.iotdb.db.engine.flush.FlushManager
|
||||
|
||||
Memtable 的 Flush 任务管理器。
|
||||
|
||||
* org.apache.iotdb.db.engine.flush.MemtableFlushTask
|
||||
|
||||
负责持久化一个 Memtable。
|
||||
|
||||
## FlushManager: 持久化管理器
|
||||
|
||||
FlushManager 可以接受 memtable 的持久化任务，提交者有两个，第一个是 TsFileProcessor，第二个是持久化子线程 FlushThread。
|
||||
|
||||
每个 TsFileProcessor 同一时刻只会有一个 flush 任务执行，一个 TsFileProcessor 可能对应多个需要持久化的 memtable
|
||||
|
||||
## MemTableFlushTask: 持久化任务
|
||||
|
||||
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/73625254-03fe2680-467f-11ea-8197-115f3a749cbd.png">
|
||||
|
||||
背景：每个 memtable 可包含多个 device，每个 device 可包含多个 measurement。
|
||||
|
||||
### 三个线程
|
||||
|
||||
一个 memtable 的持久化的过程有三个线程，只有当所有任务都完成后，主线程工作才结束。
|
||||
|
||||
* MemTableFlushTask 所在线程
|
||||
|
||||
持久化主线程兼排序线程，负责给每个 measurement 对应的 chunk 排序。
|
||||
|
||||
* encodingTask 线程
|
||||
|
||||
编码线程，负责给每个 Chunk 进行编码，编码成字节数组。
|
||||
|
||||
* ioTask 线程
|
||||
|
||||
IO线程，负责将编码好的 Chunk 持久化到磁盘的 TsFile 文件上。
|
||||
|
||||
### 两个任务队列
|
||||
|
||||
三个线程之间通过两个任务队列交互
|
||||
|
||||
* encodingTaskQueue: 排序线程->编码线程，包括三种任务
|
||||
|
||||
* StartFlushGroupIOTask：开始持久化一个 device (ChunkGroup)， encoding 不处理这个命令，直接发给 IO 线程。
|
||||
|
||||
* Pair\<TVList, MeasurementSchema\>：编码一个 Chunk
|
||||
|
||||
* EndChunkGroupIoTask：结束一个 device (ChunkGroup) 的持久化，encoding 不处理这个命令，直接发给 IO 线程。
|
||||
|
||||
* ioTaskQueue: 编码线程->IO线程，包括三种任务
|
||||
|
||||
* StartFlushGroupIOTask：开始持久化一个 device (ChunkGroup)。
|
||||
|
||||
* IChunkWriter：持久化一个 Chunk 到磁盘上
|
||||
|
||||
* EndChunkGroupIoTask：结束一个 device (ChunkGroup) 的持久化。
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 文件合并机制
|
||||
|
||||
## 相关代码
|
||||
|
||||
* org.apache.iotdb.db.engine.merge.*
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# 数据查询
|
||||
|
||||
* 查询种类
|
||||
* 原始数据查询
|
||||
* 聚合查询
|
||||
* 降采样查询
|
||||
* 单点补空值查询
|
||||
* 最新数据查询等
|
||||
|
||||
## 查询计划
|
||||
|
||||
主要包括两种查询框架：
|
||||
|
||||
* 不带值过滤条件的（如，select f1, f2, f3 from root.d1）
|
||||
|
||||
可以按照多路归并的方式进行查询，分别查询三列的结果，最后按照时间进行对齐。
|
||||
|
||||
* 带值过滤条件的（如，select f1, f2, f3 from root.d1 where f1 = 1 and f2 = 2）
|
||||
|
||||
先根据过滤条件生成结果集的时间戳，再根据时间戳查询其他列的值。
|
||||
|
||||
## 查询组件
|
||||
|
||||
为了实现任意过滤条件下的多种查询，有以下查询组件
|
||||
|
||||
* SeriesReaderWithoutValueFilter
|
||||
|
||||
一个时间序列的 reader，不带任何值过滤条件，但是可能带时间过滤条件。
|
||||
|
||||
* SeriesReaderWithValueFilter
|
||||
|
||||
一个时间序列的 reader，带值过滤条件（也可能包含时间过滤条件）。
|
||||
|
||||
* SeriesReaderByTimestamp
|
||||
|
||||
一个时间序列的 reader，每给一个时间戳，返回此时间戳对应的值，若没有则返回空。
|
||||
|
||||
* TimeGenerator
|
||||
* 功能：根据过滤条件构造表达式的树形结构，生成结果集的时间戳，如 f1 = 1 and f2 = 2，构造一棵树 AND（leaf(f1=1), leaf(f2=2)）。
|
||||
* 叶子节点：对应一个 SeriesReaderWithValueFilter，返回满足条件的结果的时间戳。
|
||||
* 中间节点：按照此节点的谓词逻辑对时间戳做计算。
|
||||
* 根节点：返回最终结果集的时间戳。
|
||||
|
||||
## AbstractSeriesReader
|
||||
|
||||
单个时间序列的基础 Reader 类。输入为一个 QueryDataSource，包括顺序和乱序文件列表。
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
# Chapter 0: Architecture
|
||||
* 1-Architecture
|
||||
# Chapter 1: TsFile
|
||||
* 1-TsFile
|
||||
* 2-Format
|
||||
* 3-Write
|
||||
* 4-Read
|
||||
# Chapter 2: QueryEngine
|
||||
* 1-QueryEngine
|
||||
* 2-Planner
|
||||
* 3-PlanExecutor
|
||||
# Chapter 3: SchemaManager
|
||||
* 1-SchemaManager
|
||||
# Chapter 4: StorageEngine
|
||||
* 1-StorageEngine
|
||||
* 2-WAL
|
||||
* 3-FlushManager
|
||||
* 4-MergeManager
|
||||
# Chapter 5: DataQuery
|
||||
* 1-DataQuery
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# TsFile
|
||||
|
||||
TsFile is the data file of IoTDB.
|
||||
|
||||
## Related
|
||||
|
||||
* [File Format](2-Format.md)
|
||||
* [Write Process](3-Write.md)
|
||||
* [Read Process](4-Read.md)
|
||||
|
|
@ -7,9 +7,9 @@
|
|||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
|
|
@ -19,7 +19,7 @@
|
|||
|
||||
-->
|
||||
|
||||
# Chapter 7: System Design
|
||||
# TsFile Format
|
||||
|
||||
## 1. TsFile Design
|
||||
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# TsFile Write Process
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
<!--
|
||||
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
-->
|
||||
|
||||
# TsFile Read Process
|
||||
|
|
@ -65,8 +65,7 @@
|
|||
* 3-Spark TsFile
|
||||
* 4-Spark IoTDB
|
||||
* 5-Hive TsFile
|
||||
# Chapter 8: System Design (Developer)
|
||||
* 1-Hierarchy
|
||||
* 2-Files
|
||||
* 3-Writing Data on HDFS
|
||||
* 4-Shared Nothing Cluster
|
||||
# Chapter 8: Architecture
|
||||
* 1-Files
|
||||
* 2-Writing Data on HDFS
|
||||
* 3-Shared Nothing Cluster
|
||||
|
|
|
|||
4
pom.xml
4
pom.xml
|
|
@ -507,6 +507,10 @@
|
|||
<exclude>licenses/*</exclude>
|
||||
<!-- generated by Github -->
|
||||
<exclude>.github/**</exclude>
|
||||
<!-- figures -->
|
||||
<exclude>**/.eps</exclude>
|
||||
<exclude>**/.png</exclude>
|
||||
<exclude>**/.jpg</exclude>
|
||||
<!--Generated by Apache Release -->
|
||||
<exclude>local-snapshots-dir/**</exclude>
|
||||
<!-- JSON can't contain comments and therefore no Apache header -->
|
||||
|
|
|
|||
Loading…
Reference in New Issue