add update pubkey in actuator;add README.en-US.md
This commit is contained in:
kaichao
parent
cbeca1120c
commit
faa72428de
|
|
@ -0,0 +1,82 @@
|
|||
# Scalebox - A cloud-native streaming computing engine
|
||||
|
||||
Scalebox is a cloud-native streaming computing engine that can run containerized stand-alone user algorithms on distributed and heterogeneous computing clusters, organize large-scale parallel processing at the module level with pipelines, and support task-level fault tolerance. Compared with existing frameworks such as big data processing and parallel computing, its technical characteristics are especially suitable for application scenarios such as data distribution, computing power resource distribution, and complex algorithms.
|
||||
|
||||
Scalebox has the following important properties:
|
||||
|
||||
- **Cloud-native design**: Encapsulate all algorithm modules and transmission modules in containers, and embed them into the data processing pipeline designed for the cloud environment through the sidecar model; the software basic platform is completely based on the cloud-native design; the platform will control messages, The data channel is separated, and the front and back modules are connected by a message bus to realize non-intrusive parallel programming, which greatly simplifies the difficulty of parallel computing.
|
||||
- [**Cross-cluster computing**](./tests/cross-cluster-primes/): Normalized processing algorithm module, transmission module, unified processing of intra-cluster/cross-cluster data through the pipeline, shielding data and computing Cross-cluster differences; cross-cluster message-driven stream processing supports the deployment of a single pipeline application on multiple heterogeneous computing power clusters.
|
||||
- [**Task-level fault tolerance**](./tests/retry_test/): For sporadic errors caused by hardware failures, software bugs, network problems, data anomalies, etc., automatic fault-tolerant processing is implemented based on rules; fine-grained task-level Fault tolerance for trusted data analytics on unreliable hardware.
|
||||
- **Location-aware scheduling**: The IP address of the sender can be configured in the message body, which supports local cascading processing between front and rear modules; separates the processing layer and storage layer to reduce coupling, and messages in the horizontal direction (processing layer) drive the vertical direction Data reading and writing (from the processing layer to the storage layer) reduces the east-west network traffic in the computing cluster and eliminates the I/O bottleneck of the cluster storage; then realizes local computing without shared storage, pure local loading of large files, and effectively supports horizontal expansion.
|
||||
- [**Task Perspective**](./tests/task-perspective/): The computing task is a message-driven process, and the task perspective records the detailed running status of each task in detail; including user program return code, standard output, Standard error, program custom text, number of bytes read and written by user programs, etc., also includes various system-level and user-defined time stamps on the computing container side and control side during the task execution cycle (message generation/distribution/processing, result recording) . Task perspective provides basic support for precise positioning, application optimization, and data statistics in troubleshooting.
|
||||
|
||||
- **Multiple parallelization methods**
|
||||
- Algorithm parallelism within the module
|
||||
- Module-level data parallelism
|
||||
- Cross-module pipeline parallelism
|
||||
|
||||
- **multi-computing backend**
|
||||
- Multiple types of computing clusters (self-managed clusters, HPC clusters, k8s container clusters, etc.)
|
||||
- Various container engines
|
||||
- docker: the default container engine
|
||||
- [singularity](./tests/hello-scalebox-singularity/)
|
||||
- k8s: TODO
|
||||
|
||||
This repository contains:
|
||||
|
||||
1. Scalebox server environment based on docker-compose ([Service Environment](./server/README.md))
|
||||
2. Dockerfile definition for scalebox standard modules([standard module](./dockerfiles/README.md))
|
||||
3. Application example of scalebox ([Application Example](./examples/README.md))
|
||||
4. Test of the main features of scalebox ([feature test](./tests/README.md))
|
||||
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Scalebox - A cloud-native streaming computing engine](#scalebox---a-cloud-native-streaming-computing-engine)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Background](#background)
|
||||
- [Install](#install)
|
||||
- [Usage](#usage)
|
||||
- [Examples](#examples)
|
||||
- [Feature tests](#feature-tests)
|
||||
- [Related Softwares](#related-softwares)
|
||||
- [Maintainers](#maintainers)
|
||||
- [Contributing](#contributing)
|
||||
- [License](#license)
|
||||
|
||||
## Background
|
||||
|
||||
## Install
|
||||
|
||||
## Usage
|
||||
|
||||
## Examples
|
||||
|
||||
## [Feature tests](./tests)
|
||||
|
||||
- [Fault tolerance Setup](tests/retry_test/)
|
||||
- [Timeout-setup](tests/timeout-gen/)
|
||||
- [Flow control management](tests/check_test/)
|
||||
- [Task-perspective](tests/task-perspective/)
|
||||
- [Cross-cluster-computing](tests/cross-cluster-primes/)
|
||||
- [singularity](tests/hello-scalebox-singularity/)
|
||||
|
||||
## Related Softwares
|
||||
|
||||
- [PostgreSQL Database Management System](https://github.com/postgres/postgres) — Scalebox backend database
|
||||
- [gRPC – An RPC library and framework](https://github.com/grpc/grpc) — Efficient communication protocol between different software modules
|
||||
- [The Go Programming Language](https://github.com/golang/go) — Programming language for cloud-native applications
|
||||
- [Pony ORM ER Diagram Editor](https://editor.ponyorm.com/) - Magical ER Diagram Tool
|
||||
|
||||
|
||||
## Maintainers
|
||||
|
||||
[@Kaichao](https://github.com/kaichao).
|
||||
|
||||
## Contributing
|
||||
|
||||
Feel free to dive in! [Open an issue](https://github.com/kaichao/docker-scalebox/issues/new) or submit Pull Requests.
|
||||
|
||||
## License
|
||||
|
||||
[Apache](LICENSE) © Kaichao Wu
|
||||
26
README.md
26
README.md
|
|
@ -2,12 +2,17 @@
|
|||
|
||||
Scalebox是一种云原生的流式计算引擎，可在分布式、异构计算集群上运行容器化的单机用户算法，以流水线组织模块层级上大规模并行处理，支持任务级容错。与已有大数据处理、并行计算等框架相比，其技术特点特别适用于数据分布、算力资源分布、算法复杂等应用场景。
|
||||
|
||||
scalebox具有以下重要特性：
|
||||
- **云原生设计**：所有算法模块、传输模块等都采用容器化封装，以边车模式嵌入到面向云环境设计的数据处理流水线；软件基础平台也完全基于云原生设计实现；平台将控制通道、数据通道分离，前后模块间通过消息总线关联，实现非侵入式并行编程，大大简化计算并行化的难度。
|
||||
- [**跨集群计算**](./tests/cross-cluster-primes/)：算法模块、传输模块的的归一化处理，通过流水线统一处理集群内/跨集群的数据，屏蔽数据和计算的跨集群差别；跨集群消息驱动流式处理，支持单流水线应用在多个异构算力集群上的部署。
|
||||
- [**任务级容错**](./tests/retry_test/)：对于硬件故障、软件bug、网络问题、数据异常等原因导致的偶发性出错，支持基于规则的自动容错处理；细粒度的任务级容错，可在不可靠硬件上实现可信的数据分析。
|
||||
- **位置感知调度**：消息体内可包含发送端IP地址，使得前后模块间可实现本地级联处理；将处理层、存储层分离，降低耦合，水平方向（处理层）的消息驱动垂直方向（处理层到存储层）的数据读写，降低计算集群内东西向网络流量，消除集群存储I/O瓶颈；进而实现无共享存储的本地计算、大文件的纯本地加载，有效支持横向扩展。
|
||||
- [**任务透视**](./tests/task-perspective/)：计算任务为消息驱动的处理过程，任务透视详细纪录每个任务的运行状态；包括用户程序返回码、标准输出、标准错误、程序定义返回文本、用户程序读写字节数等，还包括任务执行周期内（消息产生/分发/处理、结果纪录）计算容器端、控制端的各类系统级及用户定制的时间戳。任务透视支持问题排查的精准定位，并为应用优化、数据统计提供基础支撑。
|
||||
Scalebox具有以下重要特性：
|
||||
- **云原生设计**：以容器化封装所有算法模块、传输模块，通过边车模式嵌入到面向云环境设计的数据处理流水线；软件基础平台完全基于云原生设计实现；平台将控制消息、数据通道分离，前后模块间以消息总线关联，实现多语言的非侵入式并行编程，大大简化计算并行化的难度。
|
||||
|
||||
- [**跨集群计算**](./tests/cross-cluster-primes/)：归一化处理算法模块、传输模块，通过流水线统一处理集群内/跨集群的数据，屏蔽数据和计算的跨集群差异；跨集群消息驱动流式处理，支持单流水线应用在多个异构算力集群上的部署。
|
||||
|
||||
- [**任务级容错**](./tests/retry_test/)：对于硬件故障、软件bug、网络问题、数据异常等原因导致的偶发性出错，基于规则实现自动容错处理；细粒度的任务级容错，可在不可靠硬件上实现可信的数据分析。
|
||||
|
||||
- **位置感知调度**：消息体内可配置发送端IP地址，支持前后模块间的本地级联处理；将处理层、存储层分离，降低耦合，水平方向（处理层）的消息驱动垂直方向（处理层到存储层）的数据读写，降低计算集群内东西向网络流量，消除集群存储的I/O瓶颈；进而实现无共享存储的本地计算、大文件的纯本地加载，有效支持横向扩展。
|
||||
|
||||
- [**任务透视**](./tests/task-perspective/)：计算任务为消息驱动的处理过程，任务透视详细纪录每个任务的运行详细状态；包括用户程序返回码、标准输出、标准错误、程序自定义文本、用户程序读写字节数等，还包括任务执行周期内（消息产生/分发/处理、结果纪录）计算容器端、控制端的各类系统级及用户定制的时间戳。任务透视为问题排查中精准定位、应用优化、数据统计提供基础支撑。
|
||||
|
||||
- **多并行化方式**
|
||||
- 模块内算法并行
|
||||
- 模块级数据并行
|
||||
|
|
@ -16,9 +21,9 @@ scalebox具有以下重要特性：
|
|||
- **多计算后端**
|
||||
- 多种计算集群类型（自管理集群、HPC集群、k8s容器集群等）
|
||||
- 多种容器引擎
|
||||
- docker
|
||||
- docker：缺省容器引擎
|
||||
- [singularity](./tests/hello-scalebox-singularity/)
|
||||
- k8s
|
||||
- k8s：TODO
|
||||
|
||||
本仓库主要包含以下内容：
|
||||
|
||||
|
|
@ -100,7 +105,7 @@ macos主要用于单节点集群的开发测试。
|
|||
|
||||
## [应用示例](examples/)
|
||||
|
||||
scalebox应用的初级示例，包括：
|
||||
scalebox初级应用的示例，包括：
|
||||
- [hello-scalebox](examples/hello-scalebox/)：scalebox的第一个入门应用
|
||||
- [app-primes](examples/app-primes/)：计算区间内质数总数量
|
||||
- [app-copy](examples/app-copy/)：演示最常用的跨集群数据拷贝
|
||||
|
|
@ -112,7 +117,8 @@ scalebox应用的初级示例，包括：
|
|||
- 超时设置：[timeout-gen](tests/timeout-gen/)
|
||||
- 流控管理：[check_test](tests/check_test/)
|
||||
- 任务透视：[task-perspective](tests/task-perspective/)
|
||||
|
||||
- 跨集群计算：[cross-cluster-primes](tests/cross-cluster-primes/)
|
||||
- singularity：[singularity](tests/hello-scalebox-singularity/)
|
||||
|
||||
## 相关软件
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,6 @@
|
|||
FROM hub.cstcloud.cn/scalebox/actuator
|
||||
|
||||
RUN \
|
||||
echo "auto keygen in actuator ..." ; \
|
||||
rm -f /root/.ssh/id_rsa ; \
|
||||
ssh-keygen -q -N "" -f /root/.ssh/id_rsa ;
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
IMAGE_NAME:=my-actuator
|
||||
|
||||
build:
|
||||
docker build --network=host -t $(IMAGE_NAME) .
|
||||
|
||||
clean:
|
||||
docker rmi $(IMAGE_NAME)
|
||||
|
|
@ -16,7 +16,7 @@
|
|||
}
|
||||
},
|
||||
"sourceIP": {
|
||||
"description": "ip_addr for source job",
|
||||
"description": "ip_addr for source task",
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,11 +6,11 @@
|
|||
"required": [ "statusCode" ],
|
||||
"properties": {
|
||||
"statusCode": {
|
||||
"description": "Formatted Name",
|
||||
"type": "string"
|
||||
"description": "Exit status code for user code completion",
|
||||
"type": "integer"
|
||||
},
|
||||
"inputBytes": {
|
||||
"type": "string"
|
||||
"type": "integer"
|
||||
},
|
||||
"inputFiles": {
|
||||
"type": "array",
|
||||
|
|
@ -19,7 +19,7 @@
|
|||
}
|
||||
},
|
||||
"outputBytes": {
|
||||
"type": "string"
|
||||
"type": "integer"
|
||||
},
|
||||
"outputFiles": {
|
||||
"type": "array",
|
||||
|
|
|
|||
|
|
@ -95,3 +95,13 @@ pull-all:
|
|||
echo $$i; \
|
||||
docker pull hub.cstcloud.cn/scalebox/$$i; \
|
||||
done
|
||||
|
||||
auto-keygen-actuator:
|
||||
@ make -C ../dockerfiles/actuator build
|
||||
@ docker tag my-actuator hub.cstcloud.cn/scalebox/actuator
|
||||
@ docker rmi my-actuator
|
||||
|
||||
update-pubkey: auto-keygen-actuator
|
||||
@ cid=`docker create hub.cstcloud.cn/scalebox/actuator` \
|
||||
&& docker cp $$cid:/root/.ssh/id_rsa.pub ${PWD} \
|
||||
&& docker rm -v $$cid
|
||||
|
|
|
|||
Loading…
Reference in New Issue