forked from beimingwu/beimingwu
perf(docs): polish contents in docs and README
This commit is contained in:
parent
c448e3d910
commit
94ff7484e2
|
|
@ -60,9 +60,9 @@ In addition, the Beimingwu system also has the following features:
|
|||
|
||||
- **Learnware Specification Generation**: The Beimingwu system provides specification generation interfaces in the `learnware` Python package, supporting various data types (tables, images, and text) for efficient local generation.
|
||||
- **Learnware Quality Inspection**: The Beimingwu system includes multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system also supports the search for heterogeneous table learnwares.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system preliminarily supports the search for heterogeneous table learnwares.
|
||||
- **Local Learnware Deployment**: The Beimingwu system provides a unified user interface for learnware deployment and reuse in the `learnware` Python package, facilitating users' convenient and secure deployment and reuse of arbitrary learnwares.
|
||||
- **Not Disclosing Original Data**: The Beimingwu system operations, including learnware submission, identification, and deployment, do not require users to upload raw data. All relevant statistical specifications are generated locally by users using API.
|
||||
- **Raw Data Protection**: The Beimingwu system operations, including learnware submission, identification, and deployment, do not require users to upload raw data. All relevant statistical specifications are generated locally by users using open-source API.
|
||||
- **Open Source System**: The Beimingwu system's source code is open-source, including the `learnware` Python package and frontend/backend code. The `learnware` package is highly extensible, making it easy to integrate new specification designs, learnware system designs, and learnware reuse methods in the future.
|
||||
|
||||
## How is Beimingwu organized?
|
||||
|
|
|
|||
|
|
@ -60,7 +60,7 @@
|
|||
|
||||
- **学件规约生成**:北冥坞系统在 `learnware` Python 包中提供规约生成接口,支持多种数据类型(表格、图像和文本),可以在本地高效生成。
|
||||
- **学件质量检测**:北冥坞系统内置了多重检测机制,以确保系统中每个学件的质量。
|
||||
- **学件多样查搜**:北冥坞系统同时支持语义规约和统计规约的查搜,覆盖的数据类型包括表格、图像、文本。另外,对于表格型任务,系统额外支持异构表格学件的查搜。
|
||||
- **学件多样查搜**:北冥坞系统同时支持语义规约和统计规约的查搜,覆盖的数据类型包括表格、图像、文本。另外,对于表格型任务,系统初步支持异构表格学件的查搜。
|
||||
- **学件本地部署**:北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口,帮助用户以统一的方式便携地部署与复用学件。
|
||||
- **保护原始数据**:北冥坞系统所涉及的学件上传、查搜、部署均无需用户泄露原始数据,生成统计规约的过程在用户本地进行且代码公开。
|
||||
- **面向社区开源**:北冥坞系统面向社区开源,包括 `learnware` Python 包与前后端代码。其中 `leanrware` 包高度可扩展,未来新的规约设计、学件系统设计、学件查搜和复用方法都能轻松集成进来。
|
||||
|
|
@ -119,7 +119,7 @@ spec.save("stat.json")
|
|||
|
||||

|
||||
|
||||
某些情况下,组合多个学件的统计信息可能更加贴合您的任务需求,系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
某些情况下,组合多个学件的统计规约可能更加贴合您的任务需求,系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
|
||||

|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
### Q1: How does the Beimingwu system protect user data privacy?
|
||||
|
||||
A1: In the Beimingwu system, the uploading, searching, and deployment of learnwares do not require users to upload local data. The statistical specifications are generated locally by the user using API and do not compromise the raw data's security.
|
||||
A1: In the Beimingwu system, the uploading, searching, and deployment of learnwares do not require users to upload local data. The statistical specifications are generated locally by the user using open-source API and do not compromise the raw data's security.
|
||||
|
||||
|
||||
### Q2: How does the Beimingwu system ensure the safety of deploying learnware?
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ features:
|
|||
- title: Local Learnware Deployment
|
||||
details: The Beimingwu system allows for effortless and safe deployment and reuse of arbitrary learnwares in a unified way.
|
||||
icon: { src: "/icons/local-learnware-deployment.svg" }
|
||||
- title: Not Disclosing Original Data
|
||||
- title: Raw Data Protection
|
||||
details: The learnware submission, identification, and deployment of the Beimingwu system doesn't require users to upload raw data.
|
||||
icon: { src: "/icons/data-privacy-protection.svg" }
|
||||
- title: Open Source System
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ Beimingwu is the first systematic open-source implementation of learnware dock s
|
|||
|
||||
## What is Learnware?
|
||||
|
||||
A learnware consists of high-performance machine learning models and specifications that characterize the models, i.e., "Learnware = Model + Specification."
|
||||
A learnware consists of high-performing machine learning models and specifications that characterize the models, i.e., "Learnware = Model + Specification."
|
||||
|
||||
The learnware specification consists of "semantic specification" and "statistical specification":
|
||||
- semantic specification describes the type and functionality of the model through text.
|
||||
|
|
@ -19,22 +19,22 @@ Learnware specifications describe the model's capabilities, enabling the model t
|
|||
|
||||
## Why Do We Need the Learnware Dock System?
|
||||
|
||||
Machine learning has achieved great success in many fields but still faces various challenges, such as the need for extensive training data and advanced training techniques, the difficulty of continuous learning, the risk of catastrophic forgetting, and the leakage of data privacy.
|
||||
Machine learning has achieved great success in many fields but still faces various challenges, such as the need for extensive training data and advanced training techniques, the difficulty of continuous learning, the risk of catastrophic forgetting, and the risk of data privacy breach.
|
||||
|
||||
Although many efforts focus on one of these issues separately, these efforts pay less attention to the fact that most issues are entangled in practice, and solving one may exacerbate others.
|
||||
|
||||
The learnware dock system aims to address many of these challenges through a unified framework:
|
||||
- **Lack of Training Data/Skills**: Even for ordinary users with limited data and machine learning knowledge, they can obtain powerful machine learning models from the learnware dock system. Users can acquire high-performance learnwares and further customize or improve them without starting from scratch.
|
||||
- **Continuous Learning**: As high-performance learnware is continually submitted for various tasks, the knowledge in the learnware dock system will naturally accumulate, achieving continuous and lifelong learning.
|
||||
- **Catastrophic Forgetting**: Once learnware is accepted, it will always be retained in the learnware dock system unless it can be replaced by other learnware in all aspects. Therefore, old knowledge in the learnware dock system is always preserved and never forgotten.
|
||||
- **Data Privacy/Ownership**: Developers only submit models without sharing private data, thus protecting data privacy and ownership. Although the possibility of reverse engineering the model cannot be completely eliminated, the risk of privacy leakage in the learnware dock system is minimal compared to many other privacy protection schemes.
|
||||
- **Lack of Training Data/Skills**: Even for ordinary users with limited data and machine learning knowledge, they can obtain powerful machine learning models from the learnware dock system. Users can acquire high-performing learnwares and further customize or improve them without starting from scratch.
|
||||
- **Continuous Learning**: As high-performing learnwares are continually submitted for various tasks, the learnware doc system will continuously expand its knowledge base and improve its capabilities, naturally realizing lifelong learning.
|
||||
- **Catastrophic Forgetting**: Once a learnware passes the quality checks, it will always be stored in the learnware dock system. Therefore, old knowledge in the learnware dock system is always preserved and never forgotten.
|
||||
- **Data Privacy/Ownership**: Developers freely submit their high-performing models to the learnware doc system, without revealing their training data. The learnware paradigm enables developers to share their models that can be adequately identified and reused by future users without disclosing developer's original data.
|
||||
|
||||
## How Does the Learnware Dock System Work?
|
||||
|
||||
The learnware dock system is the core entity in the Learnware paradigm. In the learnware paradigm, there are three essential entities:
|
||||
- **Developers**: Typically machine learning experts who produce and wish to share/sell their high-performance machine learning models.
|
||||
- **Developers**: Typically machine learning experts who produce and wish to share/sell their high-performing machine learning models.
|
||||
- **Users**: In need of machine learning services but often have limited data and lack machine learning knowledge and skills.
|
||||
- **Learnware Dock System**: Receives high-performance machine learning models from developers, incorporates them into the system, and provides services to users by identifying and reusing learnware to help users solve current tasks.
|
||||
- **Learnware Dock System**: Receives high-performing machine learning models from developers, incorporates them into the system, and provides services to users by identifying and reusing learnware to help users solve current tasks.
|
||||
|
||||
As shown in the diagram below, the system workflow consists of two stages:
|
||||
|
||||
|
|
@ -80,9 +80,9 @@ In addition, the Beimingwu system also has the following features:
|
|||
|
||||
- **Learnware Specification Generation**: The Beimingwu system provides specification generation interfaces in the `learnware` Python package, supporting various data types (tables, images, and text) for efficient local generation.
|
||||
- **Learnware Quality Inspection**: The Beimingwu system includes multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system also supports the search for heterogeneous table learnwares.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system preliminarily supports the search for heterogeneous table learnwares.
|
||||
- **Local Learnware Deployment**: The Beimingwu system provides a unified user interface for learnware deployment and reuse in the `learnware` Python package, facilitating users' convenient and secure deployment and reuse of arbitrary learnwares.
|
||||
- **Not Disclosing Original Data**: The Beimingwu system operations, including learnware submission, identification, and deployment, do not require users to upload raw data. All relevant statistical specifications are generated locally by users using API.
|
||||
- **Raw Data Protection**: The Beimingwu system operations, including learnware submission, identification, and deployment, do not require users to upload raw data. All relevant statistical specifications are generated locally by users using open-source API.
|
||||
- **Open Source System**: The Beimingwu system's source code is open-source, including the `learnware` Python package and frontend/backend code. The `learnware` package is highly extensible, making it easy to integrate new specification designs, learnware system designs, and learnware reuse methods in the future.
|
||||
|
||||
Building the learnware paradigm requires collective efforts from the community. As the first learnware dock system, Beimingwu is still in its early stages, with much room for improvement in related technologies. We sincerely invite the community to upload models, collaborate in system development, and engage in research and enhancements in learnware algorithms. Your valuable feedback is essential for the continuous improvement of the system.
|
||||
|
|
@ -69,11 +69,11 @@ predict_y = learnware.predict(test_x)
|
|||
|
||||
## Homogeneous Learnware Reuse Methods
|
||||
|
||||
In addition to using learnwares directly, users can further make predictions on unlabeled data using learnware reuse methods provided by the system.
|
||||
In addition to using learnwares directly, users can further make predictions on unlabeled data using basic learnware reuse methods provided by the system.
|
||||
|
||||
There are two main categories of reuse methods: (1) direct reuse and (2) reuse based on a small amount of labeled data.
|
||||
There are two main categories of reuse methods: (1) data-free reusers which reuse learnwares directly and (2) data-dependent reusers which reuse learnwares with a small amount of labeled data.
|
||||
|
||||
### Direct Reuse of Learnware
|
||||
### Data-Free Reusers
|
||||
|
||||
Two methods for direct reuse of learnwares are provided: `JobSelectorReuser` and `AveragingReuser`.
|
||||
|
||||
|
|
@ -109,9 +109,9 @@ ensemble_predict_y = reuse_ensemble.predict(user_data=test_x)
|
|||
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Data-Free Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#direct-reuse-of-learnware).
|
||||
|
||||
### Reusing Learnware with Labeled Data
|
||||
### Data-Dependent Reusers
|
||||
|
||||
When users have a small amount of labeled data, the system provides two methods: `EnsemblePruningReuser` and `FeatureAugmentReuser` to help reuse learnwares.
|
||||
When users have a small amount of labeled data, the system provides two methods: `EnsemblePruningReuser` and `FeatureAugmentReuser` to help adapt the learnwares.
|
||||
|
||||
- `EnsemblePruningReuser` selects a subset of suitable learnwares using a multi-objective evolutionary algorithm and uses an average ensemble for prediction:
|
||||
|
||||
|
|
@ -129,9 +129,7 @@ reuse_ensemble_pruning.fit(val_X=val_X, val_y=val_y)
|
|||
predict_y = reuse_job_selector.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
- `FeatureAugmentReuser` helps users reuse learnwares by augmenting features. The output of the original learnware is concatenated with the user's task features, and a simple model is trained on the labeled data (logistic regression for classification tasks and ridge regression for regression tasks):
|
||||
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Data-Dependent Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#reuse-learnware-with-labeled-data).
|
||||
- `FeatureAugmentReuser` enhances user task features by incorporating predictions from learnwares, subsequently training a simple model (logistic regression for classification tasks and ridge regression for regression tasks):
|
||||
|
||||
```python
|
||||
from learnware.reuse import FeatureAugmentReuser
|
||||
|
|
@ -147,6 +145,8 @@ augment_reuser.fit(val_X, val_y)
|
|||
predict_y = augment_reuser.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Data-Dependent Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#reuse-learnware-with-labeled-data).
|
||||
|
||||
## Heterogeneous Learnware Reuse Methods
|
||||
|
||||
The system provides the `HeteroMapAlignLearnware` class to help align heterogeneous learnware with the user's task, including two steps: input space alignment and output space alignment.
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@ semantic_spec = generate_semantic_spec(
|
|||
task_type="Classification",
|
||||
library_type="Scikit-learn",
|
||||
scenarios=["Business", "Financial"],
|
||||
license=["Apache-2.0"],
|
||||
input_description=input_description,
|
||||
output_description=output_description,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ The system is designed to help users efficiently solve machine learning tasks wi
|
|||
2. **Learnware Quality Detection**: The Beimingwu system has multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
3. **Diverse Learnware Search**: The Beimingwu system supports both semantic specification and statistical specification search, covering data types including tables, images, and text. Additionally, for tabular tasks, the system preliminarily supports the search of heterogeneous table learnwares.
|
||||
4. **Local Learnware Deployment**: The Beimingwu system provides a unified user interface for learnware deployment and reuse in the `learnware` Python package, helping users to deploy and reuse arbitrary learnwares conveniently and securely.
|
||||
5. **Not Disclosing Original Data**: The learnware submission, identification, and deployment of the Beimingwu system doesn't require users to upload raw data, and all involved statistical specifications are generated locally by users using API.
|
||||
5. **Raw Data Protection**: The learnware submission, identification, and deployment of the Beimingwu system doesn't require users to upload raw data, and all involved statistical specifications are generated locally by users using open-source API.
|
||||
6. **Open Source System**: The source code of the Beimingwu system, including the `learnware` Python package and frontend and backend codes, are open source.
|
||||
|
||||
### v1.0 R&D Team
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ features:
|
|||
details: 北冥坞系统内置了多重检测机制,以确保系统中每个学件的质量符合要求。
|
||||
icon: { src: "/icons/learnware-quality-detector.svg" }
|
||||
- title: 学件多样查搜
|
||||
details: 北冥坞系统同时支持语义规约和统计规约的查搜,支持多种数据类型,表格数据额外支持异构查搜。
|
||||
details: 北冥坞系统同时支持语义规约和统计规约的查搜,支持多种数据类型,表格数据初步支持异构查搜。
|
||||
icon: { src: "/icons/diverse-learnware-search.svg" }
|
||||
- title: 学件本地部署
|
||||
details: 北冥坞系统提供学件部署与复用工具,帮助用户以统一方式便捷地部署与复用学件。
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ spec.save("stat.json")
|
|||

|
||||
|
||||
|
||||
某些情况下,组合多个学件的统计信息可能更加贴合您的任务需求,系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
某些情况下,组合多个学件的统计规约可能更加贴合您的任务需求,系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
|
||||

|
||||
|
||||
|
|
|
|||
|
|
@ -80,7 +80,7 @@
|
|||
|
||||
- **学件规约生成**:北冥坞系统在 `learnware` Python 包中提供规约生成接口,支持多种数据类型(表格、图像和文本),可以在本地高效生成。
|
||||
- **学件质量检测**:北冥坞系统内置了多重检测机制,以确保系统中每个学件的质量。
|
||||
- **学件多样查搜**:北冥坞系统同时支持语义规约和统计规约的查搜,覆盖的数据类型包括表格、图像、文本。另外,对于表格型任务,系统额外支持异构表格学件的查搜。
|
||||
- **学件多样查搜**:北冥坞系统同时支持语义规约和统计规约的查搜,覆盖的数据类型包括表格、图像、文本。另外,对于表格型任务,系统初步支持异构表格学件的查搜。
|
||||
- **学件本地部署**:北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口,帮助用户以统一的方式便携地部署与复用学件。
|
||||
- **保护原始数据**:北冥坞系统所涉及的学件上传、查搜、部署均无需用户泄露原始数据,生成统计规约的过程在用户本地进行且代码公开。
|
||||
- **面向社区开源**:北冥坞系统面向社区开源,包括 `learnware` Python 包与前后端代码。其中 `leanrware` 包高度可扩展,未来新的规约设计、学件系统设计、学件查搜和复用方法都能轻松集成进来。
|
||||
|
|
|
|||
|
|
@ -126,7 +126,7 @@ reuse_ensemble_pruning.fit(val_X=val_X, val_y=val_y)
|
|||
predict_y = reuse_job_selector.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
- `FeatureAugmentReuser` 通过特征增广的方式帮助用户复用学件,原始学件的输出会拼接到用户任务的特征上,并基于有标记数据训练一个简单的模型(分类任务为logistics regression,回归任务为ridge):
|
||||
- `FeatureAugmentReuser` 通过特征增广的方式帮助用户复用学件,原始学件的输出会拼接到用户任务的特征上,并使用增广后的有标记数据训练一个简单的模型(分类任务为logistics regression,回归任务为ridge):
|
||||
|
||||
```python
|
||||
from learnware.reuse import FeatureAugmentReuser
|
||||
|
|
|
|||
|
|
@ -48,6 +48,7 @@ semantic_spec = generate_semantic_spec(
|
|||
task_type="Classification",
|
||||
library_type="Scikit-learn",
|
||||
scenarios=["Business", "Financial"],
|
||||
license=["Apache-2.0"],
|
||||
input_description=input_description,
|
||||
output_description=output_description,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -8,8 +8,8 @@
|
|||
|
||||
1. **学件规约生成**:北冥坞系统在 `learnware` Python 包中提供规约生成接口,支持多种数据类型(表格、图像和文本),可以在本地高效生成。
|
||||
2. **学件质量检测**:北冥坞系统内置了多重检测机制,以确保系统中每个学件的质量。
|
||||
3. **学件多样查搜**:北冥坞系统同时支持语义规约和统计规约的查搜,覆盖的数据类型包括表格、图像、文本。另外,对于表格型任务,系统额外支持异构表格学件的查搜。
|
||||
4. **学件本地部署**:北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口,帮助用户便捷、安全的部署与复用学件。
|
||||
3. **学件多样查搜**:北冥坞系统同时支持语义规约和统计规约的查搜,覆盖的数据类型包括表格、图像、文本。另外,对于表格型任务,系统初步支持异构表格学件的查搜。
|
||||
4. **学件本地部署**:北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口,帮助用户以统一的方式便携地部署与复用学件。
|
||||
5. **保护原始数据**:北冥坞系统所涉及的学件上传、查搜、部署均无需用户上传本地数据,生成统计规约的过程在用户本地进行且代码公开。
|
||||
6. **面向社区开源**:北冥坞系统面向社区开源,包括 `learnware` Python 包与前后端代码。
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue