forked from beimingwu/beimingwu
Compare commits
1 Commits
| Author | SHA1 | Date |
|---|---|---|
 Gene
|
bf155b6c79 |
47
README.md
47
README.md
|
|
@ -49,8 +49,8 @@ A learnware is a well-performed trained model with a specification that describe
|
|||
|
||||
As shown in the diagram below, the Beimingwu learnware dock system, serving as a preliminary research platform for learnware, systematically implements the core processes of the learnware paradigm for the first time:
|
||||
|
||||
- **Submitting Stage**: The system includes multiple detection mechanisms to ensure the quality of uploaded learnwares. Additionally, the system trains a heterogeneous engine based on existing learnware specifications in the system to merge different specification islands and assign new specifications to learnwares. With the submission of more learnwares, the heterogeneous engine will continually update, aiming to construct a more precise specification world through the constant iteration of learnware specifications.
|
||||
- **Deploying Stage**: After users upload task requirements, the system automatically selects whether to recommend a single learnware or multiple learnware combinations and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers baseline learnware reuse methods in a uniform format for convenient usage.
|
||||
- **Submitting Stage**: The system includes multiple detection mechanisms to ensure the quality of uploaded learnwares. Additionally, the system trains a heterogeneous engine based on existing learnware specifications in the system to merge different specification islands and assign new specifications to learnwares. With more learnwares are submitted, the heterogeneous engine will continue to update, achieving continuous iteration of learnware specifications and building a more precise specification world.
|
||||
- **Deploying Stage**: After users upload task requirements, the system automatically selects whether to recommend a single learnware or multiple learnware combinations and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers convenient learnware reuse tools.
|
||||
|
||||
<div align=center>
|
||||
<img src="docs/content/public/overview/learnware-workflow-en.svg" width="700" height="auto" style="max-width: 100%;"/>
|
||||
|
|
@ -60,9 +60,9 @@ In addition, the Beimingwu system also has the following features:
|
|||
|
||||
- **Learnware Specification Generation**: The Beimingwu system provides specification generation interfaces in the `learnware` Python package, supporting various data types (tables, images, and text) for efficient local generation.
|
||||
- **Learnware Quality Inspection**: The Beimingwu system includes multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system preliminarily supports the search for heterogeneous table learnwares.
|
||||
- **Local Learnware Deployment**: The Beimingwu system provides a unified user interface for learnware deployment and reuse in the `learnware` Python package, facilitating users' convenient deployment and reuse of arbitrary learnwares.
|
||||
- **Raw Data Protection**: The Beimingwu system operations, including learnware submission, identification, and deployment, do not require users to upload raw data. All relevant statistical specifications are generated locally by users using open-source API.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system also supports the search for heterogeneous table learnwares.
|
||||
- **Local Learnware Deployment**: The Beimingwu system provides interfaces for learnware deployment and learnware reuse in the `learnware` Python package, facilitating users' convenient and secure learnware deployment.
|
||||
- **Data Privacy Protection**: The Beimingwu system operations, including learnware upload, search, and deployment, do not require users to upload local data. All relevant statistical specifications are generated locally by users, ensuring data privacy.
|
||||
- **Open Source System**: The Beimingwu system's source code is open-source, including the `learnware` Python package and frontend/backend code. The `learnware` package is highly extensible, making it easy to integrate new specification designs, learnware system designs, and learnware reuse methods in the future.
|
||||
|
||||
## How is Beimingwu organized?
|
||||
|
|
@ -84,15 +84,15 @@ The functionalities of the four layers are described as follows:
|
|||
### Project Structure
|
||||
|
||||
Based on the system architecture, Beimingwu is developed with five sub-projects:
|
||||
- [`Engine`](https://github.com/Learnware-LAMDA/Learnware): Encompassing core components and algorithms within the learnware paradigm, and providing a command-line client for user interaction, it has been releasead as the [learnware](https://pypi.org/project/learnware/) package.
|
||||
- [`Frontend`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/frontend): Provide the interface and functionality for user interaction with the learnware dock system, including the main system and administrator system.
|
||||
- [`Backend`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/backend): Responsible for handling the dock system's operation logic and data operations, it ensures system stability and high performance.
|
||||
- [`Docs`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/docs): Maintain system documentation, including user guides, development guides, etc., ensuring system usability.
|
||||
- [`Deploy`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/deploy): Manage the system deployment configuration, including frontend and backend deployment files.
|
||||
- [`Engine`](https://www.gitlink.org.cn/beimingwu/learnware): Encompassing core components and algorithms within the learnware paradigm, and providing a command-line client for user interaction, it has been releasead as the [learnware](https://pypi.org/project/learnware/) package.
|
||||
- [`Frontend`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/frontend): Provide the interface and functionality for user interaction with the learnware dock system, including the main system and administrator system.
|
||||
- [`Backend`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/backend): Responsible for handling the dock system's operation logic and data operations, it ensures system stability and high performance.
|
||||
- [`Docs`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/docs): Maintain system documentation, including user guides, development guides, etc., ensuring system usability.
|
||||
- [`Deploy`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/deploy): Manage the system deployment configuration, including frontend and backend deployment files.
|
||||
|
||||
# Quick Start
|
||||
|
||||
Welcome to experience [Beimingwu](https://bmwu.cloud/). The following instructions will assist you in quickly exploring the search functionality on the system website and provide two demo cases from learnware search to learnware deployment using the [learnware](https://github.com/Learnware-LAMDA/Learnware) package.
|
||||
Welcome to experience [Beimingwu](https://bmwu.cloud/). The following instructions will assist you in quickly exploring the search functionality on the system website and provide two demo cases from learnware search to learnware deployment using the [learnware](https://www.gitlink.org.cn/beimingwu/learnware) package.
|
||||
|
||||
The installation instructions for the learnware package can be found here: [Installation Guide](docs/content/en/overview/installation.md).
|
||||
|
||||
|
|
@ -104,7 +104,7 @@ When searching with semantic information, you can fill in the information about
|
|||
|
||||
|
||||
|
||||
When searching with statistical information, you need to generate and submit a statistical specification, which captures the data distribution while not disclosing your original data. Using the API we provided, you can easily generate this statistical specification locally.
|
||||
When searching with statistical information, you need to submit the statistical information of the task. The tools we provide will generate approximate statistical information for your task locally while protecting your data privacy. With the following code, you can easily generate approximate statistical information for your task.
|
||||
|
||||
```python
|
||||
from learnware.specification import generate_stat_spec
|
||||
|
|
@ -118,7 +118,7 @@ By uploading the JSON file containing statistical information, the system will m
|
|||
|
||||
|
||||
|
||||
In some cases, assembling multiple helpful learnwares may be more beneficial for your task. The system will accordingly recommend a combination of these learnwares as a package. You can download the package using the "Download All" button in the upper right corner.
|
||||
In some cases, combining multiple learnwares with similar statistical information may be more suitable for your task. The system will recommend these learnwares as a package to you. You can download all of them with the "Download All" button in the upper right corner.
|
||||
|
||||
|
||||
|
||||
|
|
@ -196,24 +196,19 @@ print(f"Classification accuracy: {accuracy_score(target, y_pred)}")
|
|||
If you use our project in your research or work, we kindly request that you cite the following papers:
|
||||
|
||||
```bibtex
|
||||
@article{zhou2024learnware,
|
||||
title = {Learnware: Small models do big},
|
||||
@article{zhou2022learnware,
|
||||
author = {Zhou, Zhi-Hua and Tan, Zhi-Hao},
|
||||
journal = {Science China Information Sciences},
|
||||
title = {Learnware: Small Models Do Big},
|
||||
journal = {SCIENCE CHINA Information Sciences},
|
||||
year = {2024},
|
||||
volume = {67},
|
||||
number = {1},
|
||||
pages = {112102},
|
||||
year = {2024}
|
||||
}
|
||||
|
||||
@article{tan2024beimingwu,
|
||||
title = {Beimingwu: A learnware dock system},
|
||||
author = {Tan, Zhi-Hao and Liu, Jian-Dong and Bi, Xiao-Dong and Tan, Peng and Zheng, Qin-Cheng and Liu, Hai-Tian and Xie, Yi and Zou, Xiao-Chuan and Yu, Yang and Zhou, Zhi-Hua},
|
||||
journal = {arXiv preprint arXiv:2401.14427},
|
||||
year = {2024}
|
||||
pages = {1--12},
|
||||
}
|
||||
```
|
||||
|
||||
Please acknowledge the use of our project by citing these papers in your work. Thank you for your support!
|
||||
|
||||
# About
|
||||
|
||||
## How to Contribute
|
||||
|
|
@ -222,4 +217,4 @@ Building the learnware paradigm requires collective efforts from the community.
|
|||
|
||||
## About Us
|
||||
|
||||
The Beimingwu repository is developed and maintained by the LAMDA Beimingwu R&D (Research and Development) Team. To learn more about our team, please visit the [Team Overview](https://docs.bmwu.cloud/en/about-us.html).
|
||||
The Beimingwu repository is developed and maintained by the LAMDA Beimingwu R&D Team. To learn more about our team, please visit the [Team Overview](https://docs.bmwu.cloud/en/about-us.html).
|
||||
60
README_zh.md
60
README_zh.md
|
|
@ -34,21 +34,14 @@
|
|||
</h3>
|
||||
</div>
|
||||
|
||||
| 会议名 | 截稿时间 | 录用通知时间 | 开会时间 | 官网| 等级 |
|
||||
|:------------:|:------------:|:------------:|:--------:|:------------:|:------------: |
|
||||
|DASFAA | 2022-11-04 | 2022-01-09 | 2023-04-17 | http://www.tjudb.cn/dasfaa2023/ | CCF B |
|
||||
| | |
|
||||
| ------------ | ------------ |
|
||||
| | |
|
||||
| | |
|
||||
|
||||
|
||||
# 简介
|
||||
|
||||
学件由周志华教授在 2016 年提出 [1, 2]。在学件范式下，世界各地的开发者可分享模型至学件基座系统，系统通过有效查搜和复用学件帮助用户高效解决机器学习任务，而无需从零开始构建机器学习模型。
|
||||
|
||||
北冥坞是学件的第一个系统性开源实现，为学件相关研究提供了一个初步科研平台。有分享意愿的开发者可自由提交模型，学件坞协助产生规约形成学件存放在学件坞中，开发者在这个过程中无需向学件坞泄露自己的训练数据。未来的用户可以通过向学件坞提交需求，在学件坞协助下查搜复用学件来完成自己的机器学习任务，且用户可以不向学件坞泄露自有数据。预计在学件坞拥有数以百万计的学件后，将可能出现“涌现”行为：以往没有专门开发过模型的机器学习任务，可能通过复用若干个现有学件而解决。
|
||||
|
||||
sssssss
|
||||
|
||||
学件由性能优良的机器学习模型和描述模型的规约组成。规约刻画了模型的能力，使得模型在未来能够根据用户需求被充分识别和复用。规约由两部分构成：语义规约通过文本描述模型的功能，而统计规约刻画模型所蕴含的统计信息。
|
||||
|
||||
[1] Zhi-Hua Zhou. Learnware: on the future of machine learning. Frontiers of Computer Science, 2016, 10(4): 589–590 <br/>
|
||||
|
|
@ -56,9 +49,9 @@
|
|||
|
||||
## 北冥坞系统有哪些特性？
|
||||
|
||||
如下图所示，北冥坞学件基座系统作为学件的初步科研平台，首次实现了学件范式中的核心流程：
|
||||
如下图所示，北冥坞学件基座系统作为学件的初步科研平台，首次系统性地实现了学件范式中的核心流程：
|
||||
|
||||
- **提交阶段**：系统内置了多重检测机制，以确保上传学件的质量。另外，系统会根据已有的学件规约，训练一个异构引擎，用于合并不同的规约岛屿，以及为学件赋予新规约。随着更多学件的上传，系统的异构引擎将持续更新，目标是通过系统内学件规约的持续迭代，构建更加精准的规约世界。
|
||||
- **提交阶段**：系统内置了多重检测机制，以确保上传学件的质量。另外，系统会根据已有的学件规约，训练一个异构引擎，用于合并不同的规约岛屿，以及为学件赋予新规约。随着更多学件的上传，异构引擎将持续更新，实现学件规约的持续迭代，构建更精准的规约世界。
|
||||
- **部署阶段**：用户上传任务需求后，系统会自动选择是推荐单学件还是多学件组合，并提供高效的部署方式。无论是单个学件还是多学件组合，系统均提供了便捷的学件复用接口。
|
||||
|
||||
<div align=center>
|
||||
|
|
@ -69,10 +62,10 @@
|
|||
|
||||
- **学件规约生成**：北冥坞系统在 `learnware` Python 包中提供规约生成接口，支持多种数据类型（表格、图像和文本），可以在本地高效生成。
|
||||
- **学件质量检测**：北冥坞系统内置了多重检测机制，以确保系统中每个学件的质量。
|
||||
- **学件多样查搜**：北冥坞系统同时支持语义规约和统计规约的查搜，覆盖的数据类型包括表格、图像、文本。另外，对于表格型任务，系统额外支持初步的异构表格学件的查搜。
|
||||
- **学件本地部署**：北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口，帮助用户以统一的方式便携地部署与复用学件。
|
||||
- **保护原始数据**：北冥坞系统所涉及的学件上传、查搜、部署均无需用户泄露原始数据，生成统计规约的过程在用户本地进行且代码公开。
|
||||
- **面向社区开源**：北冥坞系统面向社区开源，包括 `learnware` Python 包与前后端代码。其中 `leanrware` 包高度可扩展，未来新的规约设计、学件系统设计、学件查搜和复用方法都能轻松集成进来。
|
||||
- **学件多样查搜**：北冥坞系统同时支持语义规约和统计规约的查搜，覆盖的数据类型包括表格、图像、文本。另外，对于表格型任务，系统额外支持异构表格学件的查搜。
|
||||
- **学件本地部署**：北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口，帮助用户便捷、安全的部署与复用学件。
|
||||
- **数据隐私保护**：北冥坞系统所涉及的学件上传、查搜、部署均无需用户上传本地数据，所有涉及的统计规约均由用户本地生成，确保用户数据隐私。
|
||||
- **面向社区开源**：北冥坞系统面向社区开源，包括 `learnware` Python 包与前后端代码。其中 `leanrware` 包高度可扩展，未来新的规约设计、学件系统设计、学件复用方法都能轻松集成进来。
|
||||
|
||||
## 北冥坞系统是如何组织的？
|
||||
|
||||
|
|
@ -94,15 +87,15 @@
|
|||
### 项目结构
|
||||
|
||||
基于上述系统架构，北冥坞项目一共包含如下五个子项目：
|
||||
- [`系统引擎`](https://github.com/Learnware-LAMDA/Learnware)：实现了学件范式中的核心组件和算法，并提供了一个基于命令行的客户端以便于用户交互，同时将其作为 [learnware](https://pypi.org/project/learnware/) 包发布。
|
||||
- [`系统前端`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/frontend)：提供了用户与系统交互的界面和功能，包括主系统和管理员系统。
|
||||
- [`系统后端`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/backend)：负责处理系统的运行逻辑和数据操作，确保系统的稳定性和高性能。
|
||||
- [`系统文档`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/docs)：维护系统的文档，包括用户指南、开发指南等，确保系统的易用性。
|
||||
- [`系统部署`](https://github.com/Learnware-LAMDA/Beimingwu/tree/main/deploy)：负责管理系统的部署配置，包括前后端的部署文件。
|
||||
- [`系统引擎`](https://www.gitlink.org.cn/beimingwu/learnware)：实现了学件范式中的核心组件和算法，并提供了一个基于命令行的客户端以便于用户交互，同时将其作为 [learnware](https://pypi.org/project/learnware/) 包发布。
|
||||
- [`系统前端`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/frontend)：提供了用户与系统交互的界面和功能，包括主系统和管理员系统。
|
||||
- [`系统后端`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/backend)：负责处理系统的运行逻辑和数据操作，确保系统的稳定性和高性能。
|
||||
- [`系统文档`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/docs)：维护系统的文档，包括用户指南、开发指南等，确保系统的易用性。
|
||||
- [`系统部署`](https://www.gitlink.org.cn/beimingwu/beimingwu/tree/main/deploy)：负责管理系统的部署配置，包括前后端的部署文件。
|
||||
|
||||
# 快速上手
|
||||
|
||||
欢迎体验[北冥坞系统](https://bmwu.cloud/)！下述内容将帮你快速探索系统网站上的学件查搜功能，并基于 [learnware](https://github.com/Learnware-LAMDA/Learnware) 包提供两个从学件查搜至学件部署的应用案例。
|
||||
欢迎体验[北冥坞系统](https://bmwu.cloud/)！下述内容将帮你快速探索系统网站上的学件查搜功能，并基于 [learnware](https://www.gitlink.org.cn/beimingwu/learnware) 包提供两个从学件查搜至学件部署的应用案例。
|
||||
|
||||
learnware 包的安装可参考：[环境安装](docs/content/zh-CN/overview/installation.md)。
|
||||
|
||||
|
|
@ -114,7 +107,7 @@ learnware 包的安装可参考：[环境安装](docs/content/zh-CN/overview/ins
|
|||
|
||||
|
||||
|
||||
通过统计信息进行查搜时，您需要提交任务的统计规约。统计规约可以在保护您的原始数据不被泄露的情况下提供任务的统计信息。您可以使用系统提供的工具，通过下列代码在本地生成统计规约。
|
||||
通过统计信息进行查搜时，你需要提交任务的统计信息。我们提供的工具将在保护数据隐私的情况下在本地为你生成任务的近似统计信息。通过下列代码，你可以轻松生成任务的近似统计信息。
|
||||
|
||||
```python
|
||||
from learnware.specification import generate_stat_spec
|
||||
|
|
@ -128,7 +121,7 @@ spec.save("stat.json")
|
|||
|
||||
|
||||
|
||||
某些情况下，组合多个学件的统计规约可能更加贴合您的任务需求，系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
某些情况下，多个学件组合在一起的统计信息会更加接近你的任务，系统会将这些学件打包推荐给你。你可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
|
||||
|
||||
|
||||
|
|
@ -203,24 +196,17 @@ print(f"Classification accuracy: {accuracy_score(target, y_pred)}")
|
|||
|
||||
# 引用
|
||||
|
||||
如果您在研究或工作中使用了我们的项目，请引用下述论文，感谢你的支持！
|
||||
如果你在研究或工作中使用了我们的项目，请引用下述论文，感谢你的支持！
|
||||
|
||||
```bibtex
|
||||
@article{zhou2024learnware,
|
||||
title = {Learnware: Small models do big},
|
||||
@article{zhou2022learnware,
|
||||
author = {Zhou, Zhi-Hua and Tan, Zhi-Hao},
|
||||
journal = {Science China Information Sciences},
|
||||
title = {Learnware: Small Models Do Big},
|
||||
journal = {SCIENCE CHINA Information Sciences},
|
||||
year = {2024},
|
||||
volume = {67},
|
||||
number = {1},
|
||||
pages = {112102},
|
||||
year = {2024}
|
||||
}
|
||||
|
||||
@article{tan2024beimingwu,
|
||||
title = {Beimingwu: A learnware dock system},
|
||||
author = {Tan, Zhi-Hao and Liu, Jian-Dong and Bi, Xiao-Dong and Tan, Peng and Zheng, Qin-Cheng and Liu, Hai-Tian and Xie, Yi and Zou, Xiao-Chuan and Yu, Yang and Zhou, Zhi-Hua},
|
||||
journal = {arXiv preprint arXiv:2401.14427},
|
||||
year = {2024}
|
||||
pages = {1--12},
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -232,4 +218,4 @@ print(f"Classification accuracy: {accuracy_score(target, y_pred)}")
|
|||
|
||||
## 关于我们
|
||||
|
||||
北冥坞仓库由 LAMDA 北冥坞研发团队开发和维护，更多信息可参考：[团队简介](https://docs.bmwu.cloud/zh-CN/about-us.html)。
|
||||
北冥坞仓库由 LAMDA 北冥坞研发团队开发和维护，更多信息可参考：[团队简介](https://docs.bmwu.cloud/zh-CN/about-us.html)。
|
||||
|
|
@ -118,5 +118,4 @@ _DEFAULT_CONFIG = {
|
|||
"datasets_path": os.path.join(DATA_PATH, "datasets"),
|
||||
"env_path": os.path.join(DATA_PATH, "envs"),
|
||||
"learnware_checker_type": "conda",
|
||||
"log_target": "console",
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ import concurrent_log_handler
|
|||
database: Database = None
|
||||
engine = None
|
||||
engine_config = None
|
||||
redis_pool = None
|
||||
redis_client = None
|
||||
stats = 0
|
||||
sensitive_pattern = None
|
||||
|
||||
|
|
@ -85,22 +85,15 @@ def init_backend():
|
|||
|
||||
|
||||
def init_redis():
|
||||
global config, redis_pool
|
||||
redis_pool = redis.ConnectionPool(
|
||||
host=config.redis["host"], port=config.redis["port"], decode_responses=True, max_connections=256
|
||||
)
|
||||
global config, redis_client
|
||||
|
||||
redis_client = redis.Redis(host=config.redis["host"], port=config.redis["port"], decode_responses=True)
|
||||
pass
|
||||
|
||||
|
||||
def get_redis_client():
|
||||
global redis_pool
|
||||
return redis.Redis(connection_pool=redis_pool, decode_responses=True)
|
||||
|
||||
|
||||
def init_logger():
|
||||
def init_logger(target="console"):
|
||||
global config, logger
|
||||
|
||||
target = config["log_target"]
|
||||
if len(logger.handlers) > 0:
|
||||
pass
|
||||
elif target == "console":
|
||||
|
|
|
|||
|
|
@ -1,48 +1,34 @@
|
|||
import context
|
||||
import time
|
||||
import redis
|
||||
|
||||
|
||||
def subscribe():
|
||||
while True:
|
||||
try:
|
||||
redis_client = redis.Redis(
|
||||
host=context.config["redis"]["host"], port=context.config["redis"]["port"], decode_responses=True
|
||||
)
|
||||
channel = redis_client.pubsub()
|
||||
channel.subscribe("learnware")
|
||||
channel = context.redis_client.pubsub()
|
||||
channel.subscribe("learnware")
|
||||
|
||||
for message in channel.listen():
|
||||
if message["type"] == "message":
|
||||
message_data = message["data"]
|
||||
if message_data.startswith("reload"):
|
||||
learnware_id = message_data.split(" ")[1]
|
||||
context.logger.info(f"sub Reload learnware {learnware_id}")
|
||||
context.engine.reload_learnware(learnware_id)
|
||||
pass
|
||||
elif message_data.startswith("delete"):
|
||||
learnware_id = message_data.split(" ")[1]
|
||||
context.logger.info(f"sub Delete learnware {learnware_id}")
|
||||
context.engine.delete_learnware(learnware_id)
|
||||
pass
|
||||
for message in channel.listen():
|
||||
if message["type"] == "message":
|
||||
message_data = message["data"]
|
||||
if message_data.startswith("reload"):
|
||||
learnware_id = message_data.split(" ")[1]
|
||||
context.logger.info(f"sub Reload learnware {learnware_id}")
|
||||
context.engine.reload_learnware(learnware_id)
|
||||
pass
|
||||
except Exception as e:
|
||||
elif message_data.startswith("delete"):
|
||||
learnware_id = message_data.split(" ")[1]
|
||||
context.logger.info(f"sub Delete learnware {learnware_id}")
|
||||
context.engine.delete_learnware(learnware_id)
|
||||
pass
|
||||
context.logger.info("redis subscribe connection is broken, wait 10 seconds to reconnect")
|
||||
time.sleep(10)
|
||||
pass
|
||||
pass
|
||||
|
||||
|
||||
def publish_reload_learnware(learnware_id):
|
||||
redis_client = context.get_redis_client()
|
||||
redis_client.publish("learnware", f"reload {learnware_id}")
|
||||
context.redis_client.publish("learnware", f"reload {learnware_id}")
|
||||
pass
|
||||
|
||||
|
||||
def publish_delete_learnware(learnware_id):
|
||||
redis_client = context.get_redis_client()
|
||||
redis_client.publish("learnware", f"delete {learnware_id}")
|
||||
context.redis_client.publish("learnware", f"delete {learnware_id}")
|
||||
pass
|
||||
|
||||
|
||||
|
|
@ -53,15 +39,13 @@ def add_learnware_download_token(learnware_ids, token):
|
|||
|
||||
key = f"learnware:download_token:{token}"
|
||||
value = "|".join(learnware_ids)
|
||||
|
||||
redis_client = context.get_redis_client()
|
||||
redis_client.set(key, value, ex=60 * 30)
|
||||
context.redis_client.set(key, value, ex=60 * 30)
|
||||
pass
|
||||
|
||||
|
||||
def get_learnware_id_from_download_token(token):
|
||||
key = f"learnware:download_token:{token}"
|
||||
learnware_id = context.get_redis_client().get(key)
|
||||
learnware_id = context.redis_client.get(key)
|
||||
if learnware_id is None:
|
||||
return None
|
||||
return learnware_id.split("|")
|
||||
|
|
@ -69,15 +53,13 @@ def get_learnware_id_from_download_token(token):
|
|||
|
||||
def delete_learnware_download_token(token):
|
||||
key = f"learnware:download_token:{token}"
|
||||
redis_client = context.get_redis_client()
|
||||
redis_client.delete(key)
|
||||
context.redis_client.delete(key)
|
||||
pass
|
||||
|
||||
|
||||
def set_key(key, value, expire=60 * 60):
|
||||
key = f"learnware:{key}"
|
||||
redis_client = context.get_redis_client()
|
||||
redis_client.set(
|
||||
context.redis_client.set(
|
||||
key,
|
||||
value,
|
||||
ex=expire,
|
||||
|
|
@ -87,8 +69,7 @@ def set_key(key, value, expire=60 * 60):
|
|||
|
||||
def get_key(key, default=None):
|
||||
key = f"learnware:{key}"
|
||||
redis_client = context.get_redis_client()
|
||||
value = redis_client.get(key)
|
||||
value = context.redis_client.get(key)
|
||||
if value is None:
|
||||
return default
|
||||
return value
|
||||
|
|
|
|||
|
|
@ -79,7 +79,7 @@ class RegisterApi(flask_restful.Resource):
|
|||
email, secret_key=context.config["app_secret_key"]
|
||||
)
|
||||
# send email
|
||||
utils.send_verification_email(email, username, verification_code, email_config=context.config["email"])
|
||||
utils.send_verification_email(email, verification_code, email_config=context.config["email"])
|
||||
pass
|
||||
else:
|
||||
database.update_email_confirm_time(email=email)
|
||||
|
|
@ -104,12 +104,8 @@ class ResendEmailConfirmApi(flask_restful.Resource):
|
|||
return {"code": 21, "msg": "Request parameters error."}, 200
|
||||
|
||||
email = body["email"]
|
||||
user_info = database.get_user_info(by="email", value=email)
|
||||
if user_info is None:
|
||||
return {"code": 51, "msg": "Your email not exist. Please re-register"}, 200
|
||||
username = user_info["username"]
|
||||
verification_code = utils.generate_email_verification_code(email, secret_key=context.config["app_secret_key"])
|
||||
utils.send_verification_email(email, username, verification_code, email_config=context.config["email"])
|
||||
utils.send_verification_email(email, verification_code, email_config=context.config["email"])
|
||||
|
||||
result = {"code": 0, "msg": "success", "data": {}}
|
||||
|
||||
|
|
@ -155,11 +151,8 @@ class SendResetPasswordEmailApi(flask_restful.Resource):
|
|||
if user_info is None:
|
||||
return {"code": 51, "msg": "Your email not exist. Please register first"}, 200
|
||||
user_id = user_info["id"]
|
||||
username = user_info["username"]
|
||||
verification_code = utils.generate_email_verification_code(email, secret_key=context.config["app_secret_key"])
|
||||
utils.send_reset_password_email(
|
||||
email, username, verification_code, str(user_id), email_config=context.config["email"]
|
||||
)
|
||||
utils.send_reset_password_email(email, verification_code, str(user_id), email_config=context.config["email"])
|
||||
|
||||
result = {"code": 0, "msg": "success", "data": {}}
|
||||
|
||||
|
|
|
|||
|
|
@ -4,12 +4,10 @@ from learnware import learnware
|
|||
import lib.database_operations as dbops
|
||||
import itsdangerous
|
||||
import smtplib
|
||||
from email.message import EmailMessage
|
||||
import ssl
|
||||
import multiprocessing.dummy as mp
|
||||
import multiprocessing as mp
|
||||
import socks
|
||||
import socket
|
||||
import context as backend_context
|
||||
|
||||
|
||||
__all__ = ["get_parameters", "generate_random_str", "dump_learnware"]
|
||||
|
|
@ -71,30 +69,26 @@ def send_email_worker(sender_email, password, receiver_email, message, smtp_serv
|
|||
if len(smtp_server) == 0:
|
||||
return
|
||||
|
||||
try:
|
||||
if len(proxy_host) > 0 and proxy_port > 0:
|
||||
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, proxy_host, proxy_port)
|
||||
original_socket = socket.socket
|
||||
socket.socket = socks.socksocket
|
||||
pass
|
||||
if len(proxy_host) > 0 and proxy_port > 0:
|
||||
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, proxy_host, proxy_port)
|
||||
original_socket = socket.socket
|
||||
socket.socket = socks.socksocket
|
||||
pass
|
||||
|
||||
context = ssl.create_default_context()
|
||||
context = ssl.create_default_context()
|
||||
|
||||
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
|
||||
server.login(sender_email, password)
|
||||
# server.sendmail(sender_email, receiver_email, message)
|
||||
server.send_message(message)
|
||||
pass
|
||||
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
|
||||
server.login(sender_email, password)
|
||||
server.sendmail(sender_email, receiver_email, message)
|
||||
pass
|
||||
|
||||
if len(proxy_host) > 0 and proxy_port > 0:
|
||||
socket.socket = original_socket
|
||||
pass
|
||||
except Exception as e:
|
||||
backend_context.logger.exception(e)
|
||||
if len(proxy_host) > 0 and proxy_port > 0:
|
||||
socket.socket = original_socket
|
||||
pass
|
||||
pass
|
||||
|
||||
|
||||
def send_verification_email(email: str, username: str, verification_code: str, email_config: dict) -> bool:
|
||||
def send_verification_email(email: str, verification_code: str, email_config: dict) -> bool:
|
||||
port = email_config["smtp_port"]
|
||||
smtp_server = email_config["smtp_server"]
|
||||
sender_email = email_config["sender_email"]
|
||||
|
|
@ -104,110 +98,48 @@ def send_verification_email(email: str, username: str, verification_code: str, e
|
|||
proxy_host = email_config.get("proxy_host", "")
|
||||
proxy_port = email_config.get("proxy_port", 0)
|
||||
|
||||
message = EmailMessage()
|
||||
message["From"] = sender_email
|
||||
message["To"] = receiver_email
|
||||
message["Subject"] = "[bmwu]Please activate your account"
|
||||
message.set_content(
|
||||
f"""\
|
||||
<html>
|
||||
<body>
|
||||
<div style="margin:0;">
|
||||
Dear {username},
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
<br />
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
Welcome to Beimingwu! Thanks for signing up. Please follow this link to activate your account:
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
<br />
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
<div style="margin:0;">
|
||||
{ confirm_url }
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
Cheers!
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
Beimingwu Group
|
||||
</div>
|
||||
</div>
|
||||
<br />
|
||||
</body>
|
||||
</html>
|
||||
""",
|
||||
subtype="html",
|
||||
)
|
||||
message = f"""\
|
||||
From: {sender_email}\r\n\
|
||||
Subject: Please activate your account\r\n\
|
||||
\r\n
|
||||
Welcome! Thanks for signing up. Please follow this link to activate your account:
|
||||
|
||||
{ confirm_url }
|
||||
|
||||
Cheers!
|
||||
Beimingwu Group
|
||||
"""
|
||||
|
||||
thread = mp.Process(
|
||||
target=send_email_worker,
|
||||
args=(sender_email, password, receiver_email, message, smtp_server, port, proxy_host, proxy_port),
|
||||
)
|
||||
thread.start()
|
||||
|
||||
return thread
|
||||
# send_email_worker(sender_email, password, receiver_email, message, smtp_server, port, proxy_host, proxy_port)
|
||||
# return None
|
||||
|
||||
|
||||
def send_reset_password_email(
|
||||
email: str, username: str, verification_code: str, user_id: str, email_config: dict
|
||||
) -> bool:
|
||||
def send_reset_password_email(email: str, verification_code: str, user_id: str, email_config: dict) -> bool:
|
||||
port = email_config["smtp_port"]
|
||||
smtp_server = email_config["smtp_server"]
|
||||
sender_email = email_config["sender_email"]
|
||||
receiver_email = email
|
||||
password = email_config["smtp_password"]
|
||||
reset_url = email_config["reset_password_url"] + "?code=" + verification_code + "&user_id=" + user_id
|
||||
confirm_url = email_config["reset_password_url"] + "?code=" + verification_code + "&user_id=" + user_id
|
||||
proxy_host = email_config.get("proxy_host", "")
|
||||
proxy_port = email_config.get("proxy_port", 0)
|
||||
|
||||
message = EmailMessage()
|
||||
message["From"] = sender_email
|
||||
message["To"] = receiver_email
|
||||
message["Subject"] = "[bmwu]Reset your password"
|
||||
message.set_content(
|
||||
f"""\
|
||||
<html>
|
||||
<body>
|
||||
<div style="margin:0;">
|
||||
Dear {username},
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
<br />
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
Please follow this link to reset your password:
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
<br />
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
<div style="margin:0;">
|
||||
{ reset_url }
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
Cheers!
|
||||
</div>
|
||||
<div style="margin:0;">
|
||||
Beimingwu Group
|
||||
</div>
|
||||
</div>
|
||||
<br />
|
||||
</body>
|
||||
</html>
|
||||
""",
|
||||
subtype="html",
|
||||
)
|
||||
message = f"""\
|
||||
From: {sender_email}\r\n\
|
||||
Subject: Reset your password\r\n\
|
||||
\r\n
|
||||
Please follow this link to reset your password:
|
||||
|
||||
{ confirm_url }
|
||||
|
||||
Cheers!
|
||||
|
||||
Beimingwu Group
|
||||
"""
|
||||
|
||||
thread = mp.Process(
|
||||
target=send_email_worker,
|
||||
|
|
|
|||
|
|
@ -34,8 +34,6 @@ CORS(app)
|
|||
bcrypt = flask_bcrypt.Bcrypt(app)
|
||||
jwt = flask_jwt_extended.JWTManager(app)
|
||||
Markdown(app)
|
||||
# fix for gunicorn bug
|
||||
app.config["PROPAGATE_EXCEPTIONS"] = True
|
||||
|
||||
|
||||
@jwt.unauthorized_loader
|
||||
|
|
@ -64,7 +62,7 @@ def create_app():
|
|||
context.init_engine()
|
||||
|
||||
# Init logger
|
||||
context.init_logger()
|
||||
context.init_logger(target="file")
|
||||
|
||||
# Init redis
|
||||
context.init_redis()
|
||||
|
|
|
|||
|
|
@ -222,7 +222,7 @@ def worker_process_func(q: queue.Queue, env: dict):
|
|||
|
||||
def main(num_worker):
|
||||
context.init_database()
|
||||
context.init_logger()
|
||||
context.init_logger(target="file")
|
||||
context.init_engine()
|
||||
context.init_redis()
|
||||
context.init_sensitive_words()
|
||||
|
|
|
|||
|
|
@ -34,7 +34,6 @@ COPY ./Learnware /learnware/
|
|||
WORKDIR /learnware-backend
|
||||
COPY ./config.json* /learnware-backend/
|
||||
COPY ./swords.txt* /learnware-backend/swords.txt
|
||||
COPY ./models--bert-base-uncased* /root/.learnware/cache/
|
||||
|
||||
RUN python3 -m pip install -e /learnware/
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ mkdir -p Beimingwu/deploy/
|
|||
cp -r ../../backend Beimingwu/
|
||||
cp -r ../../frontend Beimingwu/
|
||||
cp -r ../static Beimingwu/deploy/
|
||||
git clone https://github.com/Learnware-LAMDA/Learnware.git
|
||||
git clone https://gitlink.org.cn/beimingwu/learnware.git
|
||||
```
|
||||
|
||||
## Build Image
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ http {
|
|||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass http://learnware-backend:8088/;
|
||||
proxy_pass http://backend:8088/;
|
||||
}
|
||||
|
||||
location /static {
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ http {
|
|||
}
|
||||
|
||||
location /api {
|
||||
proxy_pass http://learnware-backend:8088/;
|
||||
proxy_pass http://backend:8088/;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -108,10 +108,7 @@ const en: LocaleSpecificConfig & {
|
|||
|
||||
outline: { label: "Outline" },
|
||||
|
||||
socialLinks: [
|
||||
{ icon: "git", link: "https://www.gitlink.org.cn/beimingwu/beimingwu" },
|
||||
{ icon: "github", link: "https://github.com/learnware-lamda/Beimingwu" },
|
||||
],
|
||||
socialLinks: [{ icon: "git", link: "https://www.gitlink.org.cn/beimingwu/beimingwu" }],
|
||||
},
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -108,10 +108,7 @@ const zhCN: LocaleSpecificConfig & {
|
|||
|
||||
outline: { label: "大纲" },
|
||||
|
||||
socialLinks: [
|
||||
{ icon: "git", link: "https://www.gitlink.org.cn/beimingwu/beimingwu" },
|
||||
{ icon: "github", link: "https://github.com/learnware-lamda/Beimingwu" },
|
||||
],
|
||||
socialLinks: [{ icon: "git", link: "https://www.gitlink.org.cn/beimingwu/beimingwu" }],
|
||||
},
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
# About Us
|
||||
|
||||
The R&D (Research and Development) Team of Beimingwu is from [Nanjing University's LAMDA Research Institute](https://www.lamda.nju.edu.cn/MainPage.ashx), with Prof. [Zhi-Hua Zhou](http://cs.nju.edu.cn/zhouzh) serving as the Founding Director.
|
||||
The R&D Team of Beimingwu is from [Nanjing University's LAMDA Research Institute](https://www.lamda.nju.edu.cn/MainPage.ashx), with Prof. [Zhi-Hua Zhou](http://cs.nju.edu.cn/zhouzh) serving as the Founding Director.
|
||||
|
||||
LAMDA is affiliated with the National Key Laboratory for Novel Software Technology, the Department of Computer Science & Technology and the School of Artificial Intelligence, Nanjing University, China. It locates at Computer Science and Technology Building in the Xianlin campus of Nanjing University, mainly in Rm910.
|
||||
|
||||
"LAMDA" means "Learning And Mining from DatA". The main research interests of LAMDA include machine learning, data mining, pattern recognition, information retrieval, evolutionary computation, neural computation, and some other related areas. Currently our research mainly involves: ensemble learning, semi-supervised and active learning, multi-instance and multi-label learning, cost-sensitive and class-imbalance learning, metric learning, dimensionality reduction and feature selection, structure learning and clustering, theoretical foundations of evolutionary computation, improving comprehensibility, content-based image retrieval, web search and mining, face recognition, computer-aided medical diagnosis, bioinformatics, etc.
|
||||
|
||||
The Beimingwu system is currently maintained by the LAMDA Beimingwu R&D Team, consisting of: <a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;" target="_blank">Zhi-Hua Zhou</a>, <a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;" target="_blank">Yang Yu</a>, <a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;" target="_blank">Zhi-Hao Tan</a>, <a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;" target="_blank">Jian-Dong Liu</a>, <a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;" target="_blank">Peng Tan</a>, <a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;" target="_blank">Xiao-Dong Bi</a>, <a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;" target="_blank">Qin-Cheng Zheng</a>, Xiao-Chuan Zou, <a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;" target="_blank">Yi Xie</a>, Hai-Tian Liu, Hao-Yu Shi, Xin-Yu Zhang and others.
|
||||
The Beimingwu system is currently maintained by the LAMDA Beimingwu R&D Team, consisting of: <a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;">Zhi-Hua Zhou</a>, <a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;">Yang Yu</a>, <a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;">Zhi-Hao Tan</a>, <a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;">Jian-Dong Liu</a>, <a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;">Peng Tan</a>, <a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;">Xiao-Dong Bi</a>, <a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;">Qin-Cheng Zheng</a>, Xiao-Chuan Zou, <a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;">Yi Xie</a>, Hai-Tian Liu, Hao-Yu Shi, Xin-Yu Zhang and others.
|
||||
|
||||
### Contact
|
||||
|
||||
|
|
|
|||
|
|
@ -2,12 +2,12 @@
|
|||
|
||||
### Q1: How does the Beimingwu system protect user data privacy?
|
||||
|
||||
A1: In the Beimingwu system, the uploading, searching, and deployment of learnwares do not require users to upload local data. The statistical specifications are generated locally by the user using open-source API and do not compromise the raw data's security.
|
||||
A1: In the Beimingwu system, the uploading, searching, and deployment of learnwares do not require users to upload local data. The statistical specifications are generated locally by the user and do not expose original data, ensuring data privacy.
|
||||
|
||||
|
||||
### Q2: How does the Beimingwu system ensure the safety of deploying learnware?
|
||||
|
||||
A2: We make every effort to verify the safety of each learnware and provide interfaces for deploying learnware within a docker container.
|
||||
A2: We make every effort to verify the safety of each learnware and provide tools for deploying learnware within a docker container.
|
||||
|
||||
|
||||
### Q3: What types of data does the Beimingwu system support for learnware?
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/${R
|
|||
## Deploy Backend
|
||||
### Download Deployment Files
|
||||
```shell
|
||||
git clone https://github.com/Learnware-LAMDA/Beimingwu.git
|
||||
git clone https://gitlink.org.cn/beimingwu/beimingwu.git
|
||||
cd Beimingwu/deploy/kubernetes
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ Visit the [Docker official website](https://docs.docker.com/compose/install/#ins
|
|||
|
||||
## Deploy Using Docker Compose
|
||||
|
||||
Download the [backend code](https://github.com/Learnware-LAMDA/Beimingwu/tree/main) and navigate to the `deploy/docker_compose` directory.
|
||||
Download the [backend code](https://www.gitlink.org.cn/beimingwu/beimingwu) and navigate to the `deploy/docker_compose` directory.
|
||||
|
||||
## Prepare Files
|
||||
Prepare miniconda installation file
|
||||
|
|
@ -19,7 +19,7 @@ mkdir -p Beimingwu/deploy/
|
|||
cp -r ../../backend Beimingwu/
|
||||
cp -r ../../frontend Beimingwu/
|
||||
cp -r ../static Beimingwu/deploy/
|
||||
git clone https://github.com/Learnware-LAMDA/Learnware.git
|
||||
git clone https://gitlink.org.cn/beimingwu/learnware.git
|
||||
```
|
||||
|
||||
## Build Image
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Beimingwu learnware dock system is developed with five sub-projects:
|
|||
- **Docs**: Maintain system documentation, including user guides, development guides, etc., ensuring system usability.
|
||||
- **Deploy**: Manage the system deployment configuration, including frontend and backend deployment files.
|
||||
|
||||
The `Engine` is implemented in a separate `Learnware` [code repository](https://github.com/Learnware-LAMDA/Learnware) and is configured with its own [project documentation](https://learnware.readthedocs.io/en/latest/).
|
||||
The `Engine` is implemented in a separate `Learnware` [code repository](https://www.gitlink.org.cn/beimingwu/learnware) and is configured with its own [project documentation](https://learnware.readthedocs.io/en/latest/).
|
||||
|
||||
The remaining four sub-projects are implemented within the `Beimingwu` [code repository](https://github.com/Learnware-LAMDA/Beimingwu) and are managed using a `Monorepo` approach. The following sections will provide details on the specific structure and development guidelines for the `Beimingwu` code repository.
|
||||
The remaining four sub-projects are implemented within the `Beimingwu` [code repository](https://www.gitlink.org.cn/beimingwu/beimingwu) and are managed using a `Monorepo` approach. The following sections will provide details on the specific structure and development guidelines for the `Beimingwu` code repository.
|
||||
|
||||
## Beimingwu Project Structure
|
||||
|
||||
|
|
|
|||
|
|
@ -23,13 +23,13 @@ features:
|
|||
details: The Beimingwu system incorporates multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
icon: { src: "/icons/learnware-quality-detector.svg" }
|
||||
- title: Diverse Learnware Search
|
||||
details: The Beimingwu system facilitates semantic & statistical search across various data types and also preliminarily supports heterogeneous tabular learnware search.
|
||||
details: The Beimingwu system supports semantic & statistical search on various data types, enabling heterogeneous tabular learnware search.
|
||||
icon: { src: "/icons/diverse-learnware-search.svg" }
|
||||
- title: Local Learnware Deployment
|
||||
details: The Beimingwu system allows for effortless and safe deployment and reuse of arbitrary learnwares in a unified way.
|
||||
details: The Beimingwu system provides tools for learnware deployment, helping users to deploy and reuse learnwares conveniently and securely.
|
||||
icon: { src: "/icons/local-learnware-deployment.svg" }
|
||||
- title: Raw Data Protection
|
||||
details: The learnware submission, identification, and deployment of the Beimingwu system doesn't require users to upload raw data.
|
||||
- title: Data Privacy Protection
|
||||
details: The Beimingwu system ensures data privacy as the uploading, searching, and deployment of learnware doesn't require uploading local data.
|
||||
icon: { src: "/icons/data-privacy-protection.svg" }
|
||||
- title: Open Source System
|
||||
details: The Beimingwu system is open source and the learnware package is highly extensible and easy to integrate with new features.
|
||||
|
|
|
|||
|
|
@ -16,9 +16,9 @@ pip install learnware==0.3.2 -i https://pypi.tuna.tsinghua.edu.cn/simple
|
|||
|
||||
## Install from Source Code
|
||||
|
||||
The source code for the `learnware` package is concurrently released on both [GitLink](https://www.gitlink.org.cn/beimingwu/learnware) and [Github](https://github.com/Learnware-LAMDA/Learnware) platforms. Taking GitHub as an example, users can install it through the source code in the following manner:
|
||||
The source code of the `learnware` package is available on the [GitLink](https://www.gitlink.org.cn/beimingwu/learnware) platform, and users can download the source code for installation using the following code:
|
||||
```bash
|
||||
git clone https://github.com/Learnware-LAMDA/Learnware.git
|
||||
git clone https://gitlink.org.cn/beimingwu/learnware.git
|
||||
cd Learnware
|
||||
git fetch origin main
|
||||
git checkout main
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Quick Start
|
||||
|
||||
Welcome to Beimingwu learnware dock system! The following content will help you quickly experience the system, mainly including learnware search and deployment.
|
||||
Welcome to Beimingwu learnware dock system! The following content will help you quickly experience the system, including learnware search and deployment.
|
||||
|
||||
## Learnware Search
|
||||
|
||||
|
|
@ -10,7 +10,7 @@ When searching with semantic information, you can fill in the information about
|
|||
|
||||
|
||||
|
||||
When searching with statistical information, you need to generate and submit a statistical specification, which captures the data distribution while not disclosing your original data. Using the API we provided, you can easily generate this statistical specification locally.
|
||||
When searching with statistical information, you need to submit the statistical information of the task. The tools we provide will generate approximate statistical information for your task locally while protecting your data privacy. With the following code, you can easily generate approximate statistical information for your task.
|
||||
|
||||
```python
|
||||
from learnware.specification import generate_stat_spec
|
||||
|
|
@ -24,7 +24,7 @@ By uploading the JSON file containing statistical information, the system will m
|
|||
|
||||
|
||||
|
||||
In some cases, assembling multiple helpful learnwares may be more beneficial for your task. The system will accordingly recommend a combination of these learnwares as a package. You can download the package using the "Download All" button in the upper right corner.
|
||||
In some cases, combining multiple learnwares with similar statistical information may be more suitable for your task. The system will recommend these learnwares as a package to you. You can download all of them with the "Download All" button in the upper right corner.
|
||||
|
||||
|
||||
|
||||
|
|
@ -43,22 +43,4 @@ learnware = client.load_learnware(
|
|||
|
||||
# Use the learnware to make predictions on data
|
||||
pred_y = learnware.predict(test_x)
|
||||
```
|
||||
|
||||
<!--
|
||||
|
||||
## Demo Videos
|
||||
|
||||
The following videos provide a more comprehensive demonstration of how to use the Beiming Wu system to solve a machine learning task:
|
||||
|
||||
<div style="padding: 56.25% 0 0 0; position: relative"><div style="height:100%;left:0;position:absolute;top:0;width:100%"><iframe height="100%" width="100%;" src="https://embed.wave.video/S2zG1ZbUaRpEo8UG" frameborder="0" allow="autoplay; fullscreen" scrolling="no"></iframe></div></div>
|
||||
|
||||
For more information, you can refer to the "[Learnware Search](https://docs.beiming.cloud/en/user-guide/learnware-search.html)" and "[Learnware Deployment](https://docs.beiming.cloud/en/user-guide/learnware-deploy.html)" sections.
|
||||
|
||||
If you're interested in sharing your model with the Beiming Wu system, you can quickly get started with the following video:
|
||||
|
||||
<div style="padding: 56.25% 0 0 0; position: relative"><div style="height:100%;left:0;position:absolute;top:0;width:100%"><iframe height="100%" width="100%;" src="https://embed.wave.video/JcMSmDcgTJep5zPo" frameborder="0" allow="autoplay; fullscreen" scrolling="no"></iframe></div></div>
|
||||
|
||||
For more details, please refer to the "[Learnware Preparation](https://docs.beiming.cloud/en/user-guide/learnware-upload/prepare.html)," "[Uploading via Web Interface](https://docs.beiming.cloud/en/user-guide/learnware-upload/upload-web.html)," and "[Uploading via Client Interface](https://docs.beiming.cloud/en/user-guide/learnware-upload/upload-client.html)" sections.
|
||||
|
||||
-->
|
||||
```
|
||||
|
|
@ -9,7 +9,7 @@ Beimingwu is the first systematic open-source implementation of learnware dock s
|
|||
|
||||
## What is Learnware?
|
||||
|
||||
A learnware consists of high-performing machine learning models and specifications that characterize the models, i.e., "Learnware = Model + Specification."
|
||||
A learnware consists of high-performance machine learning models and specifications that characterize the models, i.e., "Learnware = Model + Specification."
|
||||
|
||||
The learnware specification consists of "semantic specification" and "statistical specification":
|
||||
- semantic specification describes the type and functionality of the model through text.
|
||||
|
|
@ -19,33 +19,33 @@ Learnware specifications describe the model's capabilities, enabling the model t
|
|||
|
||||
## Why Do We Need the Learnware Dock System?
|
||||
|
||||
Machine learning has achieved great success in many fields but still faces various challenges, such as the need for extensive training data and advanced training techniques, the difficulty of continuous learning, the risk of catastrophic forgetting, and the risk of data privacy breach.
|
||||
Machine learning has achieved great success in many fields but still faces various challenges, such as the need for extensive training data and advanced training techniques, the difficulty of continuous learning, the risk of catastrophic forgetting, and the leakage of data privacy.
|
||||
|
||||
Although many efforts focus on one of these issues separately, these efforts pay less attention to the fact that most issues are entangled in practice, and solving one may exacerbate others.
|
||||
Although there are many efforts focusing on one of these issues separately, they are entangled, and solving one problem may exacerbate others.
|
||||
|
||||
The learnware dock system aims to address many of these challenges through a unified framework:
|
||||
- **Lack of Training Data/Skills**: Even for ordinary users with limited data and machine learning knowledge, they can obtain powerful machine learning models from the learnware dock system. Users can acquire high-performing learnwares and further customize or improve them without starting from scratch.
|
||||
- **Continuous Learning**: As high-performing learnwares are continually submitted for various tasks, the learnware doc system will continuously expand its knowledge base and improve its capabilities, naturally realizing lifelong learning.
|
||||
- **Catastrophic Forgetting**: Once a learnware passes the quality checks, it will always be stored in the learnware dock system. Therefore, old knowledge in the learnware dock system is always preserved and never forgotten.
|
||||
- **Data Privacy/Ownership**: Developers freely submit their high-performing models to the learnware doc system, without revealing their training data. The learnware paradigm enables developers to share their models that can be adequately identified and reused by future users without disclosing developer's original data.
|
||||
- **Lack of Training Data/Skills**: Even for ordinary users with limited data and machine learning knowledge, they can obtain powerful machine learning models from the learnware dock system. Users can acquire high-performance learnwares and further customize or improve them without starting from scratch.
|
||||
- **Continuous Learning**: As high-performance learnware is continually submitted for various tasks, the knowledge in the learnware dock system will naturally accumulate, achieving continuous and lifelong learning.
|
||||
- **Catastrophic Forgetting**: Once learnware is accepted, it will always be retained in the learnware dock system unless it can be replaced by other learnware in all aspects. Therefore, old knowledge in the learnware dock system is always preserved and never forgotten.
|
||||
- **Data Privacy/Ownership**: Developers only submit models without sharing private data, thus protecting data privacy and ownership. Although the possibility of reverse engineering the model cannot be completely eliminated, the risk of privacy leakage in the learnware dock system is minimal compared to many other privacy protection schemes.
|
||||
|
||||
## How Does the Learnware Dock System Work?
|
||||
|
||||
The learnware dock system is the core entity in the Learnware paradigm. In the learnware paradigm, there are three essential entities:
|
||||
- **Developers**: Typically machine learning experts who produce and wish to share/sell their high-performing machine learning models.
|
||||
- **Developers**: Typically machine learning experts who produce and wish to share/sell their high-performance machine learning models.
|
||||
- **Users**: In need of machine learning services but often have limited data and lack machine learning knowledge and skills.
|
||||
- **Learnware Dock System**: Receives high-performing machine learning models from developers, incorporates them into the system, and provides services to users by identifying and reusing learnware to help users solve current tasks.
|
||||
- **Learnware Dock System**: Receives high-performance machine learning models from developers, incorporates them into the system, and provides services to users by identifying and reusing learnware to help users solve current tasks.
|
||||
|
||||
As shown in the diagram below, the system workflow consists of two stages:
|
||||
|
||||
- **Submitting Stage**: Developers voluntarily submit various learnwares to the learnware dock system, and the system will conduct quality checks and further organization of these learnwares.
|
||||
- **Deploying Stage**: The user submits her task requirement to the learnware doc system, and the system will identify and return some helpful learnwares to the user based on specifications, which can be further reused on user data.
|
||||
- **Submitting Stage**: Developers voluntarily submit various learnwares to the learnware dock system, and the system conducts quality checks and further organization of these learnwares.
|
||||
- **Deploying Stage**: When users submit task requirements, the learnware dock system automatically selects whether to recommend a single learnware or a combination of multiple learnwares and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers convenient learnware reuse interfaces.
|
||||
|
||||
|
||||
|
||||
### Specification World
|
||||
|
||||
Specification is the central component of the learnware dock system, linking all processes about learnwares, including uploading, organizing, searching, deploying, and reusing.
|
||||
Specification is the core component of the learnware dock system, linking all processes about learnwares, including uploading, organizing, searching, deploying, and reusing.
|
||||
|
||||
Learnwares from different feature/label spaces form numerous islands of specifications, and all these islands together constitute the "specification world" in the learnware dock system. In the specification world, if connections between different islands can be discovered and established, the corresponding islands of specification can be merged.
|
||||
|
||||
|
|
@ -53,7 +53,7 @@ Learnwares from different feature/label spaces form numerous islands of specific
|
|||
|
||||
When searching in the learnware dock system, the system first identifies specific islands of specifications based on semantic specifications in user requirements, and then it accurately recognizes learnwares on the specification islands through statistical specifications in user requirements. Merging different specification islands means that the corresponding learnwares can be used in tasks with different feature/label spaces, i.e., they can be reused in tasks beyond their original purposes.
|
||||
|
||||
The learnware paradigm leverages the community-shared machine learning models to create a unified specification space, efficiently addressing new user tasks in a standardized manner. As more learnwares are submitted and the algorithms advance, the doc system will continuously expand its knowledge base and improve its capability to tackle more varied user tasks.
|
||||
The learnware paradigm leverages the community-shared machine learning models to create a unified specification space, efficiently addressing new user tasks in a standardized manner. As more learnwares are submitted, effectively organizing learnwares will significantly enhance the task-solving abilities of the learnware dock system.
|
||||
|
||||
## How is Beimingwu System Organized?
|
||||
|
||||
|
|
@ -73,16 +73,16 @@ The functionalities of the four layers are described as follows:
|
|||
|
||||
The Beimingwu learnware dock system, serving as a preliminary research platform for learnware, systematically implements the core processes of the learnware paradigm for the first time:
|
||||
|
||||
- **Submitting Stage**: The system includes multiple detection mechanisms to ensure the quality of uploaded learnwares. Additionally, the system trains a heterogeneous engine based on existing learnware specifications in the system to merge different specification islands and assign new specifications to learnwares. With the submission of more learnwares, the heterogeneous engine will continually update, aiming to construct a more precise specification world through the constant iteration of learnware specifications.
|
||||
- **Deploying Stage**: After users upload task requirements, the system automatically selects whether to recommend a single learnware or multiple learnware combinations and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers baseline learnware reuse methods in a uniform format for convenient usage.
|
||||
- **Submitting Stage**: The system includes multiple detection mechanisms to ensure the quality of uploaded learnwares. Additionally, the system trains a heterogeneous engine based on existing learnware specifications in the system to merge different specification islands and assign new specifications to learnwares. With more learnwares are submitted, the heterogeneous engine will continue to update, achieving continuous iteration of learnware specifications and building a more precise specification world.
|
||||
- **Deploying Stage**: After users upload task requirements, the system automatically selects whether to recommend a single learnware or multiple learnware combinations and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers convenient learnware reuse tools.
|
||||
|
||||
In addition, the Beimingwu system also has the following features:
|
||||
|
||||
- **Learnware Specification Generation**: The Beimingwu system provides specification generation interfaces in the `learnware` Python package, supporting various data types (tables, images, and text) for efficient local generation.
|
||||
- **Learnware Quality Inspection**: The Beimingwu system includes multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system preliminarily supports the search for heterogeneous table learnwares.
|
||||
- **Local Learnware Deployment**: The Beimingwu system provides a unified user interface for learnware deployment and reuse in the `learnware` Python package, facilitating users' convenient deployment and reuse of arbitrary learnwares.
|
||||
- **Raw Data Protection**: The Beimingwu system operations, including learnware submission, identification, and deployment, do not require users to upload raw data. All relevant statistical specifications are generated locally by users using open-source API.
|
||||
- **Diverse Learnware Search**: The Beimingwu system supports both semantic specifications and statistical specifications searches, covering data types such as tables, images, and text. In addition, for table-based tasks, the system also supports the search for heterogeneous table learnwares.
|
||||
- **Local Learnware Deployment**: The Beimingwu system provides interfaces for learnware deployment and learnware reuse in the `learnware` Python package, facilitating users' convenient and secure learnware deployment.
|
||||
- **Data Privacy Protection**: The Beimingwu system operations, including learnware upload, search, and deployment, do not require users to upload local data. All relevant statistical specifications are generated locally by users, ensuring data privacy.
|
||||
- **Open Source System**: The Beimingwu system's source code is open-source, including the `learnware` Python package and frontend/backend code. The `learnware` package is highly extensible, making it easy to integrate new specification designs, learnware system designs, and learnware reuse methods in the future.
|
||||
|
||||
Building the learnware paradigm requires collective efforts from the community. As the first learnware dock system, Beimingwu is still in its early stages, with much room for improvement in related technologies. We sincerely invite the community to upload models, collaborate in system development, and engage in research and enhancements in learnware algorithms. Your valuable feedback is essential for the continuous improvement of the system.
|
||||
|
|
@ -22,7 +22,7 @@ learnware = client.load_learnware(
|
|||
)
|
||||
```
|
||||
|
||||
When the user wants to load multiple learnwares according to the id list `learnware_ids`, it can be accomplished with the following code:
|
||||
When the user wants to load multiple learnwares according to the id list `learnware_ids`, this can be accomplished with the following code:
|
||||
```python
|
||||
learnware_ids = ["00000082", "00000120"]
|
||||
learnware_list = client.load_learnware(
|
||||
|
|
@ -30,17 +30,17 @@ learnware_list = client.load_learnware(
|
|||
)
|
||||
```
|
||||
|
||||
The `runnable_option` parameter includes three options (default is `None`), each corresponding to a specific learnware deployment method:
|
||||
The `runnable_option` parameter has three options (default `None`), each for a specific learnware environment loading method:
|
||||
|
||||
- `None`: Load only learnware specifications and basic information; run the learnware using the current `learnware` package's Python environment.
|
||||
- `"conda"`: Install a separate `conda` virtual environment for each learnware (automatically deleted after execution); run each learnware independently within its virtual environment.
|
||||
- `"docker"`: Install a `conda` virtual environment inside a Docker container (automatically destroyed after execution); run each learnware independently within the container (requires Docker privileges).
|
||||
|
||||
It's important to note that while the system makes every effort to ensure the security of each learnware, the `None` and `"conda"` modes are **not secure** if there are any malicious learnwares. If the user cannot guarantee the security of the learnware they want to load, it's recommended to use the **relatively secure** `"docker"` mode to load the learnware.
|
||||
It's important to note that while the system makes every effort to ensure the security of each learnware, the `None` and `"conda"` modes are **not secure** if there are any malicious learnwares. If the user cannot guarantee the security of the learnware they want to load, it's recommended to use the `"docker"` mode to load the learnware.
|
||||
|
||||
### Loading Learnware from a ZIP File
|
||||
|
||||
In addition to loading learnware by ID, users can also load a learnware from a zip file downloaded from the web frontend:
|
||||
In addition to loading learnware by ID, users can also load a learnware from a zip file downloaded from the web:
|
||||
|
||||
```python
|
||||
learnware_path = "learnware.zip"
|
||||
|
|
@ -49,7 +49,7 @@ learnware = client.load_learnware(
|
|||
)
|
||||
```
|
||||
|
||||
When the user wants to load multiple learnwares according to the zip path list `learnware_paths`, it can be achieved with the following code:
|
||||
When the user wants to load multiple learnwares according to the zip path list `learnware_paths`, this can be achieved with the following code:
|
||||
```python
|
||||
learnware_paths = ["learnware1.zip", "learnware2.zip"]
|
||||
learnware_list = client.load_learnware(
|
||||
|
|
@ -69,15 +69,15 @@ predict_y = learnware.predict(test_x)
|
|||
|
||||
## Homogeneous Learnware Reuse Methods
|
||||
|
||||
In addition to using learnwares directly, users can further make predictions on unlabeled data using basic learnware reuse methods provided by the system.
|
||||
In addition to using learnwares directly, users can further make predictions on unlabeled data using learnware reuse methods provided by the system.
|
||||
|
||||
There are two main categories of reuse methods: (1) data-free reusers which reuse learnwares directly and (2) data-dependent reusers which reuse learnwares with a small amount of labeled data.
|
||||
There are two main categories of reuse methods: (1) direct reuse and (2) reuse based on a small amount of labeled data.
|
||||
|
||||
### Data-Free Reusers
|
||||
### Direct Reuse of Learnware
|
||||
|
||||
Two methods for direct reuse of learnwares are provided: `JobSelectorReuser` and `AveragingReuser`.
|
||||
Two methods for direct reuse of learnwares are provided: `JobSelector` and `Averaging`.
|
||||
|
||||
- `JobSelectorReuser` selects different learnwares for different data by training a classifier. Here's how to use it:
|
||||
- `JobSelector` selects different learnwares for different data by training a classifier. Here's how to use it:
|
||||
|
||||
```python
|
||||
from learnware.reuse import JobSelectorReuser
|
||||
|
|
@ -90,7 +90,7 @@ reuse_job_selector = JobSelectorReuser(learnware_list=learnware_list)
|
|||
predict_y = reuse_job_selector.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
- `AveragingReuser` uses an ensemble method to make predictions. The `mode` parameter specifies the specific ensemble method:
|
||||
- `Averaging` uses an ensemble method to make predictions. The `mode` parameter specifies the specific ensemble method:
|
||||
|
||||
```python
|
||||
from learnware.reuse import AveragingReuser
|
||||
|
|
@ -107,13 +107,13 @@ reuse_ensemble = AveragingReuser(
|
|||
ensemble_predict_y = reuse_ensemble.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Data-Free Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#direct-reuse-of-learnware).
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#direct-reuse-of-learnware).
|
||||
|
||||
### Data-Dependent Reusers
|
||||
### Reusing Learnware with Labeled Data
|
||||
|
||||
When users have a small amount of labeled data, the system provides two methods: `EnsemblePruningReuser` and `FeatureAugmentReuser` to help adapt the learnwares.
|
||||
When users have a small amount of labeled data, the system provides two methods: `EnsemblePruning` and `FeatureAugmentReuser` to help reuse learnwares.
|
||||
|
||||
- `EnsemblePruningReuser` selects a subset of suitable learnwares using a multi-objective evolutionary algorithm and uses an average ensemble for prediction:
|
||||
- `EnsemblePruning` selectively integrates learnwares to choose the ones that are most suitable for the user's task:
|
||||
|
||||
```python
|
||||
from learnware.reuse import EnsemblePruningReuser
|
||||
|
|
@ -129,7 +129,9 @@ reuse_ensemble_pruning.fit(val_X=val_X, val_y=val_y)
|
|||
predict_y = reuse_job_selector.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
- `FeatureAugmentReuser` enhances user task features by incorporating predictions from learnwares, subsequently training a simple model (logistic regression for classification tasks and ridge regression for regression tasks):
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#reuse-learnware-with-labeled-data).
|
||||
|
||||
- `FeatureAugmentReuser` helps users reuse learnwares by augmenting features. The output of the original learnware is concatenated with the user's task features, and a simple model is trained on the labeled data (logistic regression for classification tasks and ridge regression for regression tasks):
|
||||
|
||||
```python
|
||||
from learnware.reuse import FeatureAugmentReuser
|
||||
|
|
@ -145,8 +147,6 @@ augment_reuser.fit(val_X, val_y)
|
|||
predict_y = augment_reuser.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
For more detailed usage and explanations, please refer to the [Learnware Package Data-Dependent Reuse Methods Introduction](https://learnware.readthedocs.io/en/latest/components/learnware.html#reuse-learnware-with-labeled-data).
|
||||
|
||||
## Heterogeneous Learnware Reuse Methods
|
||||
|
||||
The system provides the `HeteroMapAlignLearnware` class to help align heterogeneous learnware with the user's task, including two steps: input space alignment and output space alignment.
|
||||
|
|
@ -165,7 +165,7 @@ hetero_learnware.align(user_spec, val_x, val_y)
|
|||
predict_y = hetero_learnware.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
If you want to reuse multiple heterogeneous learnwares, you can combine `HeteroMapAlignLearnware` with the homogeneous reuse methods `AveragingReuser` and `EnsemblePruningReuser` as mentioned before:
|
||||
If you want to reuse multiple heterogeneous learnwares, you can combine `HeteroMapAlignLearnware` with the homogeneous reuse methods `Averaging` and `EnsemblePruning` as mentioned before:
|
||||
|
||||
```python
|
||||
hetero_learnware_list = []
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# How to Search for Learnware(s)?
|
||||
# How to Search for Learnwares?
|
||||
|
||||
In the Beimingwu system, you can search for learnware(s) through the web interface or the `learnware` package.
|
||||
In the Beimingwu system, you can search for learnwares through the web interface or the `learnware` package.
|
||||
|
||||
Next, we will introduce these two methods separately.
|
||||
|
||||
|
|
@ -67,7 +67,7 @@ Where "email" is your registered email for the system, and "token" is the token
|
|||
|
||||
Similar to the web interface search, the `learnware` package supports semantic specification search, statistical specification search, and a combination of semantic and statistical specification search.
|
||||
|
||||
You can search for learnware(s) in the learnware dock system through semantic specifications, and all learnwares that meet the semantic specifications will be returned via the API. For example, the following code retrieves all learnware in the system with a task type of "Classification":
|
||||
You can search for learnwares in the learnware dock system through semantic specifications, and all learnwares that meet the semantic specifications will be returned via the API. For example, the following code retrieves all learnware in the system with a task type of "Classification":
|
||||
|
||||
```python
|
||||
from learnware.market import BaseUserInfo
|
||||
|
|
@ -96,14 +96,14 @@ search_result = {
|
|||
}
|
||||
```
|
||||
|
||||
Moreover, you can also search for learnware(s) in the learnware dock system through statistical specifications, and more targeted learnwares for your task will be returned through the API. Using the `generate_stat_spec` function mentioned above, you can generate your task's statistical specification `stat_spec`. Then, you can use the following code to easily obtain suitable learnware(s) identified by the system for your specific task:
|
||||
What's more, you can also search for learnwares in the learnware dock system through statistical specifications, and all learnwares with similar distribution will be returned through the API. By using the `generate_stat_spec` function mentioned above, you can easily obtain the statistical specification `stat_spec` corresponding to your current task. Then, you can use the following code to retrieve learnwares in the system that satisfies the statistical specification for the same type of data as your task:
|
||||
|
||||
```python
|
||||
user_info = BaseUserInfo(stat_info={stat_spec.type: stat_spec})
|
||||
search_result = client.search_learnware(user_info)
|
||||
```
|
||||
|
||||
By combining both semantic and statistical specifications, you can perform more accurate searches. For instance, the code below demonstrates how to search for learnware(s) in tabular data that satisfy both the semantic and statistical specifications:
|
||||
By combining both semantic and statistical specifications, you can perform more accurate searches. For example, the following code searches for learnware in tabular data that meet both semantic and statistical specifications:
|
||||
|
||||
```python
|
||||
user_semantic = generate_semantic_spec(
|
||||
|
|
@ -132,7 +132,7 @@ For more detailed deployment guidance, you can refer to the "[Learnware Deployme
|
|||
|
||||
### Heterogeneous Table Search
|
||||
|
||||
For tabular tasks, if the task type is "Classification" or "Regression", and you have provided a statistical specification along with descriptions for each feature dimension in the semantic specification, the system will enable heterogeneous table search. This is designed to support searching models from different feature spaces preliminarily. The following code example shows how to perform a heterogeneous table search via the API:
|
||||
When you provide a statistical specification for tabular data, the task type is "Classification" or "Regression," and your semantic specification includes descriptions for each dimension, the system will automatically enable heterogeneous table search. It won't only search in the tabular learnwares with same dimensions. The following code will perform heterogeneous table search through the API:
|
||||
|
||||
```python
|
||||
input_description = {
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ Where "email" is your registered email address in the system, and "token" is the
|
|||
|
||||
## Prepare Semantic Specification
|
||||
|
||||
After successfully logging in, your next step is to prepare the semantic specification. For instance, here's an example of "Table Data" in a "Classification Task":
|
||||
After successfully logging in, you need to prepare the semantic specification. Here is an example of a "Table Data" for a "Classification Task":
|
||||
|
||||
```python
|
||||
from learnware.specification import generate_semantic_spec
|
||||
|
|
@ -50,7 +50,6 @@ semantic_spec = generate_semantic_spec(
|
|||
task_type="Classification",
|
||||
library_type="Scikit-learn",
|
||||
scenarios=["Business", "Financial"],
|
||||
license=["Apache-2.0"],
|
||||
input_description=input_description,
|
||||
output_description=output_description,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ If there are many dimensions, consider using a large language model to analyze t
|
|||
|
||||
|
||||
|
||||
Next, you'll select the "Task type" for your learnware. If `Classification` is chosen, you need to provide the semantics of model output labels (prediction labels start from 0), making the uploaded learnware suitable for classification tasks with heterogeneous output spaces.
|
||||
Next is selecting the "Task type" involved in the learnware. If `Classification` is chosen, you need to provide the semantics of model output labels (prediction labels start from 0), making the uploaded learnware suitable for classification tasks with heterogeneous output spaces.
|
||||
|
||||
If `Regression` is chosen, you need to specify the semantics of each dimension of the model output, making the uploaded learnware suitable for regression tasks with heterogeneous output spaces.
|
||||
|
||||
|
|
|
|||
|
|
@ -9,20 +9,22 @@ The system is designed to help users efficiently solve machine learning tasks wi
|
|||
|
||||
1. **Learnware Specification Generation**: The Beimingwu system provides a specification generation tool in the `learnware` Python package, supporting multiple data types (tables, images, and text), and can be efficiently generated locally.
|
||||
2. **Learnware Quality Detection**: The Beimingwu system has multiple detection mechanisms to ensure the quality of each learnware in the system.
|
||||
3. **Diverse Learnware Search**: The Beimingwu system supports both semantic specification and statistical specification search, covering data types including tables, images, and text. Additionally, for tabular tasks, the system preliminarily supports the search of heterogeneous table learnwares.
|
||||
4. **Local Learnware Deployment**: The Beimingwu system provides a unified user interface for learnware deployment and reuse in the `learnware` Python package, helping users to deploy and reuse arbitrary learnwares conveniently.
|
||||
5. **Raw Data Protection**: The learnware submission, identification, and deployment of the Beimingwu system doesn't require users to upload raw data, and all involved statistical specifications are generated locally by users using open-source API.
|
||||
3. **Diverse Learnware Search**: The Beimingwu system supports both semantic specification and statistical specification search, covering data types including tables, images, and text. Additionally, for tabular tasks, the system also supports the search of heterogeneous table learnwares.
|
||||
4. **Local Learnware Deployment**: The Beimingwu system provides tools for learnware deployment and reuse in the `learnware` Python package, helping users to deploy and reuse learnwares conveniently and securely.
|
||||
5. **Data Privacy Protection**: The learnware uploading, searching, and deployment of the Beimingwu system don't require users to upload local data, and all involved statistical specifications are generated locally by users, ensuring data privacy.
|
||||
6. **Open Source System**: The source code of the Beimingwu system, including the `learnware` Python package and frontend and backend codes, are open source.
|
||||
|
||||
### v1.0 R&D Team
|
||||
|
||||
The Beimingwu R&D (Research and Development) Team consists of <a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;" target="_blank">Zhi-Hua Zhou</a>, <a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;" target="_blank">Yang Yu</a>, <a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;" target="_blank">Zhi-Hao Tan</a>, <a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;" target="_blank">Jian-Dong Liu</a>, <a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;" target="_blank">Peng Tan</a>, <a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;" target="_blank">Xiao-Dong Bi</a>, <a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;" target="_blank">Qin-Cheng Zheng</a>, Xiao-Chuan Zou, <a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;" target="_blank">Yi Xie</a>, Hai-Tian Liu, Hao-Yu Shi, Xin-Yu Zhang and others. Additionally, <a href="https://www.lamda.nju.edu.cn/guolz/" style="text-decoration: none; color: inherit;" target="_blank">Lan-Zhe Guo</a>, <a href="http://www.lamda.nju.edu.cn/chenzx/" style="text-decoration: none; color: inherit;" target="_blank">Zi-Xuan Chen</a>, <a href="http://www.lamda.nju.edu.cn/zhouz/" style="text-decoration: none; color: inherit;" target="_blank">Zhi Zhou</a>, <a href="http://www.lamda.nju.edu.cn/jinyx/" style="text-decoration: none; color: inherit;" target="_blank">Yi-Xuan Jin</a>, and others also participated in the early prototype development.
|
||||
The Beimingwu R&D Team consists of <a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;">Zhi-Hua Zhou</a>, <a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;">Yang Yu</a>, <a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;">Zhi-Hao Tan</a>, <a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;">Jian-Dong Liu</a>, <a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;">Peng Tan</a>, <a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;">Xiao-Dong Bi</a>, <a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;">Qin-Cheng Zheng</a>, Xiao-Chuan Zou, <a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;">Yi Xie</a>, Hai-Tian Liu, Hao-Yu Shi, Xin-Yu Zhang and others.
|
||||
|
||||
<!-- System Principal: <a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;" target="_blank">Zhi-Hua Zhou</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;" target="_blank">Yang Yu</a>
|
||||
- System R&D Principal: <a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;" target="_blank">Zhi-Hao Tan</a>
|
||||
- System Architect & Developer: <a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;" target="_blank">Jian-Dong Liu</a>
|
||||
- Engine Architect & Developer: <a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;" target="_blank">Xiao-Dong Bi</a>
|
||||
- Engine Algorithm R&D: <a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;" target="_blank">Peng Tan</a>
|
||||
- Frontend Designer & Developer: <a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;" target="_blank">Qin-Cheng Zheng</a>
|
||||
<!-- - System Principal: <a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;">Zhi-Hua Zhou</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;">Yang Yu</a>
|
||||
- System R&D Principal: <a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;">Zhi-Hao Tan</a>
|
||||
- System Architect & Developer: <a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;">Jian-Dong Liu</a>
|
||||
- Engine Architect & Developer: <a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;">Xiao-Dong Bi</a>
|
||||
- Engine Algorithm R&D: <a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;">Peng Tan</a>
|
||||
- Frontend Designer & Developer: <a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;">Qin-Cheng Zheng</a>
|
||||
- Backend Architect & Developer: Xiao-Chuan Zou
|
||||
- Algorithm Developer & Learnware Preparation: <a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;" target="_blank">Yi Xie</a>、Hai-Tian Liu、Hao-Yu Shi、Xin-Yu Zhang -->
|
||||
- Algorithm Developer & Learnware Preparation: <a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;">Yi Xie</a>、Hai-Tian Liu、Hao-Yu Shi、Xin-Yu Zhang
|
||||
|
||||
In addition, during the early prototype development phase of the system, <a href="https://www.lamda.nju.edu.cn/guolz/" style="text-decoration: none; color: inherit;">Lan-Zhe Guo</a>, <a href="https://www.lamda.nju.edu.cn/chenzx/" style="text-decoration: none; color: inherit;">Zi-Xuan Chen</a>, <a href="https://www.lamda.nju.edu.cn/zhouz/" style="text-decoration: none; color: inherit;">Zhi Zhou</a>, and <a href="https://www.lamda.nju.edu.cn/jinyx/" style="text-decoration: none; color: inherit;">Yi-Xuan Jin</a> were also involved. -->
|
||||
|
|
@ -6,7 +6,7 @@ LAMDA 隶属于计算机软件新技术国家重点实验室，南京大学计
|
|||
|
||||
"LAMDA" 的含义是 "Learning And Mining from DatA". LAMDA 的主要研究兴趣包括机器学习、数据挖掘、模式识别、信息检索、演化计算、神经计算，以及相关的其他领域。目前的主要研究内容包括：集成学习、半监督与主动学习、多示例与多标记学习、代价敏感和类别不平衡学习、度量学习、降维与特征选择、结构学习与聚类、演化计算的理论基础、增强可理解性、基于内容的图像检索、Web 搜索与挖掘、人脸识别、 计算机辅助医疗诊断、生物信息学等。
|
||||
|
||||
北冥坞系统目前由 LAMDA 北冥坞研发团队维护，人员包括：<a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;" target="_blank">周志华</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;" target="_blank">俞扬</a>、<a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;" target="_blank">谭志豪</a>、<a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;" target="_blank">刘建东</a>、<a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;" target="_blank">谭鹏</a>、<a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;" target="_blank">毕晓栋</a>、<a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;" target="_blank">郑钦城</a>、邹晓川、<a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;" target="_blank">谢逸</a>、刘海天、史浩宇、张鑫宇等人。
|
||||
北冥坞系统目前由 LAMDA 北冥坞研发团队维护，人员包括：<a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;">周志华</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;">俞扬</a>、<a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;">谭志豪</a>、<a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;">刘建东</a>、<a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;">谭鹏</a>、<a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;">毕晓栋</a>、<a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;">郑钦城</a>、邹晓川、<a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;">谢逸</a>、刘海天、史浩宇、张鑫宇等人。
|
||||
|
||||
### 联系方式
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
# 常见问题
|
||||
|
||||
|
||||
### Q1: 北冥坞系统是否会泄露用户原始数据？
|
||||
### Q1: 北冥坞系统如何保护用户数据隐私？
|
||||
|
||||
A1: 在北冥坞系统中，学件的上传、查搜、部署均无需用户上传本地数据。生成统计规约的过程在用户本地进行且代码公开，并不会暴露原始数据。
|
||||
A1: 在北冥坞系统中，学件的上传、查搜、部署均无需用户上传本地数据。所有涉及的统计规约均由用户本地生成，并不会暴露原始数据，以确保您的数据隐私。
|
||||
|
||||
|
||||
### Q2: 北冥坞系统如何确保用户部署学件的安全？
|
||||
|
|
@ -23,4 +23,4 @@ A4: 当常规查搜返回空列表时，系统会提示您开启异构查搜。
|
|||
|
||||
### Q5: 学件上传后为什么没有在系统中显示？
|
||||
|
||||
A5: 学件上传后，系统会自动将其加入验证队列检验其是否符合规范，包括学件格式以及模型功能的检查。验证结果将在网页端的「个人信息 - 我的学件」处显示，只有通过验证的学件才会展示在系统中。为提高通过率，建议您在上传前使用 `learnware` 包对学件进行本地验证。
|
||||
A5: 学件上传后，系统会自动将其加入验证队列检验其是否符合规范，包括学件格式以及模型功能的检查。验证结果将在网页端「个人信息 - 我的学件」处显示，只有验证通过才会在系统中显示。为提高通过率，建议您在上传前使用 `learnware` 包对学件进行本地验证。
|
||||
|
|
@ -43,7 +43,7 @@ kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/${R
|
|||
## 部署后端
|
||||
### 下载部署文件
|
||||
```shell
|
||||
git clone https://github.com/Learnware-LAMDA/Beimingwu.git
|
||||
git clone https://gitlink.org.cn/beimingwu/beimingwu.git
|
||||
cd Beimingwu/deploy/kubernetes
|
||||
```
|
||||
### 设置配置文件
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
|
||||
## 使用 Docker Compose 部署
|
||||
|
||||
下载 [后端代码](https://github.com/Learnware-LAMDA/Beimingwu)，进入 `deploy/docker_compose` 目录。
|
||||
下载 [后端代码](https://www.gitlink.org.cn/beimingwu/beimingwu)，进入 `deploy/docker_compose` 目录。
|
||||
|
||||
### 准备文件
|
||||
miniconda 安装文件
|
||||
|
|
@ -18,7 +18,7 @@ mkdir -p Beimingwu/deploy/
|
|||
cp -r ../../backend Beimingwu/
|
||||
cp -r ../../frontend Beimingwu/
|
||||
cp -r ../static Beimingwu/deploy/
|
||||
git clone https://github.com/Learnware-LAMDA/Learnware.git
|
||||
git clone https://gitlink.org.cn/beimingwu/learnware.git
|
||||
```
|
||||
|
||||
### 构建镜像
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@
|
|||
- **系统文档**：维护系统的文档，包括用户指南、开发指南等，确保系统的易用性。
|
||||
- **系统部署**：负责管理系统的部署配置，包括前后端的部署文件。
|
||||
|
||||
其中「系统引擎」实现于独立的 `Learnware` [代码仓库](https://github.com/Learnware-LAMDA/Learnware)中，并配置了独立的[项目文档](https://learnware.readthedocs.io/en/latest/)。
|
||||
其中「系统引擎」实现于独立的 `Learnware` [代码仓库](https://www.gitlink.org.cn/beimingwu/learnware)中，并配置了独立的[项目文档](https://learnware.readthedocs.io/en/latest/)。
|
||||
|
||||
其余四个子项目则实现于 `Beimingwu` [代码仓库](https://github.com/Learnware-LAMDA/Beimingwu)中，并采用 `Monorepo` 的方式进行管理。下文将介绍 `Beimingwu` 代码仓库的具体结构与开发规范。
|
||||
其余四个子项目则实现于 `Beimingwu` [代码仓库](https://www.gitlink.org.cn/beimingwu/beimingwu)中，并采用 `Monorepo` 的方式进行管理。下文将介绍 `Beimingwu` 代码仓库的具体结构与开发规范。
|
||||
|
||||
## Beimingwu 项目结构
|
||||
|
||||
|
|
|
|||
|
|
@ -23,13 +23,13 @@ features:
|
|||
details: 北冥坞系统内置了多重检测机制，以确保系统中每个学件的质量符合要求。
|
||||
icon: { src: "/icons/learnware-quality-detector.svg" }
|
||||
- title: 学件多样查搜
|
||||
details: 北冥坞系统同时支持语义规约和统计规约的查搜，支持多种数据类型，表格数据初步支持异构查搜。
|
||||
details: 北冥坞系统同时支持语义规约和统计规约的查搜，支持多种数据类型，表格数据额外支持异构查搜。
|
||||
icon: { src: "/icons/diverse-learnware-search.svg" }
|
||||
- title: 学件本地部署
|
||||
details: 北冥坞系统提供学件部署与复用工具，帮助用户以统一方式便捷地部署与复用学件。
|
||||
details: 北冥坞系统提供学件部署与复用工具，帮助用户便捷、安全的部署与复用学件。
|
||||
icon: { src: "/icons/local-learnware-deployment.svg" }
|
||||
- title: 保护原始数据
|
||||
details: 北冥坞系统支持学件的上传、查搜、部署和复用，整个过程中用户无需暴露本地数据。
|
||||
- title: 数据隐私保护
|
||||
details: 北冥坞系统涉及的学件上传、查搜、部署均无需用户上传本地数据，确保用户数据隐私。
|
||||
icon: { src: "/icons/data-privacy-protection.svg" }
|
||||
- title: 面向社区开源
|
||||
details: 北冥坞系统面向社区开源，其中的 learnware 包高度可扩展，易于集成新特性和新功能。
|
||||
|
|
|
|||
|
|
@ -16,9 +16,9 @@ pip install learnware==0.3.2 -i https://pypi.tuna.tsinghua.edu.cn/simple
|
|||
|
||||
## 通过源码安装
|
||||
|
||||
`learnware` 包的源代码同时发布在 [GitLink](https://www.gitlink.org.cn/beimingwu/learnware) 和 [Github](https://github.com/Learnware-LAMDA/Learnware) 平台上。以 Github 平台为例，用户可按下述方式通过源代码安装：
|
||||
`learnware` 包的源代码发布在 [GitLink](https://www.gitlink.org.cn/beimingwu/learnware) 平台上，用户可按下述方式通过源代码进行安装：
|
||||
```bash
|
||||
git clone https://github.com/Learnware-LAMDA/Learnware.git
|
||||
git clone https://gitlink.org.cn/beimingwu/learnware.git
|
||||
cd Learnware
|
||||
git fetch origin main
|
||||
git checkout main
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# 快速上手
|
||||
|
||||
欢迎来到北冥坞：学件基座系统！下列内容将帮您快速体验系统，主要包含学件查搜与部署。
|
||||
欢迎来到北冥坞：学件基座系统！下列内容将帮您快速体验系统，包含学件查搜与部署。
|
||||
|
||||
|
||||
## 学件查搜
|
||||
|
|
@ -11,7 +11,7 @@
|
|||
|
||||
|
||||
|
||||
通过统计信息进行查搜时，您需要提交任务的统计规约。统计规约可以在保护您的原始数据不被泄露的情况下提供任务的统计信息。您可以使用系统提供的工具，通过下列代码在本地生成统计规约。
|
||||
通过统计信息进行查搜时，您需要提交任务的统计信息。我们提供的工具将在保护您数据隐私的情况下在本地为您生成任务的近似统计信息。通过下列代码，您可以轻松生成任务的近似统计信息。
|
||||
|
||||
```python
|
||||
from learnware.specification import generate_stat_spec
|
||||
|
|
@ -26,7 +26,7 @@ spec.save("stat.json")
|
|||
|
||||
|
||||
|
||||
某些情况下，组合多个学件的统计规约可能更加贴合您的任务需求，系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
某些情况下，多个学件组合在一起的统计信息会更加接近您的任务，系统会将这些学件打包推荐给您。您可以通过右上角的 “下载全部” 按钮进行一键下载。
|
||||
|
||||
|
||||
|
||||
|
|
@ -45,18 +45,4 @@ leanrware = client.load_learnware(
|
|||
|
||||
# 使用学件对数据进行预测
|
||||
pred_y = learnware.predict(test_x)
|
||||
```
|
||||
|
||||
## 演示视频
|
||||
|
||||
下列视频将更加完整地演示如何使用北冥坞系统解决一个机器学习任务：
|
||||
|
||||
<div style="padding: 56.25% 0 0 0; position: relative"><div style="height:100%;left:0;position:absolute;top:0;width:100%"><iframe height="100%" width="100%;" src="https://embed.wave.video/S2zG1ZbUaRpEo8UG" frameborder="0" allow="autoplay; fullscreen" scrolling="no"></iframe></div></div>
|
||||
|
||||
更多内容可参考 「[学件查搜](https://docs.beiming.cloud/zh-CN/user-guide/learnware-search.html)」, 「[学件部署](https://docs.beiming.cloud/zh-CN/user-guide/learnware-deploy.html)」。
|
||||
|
||||
如果您乐意分享模型至北冥坞系统，可以通过下列视频快速入手：
|
||||
|
||||
<div style="padding: 56.25% 0 0 0; position: relative"><div style="height:100%;left:0;position:absolute;top:0;width:100%"><iframe height="100%" width="100%;" src="https://embed.wave.video/JcMSmDcgTJep5zPo" frameborder="0" allow="autoplay; fullscreen" scrolling="no"></iframe></div></div>
|
||||
|
||||
更多内容可参考 「[如何准备一个学件？](https://docs.beiming.cloud/zh-CN/user-guide/learnware-upload/prepare.html)」, 「[通过网页端上传](https://docs.beiming.cloud/zh-CN/user-guide/learnware-upload/upload-web.html)」, 「[通过客户端上传](https://docs.beiming.cloud/zh-CN/user-guide/learnware-upload/upload-client.html)」。
|
||||
```
|
||||
|
|
@ -15,7 +15,7 @@
|
|||
- 语义规约通过文本对模型的类型及功能进行描述；
|
||||
- 统计规约则通过各类机器学习技术，刻画模型所蕴含的统计信息。
|
||||
|
||||
学件的规约刻画了模型的能力，使得模型在未来能够根据用户需求被充分识别和复用。
|
||||
学件的规约刻画了模型的能力，使得模型能够在未来用户事先对学件一无所知的情况下被充分识别并复用，以满足用户需求。
|
||||
|
||||
## 为什么需要学件基座系统？
|
||||
|
||||
|
|
@ -23,11 +23,11 @@
|
|||
|
||||
虽然上述每一个问题都有相应的研究，但由于问题之间是相互耦合的，在解决其中一个问题时，可能会导致其他问题变得更加严重。
|
||||
|
||||
学件基座系统期望通过一个整体框架，同时解决下述诸多问题：
|
||||
学件基座系统期望通过一个整体框架，同时解决上述诸多问题：
|
||||
- **缺乏训练数据/技能**：即使对于缺乏训练技能或数据量较小的普通用户，也可以获得强大的机器学习模型，因为用户可以从学件基座系统中获取性能优良的学件，并进一步调整或改进，而不是自己从头开始构建模型。
|
||||
- **持续学习**：随着在各种不同任务上训练得到的、性能优良的学件被不断提交，学件基座系统中的知识将不断丰富，进而自然地实现持续和终身学习。
|
||||
- **灾难性遗忘**：一旦学件被接收，它将永远被容纳在学件基座系统中，除非其各方面功能都可以被其他学件所替代。因此，学件基座系统中的旧知识总是会被保留，而不会被遗忘。
|
||||
- **数据隐私/所有权**：开发者通过仅提交模型而不共享私有数据的方式，有助于更好地保护数据隐私和所有权。尽管无法完全排除对模型进行逆向工程的可能性，但与许多其它隐私保护方案相比，学件基座系统泄露隐私的风险更低。
|
||||
- **数据隐私/所有权**：开发者只提交模型而不共享私有数据，因此数据隐私/所有权可以得到很好的保护。尽管无法完全排除对模型进行逆向工程的可能性，但与许多其它隐私保护方案相比，学件基座系统泄露隐私的风险非常小。
|
||||
|
||||
|
||||
## 学件基座系统是如何工作的？
|
||||
|
|
@ -54,7 +54,7 @@
|
|||
|
||||
学件基座系统在查搜时，首先通过用户需求中的语义规约定位到具体的规约岛屿，再通过用户需求中的统计规约对规约岛上的学件进行精确识别。不同的规约岛屿合并，则意味着相应的学件可以被用于不同特征/标记空间的任务上，即可以复用至超出其原始目的的任务中。
|
||||
|
||||
学件范式通过充分利用社区分享的机器学习模型的能力，构建统一的规约空间，以统一的方式高效地为新用户解决机器学习任务。随着算法的改进和学件数量的增多，学件基座系统的知识库将不断扩充，处理学习任务的能力也将逐步增强。
|
||||
学件范式通过充分利用社区分享的机器学习模型的能力，构建统一的规约空间，以统一的方式高效地为新用户解决机器学习任务。随着学件数量增多，通过有效地组织学件结构，学件基座系统整体解决任务的能力将显著增强。
|
||||
|
||||
## 北冥坞系统是如何组织的？
|
||||
|
||||
|
|
@ -71,24 +71,18 @@
|
|||
|
||||
## 北冥坞系统有哪些特性？
|
||||
|
||||
北冥坞学件基座系统作为学件的初步科研平台，实现了学件范式中的核心流程：
|
||||
北冥坞学件基座系统作为学件的初步科研平台，首次系统性地实现了学件范式中的核心流程：
|
||||
|
||||
- **提交阶段**：系统内置了多重检测机制，以确保上传学件的质量。另外，系统会根据已有的学件规约，训练一个异构引擎，用于合并不同的规约岛屿，以及为学件赋予新规约。随着更多学件的上传，系统的异构引擎将持续更新，目标是通过系统内学件规约的持续迭代，构建更加精准的规约世界。
|
||||
- **部署阶段**：用户上传任务需求后，系统会自动选择是推荐单学件还是多学件组合，并提供高效的部署方式。无论是单个学件还是多学件组合，系统均提供了基础的学件复用接口。
|
||||
- **提交阶段**：系统内置了多重检测机制，以确保上传学件的质量。另外，系统会根据已有的学件规约，训练一个异构引擎，用于合并不同的规约岛屿，以及为学件赋予新规约。随着更多学件的上传，异构引擎将持续更新，实现学件规约的持续迭代，构建更精准的规约世界。
|
||||
- **部署阶段**：用户上传任务需求后，系统会自动选择是推荐单学件还是多学件组合，并提供高效的部署方式。无论是单个学件还是多学件组合，系统均提供了便捷的学件复用接口。
|
||||
|
||||
此外，北冥坞系统还具备以下特性：
|
||||
|
||||
- **学件规约生成**：北冥坞系统在 `learnware` Python 包中提供规约生成接口，支持多种数据类型（表格、图像和文本），可以在本地高效生成。
|
||||
- **学件质量检测**：北冥坞系统内置了多重检测机制，以确保系统中每个学件的质量。
|
||||
- **学件多样查搜**：北冥坞系统同时支持语义规约和统计规约的查搜，覆盖的数据类型包括表格、图像、文本。另外，对于表格型任务，系统额外支持初步的异构表格学件的查搜。
|
||||
- **学件本地部署**：北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口，帮助用户以统一的方式便携地部署与复用学件。
|
||||
- **保护原始数据**：北冥坞系统所涉及的学件上传、查搜、部署均无需用户泄露原始数据，生成统计规约的过程在用户本地进行且代码公开。
|
||||
- **面向社区开源**：北冥坞系统面向社区开源，包括 `learnware` Python 包与前后端代码。其中 `leanrware` 包高度可扩展，未来新的规约设计、学件系统设计、学件查搜和复用方法都能轻松集成进来。
|
||||
- **学件多样查搜**：北冥坞系统同时支持语义规约和统计规约的查搜，覆盖的数据类型包括表格、图像、文本。另外，对于表格型任务，系统额外支持异构表格学件的查搜。
|
||||
- **学件本地部署**：北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口，帮助用户便捷、安全的部署与复用学件。
|
||||
- **数据隐私保护**：北冥坞系统所涉及的学件上传、查搜、部署均无需用户上传本地数据，所有涉及的统计规约均由用户本地生成，确保用户数据隐私。
|
||||
- **面向社区开源**：北冥坞系统面向社区开源，包括 `learnware` Python 包与前后端代码。其中 `leanrware` 包高度可扩展，未来新的规约设计、学件系统设计、学件复用方法都能轻松集成进来。
|
||||
|
||||
学件范式的构建需要社区的共同努力。作为首个学件基座系统，北冥坞还很年轻，相关技术仍有很大的改进空间。我们诚邀大家基于北冥坞系统上传模型、共同开发完善系统，或进行学件相关算法的研究与改进，为系统的持续改进提出宝贵意见。
|
||||
|
||||
## 案例演示
|
||||
|
||||
在下列视频中，我们将展示北冥坞系统如何在一个实际应用场景中提供可能的解决方案：
|
||||
|
||||
<div style="padding: 56.25% 0 0 0; position: relative"><div style="height:100%;left:0;position:absolute;top:0;width:100%"><iframe height="100%" width="100%;" src="https://embed.wave.video/s8B9PykeyMwhUX5F" frameborder="0" allow="autoplay; fullscreen" scrolling="no"></iframe></div></div>
|
||||
学件范式的构建需要社区的共同努力。作为首个学件基座系统，北冥坞还很年轻，相关技术仍有很大的改进空间。我们诚邀大家基于北冥坞系统上传模型、共同开发完善系统，或进行学件相关算法的研究与改进，为系统的持续改进提出宝贵意见。
|
||||
|
|
@ -13,7 +13,7 @@ client = LearnwareClient()
|
|||
|
||||
### 根据学件 `id` 载入学件
|
||||
|
||||
假设用户已知需要载入的学件对应的 `id`，那么可以使用如下代码载入相应的学件和环境：
|
||||
假设用户已知需要载入的学件对应的 `id`，那么可以使用如下代码载入对应的学件和环境：
|
||||
|
||||
```python
|
||||
learnware_id = "00000082"
|
||||
|
|
@ -36,12 +36,12 @@ learnware_list = client.load_learnware(
|
|||
- `"conda"`：为每个学件安装独立的 `conda` 虚拟环境（运行结束后自动删除），在虚拟环境内独立运行每个学件；
|
||||
- `"docker"`：在 `docker` 容器内安装 `conda` 虚拟环境（运行结束后自动销毁），在容器内独立运行每个学件（用户需要有 `docker` 权限）。
|
||||
|
||||
需要注意的是，尽管系统已尽最大努力确保每个学件的安全，但如果仍有包含恶意代码的漏网之鱼，则 `None` 和 `"conda"` 两种模式是**不安全**的。如果用户不能确保需要加载的学件的安全性，请使用**较为安全** `"docker"` 模式载入学件。
|
||||
需要注意的是，尽管系统已尽最大努力确保每个学件的安全，但如果仍有包含恶意代码的漏网之鱼，则 `None` 和 `"conda"` 两种模式是**不安全**的。如果用户不能确保需要加载的学件的安全性，请使用 `"docker"` 模式载入学件。
|
||||
|
||||
|
||||
### 根据学件 `zip` 文件载入学件
|
||||
|
||||
除了根据学件 `id` 载入学件，用户还可以从网页端下载的 `zip` 文件中载入学件：
|
||||
除了根据学件 `id` 载入学件，用户还可以使用从网页端下载的 `zip` 文件载入学件：
|
||||
|
||||
```python
|
||||
learnware_path = "learnware.zip"
|
||||
|
|
@ -60,7 +60,7 @@ learnware_list = client.load_learnware(
|
|||
|
||||
### 调用学件
|
||||
|
||||
按上述方式加载学件后，您可以直接调用学件的 `predict(X)` 接口进行预测，具体代码如下：
|
||||
按上述方式加载学件后，可直接调用学件的 `predict(X)` 接口进行预测，具体代码如下：
|
||||
```python
|
||||
# test_x 是用户任务的待预测数据, predict_y 是学件的预测结果
|
||||
learnware = client.load_learnware(learnware_id=learnware_id)
|
||||
|
|
@ -76,9 +76,9 @@ predict_y = learnware.predict(test_x)
|
|||
|
||||
### 直接复用学件
|
||||
|
||||
我们提供了 `JobSelectorReuser` 和 `AveragingReuser` 两种直接复用学件的方法:
|
||||
我们提供了 `JobSelector` 和 `Averaging` 两种直接复用学件的方法:
|
||||
|
||||
- `JobSelectorReuser` 通过训练分类器为不同的数据选择合适的学件进行预测，具体代码如下：
|
||||
- `JobSelector` 通过训练分类器为不同的数据选择不同的学件进行预测，具体代码如下：
|
||||
|
||||
```python
|
||||
from learnware.reuse import JobSelectorReuser
|
||||
|
|
@ -90,7 +90,7 @@ reuse_job_selector = JobSelectorReuser(learnware_list=learnware_list)
|
|||
predict_y = reuse_job_selector.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
- `AveragingReuser` 使用平均集成的方法进行学件复用，其中 `mode` 表示具体的集成方式：
|
||||
- `Averaging` 使用平均集成的方法进行学件复用，其中 `mode` 表示具体的集成方式：
|
||||
|
||||
```python
|
||||
from learnware.reuse import AveragingReuser
|
||||
|
|
@ -104,13 +104,13 @@ reuse_ensemble = AveragingReuser(
|
|||
ensemble_predict_y = reuse_ensemble.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
更详细的使用方法和原理介绍请参考：[Learnware 包直接复用方法介绍](https://learnware.readthedocs.io/en/latest/components/learnware.html#direct-reuse-of-learnware)。
|
||||
更详细的使用方法和原理介绍请参考：[Learnware 包复用方法介绍](https://learnware.readthedocs.io/en/latest/components/learnware.html#direct-reuse-of-learnware)。
|
||||
|
||||
### 使用有标记数据复用学件
|
||||
|
||||
当用户有少量有标记数据时，我们提供了 `EnsemblePruningReuser` 和 `FeatureAugmentReuser` 两种方法帮助用户复用学件。
|
||||
当用户有少量有标记数据时，我们提供了`EnsemblePruning`和`FeatureAugmentReuser`两种方法帮助用户复用学件。
|
||||
|
||||
- `EnsemblePruningReuser` 通过多目标演化算法挑选适合用户任务的学件组，并进行平均集成：
|
||||
- `EnsemblePruning` 通过选择性集成的方法选择更适合用户任务的学件：
|
||||
|
||||
```python
|
||||
from learnware.reuse import EnsemblePruningReuser
|
||||
|
|
@ -126,7 +126,9 @@ reuse_ensemble_pruning.fit(val_X=val_X, val_y=val_y)
|
|||
predict_y = reuse_job_selector.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
- `FeatureAugmentReuser` 通过特征增广的方式帮助用户复用学件，原始学件的输出会拼接到用户任务的特征上，并使用增广后的有标记数据训练一个简单的模型（分类任务为logistics regression，回归任务为ridge）：
|
||||
更详细的使用方法和原理介绍请参考：[Learnware 包复用方法介绍](https://learnware.readthedocs.io/en/latest/components/learnware.html#reuse-learnware-with-labeled-data)。
|
||||
|
||||
- `FeatureAugmentReuser`通过特征增广的方式帮助用户复用学件，原始学件的输出会拼接到用户任务的特征上，并基于有标记数据训练一个简单的模型（分类任务为logistics regression，回归任务为ridge）：
|
||||
|
||||
```python
|
||||
from learnware.reuse import FeatureAugmentReuser
|
||||
|
|
@ -142,8 +144,6 @@ augment_reuser.fit(val_X, val_y)
|
|||
predict_y = augment_reuser.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
更详细的使用方法和原理介绍请参考：[Learnware 包使用有标记数据复用方法介绍](https://learnware.readthedocs.io/en/latest/components/learnware.html#reuse-learnware-with-labeled-data)。
|
||||
|
||||
## 异构学件复用方法
|
||||
|
||||
我们提供了 `HeteroMapAlignLearnware` 类帮助用户将异构学件与用户任务进行对齐，一共包含两步：输入空间对齐、输出空间对齐。
|
||||
|
|
@ -161,7 +161,7 @@ hetero_learnware.align(user_spec, val_x, val_y)
|
|||
predict_y = hetero_learnware.predict(user_data=test_x)
|
||||
```
|
||||
|
||||
如果想要复用多个异构学件，可以将 `HeteroMapAlignLearnware` 和上文介绍的同构复用方法 `AveragingReuser`、`EnsemblePruningReuser` 相结合：
|
||||
如果想要复用多个异构学件，可以将 `HeteroMapAlignLearnware` 和上文介绍的同构复用方法 `Averaging`、`EnsemblePruning` 相结合：
|
||||
|
||||
```python
|
||||
hetero_learnware_list = []
|
||||
|
|
|
|||
|
|
@ -70,7 +70,7 @@ client.login(email="your email", token="your token")
|
|||
|
||||
与网页端查搜相对应，learnware 包也支持语义规约查搜、统计规约查搜、语义规约 + 统计规约的混合查搜。
|
||||
|
||||
您可以通过语义规约在学件基座系统中查搜学件，所有符合语义规约的学件都将通过 API 返回。例如，下列代码将得到系统中所有任务类型为分类的学件：
|
||||
您可以通过语义规约在学件基座系统中查搜学件，所有符合语义规约的学件都将通过API返回。例如，下列代码将得到系统中所有任务类型为分类的学件：
|
||||
|
||||
```python
|
||||
from learnware.market import BaseUserInfo
|
||||
|
|
@ -98,14 +98,14 @@ search_result = {
|
|||
}
|
||||
```
|
||||
|
||||
您也可以通过统计规约在学件基座系统中查搜学件，这些模型往往具有更好的针对性，并将通过 API 返回。利用上述提到的 `generate_stat_spec` 函数，您可以生成当前任务的统计规约 `stat_spec`，随后使用下列代码获取系统中符合该统计规约的同数据类型的学件：
|
||||
另外，您也可以通过统计规约在学件基座系统中查搜学件，所有分布相似的学件都将通过 API 返回。通过上述提到的 `generate_stat_spec` 函数，您可以便捷地得到您当前任务对应的统计规约 `stat_spec`，随后通过下列代码即可得到系统中同一类型数据下满足您任务统计规约的学件：
|
||||
|
||||
```python
|
||||
user_info = BaseUserInfo(stat_info={stat_spec.type: stat_spec})
|
||||
search_result = client.search_learnware(user_info)
|
||||
```
|
||||
|
||||
通过将统计规约和语义规约结合起来，您可以进行更加细致的查搜。例如，以下代码演示了如何在表格型学件中查搜同时满足您任务语义规约和统计规约的学件：
|
||||
通过将统计规约和语义规约结合起来，您可以进行更加细致的查搜，比如下列代码将在表格型数据中查搜满足您语义规约的学件：
|
||||
|
||||
```python
|
||||
user_semantic = generate_semantic_spec(
|
||||
|
|
@ -119,7 +119,7 @@ user_info = BaseUserInfo(
|
|||
search_result = client.search_learnware(user_info)
|
||||
```
|
||||
|
||||
当学件查搜完成后，您可以通过 `search_result` 中的学件 ID 本地加载和使用学件：
|
||||
当学件查搜完成后，您可以通过 `search_result` 中的学件 ID 本地加载和使用学件，可参考下述示例：
|
||||
|
||||
```python
|
||||
learnware_id = search_result["single"]["learnware_ids"][0]
|
||||
|
|
@ -135,7 +135,7 @@ predict_y = learnware.predict(test_x)
|
|||
|
||||
### 异构表格查搜
|
||||
|
||||
对于表格型任务，如果任务类型为“分类”或者回归，并且您提供了统计规约、在语义规约中包含了每个维度特征的语义描述，系统会自动启动异构表格查搜，以匹配和推荐特征空间不对齐的模型。以下代码示例展示了如何通过 API 执行异构表格查搜：
|
||||
当您提供表格型数据的统计规约，任务类型为“分类”或者“回归”，并且在语义规约中包含了每一维特征语义维度的描述时，系统将自行开启异构表格查搜，而不仅仅是在维度相匹配的数据类型为表格的学件中进行查搜。以下代码将进行通过 API 进行异构表格查搜：
|
||||
|
||||
```python
|
||||
input_description = {
|
||||
|
|
|
|||
|
|
@ -16,9 +16,9 @@
|
|||
## 模型调用文件 `__init__.py`
|
||||
|
||||
|
||||
为了使上传的学件可以被后续用户使用，您需要在 `__init__.py` 中提供模型拟合 `fit(X, y)`、预测 `predict(X)`、微调 `finetune(X, y)` 这三种基本功能接口。请注意，`predict(X)` 接口是必须提供的，而 `fit(X, y)` 和 `finetune(X, y)` 根据模型本身的特性和功能决定是否实现。
|
||||
为了使上传的学件可以被后续用户使用，需在 \_\_init\_\_.py 中提供模型拟合 `fit(X, y)`、预测 `predict(X)`、微调 `finetune(X, y)` 的接口。三个接口中仅 `predict(X)` 为必须提供的接口，其余接口根据模型本身功能而定。
|
||||
|
||||
以下是 `__init__.py` 文件的参考模板，在准备您的模型调用文件时，请确保各个接口的输入参数格式（参数个数、参数名）与提供的模板一致：
|
||||
以下是 `__init__.py` 文件的参考模板，请确保您上传的模型调用文件中各个接口输入参数的格式（参数个数、参数名）与下述模板保持一致：
|
||||
```py
|
||||
import os
|
||||
import pickle
|
||||
|
|
@ -55,9 +55,9 @@ class MyModel(BaseModel):
|
|||
|
||||
### 文件路径
|
||||
|
||||
在 `__init__.py` 文件（以及其它相关的 Python 文件）中，如果您需要从 zip 包内导入特定文件（例如 `model.pkl`），请采用上述示例代码中获取 `model_path` 的方式：
|
||||
- 首先确定整个学件包的根目录路径 `dir_path`；
|
||||
- 然后根据文件在包内的相对位置，获取该文件的具体路径 `model_path`。
|
||||
如果在 `__init__.py` 文件（以及其它可能涉及的 Python 文件）中需要导入 zip 包内某些文件时（例如模型文件 `model.pkl`），请采用上述示例代码中获取 `model_path` 的方式：
|
||||
- 先获取整个学件包的根目录路径 `dir_path`；
|
||||
- 再根据具体文件在包内的相对位置，获取具体文件的路径 `model_path`。
|
||||
|
||||
### 模块引用
|
||||
|
||||
|
|
@ -86,7 +86,7 @@ spec.save("stat.json")
|
|||
|
||||
## 学件配置文件 `learnware.yaml`
|
||||
|
||||
该文件用于指明模型调用文件 \_\_init\_\_.py 中的类名 (`MyModel`)、生成统计规约所调用的模块 (`learnware.specification`) 以及统计规约的类型 (`RKMETableSpecification`) 和文件名 (`stat.json`)：
|
||||
该文件用于指明模型调用文件 \_\_init\_\_.py 中的类名 (`MyModel`)、生成统计规约所调用的模块 (`learnware.specification`) 以及统计规约的类别 (`RKMETableSpecification`) 与具体的文件名 (`stat.json`)：
|
||||
```yaml
|
||||
model:
|
||||
class_name: MyModel
|
||||
|
|
@ -102,11 +102,11 @@ stat_specifications:
|
|||
|
||||
## 模型运行依赖
|
||||
|
||||
为了使上传的学件可以被后续其它用户使用，您需要在上传的学件 `zip` 包中明确指定模型的运行依赖。北冥坞系统支持下述两种指定运行依赖的方式：
|
||||
为了使上传的学件可以被后续其它用户使用，上传学件的 `zip` 包中需指明模型的运行依赖。北冥坞系统支持下述两种指定运行依赖的方式：
|
||||
- 提供 `conda` 支持的 `environment.yaml` 文件；
|
||||
- 提供 `pip` 支持的 `requirements.txt` 文件。
|
||||
|
||||
您可以选择其中一种方式来指定依赖，但无论使用哪一种方式，请尽量去除不需要的依赖，使依赖项尽可能的少。
|
||||
两种方式选其一即可，但无论使用哪一种方式，请尽量去除不需要的依赖，使依赖项尽可能的少。
|
||||
|
||||
### 使用 `environment.yaml` 文件
|
||||
|
||||
|
|
@ -155,7 +155,7 @@ conda env remove --name test_env
|
|||
numpy==1.23.5
|
||||
scikit-learn==1.2.2
|
||||
```
|
||||
如果觉得手动列举比较麻烦，您也可以使用 `pipreqs` 包直接扫描整个项目，自动导出使用的包及其具体版本（但很可能会有些偏差，需要再自行检查）：
|
||||
如果觉得手动列举比较麻烦，也可以使用 `pipreqs` 包直接扫描整个项目，自动导出使用的包及其具体版本（但很可能会有些偏差，需要再自行检查）：
|
||||
```bash
|
||||
pip install pipreqs
|
||||
pipreqs ./ # 需在项目根目录执行
|
||||
|
|
@ -167,7 +167,7 @@ pipreqs ./ # 需在项目根目录执行
|
|||
|
||||
## 学件本地验证
|
||||
|
||||
上述文件准备完毕后，即可进行学件上传。一旦学件成功上传，系统后台会自动将该学件加入验证队列，以检验学件是否符合规范（包括学件格式以及模型是否可运行的检查）。这个验证过程可能需要一些时间，具体取决于学件的复杂程度以及系统后台的工作负载。验证完成后，结果将在网页端显示。
|
||||
上述文件填写完毕后，即可进行学件上传。一旦学件成功上传，系统后台会自动将该学件加入验证队列，以检验学件是否符合规范（包括学件格式以及模型是否可运行的检查）。这个验证过程可能需要一些时间，具体取决于学件的复杂程度以及系统后台的工作负载。完成验证后，结果将在网页端显示。
|
||||
|
||||
由于后台验证学件的过程相对耗时，为了提高学件通过验证的几率，我们建议您在上传之前，先使用以下代码在本地对学件进行验证：
|
||||
```py
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
在北冥坞系统中，学件既可以从网页端上传，也可以使用 `learnware` Python 包进行上传，即使用客户端上传。
|
||||
|
||||
接下来，我们将介绍如何使用 `learnware` 包上传学件。首先需要登录：
|
||||
接下来，我们将对使用 `learnware` 包上传的部分进行介绍，首先需要登录：
|
||||
```py
|
||||
from learnware.client import LearnwareClient, SemanticSpecificationKey
|
||||
|
||||
|
|
@ -14,7 +14,7 @@ client.login(email="your email", token="your token")
|
|||
|
||||
## 准备语义规约
|
||||
|
||||
成功登录后，您需要准备语义规约，此处以「表格数据」的「分类任务」为例：
|
||||
成功登录后需要准备语义规约，此处以「表格数据」的「分类任务」为例：
|
||||
```py
|
||||
from learnware.specification import generate_semantic_spec
|
||||
|
||||
|
|
@ -48,7 +48,6 @@ semantic_spec = generate_semantic_spec(
|
|||
task_type="Classification",
|
||||
library_type="Scikit-learn",
|
||||
scenarios=["Business", "Financial"],
|
||||
license=["Apache-2.0"],
|
||||
input_description=input_description,
|
||||
output_description=output_description,
|
||||
)
|
||||
|
|
@ -60,16 +59,17 @@ semantic_spec = generate_semantic_spec(
|
|||
- scenarios 必须为 `key=SemanticSpecificationKey.SENARIOES` 对应结果的子集。
|
||||
|
||||
另外，需要注意：
|
||||
- 如果 data\_type 为 `"Table"`，则需要填写模型输入数据的每一维特征语义，使上传的学件可用于特征空间不对齐的任务；
|
||||
- 如果 task\_type 为 `"Classification"`，则需要填写模型输出标记的语义（预测标记从 0 开始编号），使上传的学件可用于输出空间不对齐的分类任务；
|
||||
- 如果 task\_type 为 `"Regression"`，则需要填写模型输出的每一维语义，使上传的学件可用于输出空间不对齐的回归任务；
|
||||
- 如果 data\_type 为 `"Table"`，则需要填写模型输入数据的每一维特征语义，使上传的学件可用于异构特征空间的任务；
|
||||
- 如果 task\_type 为 `"Classification"`，则需要填写模型输出标记的语义（预测标记从 0 开始编号），使上传的学件可用于异构输出空间的分类任务；
|
||||
- 如果 task\_type 为 `"Regression"`，则需要填写模型输出的每一维语义，使上传的学件可用于异构输出空间的回归任务；
|
||||
|
||||
## 学件上传
|
||||
|
||||
接下来，填写语义规约和学件 zip 包路径，您即可完成学件上传。
|
||||
最后，填写语义规约和学件 zip 包路径，即可实现学件上传。
|
||||
|
||||
关于学件 zip 包的准备，请参考指南：[如何准备一个学件？](/zh-CN/user-guide/learnware-upload/prepare)。 在上传前，建议您先对学件进行本地验证，代码示例如下：
|
||||
学件 zip 包的准备可查看：[如何准备一个学件？](/zh-CN/user-guide/learnware-upload/prepare)
|
||||
|
||||
记得在上传前先对学件进行验证，代码示例如下：
|
||||
```py
|
||||
# Prepare your learnware zip file
|
||||
zip_path = "your learnware zip"
|
||||
|
|
@ -85,4 +85,6 @@ learnware_id = client.upload_learnware(
|
|||
)
|
||||
```
|
||||
|
||||
学件上传后，您可以在「个人信息 - 我的学件」页面看到上传的学件。系统后台会自动将该学件加入验证队列，以检验学件是否符合规范。如果学件通过验证，其标签会变为「验证成功」，并出现在系统中。
|
||||
学件上传成功后，可以在「个人信息 - 我的学件」处看到上传的学件。
|
||||
|
||||
学件上传后，后台会对学件进行检查。检查通过后，学件的标签将变为「验证成功」，且上传的学件会在系统中出现。
|
||||
|
|
@ -8,25 +8,25 @@
|
|||
|
||||
点击网站导航栏处的「[提交](https://www.bmwu.cloud/#/submit)」按钮，即可开始学件上传。
|
||||
|
||||
首先您将进入第一步：填写学件名称，具体页面如下所示：
|
||||
首先是「第一步 - 填写学件名称」，具体页面如下所示：
|
||||
|
||||
|
||||
|
||||
请注意，「学件名称」需要控制在 5 至 50 个字符之间。
|
||||
此处需要注意「学件名称」需要控制在 5 至 50 个字符之间。
|
||||
|
||||
## 步骤 2：选择学件标签
|
||||
|
||||
填写完学件名称后，您将进入第二步：填写学件标签。
|
||||
填写完学件名称后，进入第二步：填写学件标签。
|
||||
|
||||
首先是选择学件所涉及的「数据类型」，此处如果选择「表格」，那么您将需要填写模型输入数据的每一维特征语义，使得上传的表格学件可以用于特征空间不对齐的任务。
|
||||
首先是选择学件所涉及的「数据类型」，此处如果选择「表格」，则需要填写模型输入数据的每一维特征语义，使上传的学件可用于异构特征空间的任务。
|
||||
|
||||
如果特征维度较多，您可以考虑使用大型语言模型分析特征工程的代码，以生成各维度的语义信息。
|
||||
如果维度过多，可考虑使用大语言模型。通过分析特征工程的代码，生成各维度的语义。
|
||||
|
||||
|
||||
|
||||
随后是选择学件所涉及的「任务类型」，此处如果选择「分类」，那么您需要填写模型输出标记的语义（预测标记从 0 开始编号），使得上传的学件可用于输出空间不对齐的分类任务。
|
||||
随后是选择学件所涉及的「任务类型」，此处如果选择「分类」，则需要填写模型输出标记的语义（预测标记从 0 开始编号），使上传的学件可用于异构输出空间的分类任务。
|
||||
|
||||
如果选择「回归」，则您需要填写模型输出的每一维语义，使上传的学件可用于输出空间不对齐的回归任务。
|
||||
如果选择「回归」，则需要填写模型输出的每一维语义，使上传的学件可用于异构输出空间的回归任务。
|
||||
|
||||
|
||||
|
||||
|
|
@ -36,7 +36,7 @@
|
|||
|
||||
## 步骤 3：填写学件描述
|
||||
|
||||
选择完学件标签后，即可进入第三步：填写学件描述，此处描述内容支持 `Markdown` 格式。
|
||||
选择完学件标签后，即可进入第三步：填写学件描述，此处描述内容支持 `markdown` 格式。
|
||||
|
||||
另外需要注意「学件描述」需要控制在 10 至 10000 个字符之间。
|
||||
|
||||
|
|
@ -46,8 +46,10 @@
|
|||
|
||||
填写完学件描述后，即可进入最后一步：上传学件 zip 包。
|
||||
|
||||
关于如何准备一个学件 zip 包，您可以查看相关文档：[如何准备一个学件？](/zh-CN/user-guide/learnware-upload/prepare)，以获取详细信息。
|
||||
学件 zip 包的准备可查看：[如何准备一个学件？](/zh-CN/user-guide/learnware-upload/prepare)
|
||||
|
||||
|
||||
|
||||
学件上传后，您可以在「个人信息 - 我的学件」页面看到上传的学件，此时系统会在后台自动对学件进行检查。如果学件通过验证，其标签会变为「验证成功」，并出现在系统中。
|
||||
学件上传成功后，可以在「个人信息 - 我的学件」处看到上传的学件。
|
||||
|
||||
学件上传后，后台会对学件进行检查。检查通过后，学件的标签将变为「验证成功」，且上传的学件会在系统中出现。
|
||||
|
|
@ -2,26 +2,28 @@
|
|||
|
||||
## 北冥坞 v1.0 版本 (2024.01)
|
||||
|
||||
北冥坞学件基座系统 v1.0 版本是学件的第一个系统性开源实现，为学件相关研究提供了一个初步科研平台。该系统旨在通过查搜和复用学件，帮助用户便捷解决机器学习任务，而无需从零开始构建机器学习模型。
|
||||
北冥坞学件基座系统 v1.0 版本是学件的第一个系统性开源实现，为学件相关研究提供了一个初步科研平台。该系统旨在通过有效查搜和复用学件，帮助用户高效解决机器学习任务，而无需从零开始构建机器学习模型。
|
||||
|
||||
### v1.0 系统特点
|
||||
|
||||
1. **学件规约生成**：北冥坞系统在 `learnware` Python 包中提供规约生成接口，支持多种数据类型（表格、图像和文本），可以在本地高效生成。
|
||||
2. **学件质量检测**：北冥坞系统内置了多重检测机制，以确保系统中每个学件的质量。
|
||||
3. **学件多样查搜**：北冥坞系统同时支持语义规约和统计规约的查搜，覆盖的数据类型包括表格、图像、文本。另外，对于表格型任务，系统额外支持初步的异构表格学件查搜。
|
||||
4. **学件本地部署**：北冥坞系统在 `learnware` Python 包中提供统一的学件部署与复用的接口，帮助用户便捷的使用学件。
|
||||
5. **保护原始数据**：北冥坞系统所涉及的学件上传、查搜、部署均无需用户上传本地数据，生成统计规约的过程在用户本地进行且代码公开。
|
||||
3. **学件多样查搜**：北冥坞系统同时支持语义规约和统计规约的查搜，覆盖的数据类型包括表格、图像、文本。另外，对于表格型任务，系统额外支持异构表格学件的查搜。
|
||||
4. **学件本地部署**：北冥坞系统在 `learnware` Python 包中同时提供学件部署与学件复用的接口，帮助用户便捷、安全的部署与复用学件。
|
||||
5. **数据隐私保护**：北冥坞系统所涉及的学件上传、查搜、部署均无需用户上传本地数据，所有涉及的统计规约均由用户本地生成，确保用户数据隐私。
|
||||
6. **面向社区开源**：北冥坞系统面向社区开源，包括 `learnware` Python 包与前后端代码。
|
||||
|
||||
### v1.0 研发团队
|
||||
|
||||
北冥坞研发团队人员包括：<a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;" target="_blank">周志华</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;" target="_blank">俞扬</a>、<a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;" target="_blank">谭志豪</a>、<a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;" target="_blank">刘建东</a>、<a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;" target="_blank">谭鹏</a>、<a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;" target="_blank">毕晓栋</a>、<a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;" target="_blank">郑钦城</a>、邹晓川、<a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;" target="_blank">谢逸</a>、刘海天、史浩宇、张鑫宇等人，此外，<a href="https://www.lamda.nju.edu.cn/guolz/" style="text-decoration: none; color: inherit;" target="_blank">郭兰哲</a>、<a href="http://www.lamda.nju.edu.cn/chenzx/" style="text-decoration: none; color: inherit;" target="_blank">陈梓轩</a>、<a href="http://www.lamda.nju.edu.cn/zhouz/" style="text-decoration: none; color: inherit;" target="_blank">周植</a>、<a href="http://www.lamda.nju.edu.cn/jinyx/" style="text-decoration: none; color: inherit;" target="_blank">金苡萱</a>等人也参与了早期的原型研发。
|
||||
北冥坞研发团队人员包括：<a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;">周志华</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;">俞扬</a>、<a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;">谭志豪</a>、<a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;">刘建东</a>、<a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;">谭鹏</a>、<a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;">毕晓栋</a>、<a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;">郑钦城</a>、邹晓川、<a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;">谢逸</a>、刘海天、史浩宇、张鑫宇等人。
|
||||
|
||||
<!-- 系统总负责：<a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;" target="_blank">周志华</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;" target="_blank">俞扬</a>
|
||||
- 系统研发负责：<a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;" target="_blank">谭志豪</a>
|
||||
- 系统架构与开发：<a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;" target="_blank">刘建东</a>
|
||||
- 引擎架构与开发：<a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;" target="_blank">毕晓栋</a>
|
||||
- 引擎算法研发：<a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;" target="_blank">谭鹏</a>
|
||||
- 前端设计与开发：<a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;" target="_blank">郑钦城</a>
|
||||
<!-- - 系统总负责：<a href="http://cs.nju.edu.cn/zhouzh" style="text-decoration: none; color: inherit;">周志华</a>、<a href="http://www.lamda.nju.edu.cn/yuy" style="text-decoration: none; color: inherit;">俞扬</a>
|
||||
- 系统研发负责：<a href="http://www.lamda.nju.edu.cn/tanzh/" style="text-decoration: none; color: inherit;">谭志豪</a>
|
||||
- 系统架构与开发：<a href="http://www.lamda.nju.edu.cn/liujd/" style="text-decoration: none; color: inherit;">刘建东</a>
|
||||
- 引擎架构与开发：<a href="http://www.lamda.nju.edu.cn/bixd/" style="text-decoration: none; color: inherit;">毕晓栋</a>
|
||||
- 引擎算法研发：<a href="http://www.lamda.nju.edu.cn/tanp/" style="text-decoration: none; color: inherit;">谭鹏</a>
|
||||
- 前端设计与开发：<a href="http://www.lamda.nju.edu.cn/zhengqc/" style="text-decoration: none; color: inherit;">郑钦城</a>
|
||||
- 后端架构与开发：邹晓川
|
||||
- 算法开发与开源学件准备：<a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;" target="_blank">谢逸</a>、刘海天、史浩宇、张鑫宇 -->
|
||||
- 算法开发与开源学件准备：<a href="http://www.lamda.nju.edu.cn/xiey/" style="text-decoration: none; color: inherit;">谢逸</a>、刘海天、史浩宇、张鑫宇
|
||||
|
||||
此外，系统早期原型开发期间还有<a href="https://www.lamda.nju.edu.cn/guolz/" style="text-decoration: none; color: inherit;">郭兰哲</a>、<a href="http://www.lamda.nju.edu.cn/chenzx/" style="text-decoration: none; color: inherit;">陈梓轩</a>、<a href="http://www.lamda.nju.edu.cn/zhouz/" style="text-decoration: none; color: inherit;">周植</a>、<a href="http://www.lamda.nju.edu.cn/jinyx/" style="text-decoration: none; color: inherit;">金苡萱</a>参与。 -->
|
||||
|
|
@ -302,7 +302,6 @@ async function handleClickExport(): Promise<void> {
|
|||
username: filters.value.userName,
|
||||
email: filters.value.email,
|
||||
limit: pageSize.value,
|
||||
is_verified: filters.value.verifyStatus,
|
||||
page: _page - 1,
|
||||
}),
|
||||
})
|
||||
|
|
|
|||
|
|
@ -2,13 +2,7 @@ export default {
|
|||
Cover: {
|
||||
Beiming: "Beimingwu: A Learnware Dock System",
|
||||
Introduction:
|
||||
'Beimingwu is the first systematic open-source implementation of learnware dock system, providing a preliminary research platform for learnware studies. Developers worldwide can submit their models freely to the learnware dock. They can generate specifications for the model with the help of Beimingwu without disclosing their raw data, and then the model and specification can be assembled into a learnware, which will be accommodated in the learnware dock. Future users can solve their tasks by submitting their requirements and reusing helpful learnwares returned by Beimingwu, while also not disclosing their own data. It is anticipated that after Beimingwu accumulates millions of learnwares, an "emergent" behavior may occur: machine learning tasks that have never been specifically tackled may be solved by assembling and reusing some existing learnwares. For more in-depth information, you can access the complete paper here: ',
|
||||
PaperLinkDescription: "Paper Link",
|
||||
CodeRepo: {
|
||||
Description: "Beimingwu is completely open-source. The related code repositories can be accessed by clicking: ",
|
||||
Beimingwu: "System Frontend & Backend",
|
||||
Learnware: "System Engine & Toolkit",
|
||||
},
|
||||
"Beimingwu is the first systematic open-source implementation of learnware dock system, providing a preliminary research platform for learnware studies. Developers worldwide can submit their models freely to the learnware dock. They can generate specifications for the model with the help of Beimingwu without disclosing their raw data, and then the model and specification can be assembled into a learnware, which will be accommodated in the learnware dock. Future users can solve their tasks by submitting their requirements and reusing helpful learnwares returned by Beimingwu, while also not disclosing their own data. It is anticipated that after Beimingwu accumulates millions of learnwares, an \"emergent\" behavior may occur: machine learning tasks that have never been specifically tackled may be solved by assembling and reusing some existing learnwares.",
|
||||
Try: "Try It Out",
|
||||
Submit: "Be A Developer",
|
||||
LearnwareName: "Learnware",
|
||||
|
|
@ -35,8 +29,7 @@ export default {
|
|||
},
|
||||
Function: {
|
||||
Title: "What are the main functions of Beimingwu?",
|
||||
Description:
|
||||
"The Beimingwu learnware dock system, serving as a preliminary research platform for the learnware paradigm, systematically accomplishes the entire process of learnware from submission to deployment as follows:",
|
||||
Description: "The Beimingwu learnware dock system, serving as a preliminary research platform for the learnware paradigm, systematically accomplishes the entire process of learnware from submission to deployment as follows:",
|
||||
SearchAndDeploy: {
|
||||
Title: "Search and deploy learnwares: ",
|
||||
Description:
|
||||
|
|
@ -106,7 +99,6 @@ export default {
|
|||
Footer: {
|
||||
About: "About",
|
||||
Git: "GitLink",
|
||||
GitHub: "GitHub",
|
||||
ContactUs: "Contact Us",
|
||||
UserAgreement: "User Agreement",
|
||||
PrivacyPolicy: "Privacy Policy",
|
||||
|
|
|
|||
|
|
@ -6,13 +6,11 @@ export default {
|
|||
ConfirmPassword: "Confirm Password",
|
||||
SentEmail:
|
||||
"We have sent an email with a confirmation link to your email address. Please follow the link in the email.",
|
||||
NotReceived: "If you haven't received the registration email for a long time, you can contact bmwu-support{'@'}lamda.nju.edu.cn.",
|
||||
Resend: "Resend",
|
||||
Close: "Close",
|
||||
ReadAndAgree: "Read and Agree to",
|
||||
UserAgreement: "Agreement",
|
||||
PrivacyPolicy: "Privacy",
|
||||
OnlyEduEmail: "Due to limited system resources, we currently only support registration with educational email addresses. For non-educational email addresses, please contact bmwu-support{'@'}lamda.nju.edu.cn to apply.",
|
||||
Error: {
|
||||
UsernameAtLeast2Chars: "Username needs to be at least 2 characters.",
|
||||
UsernameAtMost20Chars: "Username cannot be more than 20 characters.",
|
||||
|
|
|
|||
|
|
@ -2,13 +2,7 @@ export default {
|
|||
Cover: {
|
||||
Beiming: "北冥坞：学件基座系统",
|
||||
Introduction:
|
||||
"北冥坞是学件的第一个系统性开源实现，为学件相关研究提供了一个初步科研平台。有分享意愿的开发者可自由提交模型，学件坞协助产生规约形成学件存放在学件坞中，开发者在这个过程中无需向学件坞泄露自己的训练数据。未来的用户可以通过向学件坞提交需求，在学件坞协助下查搜复用学件来完成自己的机器学习任务，且用户可以不向学件坞泄露自有数据。预计在学件坞拥有数以百万计的学件后，将可能出现“涌现”行为：以往没有专门开发过模型的机器学习任务，可能通过复用若干个现有学件而解决。如需了解更多信息，可点击查看完整论文: ",
|
||||
PaperLinkDescription: "论文链接",
|
||||
CodeRepo: {
|
||||
Description: "北冥坞所有代码完全开源，相关代码仓库可点击: ",
|
||||
Beimingwu: "系统前后端",
|
||||
Learnware: "系统引擎及工具包",
|
||||
},
|
||||
"北冥坞是学件的第一个系统性开源实现，为学件相关研究提供了一个初步科研平台。有分享意愿的开发者可自由提交模型，学件坞协助产生规约形成学件存放在学件坞中，开发者在这个过程中无需向学件坞泄露自己的训练数据。未来的用户可以通过向学件坞提交需求，在学件坞协助下查搜复用学件来完成自己的机器学习任务，且用户可以不向学件坞泄露自有数据。预计在学件坞拥有数以百万计的学件后，将可能出现“涌现”行为：以往没有专门开发过模型的机器学习任务，可能通过复用若干个现有学件而解决。",
|
||||
Try: "学件查搜",
|
||||
Submit: "学件上传",
|
||||
LearnwareName: "学件",
|
||||
|
|
@ -35,8 +29,7 @@ export default {
|
|||
},
|
||||
Function: {
|
||||
Title: "北冥坞系统的主要功能是什么？",
|
||||
Description:
|
||||
"北冥坞学件基座系统作为学件范式的初步科研平台，实现了学件从上传到部署的完整流程如下：",
|
||||
Description: "北冥坞学件基座系统作为学件范式的初步科研平台，实现了学件从上传到部署的完整流程如下：",
|
||||
SearchAndDeploy: {
|
||||
Title: "查搜部署学件：",
|
||||
Description:
|
||||
|
|
@ -106,7 +99,6 @@ export default {
|
|||
Footer: {
|
||||
About: "关于我们",
|
||||
Git: "GitLink",
|
||||
GitHub: "GitHub",
|
||||
ContactUs: "联系我们",
|
||||
UserAgreement: "用户协议",
|
||||
PrivacyPolicy: "隐私政策",
|
||||
|
|
|
|||
|
|
@ -5,13 +5,11 @@ export default {
|
|||
Password: "密码",
|
||||
ConfirmPassword: "确认密码",
|
||||
SentEmail: "我们向您的邮箱发送了一封确认邮件，请点击邮件中的链接完成注册。",
|
||||
NotReceived: "若长时间未收到注册邮件，可联系bmwu-support{'@'}lamda.nju.edu.cn。",
|
||||
Resend: "重新发送",
|
||||
Close: "关闭",
|
||||
ReadAndAgree: "我已阅读并同意",
|
||||
UserAgreement: "用户协议",
|
||||
PrivacyPolicy: "隐私政策",
|
||||
OnlyEduEmail: "由于系统资源有限，现仅支持教育邮箱注册。非教育邮箱可联系bmwu-support{'@'}lamda.nju.edu.cn 进行申请。",
|
||||
Error: {
|
||||
UsernameAtLeast2Chars: "用户名至少需要 2 个字符",
|
||||
UsernameAtMost20Chars: "用户名不能超过 20 个字符",
|
||||
|
|
|
|||
|
|
@ -167,28 +167,6 @@ watch(
|
|||
|
||||
<div class="mx-auto mt-6 max-w-7xl px-10 sm:px-20 md:px-40 lg:px-60">
|
||||
{{ t("Home.Cover.Introduction") }}
|
||||
<a
|
||||
:href="`https://arxiv.org/pdf/2401.14427.pdf`"
|
||||
class="text-white dark:text-white"
|
||||
target="_blank"
|
||||
>
|
||||
{{ t("Home.Cover.PaperLinkDescription") }}
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="mx-auto mt-1 max-w-7xl px-10 sm:px-20 md:px-40 lg:px-60">
|
||||
{{ t("Home.Cover.CodeRepo.Description") }}
|
||||
<a
|
||||
:href="`https://www.gitlink.org.cn/beimingwu`"
|
||||
class="text-white dark:text-white"
|
||||
target="_blank"
|
||||
>GitLink</a
|
||||
>, <a
|
||||
:href="`https://github.com/Learnware-LAMDA`"
|
||||
class="text-white dark:text-white"
|
||||
target="_blank"
|
||||
>Github</a
|
||||
>
|
||||
</div>
|
||||
|
||||
<div class="flex justify-center pt-10">
|
||||
|
|
|
|||
|
|
@ -19,12 +19,6 @@ const links = computed(() => [
|
|||
path: "https://www.gitlink.org.cn/beimingwu/beimingwu",
|
||||
tooltips: "",
|
||||
},
|
||||
{
|
||||
icon: "mdi-github",
|
||||
name: t("Home.Footer.GitHub"),
|
||||
path: "https://github.com/learnware-lamda/Beimingwu",
|
||||
tooltips: "",
|
||||
},
|
||||
{
|
||||
icon: "mdi-email",
|
||||
name: t("Home.Footer.ContactUs"),
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ const router = useRouter();
|
|||
const email = useField<string>({
|
||||
defaultValue: "",
|
||||
validate: (value) => {
|
||||
if (!/^[a-z.-_]+@([a-z0-9.-]+\.[a-z]+|localhost)$/i.test(value)) {
|
||||
if (!/^[a-z.-_]+@([a-z.-]+\.[a-z]+|localhost)$/i.test(value)) {
|
||||
return t("Login.Error.InvalidEmail");
|
||||
}
|
||||
return "";
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ const userName = useField<string>({
|
|||
const email = useField<string>({
|
||||
defaultValue: "",
|
||||
validate: (value) => {
|
||||
if (!/^[a-z.-_]+@[a-z0-9.-]+\.[a-z]+$/i.test(value)) return t("Register.Error.InvalidEmail");
|
||||
if (!/^[a-z.-_]+@[a-z.-]+\.[a-z]+$/i.test(value)) return t("Register.Error.InvalidEmail");
|
||||
if (value?.length > 50) return t("Register.Error.EmailAtMost50Chars");
|
||||
return "";
|
||||
},
|
||||
|
|
@ -141,12 +141,6 @@ onUnmounted(() => {
|
|||
<template #msg>
|
||||
<div class="mb-8 mt-6 text-lg">
|
||||
{{ t("Register.SentEmail") }}
|
||||
<div
|
||||
class="mb-8 mt-6 text-lg"
|
||||
style="margin-top: 0"
|
||||
>
|
||||
{{ t("Register.NotReceived") }}
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
<template #buttons>
|
||||
|
|
@ -205,11 +199,6 @@ onUnmounted(() => {
|
|||
{{ t("Register.Register") }}
|
||||
</div>
|
||||
</v-card-title>
|
||||
<v-card-text>
|
||||
<div class="mx-2">
|
||||
{{ t("Register.OnlyEduEmail") }}
|
||||
</div>
|
||||
</v-card-text>
|
||||
<v-card-text>
|
||||
<v-form>
|
||||
<v-text-field
|
||||
|
|
|
|||
Loading…
Reference in New Issue