Merge pull request #272 from Learnware-LAMDA/perf(docs)/learnware_upload
ferf(docs): add more details in learnware upload
|
|
@ -40,12 +40,16 @@ const en: LocaleSpecificConfig & {
|
|||
collapsed: true,
|
||||
items: [
|
||||
{
|
||||
text: "How to Prepare a Learnware?",
|
||||
text: "Learnware Preparation",
|
||||
link: "/en/user-guide/learnware-upload/prepare",
|
||||
},
|
||||
{
|
||||
text: "How to Upload a Learnware?",
|
||||
link: "/en/user-guide/learnware-upload/upload",
|
||||
text: "Upload via Web Interface",
|
||||
link: "/en/user-guide/learnware-upload/upload-web",
|
||||
},
|
||||
{
|
||||
text: "Upload via Client Interface",
|
||||
link: "/en/user-guide/learnware-upload/upload-client",
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
|
|||
|
|
@ -44,8 +44,12 @@ const zhCN: LocaleSpecificConfig & {
|
|||
link: "/zh-CN/user-guide/learnware-upload/prepare",
|
||||
},
|
||||
{
|
||||
text: "如何上传学件?",
|
||||
link: "/zh-CN/user-guide/learnware-upload/upload",
|
||||
text: "通过网页端上传",
|
||||
link: "/zh-CN/user-guide/learnware-upload/upload-web",
|
||||
},
|
||||
{
|
||||
text: "通过客户端上传",
|
||||
link: "/zh-CN/user-guide/learnware-upload/upload-client",
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Backend Dev Guide
|
||||
|
||||
## Backend Architecture
|
||||

|
||||

|
||||
|
||||
## Feature Development
|
||||
### Technology Stack
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ In Beimingwu, learnwares can be searched using both semantic information and sta
|
|||
|
||||
When searching with semantic information, you can fill in the information about your target learnware, and the system will search in the names and descriptions of learnwares. You can also filter by tags.
|
||||
|
||||

|
||||

|
||||
|
||||
When searching with statistical information, you need to submit the statistical information of the task. The tools we provide will generate approximate statistical information for your task locally while protecting your data privacy. With the following code, you can easily generate approximate statistical information for your task.
|
||||
|
||||
|
|
@ -22,11 +22,11 @@ spec.save("stat.json")
|
|||
|
||||
By uploading the JSON file containing statistical information, the system will match learnware with similar statistical information. You can download the learnware zip by clicking on the download button in the lower left corner of the learnware card.
|
||||
|
||||

|
||||

|
||||
|
||||
In some cases, combining multiple learnwares with similar statistical information may be more suitable for your task. The system will recommend these learnwares as a package to you. You can download all of them with the "Download All" button in the upper right corner.
|
||||
|
||||

|
||||

|
||||
|
||||
## Learnware Deployment
|
||||
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ As shown in the diagram below, the system workflow consists of two stages:
|
|||
- **Submitting Stage**: Developers voluntarily submit various learnwares to the learnware dock system, and the system conducts quality checks and further organization of these learnwares.
|
||||
- **Deploying Stage**: When users submit task requirements, the learnware dock system automatically selects whether to recommend a single learnware or a combination of multiple learnwares and provides efficient deployment methods. Whether it's a single learnware or a combination of multiple learnwares, the system offers convenient learnware reuse interfaces.
|
||||
|
||||

|
||||

|
||||
|
||||
### Specification World
|
||||
|
||||
|
|
@ -44,7 +44,7 @@ Specification is the core component of the learnware dock system, linking all pr
|
|||
|
||||
Learnwares from different feature/label spaces form numerous islands of specifications, and all these islands together constitute the "specification world" in the learnware dock system. In the specification world, if connections between different islands can be discovered and established, the corresponding islands of specification can be merged.
|
||||
|
||||

|
||||

|
||||
|
||||
When searching in the learnware dock system, the system first identifies specific islands of specifications based on semantic specifications in user requirements, and then it accurately recognizes learnwares on the specification islands through statistical specifications in user requirements.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,32 +1,8 @@
|
|||
# How to Upload a Learnwares?
|
||||
# Upload Learnware via Client Interface
|
||||
|
||||
In the Beimingwu system, you can upload a learnware both from the web interface and by using the `learnware` Python package.
|
||||
|
||||
Next, we will introduce two methods separately.
|
||||
|
||||
## Uploading through the Web Interface
|
||||
|
||||
Click the "Submit" button on the website's navigation bar at [Submit](https://www.bmwu.cloud/#/submit) to begin the process of uploading learnware.
|
||||
|
||||
The entire process is divided into the following 4 steps:
|
||||
1. Fill in the name of the learnware.
|
||||
2. Select tags for the learnware.
|
||||
3. Provide a description for the learnware.
|
||||
4. Upload the prepared learnware zip package.
|
||||
|
||||
The first three steps can be completed following the instructions on the website's pages, and the details for the fourth step can be found here: [How to Prepare Learnware](/zh-CN/user-guide/learnware-upload/prepare).
|
||||
|
||||
It's important to note that during the second step, "Select Tags":
|
||||
- If the data type is "Table," you need to specify the semantics of each dimension of the model's input data to make the uploaded learnware suitable for tasks with heterogeneous feature spaces.
|
||||
- If the task type is "Classification", you need to provide the semantics of model output labels (prediction labels start from 0), making the uploaded learnware suitable for classification tasks with heterogeneous output spaces.
|
||||
- If the task type is "Regression", you need to specify the semantics of each dimension of the model output, making the uploaded learnware suitable for regression tasks with heterogeneous output spaces.
|
||||
|
||||
If there are many dimensions, consider using a large language model to analyze the feature engineering code and generate semantics for each dimension.
|
||||
|
||||
## Uploading using the learnware Package
|
||||
|
||||
Apart from the web interface, the `learnware` package also provides an interface for uploading learnwares. First, you need to log in:
|
||||
In the Beimingwu system, learnware can be uploaded either through the web interface or by using the learnware Python package, which means uploading via a client.
|
||||
|
||||
Next, we will introduce how to upload using the learnware package, starting with the need to log in:
|
||||
```python
|
||||
from learnware.client import LearnwareClient, SemanticSpecificationKey
|
||||
|
||||
|
|
@ -37,7 +13,9 @@ client.login(email="your email", token="your token")
|
|||
|
||||
Where "email" is your registered email address in the system, and "token" is the token for accessing the learnware API, which can be generated in the web interface under "Personal Information - Client Token."
|
||||
|
||||
Next, you need to prepare semantic specifications, here is an example of a "Table Data" for a "Classification Task":
|
||||
## Prepare Semantic Specification
|
||||
|
||||
After successfully logging in, you need to prepare the semantic specification. Here is an example of a "Table Data" for a "Classification Task":
|
||||
|
||||
```python
|
||||
from learnware.specification import generate_semantic_spec
|
||||
|
|
@ -82,11 +60,18 @@ Please ensure that the input for semantic specification falls within the range g
|
|||
- "task_type" must be within `key=SemanticSpecificationKey.TASK_TYPE`.
|
||||
- "library_type" must be within `key=SemanticSpecificationKey.LIBRARY_TYPE`.
|
||||
- "scenarios" must be a subset of `key=SemanticSpecificationKey.SENARIOS`.
|
||||
- When "data_type" is "Table," you need to provide "Input Description."
|
||||
- When "task_type" is in `["Classification", "Regression"]`, you need to provide "Output Description."
|
||||
|
||||
Additionally, it's important to note:
|
||||
- If "data_type" is `"Table"`, you need to specify the semantics of each dimension of the model's input data to make the uploaded learnware suitable for tasks with heterogeneous feature spaces.
|
||||
- If "task_type" is `"Classification"`, you need to provide the semantics of model output labels (prediction labels start from 0), making the uploaded learnware suitable for classification tasks with heterogeneous output spaces.
|
||||
- If "task_type" is `"Regression"`, you need to specify the semantics of each dimension of the model output, making the uploaded learnware suitable for regression tasks with heterogeneous output spaces.
|
||||
|
||||
## Upload Learnware
|
||||
|
||||
Finally, fill in the semantic specification and the path to the learnware zip package to complete the learnware upload.
|
||||
|
||||
The preparation of the learnware zip package can be referred to: [How to Prepare a Learnware?](/en/user-guide/learnware-upload/prepare)
|
||||
|
||||
Before uploading, remember to validate your learnware. Here's an example code:
|
||||
|
||||
```python
|
||||
|
|
@ -106,4 +91,4 @@ learnware_id = client.upload_learnware(
|
|||
|
||||
Once your learnware is successfully uploaded, you can find it under "Personal Information - My Learnware."
|
||||
|
||||
After uploading, the backend will perform a check on the learnware. You can check the validation status by clicking on the learnware. Once the check passes, the "Unverified" label will disappear, and your uploaded learnware will appear in the learnware.
|
||||
After uploading, the backend will perform a check on the learnware. Once the check passes, the learnware's tag will change to "SUCCESS", and your uploaded learnware will appear in the system.
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# Upload Learnwares via Web Interface
|
||||
|
||||
In the Beimingwu system, you can upload a learnware both from the web interface and by using the `learnware` Python package.
|
||||
|
||||
Next, we will explain how to upload learnware using the web interface.
|
||||
|
||||
## Step 1: Fill in the Learnware Name
|
||||
|
||||
Click the "Submit" button on the website's navigation bar at [Submit](https://www.bmwu.cloud/#/submit) to begin the process of uploading learnware.
|
||||
|
||||
The first step is "Fill in the Learnware Name", as shown on the specific page below:
|
||||
|
||||

|
||||
|
||||
It should be noted here that the "Learnware Name" needs to be between 5 to 50 characters in length.
|
||||
|
||||
## Step 2:Choose the Learnware Tags
|
||||
|
||||
After filling in the learnware name, proceed to the second step: choose the learnware tags.
|
||||
|
||||
First, select the "Data type" involved in the learnware. If `Table` is chosen, you need to specify the semantics of each dimension of the model's input data to make the uploaded learnware suitable for tasks with heterogeneous feature spaces.
|
||||
|
||||
If there are many dimensions, consider using a large language model to analyze the feature engineering code and generate semantics for each dimension.
|
||||
|
||||

|
||||
|
||||
Next is selecting the "Task type" involved in the learnware. If `Classification` is chosen, you need to provide the semantics of model output labels (prediction labels start from 0), making the uploaded learnware suitable for classification tasks with heterogeneous output spaces.
|
||||
|
||||
If `Regression` is chosen, you need to specify the semantics of each dimension of the model output, making the uploaded learnware suitable for regression tasks with heterogeneous output spaces.
|
||||
|
||||

|
||||
|
||||
Finally, select the "Library type", "Scenario", and "License" for the learnware, based on the specific content of the tags.
|
||||
|
||||

|
||||
|
||||
## Step 3:Fill in the Learnware Description
|
||||
|
||||
After selecting the learnware tags, you can proceed to the third step: fill in the learnware description, which supports markdown format.
|
||||
|
||||
Additionally, it's important to note that the "Learnware Description" should be between 10 to 10,000 characters in length.
|
||||
|
||||

|
||||
|
||||
## Step 4:Upload the Learnware zip Package
|
||||
|
||||
After completing the learnware description, you can proceed to the final step: upload the learnware zip package.
|
||||
|
||||
The preparation of the learnware zip package can be referred to: [How to Prepare a Learnware?](/en/user-guide/learnware-upload/prepare)
|
||||
|
||||

|
||||
|
||||
Once your learnware is successfully uploaded, you can find it under "Personal Information - My Learnware."
|
||||
|
||||
After uploading, the backend will perform a check on the learnware. Once the check passes, the learnware's tag will change to "SUCCESS", and your uploaded learnware will appear in the system.
|
||||
|
Before Width: | Height: | Size: 419 KiB After Width: | Height: | Size: 419 KiB |
|
Before Width: | Height: | Size: 348 KiB After Width: | Height: | Size: 348 KiB |
|
Before Width: | Height: | Size: 25 KiB |
|
Before Width: | Height: | Size: 579 KiB After Width: | Height: | Size: 579 KiB |
|
Before Width: | Height: | Size: 2.0 MiB After Width: | Height: | Size: 2.0 MiB |
|
After Width: | Height: | Size: 426 KiB |
|
After Width: | Height: | Size: 405 KiB |
|
After Width: | Height: | Size: 347 KiB |
|
After Width: | Height: | Size: 303 KiB |
|
After Width: | Height: | Size: 167 KiB |
|
After Width: | Height: | Size: 151 KiB |
|
Before Width: | Height: | Size: 363 KiB |
|
Before Width: | Height: | Size: 294 KiB |
|
Before Width: | Height: | Size: 202 KiB |
|
After Width: | Height: | Size: 68 KiB |
|
After Width: | Height: | Size: 66 KiB |
|
After Width: | Height: | Size: 211 KiB |
|
After Width: | Height: | Size: 189 KiB |
|
After Width: | Height: | Size: 144 KiB |
|
After Width: | Height: | Size: 123 KiB |
|
After Width: | Height: | Size: 168 KiB |
|
After Width: | Height: | Size: 160 KiB |
|
After Width: | Height: | Size: 178 KiB |
|
After Width: | Height: | Size: 168 KiB |
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 94 KiB |
|
|
@ -1,7 +1,7 @@
|
|||
# 北冥坞系统后端开发指南
|
||||
|
||||
## 后端架构图
|
||||

|
||||

|
||||
## 功能开发
|
||||
### 技术栈
|
||||
Flask + SqlAlchemy + Redis
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
|
||||
通过语义信息查搜时,您可以填写目标学件的信息,系统将在学件的名称和描述字段中进行查搜;你也可以通过标签进行筛选。
|
||||
|
||||

|
||||

|
||||
|
||||
通过统计信息进行查搜时,您需要提交任务的统计信息。我们提供的工具将在保护您数据隐私的情况下在本地为您生成任务的近似统计信息。通过下列代码,您可以轻松生成任务的近似统计信息。
|
||||
|
||||
|
|
@ -23,12 +23,12 @@ spec.save("stat.json")
|
|||
|
||||
通过上传统计信息的json文件,系统会匹配统计信息接近的学件。您可以通过学件卡片左下角的下载按钮进行学件zip包的下载。
|
||||
|
||||

|
||||

|
||||
|
||||
|
||||
某些情况下,多个学件组合在一起的统计信息会更加接近您的任务,系统会将这些学件打包推荐给您。您可以通过右上角的“下载全部”按钮进行一键下载。
|
||||
|
||||

|
||||

|
||||
|
||||
## 学件部署
|
||||
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@
|
|||
- **提交阶段**:开发者自发地提交各式各样的学件到**学件基座系统**,而系统会对这些学件进行质量检查和进一步的组织。
|
||||
- **部署阶段**:当用户提交任务需求后,**学件基座系统**会根据学件规约推荐对用户任务有帮助的学件并指导用户进行部署和复用。
|
||||
|
||||

|
||||

|
||||
|
||||
### 规约世界
|
||||
|
||||
|
|
@ -58,7 +58,7 @@
|
|||
|
||||
来自不同特征/标记空间的学件,构成众多的规约岛屿,所有规约岛屿共同构成学件基座系统中的**规约世界**。在规约世界中,如果能够发现并建立不同岛屿之间的联系,那么相对应的规约岛屿将可以进行合并。
|
||||
|
||||

|
||||

|
||||
|
||||
学件基座系统在查搜时,首先通过用户需求中的语义规约定位到具体的规约岛屿,再通过用户需求中的统计规约对规约岛上的学件进行精确识别。
|
||||
|
||||
|
|
|
|||
|
|
@ -1,32 +1,8 @@
|
|||
# 如何上传学件?
|
||||
# 通过客户端上传学件
|
||||
|
||||
在北冥坞系统中,学件既可以从网页端上传,也可以使用 `learnware` Python 包进行上传。
|
||||
在北冥坞系统中,学件既可以从网页端上传,也可以使用 `learnware` Python 包进行上传,即使用客户端上传。
|
||||
|
||||
接下来,我们将分别对这两种方式进行介绍。
|
||||
|
||||
## 通过网页端上传
|
||||
|
||||
点击网站导航栏处的「[提交](https://www.bmwu.cloud/#/submit)」按钮,即可开始学件上传。
|
||||
|
||||
整个流程分为以下 4 步:
|
||||
1. 填写学件的名称
|
||||
2. 选择学件的标签
|
||||
3. 填写学件的描述
|
||||
4. 上传准备好的学件 zip 包
|
||||
|
||||
其中前 3 步可根据网站页面指引进行操作,第 4 步可具体查看:[如何准备一个学件?](/zh-CN/user-guide/learnware-upload/prepare)
|
||||
|
||||
需要注意的是,在第 2 步「选择标签」的过程中:
|
||||
- 如果数据类型为「表格」,则需要填写模型输入数据的每一维特征语义,使上传的学件可用于异构特征空间的任务;
|
||||
- 如果任务类型为「分类」,则需要填写模型输出标记的语义(预测标记从 0 开始编号),使上传的学件可用于异构输出空间的分类任务;
|
||||
- 如果任务类型为「回归」,则需要填写模型输出的每一维语义,使上传的学件可用于异构输出空间的回归任务;
|
||||
|
||||
如果维度过多,可考虑使用大语言模型。通过分析特征工程的代码,生成各维度的语义。
|
||||
|
||||
|
||||
## 使用 learnware 包上传
|
||||
|
||||
除网页端外,learnware 包也提供学件上传的接口,首先需要登录:
|
||||
接下来,我们将对使用 `learnware` 包上传的部分进行介绍,首先需要登录:
|
||||
```py
|
||||
from learnware.client import LearnwareClient, SemanticSpecificationKey
|
||||
|
||||
|
|
@ -35,7 +11,10 @@ client = LearnwareClient()
|
|||
client.login(email="your email", token="your token")
|
||||
```
|
||||
其中 email 为系统的注册邮箱,token 为访问学件 API 的令牌,可在网页端「个人信息 - 客户端令牌」处生成。
|
||||
随后需要准备语义规约,此处以「表格数据」的「分类任务」为例:
|
||||
|
||||
## 准备语义规约
|
||||
|
||||
成功登录后需要准备语义规约,此处以「表格数据」的「分类任务」为例:
|
||||
```py
|
||||
from learnware.specification import generate_semantic_spec
|
||||
|
||||
|
|
@ -77,12 +56,19 @@ semantic_spec = generate_semantic_spec(
|
|||
- data\_type 必须在 `key=SemanticSpecificationKey.DATA_TYPE` 对应的结果中;
|
||||
- task\_type 必须在 `key=SemanticSpecificationKey.TASK_TYPE` 对应的结果中;
|
||||
- library\_type 必须在 `key=SemanticSpecificationKey.LIBRARY_TYPE` 对应的结果中;
|
||||
- scenarios 必须为 `key=SemanticSpecificationKey.SENARIOES` 对应结果的子集;
|
||||
- 当 data\_type 为 `"Table"` 时,需要填写「输入描述」;
|
||||
- 当 task\_type 在 `["Classification", "Regression"]` 中时,需要填写「输出描述」。
|
||||
- scenarios 必须为 `key=SemanticSpecificationKey.SENARIOES` 对应结果的子集。
|
||||
|
||||
另外,需要注意:
|
||||
- 如果 data\_type 为 `"Table"`,则需要填写模型输入数据的每一维特征语义,使上传的学件可用于异构特征空间的任务;
|
||||
- 如果 task\_type 为 `"Classification"`,则需要填写模型输出标记的语义(预测标记从 0 开始编号),使上传的学件可用于异构输出空间的分类任务;
|
||||
- 如果 task\_type 为 `"Regression"`,则需要填写模型输出的每一维语义,使上传的学件可用于异构输出空间的回归任务;
|
||||
|
||||
## 学件上传
|
||||
|
||||
最后,填写语义规约和学件 zip 包路径,即可实现学件上传。
|
||||
|
||||
学件 zip 包的准备可查看:[如何准备一个学件?](/zh-CN/user-guide/learnware-upload/prepare)
|
||||
|
||||
记得在上传前先对学件进行验证,代码示例如下:
|
||||
```py
|
||||
# Prepare your learnware zip file
|
||||
|
|
@ -101,4 +87,4 @@ learnware_id = client.upload_learnware(
|
|||
|
||||
学件上传成功后,可以在「个人信息 - 我的学件」处看到上传的学件。
|
||||
|
||||
学件上传后,后台会对学件进行检查。点击学件,可在「验证状态」处查看。检查通过后,学件的 `Unverified` 标签会消失,且上传的学件会在系统中出现。
|
||||
学件上传后,后台会对学件进行检查。检查通过后,学件的标签将变为「验证成功」,且上传的学件会在系统中出现。
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# 通过网页端上传学件
|
||||
|
||||
在北冥坞系统中,学件既可以从网页端上传,也可以使用 `learnware` Python 包进行上传。
|
||||
|
||||
接下来,我们将介绍如何使用网页端上传学件。
|
||||
|
||||
## 步骤 1:填写学件名称
|
||||
|
||||
点击网站导航栏处的「[提交](https://www.bmwu.cloud/#/submit)」按钮,即可开始学件上传。
|
||||
|
||||
首先是「第一步 - 填写学件名称」,具体页面如下所示:
|
||||
|
||||

|
||||
|
||||
此处需要注意「学件名称」需要控制在 5 至 50 个字符之间。
|
||||
|
||||
## 步骤 2:选择学件标签
|
||||
|
||||
填写完学件名称后,进入第二步:填写学件标签。
|
||||
|
||||
首先是选择学件所涉及的「数据类型」,此处如果选择「表格」,则需要填写模型输入数据的每一维特征语义,使上传的学件可用于异构特征空间的任务。
|
||||
|
||||
如果维度过多,可考虑使用大语言模型。通过分析特征工程的代码,生成各维度的语义。
|
||||
|
||||

|
||||
|
||||
随后是选择学件所涉及的「任务类型」,此处如果选择「分类」,则需要填写模型输出标记的语义(预测标记从 0 开始编号),使上传的学件可用于异构输出空间的分类任务。
|
||||
|
||||
如果选择「回归」,则需要填写模型输出的每一维语义,使上传的学件可用于异构输出空间的回归任务。
|
||||
|
||||

|
||||
|
||||
最后是选择学件的「库类型」、「应用场景」以及「开源协议」,根据具体标签内容选择即可。
|
||||
|
||||

|
||||
|
||||
## 步骤 3:填写学件描述
|
||||
|
||||
选择完学件标签后,即可进入第三步:填写学件描述,此处描述内容支持 `markdown` 格式。
|
||||
|
||||
另外需要注意「学件描述」需要控制在 10 至 10000 个字符之间。
|
||||
|
||||

|
||||
|
||||
## 步骤 4:上传学件 zip 包
|
||||
|
||||
填写完学件描述后,即可进入最后一步:上传学件 zip 包。
|
||||
|
||||
学件 zip 包的准备可查看:[如何准备一个学件?](/zh-CN/user-guide/learnware-upload/prepare)
|
||||
|
||||

|
||||
|
||||
学件上传成功后,可以在「个人信息 - 我的学件」处看到上传的学件。
|
||||
|
||||
学件上传后,后台会对学件进行检查。检查通过后,学件的标签将变为「验证成功」,且上传的学件会在系统中出现。
|
||||