JointCloud
/
SchedulingSimulator
1
0
Fork 2
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

Compare commits

base: JointCloud:master
Branches Tags
JointCloud:master
JointCloud:dev0322
JointCloud:dev0319
JointCloud:dev_zhouqj
JointCloud:ai4m_test
JointCloud:dev_wq
baohan:master
..
compare: baohan:master
Branches Tags
baohan:master
JointCloud:master
JointCloud:dev0322
JointCloud:dev0319
JointCloud:dev_zhouqj
JointCloud:ai4m_test
JointCloud:dev_wq

No commits in common. "master" and "master" have entirely different histories.
master ... master

30 changed files with 3996 additions and 5410 deletions
Show Stats Expand all files Collapse all files

158
README.md
View File

@ -1,158 +1,2 @@
# SchedulingSimulator 调度模拟器
# SchedulingSimulator
SchedulingSimulator 是一个任务调度模拟器，用于在不同计算环境中提交和调度任务。
## 功能特性
- **AI任务调度**：支持AI训练任务的自动与手动提交
- **云任务调度**：支持阿里云、腾讯云等云平台任务提交
- **HPC任务调度**：支持高性能计算任务如BWA、Hashcat、LAMMPS等的调度
- **智能调度策略**：支持基于资源、价格和数据位置的调度策略
- **集群资源管理**：通过MySQL数据库管理集群资源和任务状态
- **并发任务处理**：使用线程池实现任务并发提交和状态监控
## 系统架构
```
SchedulingSimulator/
├── SchedulingSimulator/ # 主程序目录
│ ├── main.py # 程序入口
│ └── threads.py # 线程实现（任务提交和状态监控）
├── config/ # 配置管理
│ ├── config.py # 全局配置和状态定义
│ ├── database.py # 数据库配置加载
│ ├── db_manager.py # 数据库操作管理
│ ├── cache_manager.py # 缓存管理
│ └── ...
├── scheduler/ # 调度器
│ ├── scheduler.py # 主调度器
│ ├── resource_scheduler.py # 资源调度策略
│ ├── price_scheduler.py # 价格调度策略
│ └── data_scheduler.py # 数据位置调度策略
├── task/ # 任务管理
│ ├── task_manager.py # 任务管理器
│ ├── ai_task_submit.py # AI任务提交
│ ├── cloud_task_submit.py # 云任务提交
│ ├── hpc_task_submit.py # HPC任务提交
│ └── ...
├── TaskSubmissionScript/ # 任务提交脚本示例
└── mysql/ # 数据库相关文件
├── simulator.sql # 数据库初始化脚本
└── simulator_new.sql # 新版数据库初始化脚本
```
## 技术选型
- **编程语言**: Python 3.x
- **数据库**: MySQL 8.0+
- **并发模型**: Python ThreadPoolExecutor
- **依赖库**:
- requests
- mysql-connector-python
- pyyaml
## 数据库配置
系统使用MySQL数据库存储配置信息和任务状态。
### 数据库表结构
- `algorithm_mapping`: 存储算法映射信息
- `cluster_resources`: 存储集群资源信息
- `api_config`: 存储API配置信息
- `cluster_prices`: 存储集群价格信息
- `file_mapping`: 存储文件映射信息
- `task_templates`: 存储任务模板信息
### 数据库配置参数
在 [config/db_manager.py](file:///C:/Users/Administrator/PycharmProjects/SchedulingSimulator/config/db_manager.py) 文件中修改以下参数：
```python
DB_CONFIG = {
'host': 'localhost',
'database': 'scheduling_simulator',
'user': '',
'password': ''
}
```
### 首次使用初始化
1. 确保MySQL服务正在运行
2. 创建数据库：
```sql
CREATE DATABASE scheduling_simulator;
```
3. 导入数据库结构和初始数据：
```bash
mysql -u root -p scheduling_simulator < mysql/simulator.sql
```
## 依赖安装
```bash
pip install requests mysql-connector-python pyyaml
```
或者使用requirements.txt（如果存在）：
```bash
pip install -r requirements.txt
```
## 使用方法
1. 安装依赖：
```bash
pip install requests mysql-connector-python pyyaml
```
2. 配置数据库连接参数
3. 初始化数据库（首次使用）：
```sql
CREATE DATABASE scheduling_simulator;
```
```bash
mysql -u root -p scheduling_simulator < mysql/simulator.sql
```
4. 运行主程序：
```bash
python SchedulingSimulator/main.py
```
## 工作流程
1. 系统启动时从数据库加载配置信息和任务模板
2. 根据任务模板生成具体任务实例
3. 启动任务提交线程和状态监控线程
4. 任务提交线程根据任务状态和调度策略选择合适的集群
5. 将任务提交到选中的集群
6. 状态监控线程定期查询已提交任务的状态
7. 根据任务执行结果更新数据库中的任务状态和集群资源
## 调度策略
系统支持三种调度策略：
1. **资源优先策略 (resource)**：选择资源最充足的集群
2. **价格优先策略 (price)**：选择成本最低的集群
3. **数据位置策略 (data)**：选择数据所在位置的集群
## 任务类型
系统支持三种任务类型：
1. **AI任务**：机器学习训练任务
2. **云任务**：云平台部署任务
3. **HPC任务**：高性能计算任务
## 扩展性
系统具有良好的扩展性，可以通过以下方式添加新功能：
1. 添加新的调度策略：实现新的调度算法并集成到调度器中
2. 支持新的任务类型：创建新的任务提交模块
3. 集成新的云平台：扩展云任务提交模块

79
SchedulingSimulator/main.py
View File

@ -1,79 +0,0 @@
import logging
# 导入数据库管理模块
# 用于获取调度后的集群 ID
# 新增导入不同任务类型的提交方法
# -------------------------- 全局配置与常量定义 --------------------------
# 日志配置
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
from config.config import logger
from config.database import load_db_config, init_dataset_map
from task.task_manager import load_tasks_to_queue, generate_task_templates
from threads import TaskMonitorThread, TaskSubmitThread
def init_system() -> None:
"""
初始化系统（使用数据库配置替换YAML配置）
"""
logger.info("开始初始化系统...")
# 从数据库加载配置（替换原有的YAML加载方式）
load_db_config()
# 初始化数据集映射
init_dataset_map()
logger.info("系统初始化完成")
def main() -> None:
"""主函数"""
# 初始化系统，使用数据库配置
init_system()
# 生成任务模板
templates = generate_task_templates()
logger.info(f"生成任务模板 {len(templates)} 个")
# 添加任务到队列
load_tasks_to_queue(templates)
from config.config import task_map
logger.info(f"添加任务到队列完成，当前任务数: {len(task_map)}")
# # 启动任务监控线程
monitor_thread = TaskMonitorThread()
monitor_thread.start()
# 启动任务提交线程
submit_thread = TaskSubmitThread(max_workers=1)
submit_thread.start()
# 等待所有线程完成
try:
# 等待提交线程完成
submit_thread.join()
logger.info("所有任务提交完成")
# # 停止监控线程
monitor_thread.join()
# logger.info("任务监控完成")
except KeyboardInterrupt:
# logger.info("收到中断信号，正在停止所有线程...")
submit_thread.stop()
# monitor_thread.stop()
submit_thread.join(timeout=5)
monitor_thread.join(timeout=5)
logger.info("线程已停止")
# -------------------------- 主程序 --------------------------
if __name__ == "__main__":
main()

264
SchedulingSimulator/threads.py
View File

@ -1,264 +0,0 @@
import concurrent.futures
import threading
from datetime import datetime
from typing import Dict
from config.auth import get_token
from scheduler.scheduler import select_cluster
from config.config import logger, TASK_STATUS, task_map_lock
from task.task_manager import submit_single_task, query_third_party_task_status
from config.db_manager import update_cluster_resources, load_config_from_db, update_task_all_fields
class TaskMonitorThread(threading.Thread):
"""监控线程：定时查询任务状态并更新"""
def __init__(self, check_interval: int = 30):
super().__init__(name="TaskMonitorThread")
self.check_interval = check_interval
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"监控线程启动 | 监控间隔: {self.check_interval}秒")
token = get_token()
while not self._stop_event.is_set():
# 从数据库加载任务信息而不是使用内存中的task_map
config = load_config_from_db()
task_templates = config.get("task_templates", [])
tasks = []
# 为每个任务模板添加必要的运行时字段
for template in task_templates:
task = template.copy()
# task.setdefault("status", TASK_STATUS["SUBMITTED"])
task.setdefault("fail_count", 0)
task.setdefault("max_fail_threshold", 3)
# task.setdefault("third_party_task_id", None)
tasks.append(task)
for task in tasks:
current_status = task["status"]
if current_status in [TASK_STATUS["SUBMITTED"], TASK_STATUS["RETRY_EXHAUSTED"], TASK_STATUS["SUCCEED"]]:
continue
else:
third_party_info = query_third_party_task_status(task["third_party_task_id"], token)
# 根据third_party_info的结构获取任务状态
sub_task_infos = third_party_info.get("subTaskInfos", [])
if not sub_task_infos:
continue
if sub_task_infos:
third_party_status = sub_task_infos[0].get("status")
else:
third_party_status = None
#云算任务状态为Running则被视为完成
if (task["type"] == "Cloud" and third_party_status == "Running") or third_party_status in ["Succeeded", "Completed","Deleted"]:
task["status"] = TASK_STATUS["SUCCEED"]
# 获取结束时间
if task["type"] == "Cloud" :
task["success_time"] = datetime.now()
else:
task["success_time"] = third_party_info.get("endTime", "")
if update_task_all_fields(task['task_name'], status=TASK_STATUS["SUCCEED"],
third_party_task_id=third_party_info.get("taskId"),
end_time=task['success_time']):
logger.info(
f"成功更新任务 {task['task_name']} 在数据库中的状态为 {TASK_STATUS['SUCCEED']}")
else:
logger.error(f"更新任务 {task['task_name']} 在数据库中的状态失败")
# 任务成功完成，确保资源已正确扣减
cluster_id = task.get("cluster_id")
if cluster_id:
# 从数据库重新加载配置以获取最新状态
from config.database import load_db_config
load_db_config()
logger.info(f"任务 {task['task_name']} 成功完成，已确认资源状态")
elif third_party_status in ["Failed", "Saved"]:
task["status"] = TASK_STATUS["FAILED"]
if task["fail_count"] >= task["max_fail_threshold"]:
task["status"] = TASK_STATUS["RETRY_EXHAUSTED"]
if update_task_all_fields(task['task_name'], status=TASK_STATUS["RETRY_EXHAUSTED"]):
logger.info(
f"成功更新任务 {task['task_name']} 在数据库中的状态为 {TASK_STATUS['RETRY_EXHAUSTED']}")
continue
else:
logger.error(f"更新任务 {task['task_name']} 在数据库中的状态失败")
continue
task["fail_count"] += 1
if update_task_all_fields(task['task_name'], status=TASK_STATUS["SUBMITTED"],fail_count=task["fail_count"]):
logger.info(f"任务 {task['task_name']} 执行失败，已经更新状态为 {TASK_STATUS['SUBMITTED']}")
# 任务失败，需要恢复资源
cluster_id = task.get("cluster_id")
task_resource = task.get("resource", {})
if cluster_id:
# 计算需要恢复的资源量
resource_changes = {}
for res_type in ["CPU", "MEMORY"]:
required = task_resource.get(res_type, 0)
resource_changes[res_type] = required
# 处理加速器资源
acc_type = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
if acc_type:
required_count = task_resource.get(acc_type, 0)
resource_changes[acc_type] = required_count
# 更新数据库中的集群资源
if update_cluster_resources(cluster_id, resource_changes):
logger.info(f"任务失败，成功从数据库恢复集群 {cluster_id} 的资源: {resource_changes}")
else:
logger.error(f"任务失败，从数据库恢复集群 {cluster_id} 的资源失败: {resource_changes}")
continue
else:
logger.error(f"更新任务 {task['task_name']} 在数据库中的状态失败")
continue
# 检查是否所有任务都已完成
all_completed = True
for task in tasks:
if task["status"] in [TASK_STATUS["SUBMITTED"], TASK_STATUS["SUBMITTING"]]:
all_completed = False
break
if task["status"] == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
all_completed = False
break
if all_completed:
logger.info("所有任务已完成")
self.stop()
self._stop_event.wait(self.check_interval)
logger.info("监控线程结束")
def stop(self) -> None:
self._stop_event.set()
class TaskSubmitThread(threading.Thread):
"""提交线程：按状态判断是否提交"""
def __init__(self, max_workers: int = 3):
super().__init__(name="TaskSubmitThread")
self.max_workers = max_workers
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"提交线程启动 | 并发数: {self.max_workers}")
# 导入缓存管理器
from config.cache_manager import load_config_from_db
token = get_token()
while not self._stop_event.is_set():
# 从缓存获取配置信息而不是每次都访问数据库
config = load_config_from_db()
task_templates = config.get("task_templates", [])
# 筛选待提交任务
pending_tasks = []
for template in task_templates:
task = template.copy()
# 添加运行时需要的字段
task.setdefault("fail_count", 0)
task.setdefault("max_fail_threshold", 3)
task.setdefault("third_party_task_id", None)
status = task["status"]
if status == TASK_STATUS["SUBMITTED"]:
pending_tasks.append(task)
elif status == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
pending_tasks.append(task)
elif status == TASK_STATUS["FAILED"]:
logger.info(f"任务 {task['task_name']} 失败次数超限，停止提交")
# 并发提交 - 只提交状态为待提交的任务
if pending_tasks:
with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:
# 从数据库中重新读取任务状态，确保获取最新状态
fresh_config = load_config_from_db()
fresh_task_templates = {task["task_name"]: task for task in fresh_config.get("task_templates", [])}
# 过滤出状态仍为"待提交"的任务
tasks_to_submit = []
for task in pending_tasks:
# 从数据库中获取该任务的最新状态
fresh_task = fresh_task_templates.get(task["task_name"])
if fresh_task and (fresh_task["status"] == TASK_STATUS["SUBMITTED"] or
(fresh_task["status"] == TASK_STATUS["FAILED"] and task["fail_count"] < task[
"max_fail_threshold"])):
# 使用数据库中的最新状态更新任务
task.update(fresh_task)
tasks_to_submit.append(task)
if tasks_to_submit:
futures = {executor.submit(self.commit_task, task, token): task for task in tasks_to_submit}
for future in concurrent.futures.as_completed(futures):
task = futures[future]
try:
future.result()
except Exception as e:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"提交异常: {str(e)}"
logger.error(f"任务 {task['task_name']} 提交异常 | 错误: {str(e)}")
# 检查是否所有任务都已完成
all_completed = True
for template in task_templates:
task = template.copy()
if task["fail_count"] !=0 and task["fail_count"] !=task["max_fail_threshold"]:
all_completed = False
break
if task["status"] == TASK_STATUS["SUBMITTED"] or task["status"] == TASK_STATUS["SUBMITTING"] :
all_completed = False
break
if all_completed:
logger.info("所有任务已完成，提交线程退出")
self.stop()
break
self._stop_event.wait(5) # 每5秒检查一次任务状态
logger.info("提交线程结束")
def commit_task(self, task: Dict, token) -> None:
"""提交任务入口：基于映射选择集群"""
logger.info(f"开始为任务 {task['task_name']} 选择集群")
cluster_id = select_cluster(task)
logger.info(f"任务 {task['task_name']} 集群选择结果: {cluster_id}")
if not cluster_id:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = "无可用集群"
logger.error(f"任务 {task['task_name']} 提交失败：无可用集群")
return
# 标记为提交中
logger.info(f"准备标记任务 {task['task_name']} 为提交中状态")
with task_map_lock:
task["status"] = TASK_STATUS["SUBMITTING"]
task["cluster_id"] = cluster_id
logger.info(
f"任务 {task['task_name']} 开始提交至集群 {cluster_id}（策略: {task['strategy']}）| 任务类型: {task['type']}")
# 执行提交
logger.info(f"开始执行任务 {task['task_name']} 的提交操作")
submit_success = submit_single_task(task, token)
# logger.info(f"任务 {task['task_name']} 提交结果: {submit_success}")
# if submit_success:
# # 任务提交成功，更新数据库中的状态为SUBMITTING
# from config.db_manager import update_task_status, update_task_all_fields
# if update_task_all_fields(task['task_name'], status=TASK_STATUS["SUBMITTING"],third_party_task_id=task['third_party_task_id'], commit_time=datetime.now()):
# logger.info(f"成功更新任务 {task['task_name']} 在数据库中的状态为 {TASK_STATUS['SUBMITTING']}")
# else:
# logger.error(f"更新任务 {task['task_name']} 在数据库中的状态失败")
# else:
# logger.warning(f"任务 {task['task_name']} 提交失败，等待重试（当前次数：{task['fail_count']}）")
def stop(self) -> None:
self._stop_event.set()

799
commit_task_jointcloud_0716.py Normal file
View File

@ -0,0 +1,799 @@
import concurrent.futures
import time
import logging
import threading
from uuid import uuid4
from typing import Dict, List, Optional
import requests
import os
import json
# -------------------------- 全局配置与常量定义 --------------------------
# 日志配置
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
logger = logging.getLogger(__name__)
# 任务状态定义
TASK_STATUS = {
"SUBMITTED": "待提交", # 初始状态
"SUBMITTING": "提交中", # 提交过程中
"SUCCEED": "提交成功", # 第三方确认成功
"FAILED": "提交失败" # 第三方确认失败
}
# 全局任务字典（key=target_id，value=任务详情）
task_map: Dict[str, Dict] = {}
task_map_lock = threading.Lock() # 任务字典线程锁
# API配置（新增任务详情查询接口）
API_CONFIG = {
"login": {
"url": "http://119.45.255.234:30180/jcc-admin/admin/login",
"timeout": 10
},
"create_package": {
"url": "http://119.45.255.234:30180/jsm/jobSet/createPackage",
"timeout": 15
},
"upload_file": {
"url": "http://119.45.255.234:30180/jcs/object/upload",
"timeout": 300
},
"notify_upload": {
"url": "http://119.45.255.234:30180/jsm/jobSet/notifyUploaded",
"timeout": 15
},
"bind_cluster": {
"url": "http://119.45.255.234:30180/jsm/jobSet/binding",
"timeout": 15
},
"query_binding": {
"url": "http://119.45.255.234:30180/jsm/jobSet/queryBinding",
"timeout": 15
},
"submit_task": {
"url": "http://119.45.255.234:30180/jsm/jobSet/submit",
"timeout": 15
},
"task_details": { # 新增任务详情查询接口配置
"url": "http://119.45.255.234:30180/pcm/v1/core/task/details",
"timeout": 15
}
}
# 集群资源配置（key=集群ID，value=总资源/可用资源）
cluster_resources: Dict[str, Dict] = {
"1790300942428540928": { # modelarts集群
"total": {"CPU": 96, "MEMORY": 1024, "NPU": 2},
"available": {"CPU": 48, "MEMORY": 512, "NPU": 1}
},
"1865927992266461184": { # openi集群
"total": {"CPU": 48, "MEMORY": 512, "DCU": 1},
"available": {"CPU": 24, "MEMORY": 256, "DCU": 1}
},
"1865927992266462181": { # 章鱼集群
"total": {"CPU": 48, "MEMORY": 512, "DCU": 1},
"available": {"CPU": 24, "MEMORY": 256, "DCU": 1}
},
"1777240145309732864": { # 曙光集群
"total": {"CPU": 48, "MEMORY": 512, "NPU": 1},
"available": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
}
cluster_lock = threading.Lock() # 集群资源线程锁
# -------------------------- 数据结构定义 --------------------------
class DatasetInfo(dict):
"""数据集信息结构"""
def __init__(self, file_location: str, name: str, size: float, **kwargs):
super().__init__()
self["file_location"] = file_location # 本地路径（主键）
self["id"] = kwargs.get("id", str(uuid4())) # 数据集唯一标识
self["name"] = name # 数据集名称
self["size"] = size # 大小（字节）
self["is_uploaded"] = kwargs.get("is_uploaded", False) # 是否已上传
self["upload_cluster"] = kwargs.get("upload_cluster", []) # 上传的集群
self["upload_time"] = kwargs.get("upload_time") # 上传时间
self["description"] = kwargs.get("description") # 描述
class AlgorithmInfo(dict):
"""算法信息结构"""
def __init__(self, cluster: str, id: str, name: str, **kwargs):
super().__init__()
self["cluster"] = cluster # 所属集群
self["id"] = id # 算法唯一标识
self["son_id"] = kwargs.get("son_id", "") # 子算法ID
self["name"] = name # 算法名称
class TaskInfo(dict):
"""任务信息结构（新增success_time字段记录成功时间）"""
def __init__(self, task_name: str, dataset_name: str, code_id: str, resource: Dict, **kwargs):
super().__init__()
self["target_id"] = kwargs.get("target_id", str(uuid4())) # 任务唯一ID
self["task_name"] = task_name # 任务名称
self["package_name"] = kwargs.get("package_name", f"{task_name.lower()}-pkg") # 文件夹名称
self["dataset_name"] = dataset_name # 关联数据集名称
self["code_id"] = code_id # 算法ID
self["son_code_id"] = "" # 子算法ID（提交时填充）
self["resource"] = resource # 资源需求（CPU/MEMORY/NPU等）
self["status"] = TASK_STATUS["SUBMITTED"] # 初始状态：待提交
self["submit_time"] = kwargs.get("submit_time", time.strftime("%Y-%m-%d %H:%M:%S")) # 提交时间
self["success_time"] = None # 成功时间（成功时填充）
self["third_party_task_id"] = "" # 第三方任务ID（提交后填充）
self["file_location"] = kwargs.get("file_location", "") # 本地文件路径
self["error_msg"] = "" # 错误信息
self["fail_count"] = 0 # 失败次数（原retry_count改为fail_count，更贴合语义）
self["max_fail_threshold"] = kwargs.get("max_fail_threshold", 3) # 最大失败阈值
self["cluster_id"] = "" # 提交的集群ID（提交时填充）
# -------------------------- 工具方法 --------------------------
def generate_task_templates() -> List[Dict]:
"""生成任务静态数据模板"""
return [
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AA",
"dataset_name": "data1.zip",
"code_id": "1164",
"file_location": "D:/数据集/cnn数据集/data1/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AB",
"dataset_name": "cifar-10-python.tar.gz",
"code_id": "1",
"file_location": "D:/数据集/cnn数据集/data2/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AC",
"dataset_name": "cifar-100-python.tar.gz",
"code_id": "1",
"file_location": "D:/数据集/cnn数据集/data3/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AD",
"dataset_name": "dev.jsonl",
"code_id": "2",
"file_location": "D:/数据集/transfomer数据集/BoolQ/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AE",
"dataset_name": "dev.jsonl",
"file_location": "D:/数据集/transfomer数据集/BoolQ/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AF",
"dataset_name": "ceval.zip",
"file_location": "D:/数据集/transfomer数据集/CEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AG",
"dataset_name": "CMMLU.zip",
"file_location": "D:/数据集/transfomer数据集/CMMLU/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AH",
"dataset_name": "mental_health.csv",
"file_location": "D:/数据集/transfomer数据集/GLUE(imdb)/imdb/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AI",
"dataset_name": "GSM8K.jsonl",
"file_location": "D:/数据集/transfomer数据集/GSM8K/GSM8K/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AJ",
"dataset_name": "human-eval.jsonl",
"file_location": "D:/数据集/transfomer数据集/HumanEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AK",
"dataset_name": "HumanEval_X.zip",
"file_location": "D:/数据集/transfomer数据集/HumanEval_X/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AF",
"dataset_name": "ceval.zip",
"file_location": "D:/数据集/transfomer数据集/CEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AG",
"dataset_name": "CMMLU.zip",
"file_location": "D:/数据集/transfomer数据集/CMMLU/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AH",
"dataset_name": "mental_health.csv",
"file_location": "D:/数据集/transfomer数据集/GLUE(imdb)/imdb/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AI",
"dataset_name": "GSM8K.jsonl",
"file_location": "D:/数据集/transfomer数据集/GSM8K/GSM8K/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AJ",
"dataset_name": "human-eval.jsonl",
"file_location": "D:/数据集/transfomer数据集/HumanEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AK",
"dataset_name": "HumanEval_X.zip",
"file_location": "D:/数据集/transfomer数据集/HumanEval_X/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
}
]
def load_tasks_to_queue(templates: List[Dict]) -> None:
"""将任务静态数据加载到任务队列（task_map）"""
global task_map
with task_map_lock:
task_map.clear()
for template in templates:
task_name = template["task_name_template"].format(prefix=template["prefix"])
task = TaskInfo(
task_name=task_name,
dataset_name=template["dataset_name"],
code_id=template["code_id"],
resource=template["resource"],
file_location=template["file_location"]
)
task_map[task["target_id"]] = task
logger.info(f"任务入队 | task_name: {task_name} | target_id: {task['target_id']}")
logger.info(f"任务队列加载完成 | 共 {len(task_map)} 个任务")
def select_cluster(task_resource: Dict) -> Optional[str]:
"""根据任务资源需求选择合适的集群"""
with cluster_lock:
for cluster_id, cluster in cluster_resources.items():
# 检查集群可用资源是否满足任务需求（支持NPU/DCU等不同加速卡类型）
resource_match = True
for res_type, required in task_resource.items():
# 集群可用资源中可能是NPU或DCU，统一检查
available = cluster["available"].get(res_type, 0)
if available < required:
resource_match = False
break
if resource_match:
return cluster_id
logger.warning(f"无满足资源需求的集群 | 任务需求: {task_resource}")
return None
# -------------------------- API调用方法 --------------------------
def get_son_code_id(cluster_id: str, code_id: str) -> str:
"""根据集群ID和算法ID查询子算法ID（模拟接口查询）"""
son_code_map = {
("1790300942428540928", "1"): "1-1",
("1790300942428540928", "2"): "2-1",
("1777240145309732864", "1"): "1-2",
("1865927992266461184", "2"): "2-2"
}
return son_code_map.get((cluster_id, code_id), f"{code_id}-default")
#def get_auth_token() -> Optional[str]:
# -------------------------- API调用方法 --------------------------
def get_token() -> Optional[str]:
"""获取认证Token"""
login_payload = {"username": "admin", "password": "Nudt@123"}
try:
config = API_CONFIG["login"]
response = requests.post(config["url"], json=login_payload, timeout=config["timeout"])
response.raise_for_status()
result = response.json()
if result.get("code") == 200 and "data" in result and "token" in result["data"]:
logger.info("Token获取成功")
return result["data"]["token"]
else:
logger.error(f"Token获取失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"登录请求异常: {str(e)}", exc_info=True)
return None
def submit_single_task(task: Dict) -> bool:
"""提交单个任务到集群（失败时更新状态为failed/error）"""
token = get_token()
if not token:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "获取Token失败"
return False
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {token}'
}
task_name = task["task_name"]
package_name = task["package_name"]
file_name = task["dataset_name"]
file_location = task["file_location"]
code_id = task["code_id"] # 修正字段名
son_code_id = get_son_code_id(task["cluster_id"], code_id) # 使用实际集群ID
file_path = os.path.join(file_location, file_name)
try:
# 第一步：创建数据集文件夹
config = API_CONFIG["create_package"]
create_payload = {
"userID": 5,
"name": package_name,
"dataType": "dataset",
"packageID": 0,
"uploadPriority": {"type": "specify", "clusters": ["1790300942428540928"]},
"bindingInfo": {
"clusterIDs": ["1790300942428540928"],
"name": package_name,
"category": "image",
"type": "dataset",
"imageID": "",
"bias": [],
"region": [],
"chip": ["ASCEND"],
"selectedCluster": [],
"modelType": "",
"env": "",
"version": "",
"packageID": 0,
"points": 0
}
}
create_resp = requests.post(config["url"], json=create_payload, headers=headers, timeout=config["timeout"])
create_resp.raise_for_status()
create_result = create_resp.json()
if create_result.get("code") != 200:
raise ValueError(f"创建文件夹失败 | API返回: {create_result}")
packageID = create_result["data"]["newPackage"]["packageID"]
logger.info(f"[{task_name}] 第一步：创建文件夹成功 | packageID: {packageID}")
# 第三步：上传数据集文件
config = API_CONFIG["upload_file"]
if not os.path.exists(file_path):
raise FileNotFoundError(f"数据集文件不存在 | path: {file_path}")
info_data = {
"userID": 5,
"packageID": packageID,
"loadTo": [3],
"loadToPath": [f"/dataset/5/{package_name}/"]
}
file_headers = {'Authorization': f'Bearer {token}'}
with open(file_path, 'rb') as f:
form_data = {"info": (None, json.dumps(info_data)), "files": f}
upload_resp = requests.post(config["url"], files=form_data, headers=file_headers, timeout=config["timeout"])
upload_resp.raise_for_status()
upload_result = upload_resp.json()
if upload_result.get("code") != 200:
raise ValueError(f"文件上传失败 | API返回: {upload_result}")
object_id = upload_result["data"]["uploadeds"][0]["objectID"]
logger.info(f"[{task_name}] 第三步：文件上传成功 | objectID: {object_id}")
# 第四步：通知上传完成
config = API_CONFIG["notify_upload"]
notify_payload = {
"userID": 5,
"packageID": packageID,
"uploadParams": {
"dataType": "dataset",
"uploadInfo": {"type": "local", "localPath": file_name, "objectIDs": [object_id]}
}
}
notify_resp = requests.post(config["url"], json=notify_payload, headers=headers, timeout=config["timeout"])
notify_resp.raise_for_status()
notify_result = notify_resp.json()
if notify_result.get("code") != 200:
raise ValueError(f"通知上传完成失败 | API返回: {notify_result}")
logger.info(f"[{task_name}] 第四步：通知上传完成成功")
# 第七步：绑定数据集到集群
config = API_CONFIG["bind_cluster"]
bind_payload = {
"userID": 5,
"info": {"type": "dataset", "packageID": packageID, "clusterIDs": ["1790300942428540928"]}
}
bind_resp = requests.post(config["url"], json=bind_payload, headers=headers, timeout=config["timeout"])
bind_resp.raise_for_status()
bind_result = bind_resp.json()
if bind_result.get("code") != 200:
raise ValueError(f"绑定集群失败 | API返回: {bind_result}")
logger.info(f"[{task_name}] 第七步：数据集绑定集群成功")
# 第八步：查询绑定ID
config = API_CONFIG["query_binding"]
query_bind_payload = {
"dataType": "dataset",
"param": {"userID": 5, "bindingID": -1, "type": "private"}
}
query_bind_resp = requests.post(config["url"], json=query_bind_payload, headers=headers, timeout=config["timeout"]).json()
if query_bind_resp.get("code") != 200:
raise ValueError(f"查询绑定失败 | API返回: {query_bind_resp}")
# 提取目标绑定ID
target_id = None
for data in query_bind_resp["data"]["datas"]:
if data["info"]["name"] == package_name:
target_id = data["ID"]
break
if not target_id:
raise ValueError(f"未找到package_name={package_name}的绑定ID")
logger.info(f"[{task_name}] 第八步：获取绑定ID成功 | target_id: {target_id}")
# 第九步：提交训练任务
config = API_CONFIG["submit_task"]
task_res = task["resource"]
submit_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"name": task_name,
"description": "自动提交的CNN训练任务",
"type": "AI",
"files": {
"dataset": {"type": "Binding", "bindingID": target_id},
"model": {"type": "Binding", "bindingID": 421},
"image": {"type": "Image", "imageID": 11}
},
"jobResources": {
"scheduleStrategy": "dataLocality",
"clusters": [
{
"clusterID": "1790300942428540928",
"runtime": {"envs": {}, "params": {}},
"code": {"type": "Binding", "bindingID": son_code_id},
"resources": [
{"type": "CPU", "name": "ARM", "number": task_res["CPU"]},
{"type": "MEMORY", "name": "RAM", "number": task_res["MEMORY"]},
{"type": "MEMORY", "name": "VRAM", "number": 32},
{"type": "STORAGE", "name": "DISK", "number": 32},
{"type": "NPU", "name": "ASCEND910", "number": task_res.get("NPU", 0)}
]
}
]
}
},
{"localJobID": "4", "type": "DataReturn", "targetLocalJobID": "1"}
]
}
}
submit_resp = requests.post(config["url"], json=submit_payload, headers=headers, timeout=config["timeout"]).json()
if submit_resp.get("code") != 200:
raise ValueError(f"任务提交失败 | API返回: {submit_resp}")
third_party_task_id = submit_resp.get('data', {}).get('jobSetID')
logger.info(f"[{task_name}] 第九步：任务提交成功 | 第三方任务ID: {third_party_task_id}")
# 更新任务状态为成功（线程安全）
with task_map_lock:
task["status"] = TASK_STATUS["SUCCEED"]
task["third_party_task_id"] = third_party_task_id # 保存第三方任务ID
return True
except Exception as e:
error_msg = f"提交失败: {str(e)}"
with task_map_lock:
# 检查是否达到最大重试次数
task["fail_count"] += 1
if task["fail_count"] >= task["max_fail_threshold"]:
task["status"] = TASK_STATUS["RETRY_EXHAUSTED"]
else:
task["status"] = TASK_STATUS["FAILED"] # 未达最大次数，标记为failed等待重试
task["error_msg"] = error_msg
logger.error(f"[{task_name}] {error_msg}", exc_info=True)
return False
def query_third_party_task_status(third_party_task_id: str) -> Optional[str]:
"""
查询云际平台任务状态（实际API调用）
返回subTaskInfos[]中第一个元素的status值
"""
if not third_party_task_id:
logger.warning("第三方任务ID为空，无法查询状态")
return None
try:
# 构建请求参数（ID作为查询参数）
config = API_CONFIG["task_details"]
params = {"id": third_party_task_id}
# 发送请求（注意：任务详情接口可能需要Token认证，此处补充认证逻辑）
token = get_token()
headers = {"Authorization": f"Bearer {token}"} if token else {}
response = requests.get(
config["url"],
params=params,
headers=headers,
timeout=config["timeout"]
)
response.raise_for_status() # 抛出HTTP错误状态码
result = response.json()
# 解析响应结果
if result.get("code") != 200:
logger.error(f"查询任务状态失败 | 任务ID: {third_party_task_id} | 响应: {result}")
return None
# 提取subTaskInfos中的status
sub_task_infos = result.get("data", {}).get("subTaskInfos", [])
if not sub_task_infos:
logger.warning(f"任务 {third_party_task_id} 未找到subTaskInfos数据")
return None
# 返回第一个子任务的status
return sub_task_infos[0].get("status")
except requests.exceptions.RequestException as e:
logger.error(f"查询任务状态请求异常 | 任务ID: {third_party_task_id} | 错误: {str(e)}", exc_info=True)
return None
except (KeyError, IndexError) as e:
logger.error(f"解析任务状态响应失败 | 任务ID: {third_party_task_id} | 错误: {str(e)}", exc_info=True)
return None
# -------------------------- 线程一：任务监控线程 --------------------------
class TaskMonitorThread(threading.Thread):
"""监控线程：专注监控任务状态，仅处理提交中任务的状态更新"""
def __init__(self, check_interval: int = 10):
super().__init__(name="TaskMonitorThread")
self.check_interval = check_interval # 监控间隔（秒）
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"监控线程启动 | 监控间隔: {self.check_interval}秒")
while not self._stop_event.is_set():
with task_map_lock:
tasks = list(task_map.values()) # 复制任务列表，避免线程安全问题
for task in tasks:
with task_map_lock:
current_status = task["status"]
# 1. 待提交状态：不处理
if current_status == TASK_STATUS["SUBMITTED"]:
continue
# 2. 提交中状态：定时查询第三方状态并更新
elif current_status == TASK_STATUS["SUBMITTING"]:
if not task["third_party_task_id"]:
logger.warning(f"任务 {task['task_name']} 无第三方ID，跳过状态查询")
continue
# 查询第三方状态
third_status = query_third_party_task_status(task["third_party_task_id"])# 根据第三方返回的id查询任务状态
with task_map_lock:
# 2.1 第三方状态为成功：更新为提交成功，记录成功时间
if third_status == "SUCCEEDED":
task["status"] = TASK_STATUS["SUCCEED"]
task["success_time"] = time.strftime("%Y-%m-%d %H:%M:%S")
logger.info(
f"任务状态更新 | task_name: {task['task_name']} | 提交成功 | 成功时间: {task['success_time']}")
# 2.2 第三方状态为失败：更新为提交失败，失败次数+1
elif third_status == "FAILED":
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"第三方任务执行失败（第{task['fail_count']}次）"
logger.warning(
f"任务状态更新 | task_name: {task['task_name']} | 提交失败 | 失败次数: {task['fail_count']}/{task['max_fail_threshold']}")
# 2.3 第三方状态为提交中：不更新状态
# 3. 提交成功状态：不处理
elif current_status == TASK_STATUS["SUCCEED"]:
continue
# 4. 提交失败状态：不处理（由提交线程判断是否重试）
elif current_status == TASK_STATUS["FAILED"]:
continue
# 检查是否所有任务已完成（成功或失败次数超阈值）
all_completed = self._check_all_tasks_completed()
if all_completed:
logger.info("所有任务已完成（成功或失败次数超过阈值）")
self.stop()
# 等待下次监控
self._stop_event.wait(self.check_interval)
logger.info("监控线程结束")
def _check_all_tasks_completed(self) -> bool:
"""检查是否所有任务已完成（成功或失败次数超阈值）"""
with task_map_lock:
for task in task_map.values():
# 待提交或提交中：未完成
if task["status"] in [TASK_STATUS["SUBMITTED"], TASK_STATUS["SUBMITTING"]]:
return False
# 提交失败但次数未超阈值：未完成（可能被提交线程重试）
if task["status"] == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
return False
return True
def stop(self) -> None:
self._stop_event.set()
# -------------------------- 线程二：任务提交线程 --------------------------
class TaskSubmitThread(threading.Thread):
"""提交线程：按状态判断是否提交，处理待提交和未超阈值的失败任务"""
def __init__(self, max_workers: int = 3):
super().__init__(name="TaskSubmitThread")
self.max_workers = max_workers # 并发提交数
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"提交线程启动 | 并发数: {self.max_workers}")
while not self._stop_event.is_set():
# 1. 筛选符合条件的任务：待提交 或 失败次数未超阈值的提交失败任务
with task_map_lock:
pending_tasks = []
for task in task_map.values():
status = task["status"]
# 1.1 待提交状态：直接提交
if status == TASK_STATUS["SUBMITTED"]:
pending_tasks.append(task)
# 1.2 提交失败状态：检查失败次数，未超阈值则提交
elif status == TASK_STATUS["FAILED"]:
if task["fail_count"] < task["max_fail_threshold"]:
pending_tasks.append(task)
else:
logger.info(
f"任务 {task['task_name']} 失败次数超阈值（{task['max_fail_threshold']}），停止提交")
# if not pending_tasks:
# logger.info("无待提交任务，等待下次检查")
# self._stop_event.wait(5)
# continue
# 2. 并发提交任务
with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:
futures = {executor.submit(self.commit_task, task): task for task in pending_tasks}
for future in concurrent.futures.as_completed(futures):
task = futures[future]
try:
future.result()
except Exception as e:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"提交过程异常: {str(e)}"
logger.error(f"任务提交异常 | task_name: {task['task_name']} | 错误: {str(e)}")
logger.info("提交线程结束")
def commit_task(self, task: Dict) -> None:
"""提交任务的入口：先选集群，再调用submit_single_task"""
# 1. 选择集群并更新任务
cluster_id = select_cluster(task["resource"])
if not cluster_id:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = "无可用集群"
logger.error(f"[{task['task_name']}] 提交失败：无可用集群")
return
# 2. 标记任务为提交中
with task_map_lock:
task["status"] = TASK_STATUS["SUBMITTING"]
task["cluster_id"] = cluster_id # 记录集群ID
logger.info(f"[{task['task_name']}] 开始提交至集群 {cluster_id}")
# 3. 调用核心提交方法
submit_success = submit_single_task(task)
if not submit_success:
logger.warning(f"[{task['task_name']}] 提交失败，等待重试（当前失败次数：{task['fail_count']}）")
# def stop(self) -> None:
# self._stop_event.set()
# -------------------------- 主程序 --------------------------
if __name__ == "__main__":
# 1. 生成任务静态数据
task_templates = generate_task_templates()
# 2. 读取任务进入队列
load_tasks_to_queue(task_templates)
# 3. 启动监控线程
monitor_thread = TaskMonitorThread(check_interval=10)
monitor_thread.start()
# 4. 启动提交线程
submit_thread = TaskSubmitThread(max_workers=3)
submit_thread.start()
# 5. 等待线程结束
monitor_thread.join()
submit_thread.join()
logger.info("所有任务处理完毕，程序退出")

1015
commit_task_jointcloud_isdataupload_0717.py Normal file
View File

File diff suppressed because it is too large Load Diff

846
commit_task_jointcloud_isdataupload_0721.py Normal file
View File

@ -0,0 +1,846 @@
import concurrent.futures
import time
import logging
import threading
from uuid import uuid4
from typing import Dict, List, Optional
import requests
import os
import json
import yaml # 用于处理YAML文件
# -------------------------- 全局配置与常量定义 --------------------------
# 日志配置
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
logger = logging.getLogger(__name__)
# 任务状态定义
TASK_STATUS = {
"SUBMITTED": "待提交", # 初始状态
"SUBMITTING": "提交中", # 提交过程中
"SUCCEED": "提交成功", # 云际确认成功
"FAILED": "提交失败", # 云际确认失败
"RETRY_EXHAUSTED": "重试耗尽" # 超过最大失败次数
}
# 全局任务字典（key=target_id，value=任务详情）
task_map: Dict[str, Dict] = {}
task_map_lock = threading.Lock() # 任务字典线程锁
# 全局数据集映射（key=file_location，value=DatasetInfo实例）
dataset_map: Dict[str, Dict] = {}
dataset_lock = threading.Lock() # 数据集映射线程锁
# 子算法-集群-数据集映射（从YAML加载）
ALGORITHM_MAPPING: Dict[int, Dict] = {} # 结构: {son_code_id: {"clusters": [], "file_location": ""}}
# 任务模板（从YAML加载）
TASK_TEMPLATES: List[Dict] = []
# API配置
API_CONFIG = {
"login": {
"url": "http://119.45.255.234:30180/jcc-admin/admin/login",
"timeout": 10
},
"create_package": {
"url": "http://119.45.255.234:30180/jsm/jobSet/createPackage",
"timeout": 15
},
"upload_file": {
"url": "http://121.36.5.116:32010/object/upload",
"timeout": 3000
},
"notify_upload": {
"url": "http://119.45.255.234:30180/jsm/jobSet/notifyUploaded",
"timeout": 15
},
"bind_cluster": {
"url": "http://119.45.255.234:30180/jsm/jobSet/binding",
"timeout": 15
},
"query_binding": {
"url": "http://119.45.255.234:30180/jsm/jobSet/queryBinding",
"timeout": 15
},
"submit_task": {
"url": "http://119.45.255.234:30180/jsm/jobSet/submit",
"timeout": 100
},
"task_details": {
"url": "http://119.45.255.234:30180/pcm/v1/core/task/details",
"timeout": 15
}
}
# 集群资源配置（key=集群ID，value=总资源/可用资源）
cluster_resources: Dict[str, Dict] = {
"1790300942428540928": { # modelarts集群
"total": {"CPU": 1024, "MEMORY": 2048, "NPU": 56},
"available": {"CPU": 1024, "MEMORY": 2048, "NPU": 56}
},
"1790300942428540929": { # 新增集群示例
"total": {"CPU": 512, "MEMORY": 1024, "NPU": 32},
"available": {"CPU": 512, "MEMORY": 1024, "NPU": 32}
}
}
cluster_lock = threading.Lock() # 集群资源线程锁
# -------------------------- 数据结构定义 --------------------------
class DatasetInfo(dict):
"""数据集信息结构"""
def __init__(self, file_location: str, name: str, size: float, **kwargs):
super().__init__()
self["file_location"] = file_location # 本地路径（主键）
self["id"] = kwargs.get("id", str(uuid4())) # 数据集唯一标识
self["name"] = name # 数据集名称
self["size"] = size # 大小（字节）
self["is_uploaded"] = kwargs.get("is_uploaded", False) # 是否已上传
self["upload_cluster"] = kwargs.get("upload_cluster", []) # 上传的集群
self["upload_time"] = kwargs.get("upload_time") # 上传时间
self["description"] = kwargs.get("description") # 描述
self["dataset_target_id"] = kwargs.get("dataset_target_id") # 绑定ID
class TaskInfo(dict):
"""任务信息结构"""
def __init__(self, task_name: str, dataset_name: str, son_code_id: int, resource: Dict, **kwargs):
super().__init__()
self["target_id"] = kwargs.get("target_id", str(uuid4())) # 任务唯一ID
self["task_name"] = task_name # 任务名称
self["package_name"] = kwargs.get("package_name", f"{task_name.lower()}-pkg") # 文件夹名称
self["dataset_name"] = dataset_name # 关联数据集名称
self["son_code_id"] = son_code_id # 子算法ID
self["resource"] = resource # 资源需求（CPU/MEMORY/NPU等）
self["status"] = TASK_STATUS["SUBMITTED"] # 初始状态：待提交
self["submit_time"] = kwargs.get("submit_time", time.strftime("%Y-%m-%d %H:%M:%S")) # 提交时间
self["success_time"] = None # 成功时间（成功时填充）
self["third_party_task_id"] = "" # 云际任务ID（提交后填充）
self["file_location"] = kwargs.get("file_location", "") # 本地文件路径（从映射获取）
self["error_msg"] = "" # 错误信息
self["fail_count"] = 0 # 失败次数
self["max_fail_threshold"] = kwargs.get("max_fail_threshold", 3) # 最大失败阈值
self["cluster_id"] = "" # 提交的集群ID（从映射获取）
# -------------------------- YAML配置文件处理 --------------------------
def load_yaml_config(yaml_path: str = "sonCode_cluster__mapping.yaml") -> None:
"""加载YAML配置文件（包含算法映射和任务模板）"""
# 加载配置文件
try:
with open(str, "r", encoding="utf-8") as f:
data = yaml.safe_load(f)
ALGORITHM_MAPPING = data.get("algorithm_mapping", {})
TASK_TEMPLATES = data.get("task_templates", [])
logger.info(
f"成功加载配置文件 | 子算法映射: {len(ALGORITHM_MAPPING)} 条 | 任务模板: {len(TASK_TEMPLATES)} 条")
except Exception as e:
logger.error(f"加载配置文件失败: {str(e)}", exc_info=True)
# -------------------------- 工具方法 --------------------------
def generate_task_templates() -> List[Dict]:
"""从全局变量返回任务模板（实际从YAML加载）"""
return TASK_TEMPLATES
def load_tasks_to_queue(templates: List[Dict]) -> None:
"""将任务静态数据加载到任务队列（从映射获取file_location）"""
global task_map
with task_map_lock:
task_map.clear()
for idx, template in enumerate(templates):
try:
# 检查必填字段
required_fields = ["task_name_template", "prefix", "dataset_name", "son_code_id", "resource"]
missing_fields = [f for f in required_fields if f not in template]
if missing_fields:
logger.warning(f"跳过无效任务模板（索引{idx}）：缺少字段 {missing_fields}")
continue
# 从映射获取file_location
son_code_id = template["son_code_id"]
mapping = ALGORITHM_MAPPING.get(son_code_id)
if not mapping:
logger.warning(f"子算法ID {son_code_id} 无映射配置，跳过任务模板（索引{idx}）")
continue
file_location = mapping["file_location"]
# 生成任务名称
task_name = template["task_name_template"].format(prefix=template["prefix"])
# 创建任务实例
task = TaskInfo(
task_name=task_name,
dataset_name=template["dataset_name"],
son_code_id=son_code_id,
resource=template["resource"],
file_location=file_location # 从映射填充
)
task_map[task["target_id"]] = task
logger.info(
f"任务入队 | task_name: {task_name} | 子算法ID: {son_code_id} | 数据集路径: {file_location}")
except Exception as e:
logger.error(f"加载任务模板失败（索引{idx}）：{str(e)}")
logger.info(f"任务队列加载完成 | 共 {len(task_map)} 个有效任务")
def select_cluster(task_resource: Dict, son_code_id: int) -> Optional[str]:
"""根据任务资源需求和子算法ID选择合适的集群（优先从映射中选择）"""
# 1. 从映射获取该子算法支持的集群
mapping = ALGORITHM_MAPPING.get(son_code_id)
if not mapping:
logger.warning(f"子算法ID {son_code_id} 无集群映射，尝试所有集群")
candidate_clusters = list(cluster_resources.keys())
else:
candidate_clusters = mapping["clusters"]
with cluster_lock:
# 2. 检查候选集群是否满足资源需求
for cluster_id in candidate_clusters:
if cluster_id not in cluster_resources:
logger.warning(f"映射中集群 {cluster_id} 不存在于资源配置中，跳过")
continue
cluster = cluster_resources[cluster_id]
# 检查资源是否满足
resource_match = True
for res_type, required in task_resource.items():
available = cluster["available"].get(res_type, 0)
if available < required:
resource_match = False
break
if resource_match:
# 占用资源
for res_type, required in task_resource.items():
if res_type in cluster["available"]:
cluster["available"][res_type] -= required
logger.info(f"选中集群 {cluster_id}（子算法 {son_code_id} 映射）| 更新后可用资源: {cluster['available']}")
return cluster_id
# 3. 若映射中的集群不满足，尝试其他集群
all_clusters = list(cluster_resources.keys())
for cluster_id in all_clusters:
if cluster_id in candidate_clusters:
continue # 已检查过
cluster = cluster_resources[cluster_id]
resource_match = True
for res_type, required in task_resource.items():
available = cluster["available"].get(res_type, 0)
if available < required:
resource_match = False
break
if resource_match:
for res_type, required in task_resource.items():
if res_type in cluster["available"]:
cluster["available"][res_type] -= required
logger.info(f"选中集群 {cluster_id}（非映射）| 更新后可用资源: {cluster['available']}")
return cluster_id
logger.warning(f"无满足资源需求的集群 | 任务需求: {task_resource} | 子算法: {son_code_id}")
return None
# -------------------------- 数据集上传判断与处理方法 --------------------------
def check_and_handle_dataset(file_location: str, dataset_name: str, cluster_id: str) -> Optional[str]:
"""检查数据集是否已上传到指定集群，未上传则执行上传"""
global dataset_map
# 步骤1: 检查数据集是否已上传
dataset = get_dataset_status(file_location, dataset_name, cluster_id)
# 步骤2: 若未上传，则执行上传
if not dataset or not dataset["is_uploaded"] or cluster_id not in dataset["upload_cluster"]:
dataset = upload_dataset(file_location, dataset_name, cluster_id)
if not dataset:
return None # 上传失败
return dataset["dataset_target_id"]
def get_dataset_status(file_location: str, dataset_name: str, cluster_id: str) -> Optional[Dict]:
"""检查数据集状态（是否已上传到指定集群）"""
with dataset_lock:
if file_location in dataset_map:
dataset = dataset_map[file_location]
# 验证是否已上传到目标集群
if dataset["is_uploaded"] and cluster_id in dataset["upload_cluster"]:
logger.info(
f"数据集 {dataset_name} 已上传到集群 {cluster_id} | target_id: {dataset['dataset_target_id']}")
return dataset
return None
def upload_dataset(file_location: str, dataset_name: str, cluster_id: str) -> Optional[Dict]:
"""执行数据集上传流程"""
dataset_path = os.path.join(file_location, dataset_name)
# 检查本地文件是否存在
if not os.path.exists(dataset_path):
logger.error(f"数据集本地文件不存在 | path: {dataset_path}")
return None
# 计算文件大小（字节）
try:
file_size = os.path.getsize(dataset_path)
except OSError as e:
logger.error(f"获取文件大小失败 | path: {dataset_path} | 错误: {str(e)}")
return None
logger.info(f"开始上传数据集 {dataset_name} 到集群 {cluster_id} | path: {dataset_path}")
try:
# 获取认证Token
token = get_token()
if not token:
logger.error("获取Token失败，无法上传数据集")
return None
headers = {'Authorization': f'Bearer {token}'}
package_name = f"dataset-{dataset_name.split('.')[0]}-{uuid4().hex[:6]}" # 生成唯一文件夹名
# 1. 创建数据集文件夹
create_payload = {
"userID": 5,
"name": package_name,
"dataType": "dataset",
"packageID": 0,
"uploadPriority": {"type": "specify", "clusters": [cluster_id]},
"bindingInfo": {
"clusterIDs": [cluster_id],
"name": package_name,
"category": "image",
"type": "dataset",
"imageID": "",
"chip": ["ASCEND"],
"packageID": 0
}
}
create_resp = requests.post(
API_CONFIG["create_package"]["url"],
json=create_payload,
headers=headers,
timeout=API_CONFIG["create_package"]["timeout"]
)
create_resp.raise_for_status()
create_result = create_resp.json()
if create_result.get("code") != "OK":
raise ValueError(f"创建文件夹失败 | 响应: {create_result}")
packageID = create_result["data"]["newPackage"]["packageID"]
logger.info(f"数据集文件夹创建成功 | packageID: {packageID}")
# 2. 上传文件
info_data = {
"userID": 5,
"packageID": packageID,
"loadTo": [3],
"loadToPath": [f"/dataset/5/{package_name}/"]
}
with open(dataset_path, 'rb') as f:
form_data = {"info": (None, json.dumps(info_data)), "files": f}
upload_resp = requests.post(
API_CONFIG["upload_file"]["url"],
files=form_data,
headers=headers,
timeout=API_CONFIG["upload_file"]["timeout"]
)
upload_resp.raise_for_status()
upload_result = upload_resp.json()
if upload_result.get("code") != "OK":
raise ValueError(f"文件上传失败 | 响应: {upload_result}")
object_id = upload_result["data"]["uploadeds"][0]["objectID"]
logger.info(f"数据集文件上传成功 | objectID: {object_id}")
# 3. 通知上传完成
notify_payload = {
"userID": 5,
"packageID": packageID,
"uploadParams": {
"dataType": "dataset",
"uploadInfo": {"type": "local", "localPath": dataset_name, "objectIDs": [object_id]}
}
}
notify_resp = requests.post(
API_CONFIG["notify_upload"]["url"],
json=notify_payload,
headers=headers,
timeout=API_CONFIG["notify_upload"]["timeout"]
)
notify_resp.raise_for_status()
# 4. 绑定到集群
bind_payload = {
"userID": 5,
"info": {"type": "dataset", "packageID": packageID, "clusterIDs": [cluster_id]}
}
bind_resp = requests.post(
API_CONFIG["bind_cluster"]["url"],
json=bind_payload,
headers=headers,
timeout=API_CONFIG["bind_cluster"]["timeout"]
)
bind_resp.raise_for_status()
# 5. 查询绑定ID（dataset_target_id）
query_resp = requests.post(
API_CONFIG["query_binding"]["url"],
json={"dataType": "dataset", "param": {"userID": 5, "bindingID": -1, "type": "private"}},
headers=headers,
timeout=API_CONFIG["query_binding"]["timeout"]
).json()
if query_resp.get("code") != "OK":
raise ValueError(f"查询绑定ID失败 | 响应: {query_resp}")
dataset_target_id = None
for item in query_resp["data"]["datas"]:
if item["info"]["name"] == package_name:
dataset_target_id = item["ID"]
break
if not dataset_target_id:
raise ValueError(f"未找到数据集 {package_name} 的绑定ID")
# 上传成功，创建并保存数据集信息
dataset = DatasetInfo(
file_location=file_location,
name=dataset_name,
size=file_size,
is_uploaded=True,
upload_cluster=[cluster_id],
upload_time=time.strftime("%Y-%m-%d %H:%M:%S"),
dataset_target_id=dataset_target_id
)
with dataset_lock:
dataset_map[file_location] = dataset
logger.info(f"数据集 {dataset_name} 上传成功 | target_id: {dataset_target_id}")
return dataset
except Exception as e:
logger.error(f"数据集上传失败 | name: {dataset_name} | 错误: {str(e)}", exc_info=True)
return None
# -------------------------- API调用方法 --------------------------
def get_token() -> Optional[str]:
"""获取认证Token"""
login_payload = {"username": "admin", "password": "Nudt@123"}
try:
config = API_CONFIG["login"]
response = requests.post(config["url"], json=login_payload, timeout=config["timeout"])
response.raise_for_status()
result = response.json()
if result.get("code") == 200 and "data" in result and "token" in result["data"]:
logger.info("Token获取成功")
return result["data"]["token"]
else:
logger.error(f"Token获取失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"登录请求异常: {str(e)}", exc_info=True)
return None
def submit_single_task(task: Dict) -> bool:
"""提交单个任务到集群"""
token = get_token()
if not token:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "获取Token失败"
return False
# 获取选择的集群ID
cluster_id = task.get("cluster_id")
if not cluster_id:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "未指定集群ID"
logger.error(f"[{task['task_name']}] 提交失败：未指定集群ID")
return False
# 数据集检查与上传
dataset_target_id = check_and_handle_dataset(
file_location=task["file_location"],
dataset_name=task["dataset_name"],
cluster_id=cluster_id
)
if not dataset_target_id:
# 数据集上传失败，释放集群资源
with cluster_lock:
if cluster_id in cluster_resources:
for res_type, required in task["resource"].items():
if res_type in cluster_resources[cluster_id]["available"]:
cluster_resources[cluster_id]["available"][res_type] += required
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "数据集上传失败"
logger.error(f"[{task['task_name']}] 提交失败：数据集处理失败")
return False
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {token}'
}
task_name = task["task_name"]
package_name = task["package_name"]
son_code_id = task["son_code_id"] # 子算法ID
try:
# 1. 创建数据集文件夹
config = API_CONFIG["create_package"]
create_payload = {
"userID": 5,
"name": package_name,
"dataType": "dataset",
"packageID": 0,
"uploadPriority": {"type": "specify", "clusters": [cluster_id]},
"bindingInfo": {
"clusterIDs": [cluster_id],
"name": package_name,
"category": "image",
"type": "dataset",
"imageID": "",
"bias": [],
"region": [],
"chip": ["ASCEND"],
"selectedCluster": [],
"modelType": "",
"env": "",
"version": "",
"packageID": 0,
"points": 0
}
}
create_resp = requests.post(config["url"], json=create_payload, headers=headers, timeout=config["timeout"])
create_resp.raise_for_status()
create_result = create_resp.json()
if create_result.get("code") != "OK":
raise ValueError(f"创建文件夹失败 | API返回: {create_result}")
packageID = create_result["data"]["newPackage"]["packageID"]
logger.info(f"[{task_name}] 第一步：创建文件夹成功 | packageID: {packageID} | 集群: {cluster_id}")
# 2. 通知上传完成
config = API_CONFIG["notify_upload"]
notify_payload = {
"userID": 5,
"packageID": packageID,
"uploadParams": {
"dataType": "dataset",
"uploadInfo": {"type": "local", "localPath": task["dataset_name"], "objectIDs": []}
}
}
notify_resp = requests.post(config["url"], json=notify_payload, headers=headers, timeout=config["timeout"])
notify_resp.raise_for_status()
logger.info(f"[{task_name}] 第二步：通知上传完成成功")
# 3. 绑定数据集到集群
config = API_CONFIG["bind_cluster"]
bind_payload = {
"userID": 5,
"info": {"type": "dataset", "packageID": packageID, "clusterIDs": [cluster_id]}
}
bind_resp = requests.post(config["url"], json=bind_payload, headers=headers, timeout=config["timeout"])
bind_resp.raise_for_status()
logger.info(f"[{task_name}] 第三步：数据集绑定集群 {cluster_id} 成功")
# 4. 提交训练任务
config = API_CONFIG["submit_task"]
task_res = task["resource"]
submit_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"name": task_name,
"description": "自动提交的训练任务",
"type": "AI",
"files": {
"dataset": {"type": "Binding", "bindingID": dataset_target_id},
"model": {"type": "Binding", "bindingID": ""},
"image": {"type": "Image", "imageID": 11}
},
"jobResources": {
"scheduleStrategy": "dataLocality",
"clusters": [
{
"clusterID": cluster_id,
"runtime": {"envs": {}, "params": {}},
"code": {"type": "Binding", "bindingID": son_code_id},
"resources": [
{"type": "CPU", "name": "ARM", "number": task_res["CPU"]},
{"type": "MEMORY", "name": "RAM", "number": task_res["MEMORY"]},
{"type": "MEMORY", "name": "VRAM", "number": 32},
{"type": "STORAGE", "name": "DISK", "number": 886},
{"type": "NPU", "name": "ASCEND910", "number": task_res.get("NPU", 0)}
]
}
]
}
}
]
}
}
response = requests.post(
config["url"],
json=submit_payload,
headers=headers,
timeout=config["timeout"]
)
response.raise_for_status()
submit_resp = response.json()
if submit_resp.get("code") != "OK":
raise ValueError(f"任务提交失败 | API返回: {submit_resp}")
third_party_task_id = submit_resp.get('data', {}).get('jobSetID')
logger.info(f"[{task_name}] 第四步：任务提交至集群 {cluster_id} 成功 | 云际任务ID: {third_party_task_id}")
# 更新任务状态为成功
with task_map_lock:
task["status"] = TASK_STATUS["SUCCEED"]
task["third_party_task_id"] = third_party_task_id
return True
except Exception as e:
error_msg = f"提交失败: {str(e)}"
# 任务失败时释放集群资源
with cluster_lock:
if cluster_id in cluster_resources:
for res_type, required in task["resource"].items():
if res_type in cluster_resources[cluster_id]["available"]:
cluster_resources[cluster_id]["available"][res_type] += required
logger.info(f"任务失败，释放集群 {cluster_id} 资源: {task['resource']}")
with task_map_lock:
task["fail_count"] += 1
if task["fail_count"] >= task["max_fail_threshold"]:
task["status"] = TASK_STATUS["RETRY_EXHAUSTED"]
else:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = error_msg
logger.error(f"[{task_name}] {error_msg}", exc_info=True)
return False
def query_third_party_task_status(third_party_task_id: str) -> Optional[Dict]:
"""查询云际平台任务状态"""
if not third_party_task_id:
logger.warning("云际任务ID为空，无法查询状态")
return None
try:
url = "http://119.45.255.234:30180/pcm/v1/core/task/list"
params = {
"pageNum": 1,
"pageSize": 10,
"type": 1
}
token = get_token()
if not token:
logger.error("获取Token失败，无法查询任务状态")
return None
headers = {"Authorization": f"Bearer {token}"} if token else {}
response = requests.get(
url,
params=params,
headers=headers,
timeout=15
)
response.raise_for_status()
result = response.json()
if result.get("code") != 200 or "data" not in result or "list" not in result["data"]:
logger.error(f"查询任务状态接口返回异常 | 响应: {result}")
return None
task_list = result["data"]["list"]
target_task = None
for task in task_list:
if task.get("id") == third_party_task_id:
target_task = task
break
if not target_task:
logger.warning(f"未找到任务 {third_party_task_id}")
return None
task_status = target_task.get("status")
end_time = target_task.get("endTime")
logger.info(f"任务 {third_party_task_id} 状态: {task_status} | 结束时间: {end_time or '未结束'}")
return {
"status": task_status,
"end_time": end_time
}
except Exception as e:
logger.error(f"查询任务状态失败 | 任务ID: {third_party_task_id} | 错误: {str(e)}", exc_info=True)
return None
# -------------------------- 线程一：任务监控线程 --------------------------
class TaskMonitorThread(threading.Thread):
"""监控线程：专注监控任务状态"""
def __init__(self, check_interval: int = 10):
super().__init__(name="TaskMonitorThread")
self.check_interval = check_interval
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"监控线程启动 | 监控间隔: {self.check_interval}秒")
while not self._stop_event.is_set():
with task_map_lock:
tasks = list(task_map.values())
for task in tasks:
with task_map_lock:
current_status = task["status"]
if current_status in [TASK_STATUS["SUBMITTED"], TASK_STATUS["RETRY_EXHAUSTED"]]:
continue
if current_status == TASK_STATUS["SUBMITTING"] and task["third_party_task_id"]:
third_party_info = query_third_party_task_status(task["third_party_task_id"])
if third_party_info:
with task_map_lock:
if third_party_info["status"] == "Succeeded":
task["status"] = TASK_STATUS["SUCCEED"]
task["success_time"] = third_party_info["end_time"]
logger.info(f"任务 {task['task_name']} 成功 | 时间: {task['success_time']}")
elif third_party_info["status"] in ["Failed", "Saved"]:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"云际任务失败（第{task['fail_count']}次）"
logger.warning(f"任务 {task['task_name']} 失败 | 次数: {task['fail_count']}")
if self._check_all_completed():
logger.info("所有任务已完成")
self.stop()
self._stop_event.wait(self.check_interval)
logger.info("监控线程结束")
def _check_all_completed(self) -> bool:
"""检查所有任务是否完成"""
with task_map_lock:
for task in task_map.values():
if task["status"] in [TASK_STATUS["SUBMITTED"], TASK_STATUS["SUBMITTING"]]:
return False
if task["status"] == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
return False
return True
def stop(self) -> None:
self._stop_event.set()
# -------------------------- 线程二：任务提交线程 --------------------------
class TaskSubmitThread(threading.Thread):
"""提交线程：按状态判断是否提交"""
def __init__(self, max_workers: int = 3):
super().__init__(name="TaskSubmitThread")
self.max_workers = max_workers
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"提交线程启动 | 并发数: {self.max_workers}")
while not self._stop_event.is_set():
# 筛选待提交任务
with task_map_lock:
pending_tasks = []
for task in task_map.values():
status = task["status"]
if status == TASK_STATUS["SUBMITTED"]:
pending_tasks.append(task)
elif status == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
pending_tasks.append(task)
elif status == TASK_STATUS["FAILED"]:
logger.info(f"任务 {task['task_name']} 失败次数超限，停止提交")
# 并发提交
with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:
futures = {executor.submit(self.commit_task, task): task for task in pending_tasks}
for future in concurrent.futures.as_completed(futures):
task = futures[future]
try:
future.result()
except Exception as e:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"提交异常: {str(e)}"
logger.error(f"任务 {task['task_name']} 提交异常 | 错误: {str(e)}")
if self._check_all_completed():
logger.info("所有任务已完成，提交线程退出")
self.stop()
break
if not pending_tasks:
logger.info("无待提交任务，等待5秒")
self._stop_event.wait(5)
logger.info("提交线程结束")
def commit_task(self, task: Dict) -> None:
"""提交任务入口：基于映射选择集群"""
son_code_id = task["son_code_id"]
cluster_id = select_cluster(task["resource"], son_code_id)
if not cluster_id:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = "无可用集群"
logger.error(f"任务 {task['task_name']} 提交失败：无可用集群")
return
# 标记为提交中
with task_map_lock:
task["status"] = TASK_STATUS["SUBMITTING"]
task["cluster_id"] = cluster_id
logger.info(f"任务 {task['task_name']} 开始提交至集群 {cluster_id}（子算法 {son_code_id}）")
# 执行提交
submit_success = submit_single_task(task)
if not submit_success:
logger.warning(f"任务 {task['task_name']} 提交失败，等待重试（当前次数：{task['fail_count']}）")
def _check_all_completed(self) -> bool:
"""检查所有任务是否完成"""
with task_map_lock:
for task in task_map.values():
if task["status"] in [TASK_STATUS["SUBMITTED"], TASK_STATUS["SUBMITTING"]]:
return False
if task["status"] == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
return False
return True
def stop(self) -> None:
self._stop_event.set()
# -------------------------- 主程序 --------------------------
if __name__ == "__main__":
# 加载YAML配置（包含算法映射和任务模板）
load_yaml_config()
# 生成任务模板并加载到队列
task_templates = generate_task_templates()
load_tasks_to_queue(task_templates)
# 启动线程
monitor_thread = TaskMonitorThread(check_interval=10)
monitor_thread.start()
submit_thread = TaskSubmitThread(max_workers=3)
submit_thread.start()
# 等待线程完成
monitor_thread.join()
submit_thread.join()
logger.info("所有任务处理完毕，程序退出")

544
commit_task_nojointcloud_0716.py Normal file
View File

@ -0,0 +1,544 @@
import concurrent.futures
import time
import logging
import threading
from uuid import uuid4
from typing import Dict, List, Optional
# -------------------------- 全局配置与常量定义 --------------------------
# 日志配置
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
logger = logging.getLogger(__name__)
# 任务状态定义
TASK_STATUS = {
"SUBMITTED": "待提交", # 初始状态
"SUBMITTING": "提交中", # 提交过程中
"SUCCEED": "提交成功", # 提交成功
"FAILED": "提交失败" # 提交失败
}
# 全局任务字典（key=target_id，value=任务详情）
task_map: Dict[str, Dict] = {}
task_map_lock = threading.Lock() # 任务字典线程锁
# 集群资源配置（key=集群ID，value=总资源/可用资源）
cluster_resources: Dict[str, Dict] = {
"1790300942428540928": { # modelarts集群
"total": {"CPU": 96, "MEMORY": 1024, "NPU": 2},
"available": {"CPU": 48, "MEMORY": 512, "NPU": 1}
},
"1865927992266461184": { # openi集群
"total": {"CPU": 48, "MEMORY": 512, "DCU": 1},
"available": {"CPU": 24, "MEMORY": 256, "DCU": 1}
},
"1865927992266462181": { # 章鱼集群
"total": {"CPU": 48, "MEMORY": 512, "DCU": 1},
"available": {"CPU": 24, "MEMORY": 256, "DCU": 1}
},
"1777240145309732864": { # 曙光集群
"total": {"CPU": 48, "MEMORY": 512, "NPU": 1},
"available": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
}
cluster_lock = threading.Lock() # 集群资源线程锁
# -------------------------- 数据结构定义 --------------------------
class DatasetInfo(dict):
"""数据集信息结构"""
def __init__(self, file_location: str, name: str, size: float, **kwargs):
super().__init__()
self["file_location"] = file_location # 本地路径（主键）
self["id"] = kwargs.get("id", str(uuid4())) # 数据集唯一标识
self["name"] = name # 数据集名称
self["size"] = size # 大小（字节）
self["is_uploaded"] = kwargs.get("is_uploaded", False) # 是否已上传
self["upload_cluster"] = kwargs.get("upload_cluster", []) # 上传的集群
self["upload_time"] = kwargs.get("upload_time") # 上传时间
self["description"] = kwargs.get("description") # 描述
class AlgorithmInfo(dict):
"""算法信息结构"""
def __init__(self, cluster: str, id: str, name: str, **kwargs):
super().__init__()
self["cluster"] = cluster # 所属集群
self["id"] = id # 算法唯一标识
self["son_id"] = kwargs.get("son_id", "") # 子算法ID
self["name"] = name # 算法名称
class TaskInfo(dict):
"""任务信息结构（新增success_time字段记录成功时间）"""
def __init__(self, task_name: str, dataset_name: str, code_id: str, resource: Dict, **kwargs):
super().__init__()
self["target_id"] = kwargs.get("target_id", str(uuid4())) # 任务唯一ID
self["task_name"] = task_name # 任务名称
self["package_name"] = kwargs.get("package_name", f"{task_name.lower()}-pkg") # 文件夹名称
self["dataset_name"] = dataset_name # 关联数据集名称
self["code_id"] = code_id # 算法ID
self["son_code_id"] = "" # 子算法ID（提交时填充）
self["resource"] = resource # 资源需求（CPU/MEMORY/NPU等）
self["status"] = TASK_STATUS["SUBMITTED"] # 初始状态：待提交
self["submit_time"] = kwargs.get("submit_time", time.strftime("%Y-%m-%d %H:%M:%S")) # 提交时间
self["success_time"] = None # 成功时间（成功时填充）
self["third_party_task_id"] = "" # 第三方任务ID（提交后填充）
self["file_location"] = kwargs.get("file_location", "") # 本地文件路径
self["error_msg"] = "" # 错误信息
self["fail_count"] = 0 # 失败次数（原retry_count改为fail_count，更贴合语义）
self["max_fail_threshold"] = kwargs.get("max_fail_threshold", 3) # 最大失败阈值
self["cluster_id"] = "" # 提交的集群ID（提交时填充）
# -------------------------- 工具方法 --------------------------
def generate_task_templates() -> List[Dict]:
"""生成任务静态数据模板"""
return [
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AA",
"dataset_name": "data1.zip",
"code_id": "1164",
"file_location": "D:/数据集/cnn数据集/data1/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AB",
"dataset_name": "cifar-10-python.tar.gz",
"code_id": "1",
"file_location": "D:/数据集/cnn数据集/data2/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AC",
"dataset_name": "cifar-100-python.tar.gz",
"code_id": "1",
"file_location": "D:/数据集/cnn数据集/data3/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AD",
"dataset_name": "dev.jsonl",
"code_id": "2",
"file_location": "D:/数据集/transfomer数据集/BoolQ/",
"resource": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AE",
"dataset_name": "dev.jsonl",
"file_location": "D:/数据集/transfomer数据集/BoolQ/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AF",
"dataset_name": "ceval.zip",
"file_location": "D:/数据集/transfomer数据集/CEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AG",
"dataset_name": "CMMLU.zip",
"file_location": "D:/数据集/transfomer数据集/CMMLU/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AH",
"dataset_name": "mental_health.csv",
"file_location": "D:/数据集/transfomer数据集/GLUE(imdb)/imdb/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AI",
"dataset_name": "GSM8K.jsonl",
"file_location": "D:/数据集/transfomer数据集/GSM8K/GSM8K/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AJ",
"dataset_name": "human-eval.jsonl",
"file_location": "D:/数据集/transfomer数据集/HumanEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AK",
"dataset_name": "HumanEval_X.zip",
"file_location": "D:/数据集/transfomer数据集/HumanEval_X/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AF",
"dataset_name": "ceval.zip",
"file_location": "D:/数据集/transfomer数据集/CEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AG",
"dataset_name": "CMMLU.zip",
"file_location": "D:/数据集/transfomer数据集/CMMLU/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AH",
"dataset_name": "mental_health.csv",
"file_location": "D:/数据集/transfomer数据集/GLUE(imdb)/imdb/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AI",
"dataset_name": "GSM8K.jsonl",
"file_location": "D:/数据集/transfomer数据集/GSM8K/GSM8K/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AJ",
"dataset_name": "human-eval.jsonl",
"file_location": "D:/数据集/transfomer数据集/HumanEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AK",
"dataset_name": "HumanEval_X.zip",
"file_location": "D:/数据集/transfomer数据集/HumanEval_X/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
}
]
def load_tasks_to_queue(templates: List[Dict]) -> None:
"""将任务静态数据加载到任务队列（task_map）"""
global task_map
with task_map_lock:
task_map.clear()
for template in templates:
task_name = template["task_name_template"].format(prefix=template["prefix"])
task = TaskInfo(
task_name=task_name,
dataset_name=template["dataset_name"],
code_id=template["code_id"],
resource=template["resource"],
file_location=template["file_location"]
)
task_map[task["target_id"]] = task
logger.info(f"任务入队 | task_name: {task_name} | target_id: {task['target_id']}")
logger.info(f"任务队列加载完成 | 共 {len(task_map)} 个任务")
def select_cluster(task_resource: Dict) -> Optional[str]:
"""根据任务资源需求选择合适的集群"""
with cluster_lock:
for cluster_id, cluster in cluster_resources.items():
# 检查集群可用资源是否满足任务需求（支持NPU/DCU等不同加速卡类型）
resource_match = True
for res_type, required in task_resource.items():
# 集群可用资源中可能是NPU或DCU，统一检查
available = cluster["available"].get(res_type, 0)
if available < required:
resource_match = False
break
if resource_match:
return cluster_id
logger.warning(f"无满足资源需求的集群 | 任务需求: {task_resource}")
return None
def get_son_code_id(cluster_id: str, code_id: str) -> str:
"""根据集群ID和算法ID查询子算法ID（模拟接口查询）"""
son_code_map = {
("1790300942428540928", "1"): "1-1",
("1790300942428540928", "2"): "2-1",
("1777240145309732864", "1"): "1-2",
("1865927992266461184", "2"): "2-2"
}
return son_code_map.get((cluster_id, code_id), f"{code_id}-default")
def get_auth_token() -> Optional[str]:
"""获取认证Token（模拟接口）"""
return "mock_valid_token"
def query_third_party_task_status(third_party_task_id: str) -> str:
"""查询云际平台任务状态（返回任务状态）"""
# 云际平台状态：SUBMITTING（提交中）、SUCCEEDED（成功）、FAILED（失败）
mock_status_map = {
"task-1001": "SUCCEEDED",
"task-1002": "FAILED",
"task-1003": "SUBMITTING",
"task-1004": "SUCCEEDED",
"task-1005": "FAILED"
}
return mock_status_map.get(third_party_task_id, "SUBMITTING")
# -------------------------- 线程一：任务监控线程 --------------------------
class TaskMonitorThread(threading.Thread):
"""监控线程：专注监控任务状态，仅处理提交中任务的状态更新"""
def __init__(self, check_interval: int = 10):
super().__init__(name="TaskMonitorThread")
self.check_interval = check_interval # 监控间隔（秒）
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"监控线程启动 | 监控间隔: {self.check_interval}秒")
while not self._stop_event.is_set():
with task_map_lock:
tasks = list(task_map.values()) # 复制任务列表，避免线程安全问题
for task in tasks:
with task_map_lock:
current_status = task["status"]
# 1. 待提交状态：不处理（由提交线程处理）
if current_status == TASK_STATUS["SUBMITTED"]:
continue
# 2. 提交中状态：定时查询第三方状态并更新
elif current_status == TASK_STATUS["SUBMITTING"]:
if not task["third_party_task_id"]:
logger.warning(f"任务 {task['task_name']} 无第三方ID，跳过状态查询")
continue
# 查询第三方状态
third_status = query_third_party_task_status(task["third_party_task_id"])
with task_map_lock:
# 2.1 第三方状态为成功：更新为提交成功，记录成功时间
if third_status == "SUCCEEDED":
task["status"] = TASK_STATUS["SUCCEED"]
task["success_time"] = time.strftime("%Y-%m-%d %H:%M:%S")
logger.info(
f"任务状态更新 | task_name: {task['task_name']} | 提交成功 | 成功时间: {task['success_time']}")
# 2.2 第三方状态为失败：更新为提交失败，失败次数+1
elif third_status == "FAILED":
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"第三方任务执行失败（第{task['fail_count']}次）"
logger.warning(
f"任务状态更新 | task_name: {task['task_name']} | 提交失败 | 失败次数: {task['fail_count']}/{task['max_fail_threshold']}")
# 2.3 第三方状态为提交中：不更新状态
# 3. 提交成功状态：不处理
elif current_status == TASK_STATUS["SUCCEED"]:
continue
# 4. 提交失败状态：不处理（由提交线程判断是否重试）
elif current_status == TASK_STATUS["FAILED"]:
continue
# 检查是否所有任务已完成（成功或失败次数超阈值）
all_completed = self._check_all_tasks_completed()
if all_completed:
logger.info("所有任务已完成（成功或失败次数超过阈值）")
self.stop()
# 等待下次监控
self._stop_event.wait(self.check_interval)
logger.info("监控线程结束")
def _check_all_tasks_completed(self) -> bool:
"""检查是否所有任务已完成（成功或失败次数超阈值）"""
with task_map_lock:
for task in task_map.values():
# 待提交或提交中：未完成
if task["status"] in [TASK_STATUS["SUBMITTED"], TASK_STATUS["SUBMITTING"]]:
return False
# 提交失败但次数未超阈值：未完成（可能被提交线程重试）
if task["status"] == TASK_STATUS["FAILED"] and task["fail_count"] < task["max_fail_threshold"]:
return False
return True
def stop(self) -> None:
self._stop_event.set()
# -------------------------- 线程二：任务提交线程 --------------------------
class TaskSubmitThread(threading.Thread):
"""提交线程：按状态判断是否提交，处理待提交和未超阈值的失败任务"""
def __init__(self, max_workers: int = 3):
super().__init__(name="TaskSubmitThread")
self.max_workers = max_workers # 并发提交数
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"提交线程启动 | 并发数: {self.max_workers}")
while not self._stop_event.is_set():
# 1. 筛选符合条件的任务：待提交 或 失败次数未超阈值的提交失败任务
with task_map_lock:
pending_tasks = []
for task in task_map.values():
status = task["status"]
# 1.1 待提交状态：直接提交
if status == TASK_STATUS["SUBMITTED"]:
pending_tasks.append(task)
# 1.2 提交失败状态：检查失败次数，未超阈值则提交
elif status == TASK_STATUS["FAILED"]:
if task["fail_count"] < task["max_fail_threshold"]:
pending_tasks.append(task)
else:
logger.info(
f"任务 {task['task_name']} 失败次数超阈值（{task['max_fail_threshold']}），停止提交")
if not pending_tasks:
logger.info("无待提交任务，等待下次检查")
self._stop_event.wait(5)
continue
# 2. 并发提交任务
with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:
futures = {executor.submit(self.commit_task, task): task for task in pending_tasks}
for future in concurrent.futures.as_completed(futures):
task = futures[future]
try:
future.result()
except Exception as e:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"提交过程异常: {str(e)}"
logger.error(f"任务提交异常 | task_name: {task['task_name']} | 错误: {str(e)}")
logger.info("提交线程结束")
def commit_task(self, task: Dict) -> None:
"""提交单个任务（核心提交逻辑）"""
# 1. 标记任务状态为提交中
with task_map_lock:
task["status"] = TASK_STATUS["SUBMITTING"]
logger.info(f"开始提交任务 | task_name: {task['task_name']} | 当前状态: {task['status']}")
# 1.1 选择集群
cluster_id = select_cluster(task["resource"])
if not cluster_id:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "无满足资源需求的集群"
logger.error(f"任务提交失败 | task_name: {task['task_name']} | 原因: 无可用集群")
return
# 1.2 根据集群ID和算法ID查询子算法ID
son_code_id = get_son_code_id(cluster_id, task["code_id"])
if not son_code_id:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "未查询到子算法ID"
logger.error(f"任务提交失败 | task_name: {task['task_name']} | 原因: 子算法ID不存在")
return
# 1.3 获取认证Token
token = get_auth_token()
if not token:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = "获取认证Token失败"
logger.error(f"任务提交失败 | task_name: {task['task_name']} | 原因: Token获取失败")
return
# 2. 模拟调用第三方接口提交任务（实际场景替换为真实API）
try:
# 生成第三方任务ID（模拟接口返回）
third_party_task_id = f"task-{hash(task['target_id'])}"
logger.info(f"第三方任务提交成功 | task_name: {task['task_name']} | 第三方ID: {third_party_task_id}")
# 3. 更新任务信息（集群ID、子算法ID、第三方ID）
with task_map_lock:
task["cluster_id"] = cluster_id
task["son_code_id"] = son_code_id
task["third_party_task_id"] = third_party_task_id
logger.info(
f"任务提交信息更新 | task_name: {task['task_name']} | 集群ID: {cluster_id} | 子算法ID: {son_code_id}")
except Exception as e:
with task_map_lock:
task["status"] = TASK_STATUS["FAILED"]
task["fail_count"] += 1
task["error_msg"] = f"第三方接口调用失败: {str(e)}"
logger.error(f"任务提交失败 | task_name: {task['task_name']} | 原因: {str(e)}")
def stop(self) -> None:
self._stop_event.set()
# -------------------------- 主程序 --------------------------
if __name__ == "__main__":
# 1. 生成任务静态数据
task_templates = generate_task_templates()
# 2. 读取任务进入队列
load_tasks_to_queue(task_templates)
# 3. 启动监控线程
monitor_thread = TaskMonitorThread(check_interval=10)
monitor_thread.start()
# 4. 启动提交线程
submit_thread = TaskSubmitThread(max_workers=3)
submit_thread.start()
# 5. 等待线程结束
monitor_thread.join()
submit_thread.join()
logger.info("所有任务处理完毕，程序退出")

684
commit_tasks_0715.py Normal file
View File

@ -0,0 +1,684 @@
import requests
import json
import concurrent.futures
import time
import logging
import os
import threading
from uuid import uuid4
from typing import Dict, List, Optional
from datetime import datetime
# 日志配置
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
logger = logging.getLogger(__name__)
# 全局任务字典（key=target_id，value=任务详情）
task_map: Dict[str, Dict] = {}
# 线程锁
task_map_lock = threading.Lock()
# 集群资源
cluster_resources: Dict[str, Dict] = {
# modelarts集群 （ID：1790300942428540928）
"1790300942428540928": {
"total": {"CPU": 96, "MEMORY": 1024, "NPU": 4},
"available": {"CPU": 48, "MEMORY": 512, "NPU": 2}
},
# openi（集群ID：1865927992266461184）
"1865927992266461184": {
"total": {"CPU": 48, "MEMORY": 512, "NPU": 1},
"available": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
# octopus-189（章鱼集群，ID：1865927992266462181）
"1865927992266462181": {
"total": {"CPU": 48, "MEMORY": 512, "NPU": 1},
"available": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
# shuguangAi（曙光集群，ID：1777240145309732864）
"1777240145309732864": {
"total": {"CPU": 48, "MEMORY": 512, "NPU": 1},
"available": {"CPU": 24, "MEMORY": 256, "NPU": 1}
},
}
# 数据集
class DatasetInfo(Dict):
id: str # 数据集唯一标识
name: str # 数据集名称
size: float # 数据集大小（字节)
status: str # 状态（如："uploaded"）
is_uploaded: bool # 是否已上传
upload_time: Optional[datetime] # 上传时间（未上传时为None）
description: Optional[str] # 可选描述信息
# 算法
class AlgorithmInfo(Dict):
id: str # 算法唯一标识
name: str # 算法名称
size: float # 数据集大小（字节)
status: str # 状态（如："uploaded"）
is_uploaded: bool # 是否已上传
upload_time: Optional[datetime] # 上传时间（未上传时为None）
description: Optional[str] # 可选描述信息
# 集群数据
class ClusterDatas(Dict):
datasets: Dict[str, DatasetInfo] # Key is dataset ID
algorithms: Dict[str, AlgorithmInfo] # Key is algorithm ID
from typing import Dict, Optional
from datetime import datetime
class TaskTemplate(Dict):
"""集群任务模板数据结构（继承Dict，支持字典操作和类型注解）"""
# 任务核心标识
target_id: str # 任务唯一标识
task_name: str # 任务名称
package_name: str # 任务包名称
# 关联资源
dataset_name: str # 关联数据集名称
code_Id: str # 关联算法ID（原模板字段名保留）
# 资源配置（嵌套字典：CPU/MEMORY/NPU）
resource: Dict[str, int] # 资源配置字典，键为"CPU"/"MEMORY"/"NPU"，值为整数
# 任务状态与时间
status: str # 任务状态（如："submitted"）
submit_time: str # 提交时间（字符串格式，如："2025-07-15 19:42:03"）
# 附加信息
file_location: str # 任务文件存储路径
error_msg: str # 错误信息（为空表示无错误）
retry_count: int # 当前重试次数
max_retries: int # 最大重试次数
def __init__(self, **kwargs):
default_values = {
"target_id": "",
"task_name": "",
"package_name": "",
"dataset_name": "",
"code_Id": "",
"resource": {"CPU": 0, "MEMORY": 0, "NPU": 0}, # 资源默认值（整数类型）
"status": "submitted", # 初始状态：待提交
"submit_time": "",
"file_location": "",
"error_msg": "",
"retry_count": 0,
"max_retries": 5
}
# 调用父类Dict的初始化，确保支持字典操作（如task_template["status"]）
super().__init__(default_values)
# 将字段绑定为实例属性，支持点语法访问（如task_template.status）
for key, value in default_values.items():
setattr(self, key, value)
class TaskMonitorThread(threading.Thread):
"任务监控线程：轮询任务状态，重置可重试任务"
def __init__(self, check_interval: int = 30, name: Optional[str] = None):
super().__init__(name=name or "TaskMonitorThread")
self.check_interval = check_interval # 轮询间隔（秒）
self._stop_event = threading.Event()
self.all_tasks_completed = threading.Event() # 通知所有任务完成的事件
def run(self) -> None:
logger.info(f"监控线程启动 | 轮询间隔: {self.check_interval}秒 | 线程ID: {self.ident}")
while not self._stop_event.is_set():
with task_map_lock:
all_completed = True # 标记是否所有任务都已完成
retry_tasks = [] # 可重试任务列表
# 遍历所有任务检查状态
for task in task_map.values():
status = task["status"]
# 忽略已完成状态（成功）
if status in ["succeed"]:
continue
# 非完成状态，检查是否可重试
if status in ["failed", "error"]:
if task["retry_count"] < task["max_retries"]:
retry_tasks.append(task) # 加入重试列表
all_completed = False
else:
# 达到最大重试次数，标记为重试耗尽
task["status"] = "retry_exhausted"
logger.warning(f"任务 {task['task_name']} 达到最大重试次数({task['max_retries']})，停止重试")
all_completed = False
else:
# 其他状态（如submitted），未完成
all_completed = False
# 处理可重试任务（重置状态为submitted）
if retry_tasks:
logger.info(f"发现 {len(retry_tasks)} 个可重试任务，重置状态为submitted")
for task in retry_tasks:
task["status"] = "submitted"
task["retry_count"] += 1
logger.info(
f"任务 {task['task_name']} (target_id: {task['target_id']}) | "
f"重试次数: {task['retry_count']}/{task['max_retries']} | 状态已重置"
)
# 所有任务进入最终状态，通知提交线程停止
if all_completed:
logger.info("所有任务已进入最终状态（成功或重试耗尽）")
self.all_tasks_completed.set()
break
# 等待下一次轮询
self._stop_event.wait(self.check_interval)
logger.info(f"监控线程结束 | 线程ID: {self.ident}")
def stop(self) -> None:
"""停止监控线程"""
self._stop_event.set()
class TaskSubmitThread(threading.Thread):
"""任务提交线程：循环提交待处理任务，响应监控线程信号"""
def __init__(self, max_workers: int = 3, name: Optional[str] = None):
super().__init__(name=name or "TaskSubmitThread")
self.max_workers = max_workers # 并发提交数
self.monitor_thread = monitor_thread # 关联监控线程，获取完成信号
self._stop_event = threading.Event()
def run(self) -> None:
logger.info(f"任务提交线程启动 | 并发数: {self.max_workers} | 线程ID: {self.ident}")
try:
# 循环提交任务，直到监控线程通知所有任务完成
while not self._stop_event.is_set() and not self.monitor_thread.all_tasks_completed.is_set():
# 1. 查询集群资源
available_resources = search_resource()
if not available_resources:
logger.error("未获取到集群资源信息，10秒后重试...")
self._stop_event.wait(10)
continue
# 2. 提交满足条件的任务
commit_tasks(available_resources, self.max_workers)
# 3. 等待下次检查（避免高频查询）
self._stop_event.wait(5) # 5秒后再次检查任务队列
except Exception as e:
logger.error(f"任务提交线程异常终止: {str(e)}", exc_info=True)
finally:
logger.info(f"任务提交线程结束 | 线程ID: {self.ident}")
def stop(self) -> None:
"""停止提交线程"""
self._stop_event.set()
def generate_tasks() -> List[Dict]:
"""生成任务模板列表（包含差异化配置）"""
base_tasks = [
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AA",
"dataset_name": "data1.zip",
"code_Id": 1,
"file_location": "D:/数据集/cnn数据集/data1/",
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AB",
"dataset_name": "cifar-10-python.tar.gz",
"file_location": "D:/数据集/cnn数据集/data2/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AC",
"dataset_name": "cifar-100-python.tar.gz",
"file_location": "D:/数据集/cnn数据集/data3/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AD",
"dataset_name": "cifar-100-python.tar.gz",
"file_location": "D:/数据集/cnn数据集/data3/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AE",
"dataset_name": "dev.jsonl",
"file_location": "D:/数据集/transfomer数据集/BoolQ/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AF",
"dataset_name": "ceval.zip",
"file_location": "D:/数据集/transfomer数据集/CEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AG",
"dataset_name": "CMMLU.zip",
"file_location": "D:/数据集/transfomer数据集/CMMLU/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AH",
"dataset_name": "mental_health.csv",
"file_location": "D:/数据集/transfomer数据集/GLUE(imdb)/imdb/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AI",
"dataset_name": "GSM8K.jsonl",
"file_location": "D:/数据集/transfomer数据集/GSM8K/GSM8K/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AJ",
"dataset_name": "human-eval.jsonl",
"file_location": "D:/数据集/transfomer数据集/HumanEval/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
},
{
"task_name_template": "{prefix}-jointCloudAi-trainingtask",
"prefix": "AK",
"dataset_name": "HumanEval_X.zip",
"file_location": "D:/数据集/transfomer数据集/HumanEval_X/",
"code_Id": 1,
"CPU": 24,
"MEMORY": 256,
"NPU": 1
}
]
logger.info(f"成功生成 {len(base_tasks)} 个任务模板")
return base_tasks
from typing import List, Dict
import uuid
import time
import logging
from threading import Lock
# 全局变量定义
task_map = {}
task_map_lock = Lock()
# 定义任务模板结构（复用之前定义的）
task_template = {
"target_id": "",
"task_name": "",
"package_name": "",
"dataset_name": "",
"code_Id": "",
"resource": {
"CPU": 0,
"MEMORY": 0,
"NPU": 0
},
"status": "submitted",
"submit_time": "",
"file_location": "",
"error_msg": "",
"retry_count": 0,
"max_retries": 5
}
def read_tasks(templates: List[Dict]) -> None:
"""将任务模板转换为可提交的任务字典（新增重试相关字段）"""
global task_map, task_template
with task_map_lock:
task_map = {} # 清空历史任务
if not templates:
logger.warning("任务模板为空，跳过任务创建")
return
for template in templates:
try:
# 重置任务模板
task = task_template.copy()
target_id = str(uuid4()) # 生成唯一任务ID
submit_time_str = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
# 设置任务属性
task["target_id"] = target_id
task["task_name"] = template["task_name_template"].format(prefix=template["prefix"])
task["package_name"] = f"{template['prefix'].lower()}-training-pkg"
task["dataset_name"] = template["dataset_name"]
task["code_Id"] = template["code_Id"]
task["resource"] = {
"CPU": template["CPU"],
"MEMORY": template["MEMORY"],
"NPU": template["NPU"]
}
task["status"] = "submitted"
task["submit_time"] = submit_time_str
task["file_location"] = template["file_location"]
task_map[target_id] = task
logger.info(f"任务创建成功 | task_name: {task['task_name']} | target_id: {target_id}")
except KeyError as e:
logger.error(f"任务模板缺少字段: {e} | 模板内容: {template}")
except Exception as e:
logger.error(f"创建任务失败: {str(e)} | 模板内容: {template}")
def search_resource() -> Dict[str, Dict]:
"""查询集群资源（返回全局集群资源字典，包含总资源和可用资源）"""
global cluster_resources
logger.info(f"查询到集群资源: {cluster_resources}")
return cluster_resources
def get_token() -> Optional[str]:
"""获取认证Token"""
login_url = "http://119.45.255.234:30180/jcc-admin/admin/login"
login_payload = {"username": "admin", "password": "Nudt@123"}
try:
response = requests.post(login_url, json=login_payload, timeout=10)
response.raise_for_status()
result = response.json()
if result.get("code") == 200 and "data" in result and "token" in result["data"]:
logger.info("Token获取成功")
return result["data"]["token"]
else:
logger.error(f"Token获取失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"登录请求异常: {str(e)}", exc_info=True)
return None
def submit_single_task(task: Dict) -> bool:
"""提交单个任务到集群（失败时更新状态为failed/error）"""
token = get_token()
if not token:
with task_map_lock:
task["status"] = "failed"
task["error_msg"] = "获取Token失败"
return False
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {token}'
}
task_name = task["task_name"]
package_name = task["package_name"]
file_name = task["dataset_name"]
file_location = task["file_location"]
son_code_Id = task["code_Id"]
file_path = os.path.join(file_location, file_name)
try:
# 第一步：创建数据集文件夹
create_url = "http://119.45.255.234:30180/jsm/jobSet/createPackage"
create_payload = {
"userID": 5,
"name": package_name,
"dataType": "dataset",
"packageID": 0,
"uploadPriority": {"type": "specify", "clusters": ["1790300942428540928"]},
"bindingInfo": {
"clusterIDs": ["1790300942428540928"],
"name": package_name,
"category": "image",
"type": "dataset",
"imageID": "",
"bias": [],
"region": [],
"chip": ["ASCEND"],
"selectedCluster": [],
"modelType": "",
"env": "",
"version": "",
"packageID": 0,
"points": 0
}
}
create_resp = requests.post(create_url, json=create_payload, headers=headers, timeout=15)
create_resp.raise_for_status()
create_result = create_resp.json()
if create_result.get("code") != 200:
raise ValueError(f"创建文件夹失败 | API返回: {create_result}")
packageID = create_result["data"]["newPackage"]["packageID"]
logger.info(f"[{task_name}] 第一步：创建文件夹成功 | packageID: {packageID}")
# 第三步：上传数据集文件
upload_url = "http://119.45.255.234:30180/jcs/object/upload"
if not os.path.exists(file_path):
raise FileNotFoundError(f"数据集文件不存在 | path: {file_path}")
info_data = {
"userID": 5,
"packageID": packageID,
"loadTo": [3],
"loadToPath": [f"/dataset/5/{package_name}/"]
}
file_headers = {'Authorization': f'Bearer {token}'}
with open(file_path, 'rb') as f:
form_data = {"info": (None, json.dumps(info_data)), "files": f}
upload_resp = requests.post(upload_url, files=form_data, headers=file_headers, timeout=300)
upload_resp.raise_for_status()
upload_result = upload_resp.json()
if upload_result.get("code") != 200:
raise ValueError(f"文件上传失败 | API返回: {upload_result}")
object_id = upload_result["data"]["uploadeds"][0]["objectID"]
logger.info(f"[{task_name}] 第三步：文件上传成功 | objectID: {object_id}")
# 第四步：通知上传完成
notify_url = "http://119.45.255.234:30180/jsm/jobSet/notifyUploaded"
notify_payload = {
"userID": 5,
"packageID": packageID,
"uploadParams": {
"dataType": "dataset",
"uploadInfo": {"type": "local", "localPath": file_name, "objectIDs": [object_id]}
}
}
notify_resp = requests.post(notify_url, json=notify_payload, headers=headers, timeout=15)
notify_resp.raise_for_status()
notify_result = notify_resp.json()
if notify_result.get("code") != 200:
raise ValueError(f"通知上传完成失败 | API返回: {notify_result}")
logger.info(f"[{task_name}] 第四步：通知上传完成成功")
# 第七步：绑定数据集到集群
bind_url = "http://119.45.255.234:30180/jsm/jobSet/binding"
bind_payload = {
"userID": 5,
"info": {"type": "dataset", "packageID": packageID, "clusterIDs": ["1790300942428540928"]}
}
bind_resp = requests.post(bind_url, json=bind_payload, headers=headers, timeout=15)
bind_resp.raise_for_status()
bind_result = bind_resp.json()
if bind_result.get("code") != 200:
raise ValueError(f"绑定集群失败 | API返回: {bind_result}")
logger.info(f"[{task_name}] 第七步：数据集绑定集群成功")
# 第八步：查询绑定ID
query_bind_url = "http://119.45.255.234:30180/jsm/jobSet/queryBinding"
query_bind_payload = {
"dataType": "dataset",
"param": {"userID": 5, "bindingID": -1, "type": "private"}
}
query_bind_resp = requests.post(query_bind_url, json=query_bind_payload, headers=headers, timeout=15).json()
if query_bind_resp.get("code") != 200:
raise ValueError(f"查询绑定失败 | API返回: {query_bind_resp}")
# 提取目标绑定ID
target_id = None
for data in query_bind_resp["data"]["datas"]:
if data["info"]["name"] == package_name:
target_id = data["ID"]
break
if not target_id:
raise ValueError(f"未找到package_name={package_name}的绑定ID")
logger.info(f"[{task_name}] 第八步：获取绑定ID成功 | target_id: {target_id}")
# 第九步：提交训练任务
submit_url = "http://119.45.255.234:30180/jsm/jobSet/submit"
task_res = task["resource"]
submit_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"name": task_name,
"description": "自动提交的CNN训练任务",
"type": "AI",
"files": {
"dataset": {"type": "Binding", "bindingID": target_id},
"model": {"type": "Binding", "bindingID": 421},
"image": {"type": "Image", "imageID": 11}
},
"jobResources": {
"scheduleStrategy": "dataLocality",
"clusters": [
{
"clusterID": "1790300942428540928",
"runtime": {"envs": {}, "params": {}},
"code": {"type": "Binding", "bindingID": son_code_Id},
"resources": [
{"type": "CPU", "name": "ARM", "number": task_res["CPU"]},
{"type": "MEMORY", "name": "RAM", "number": task_res["MEMORY"]},
{"type": "MEMORY", "name": "VRAM", "number": 32},
{"type": "STORAGE", "name": "DISK", "number": 32},
{"type": "NPU", "name": "ASCEND910", "number": task_res["NPU"]}
]
}
]
}
},
{"localJobID": "4", "type": "DataReturn", "targetLocalJobID": "1"}
]
}
}
submit_resp = requests.post(submit_url, json=submit_payload, headers=headers, timeout=15).json()
if submit_resp.get("code") != 200:
raise ValueError(f"任务提交失败 | API返回: {submit_resp}")
logger.info(f"[{task_name}] 第九步：任务提交成功 | 任务ID: {submit_resp.get('data', {}).get('jobSetID')}")
# 更新任务状态为成功（线程安全）
with task_map_lock:
task["status"] = "succeed"
return True
except Exception as e:
error_msg = f"提交失败: {str(e)}"
with task_map_lock:
# 检查是否达到最大重试次数
if task["retry_count"] >= task["max_retries"]:
task["status"] = "retry_exhausted"
else:
task["status"] = "failed" # 未达最大次数，标记为failed等待重试
task["error_msg"] = error_msg
logger.error(f"[{task_name}] {error_msg}", exc_info=True)
return False
def commit_tasks(available_resources: Dict[str, int], max_workers: int = 3) -> None:
"""提交满足资源条件的任务（并发处理）"""
global task_map
with task_map_lock:
if not task_map:
logger.warning("无待提交任务，退出提交流程")
return
current_tasks = list(task_map.values()) # 复制当前任务列表，避免线程执行中被修改
# 筛选可提交任务（状态为submitted且资源满足）
eligible_tasks = []
for task in current_tasks:
if task["status"] != "submitted":
logger.info(f"任务 {task['task_name']} 状态为 {task['status']}，跳过提交")
continue
task_res = task["resource"]
# 检查CPU、内存、NPU是否均满足
if (task_res["CPU"] <= available_resources["CPU"] and
task_res["MEMORY"] <= available_resources["MEMORY"] and
task_res["NPU"] <= available_resources["NPU"]):
eligible_tasks.append(task)
logger.info(f"任务 {task['task_name']} 资源满足，加入提交队列")
else:
logger.warning(f"任务 {task['task_name']} 资源不足 | 需求: {task_res} | 可用: {available_resources}")
if not eligible_tasks:
logger.info("无满足资源条件的任务，提交流程结束")
return
# 并发提交任务
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {executor.submit(submit_single_task, task): task for task in eligible_tasks}
for future in concurrent.futures.as_completed(futures):
task = futures[future]
try:
future.result() # 触发可能的异常
except Exception as e:
with task_map_lock:
task["status"] = "error"
task["error_msg"] = f"执行异常: {str(e)}"
logger.error(f"任务 {task['task_name']} 执行异常: {str(e)}", exc_info=True)
if __name__ == "__main__":
# 1. 任务静态数据信息
task_templates = generate_tasks()
# 2. 读取任务数据信息，队列列表（存储到全局task_map，包含重试字段）
read_tasks(task_templates)
# 3. 创建监控任务状态线程（轮询间隔30秒，可调整）
monitor_thread = TaskMonitorThread(check_interval=30)
monitor_thread.start()
# 4. 创建任务提交线程（关联监控线程，并发数30）
submit_thread = TaskSubmitThread(max_workers=30, monitor_thread=monitor_thread)
submit_thread.start()
# 5. 等待监控线程完成
monitor_thread.join()
submit_thread.join()

33
config/auth.py
View File

@ -1,33 +0,0 @@
import requests
from typing import Optional
from config.db_manager import load_config_from_db
from config.config import logger
def get_token() -> Optional[str]:
"""获取认证Token"""
login_payload = {"username": "admin", "password": "Nudt@123"}
config = load_config_from_db()
try:
# 检查API配置是否已加载
response = requests.post(
config["api_config"]["login"]["url"],
json=login_payload,
timeout=config["api_config"]["login"]["timeout"]
)
response.raise_for_status()
result = response.json()
if result.get("code") == 200 and "data" in result and "token" in result["data"]:
logger.info("Token获取成功")
return result["data"]["token"]
else:
logger.error(f"Token获取失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"登录请求异常: {str(e)}", exc_info=True)
return None
except Exception as e:
logger.error(f"获取Token时发生未知异常: {str(e)}", exc_info=True)
return None

95
config/cache_manager.py
View File

@ -1,95 +0,0 @@
import time
from typing import Dict, Any, Optional
from config.db_manager import load_config_from_db, load_dataset_info_from_db
from config.config import logger
# 缓存配置
CACHE_EXPIRY_TIME = 30 # 缓存过期时间（秒）
# 全局缓存变量
_config_cache: Optional[Dict[str, Any]] = None
_config_cache_time: float = 0
_dataset_cache: Optional[Dict[str, Any]] = None
_dataset_cache_time: float = 0
def get_cached_config() -> Dict[str, Any]:
"""
获取缓存的配置信息，如果缓存过期则从数据库重新加载
:return: 配置信息字典
"""
global _config_cache, _config_cache_time
current_time = time.time()
# 检查缓存是否过期
if _config_cache is None or (current_time - _config_cache_time) > CACHE_EXPIRY_TIME:
logger.debug("配置缓存过期或不存在，从数据库重新加载")
try:
_config_cache = load_config_from_db()
_config_cache_time = current_time
logger.debug("配置缓存更新成功")
except Exception as e:
logger.error(f"从数据库加载配置时出错: {e}")
# 如果加载失败但有缓存，继续使用旧缓存
if _config_cache is None:
_config_cache = {}
return _config_cache
def get_cached_dataset_info() -> Dict[str, Any]:
"""
获取缓存的数据集信息，如果缓存过期则从数据库重新加载
:return: 数据集信息列表
"""
global _dataset_cache, _dataset_cache_time
current_time = time.time()
# 检查缓存是否过期
if _dataset_cache is None or (current_time - _dataset_cache_time) > CACHE_EXPIRY_TIME:
logger.debug("数据集缓存过期或不存在，从数据库重新加载")
try:
dataset_info_list = load_dataset_info_from_db()
# 将数据集信息转换为字典
_dataset_cache = {dataset["file_location"]: dataset for dataset in dataset_info_list}
_dataset_cache_time = current_time
logger.debug(f"数据集缓存更新成功，加载了 {len(dataset_info_list)} 条记录")
except Exception as e:
logger.error(f"从数据库加载数据集信息时出错: {e}")
# 如果加载失败但有缓存，继续使用旧缓存
if _dataset_cache is None:
_dataset_cache = {}
return _dataset_cache
def refresh_config_cache() -> None:
"""
强制刷新配置缓存
"""
global _config_cache, _config_cache_time
try:
_config_cache = load_config_from_db()
_config_cache_time = time.time()
logger.debug("配置缓存强制刷新成功")
except Exception as e:
logger.error(f"强制刷新配置缓存时出错: {e}")
def refresh_dataset_cache() -> None:
"""
强制刷新数据集缓存
"""
global _dataset_cache, _dataset_cache_time
try:
dataset_info_list = load_dataset_info_from_db()
# 将数据集信息转换为字典
_dataset_cache = {dataset["file_location"]: dataset for dataset in dataset_info_list}
_dataset_cache_time = time.time()
logger.debug(f"数据集缓存强制刷新成功，加载了 {len(dataset_info_list)} 条记录")
except Exception as e:
logger.error(f"强制刷新数据集缓存时出错: {e}")

98
config/config.py
View File

@ -1,98 +0,0 @@
import logging
import threading
from typing import Dict
# 日志配置
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[logging.StreamHandler()]
)
logger = logging.getLogger(__name__)
# 任务状态定义
TASK_STATUS = {
"SUBMITTED": "待提交", # 初始状态
"SUBMITTING": "提交中", # 提交过程中
"SUCCEED": "提交成功", # 提交成功
"FAILED": "提交失败", # 提交失败
"RETRY_EXHAUSTED": "重试耗尽" # 超过最大失败次数
}
# 全局变量
task_map = {}
task_map_lock = threading.Lock() # 任务字典线程锁
# 全局数据集映射（key=file_location，value=DatasetInfo实例）
dataset_map = {}
dataset_lock = threading.Lock() # 数据集映射线程锁
# 子算法-集群-数据集映射（从数据库加载）
ALGORITHM_MAPPING = {} # 结构: {son_code_id: {"clusters": [], "file_location": ""}}
# 任务模板（从数据库加载）
task_templates = []
# 集群资源配置（从数据库加载）
cluster_resources = {} # 结构同YAML定义
cluster_lock = threading.Lock() # 集群资源线程锁
# API配置（从数据库加载）
API_CONFIG = {}
# 集群价格配置（从数据库加载）
CLUSTER_PRICES = {}
def _load_db_config():
"""私有方法：加载数据库配置"""
from database import load_db_config
load_db_config()
def _init_dataset_map():
"""私有方法：初始化数据集映射"""
from database import init_dataset_map
init_dataset_map()
def get_algorithm_mapping():
"""从数据库获取算法映射"""
_load_db_config()
global ALGORITHM_MAPPING
return ALGORITHM_MAPPING
def get_task_templates():
"""从数据库获取任务模板"""
_load_db_config()
global task_templates
return task_templates
def get_cluster_resources():
"""从数据库获取集群资源"""
_load_db_config()
global cluster_resources
return cluster_resources
def get_api_config():
"""从数据库获取API配置"""
_load_db_config()
global API_CONFIG
return API_CONFIG
def get_cluster_prices():
"""从数据库获取集群价格"""
_load_db_config()
global CLUSTER_PRICES
return CLUSTER_PRICES
def get_dataset_map():
"""从数据库获取数据集映射"""
_init_dataset_map()
global dataset_map
return dataset_map

39
config/database.py
View File

@ -1,39 +0,0 @@
from config.db_manager import load_config_from_db, load_dataset_info_from_db, get_db_connection
from config.cache_manager import refresh_config_cache, refresh_dataset_cache
from config.config import logger
import threading
dataset_lock = threading.Lock()
def load_db_config() -> None:
"""
从数据库加载配置到缓存中
"""
try:
# 强制刷新配置缓存，确保获取最新的数据库配置
refresh_config_cache()
logger.info("数据库配置加载成功")
except Exception as e:
logger.error(f"从数据库加载配置时出错: {e}", exc_info=True)
def init_dataset_map() -> None:
"""
初始化数据集映射到缓存中
"""
try:
# 强制刷新数据集缓存，确保获取最新的数据集信息
refresh_dataset_cache()
logger.info("数据集映射初始化成功")
except Exception as e:
logger.error(f"初始化数据集映射时出错: {e}", exc_info=True)
def test_db_connection() -> bool:
"""测试数据库连接"""
connection = get_db_connection()
if connection:
connection.close()
return True
return False

17
config/dataset_info.py
View File

@ -1,17 +0,0 @@
import uuid
from typing import Optional, Any
class DatasetInfo(dict):
"""数据集信息结构"""
def __init__(self, file_location: str, name: str, size: float, **kwargs):
super().__init__()
self["file_location"] = file_location # 本地路径（主键）
self["id"] = kwargs.get("id", str(uuid.uuid4())) # 数据集唯一标识
self["name"] = name # 数据集名称
self["size"] = size # 大小（字节）
self["is_uploaded"] = kwargs.get("is_uploaded", False) # 是否已上传
self["upload_cluster"] = kwargs.get("upload_cluster", []) # 上传的集群
self["upload_time"] = kwargs.get("upload_time") # 上传时间
self["description"] = kwargs.get("description") # 描述
self["dataset_target_id"] = kwargs.get("dataset_target_id") # 绑定ID

626
config/db_manager.py
View File

@ -1,626 +0,0 @@
import mysql.connector
from mysql.connector import Error, pooling
from typing import Dict, List, Any, Optional
import json
import logging
# 配置数据库连接参数
DB_CONFIG = {
'host': '119.45.255.234',
'database': 'scheduling_simulator',
'user': 'root',
'password': 'uJpLd6u-J?HC1'
}
# 创建连接池
try:
db_pool = pooling.MySQLConnectionPool(
pool_name="scheduling_pool",
pool_size=10,
pool_reset_session=True,
**DB_CONFIG
)
logger = logging.getLogger(__name__)
logger.info("数据库连接池创建成功")
except Error as e:
logger = logging.getLogger(__name__)
logger.error(f"创建数据库连接池时出错: {e}")
db_pool = None
def get_db_connection():
"""
从连接池获取数据库连接
:return: 数据库连接对象
"""
try:
if db_pool:
connection = db_pool.get_connection()
if connection.is_connected():
logger.debug("成功从连接池获取数据库连接")
return connection
else:
logger.error("数据库连接池未初始化")
return None
except Error as e:
logger.error(f"从连接池获取数据库连接时出错: {e}")
return None
def get_file_location_by_dataset_id(dataset_id: str) -> Optional[str]:
"""
根据dataset_id从file_mapping表中查询file_location
:param dataset_id: 数据集ID
:return: 对应的文件路径，如果未找到则返回None
"""
connection = get_db_connection()
if not connection:
return None
cursor = connection.cursor()
try:
# 根据dataset_id查询file_location
cursor.execute("""
SELECT file_location
FROM file_mapping
WHERE id = %s
""", (dataset_id,))
result = cursor.fetchone()
if result:
return result[0]
else:
logger.warning(f"未找到dataset_id为 {dataset_id} 的记录")
return None
except Error as e:
logger.error(f"查询file_location时出错: {e}")
return None
finally:
cursor.close()
connection.close()
def update_cluster_resources(cluster_id: str, resource_changes: Dict[str, Any]) -> bool:
"""
更新集群资源（扣减或恢复资源）
:param cluster_id: 集群ID
:param resource_changes: 资源变化量，例如 {"CPU": -8, "MEMORY": -16, "NPU": -1} 表示扣减资源
或 {"CPU": 8, "MEMORY": 16, "NPU": 1} 表示恢复资源
:return: 是否更新成功
"""
connection = get_db_connection()
if not connection:
return False
cursor = connection.cursor()
try:
# 开始事务
connection.start_transaction()
# 先获取当前集群资源信息
cursor.execute('''
SELECT available_cpu, available_memory, available_card_json
FROM cluster_resources
WHERE cluster_id = %s
''', (cluster_id,))
result = cursor.fetchone()
if not result:
logger.error(f"未找到集群 {cluster_id} 的资源信息")
return False
available_cpu, available_memory, available_card_json = result
available_card_info = json.loads(available_card_json) if available_card_json else {}
# 更新资源
new_cpu = available_cpu + resource_changes.get("CPU", 0)
new_memory = available_memory + resource_changes.get("MEMORY", 0)
# 处理加速器资源变化
required_accelerator = next((key for key in resource_changes if key not in ["CPU", "MEMORY"]), None)
if required_accelerator:
# 处理card可能是列表或字典的情况
if isinstance(available_card_info, list):
# 列表形式：查找匹配类型的加速器并更新
for card in available_card_info:
if card.get("Type") == required_accelerator:
card["Num"] = card.get("Num", 0) + resource_changes.get(required_accelerator, 0)
break
else:
# 字典形式：直接更新
if available_card_info.get("Type") == required_accelerator:
available_card_info["Num"] = available_card_info.get("Num", 0) + resource_changes.get(
required_accelerator, 0)
# 将更新后的card信息转换为JSON字符串
updated_card_json = json.dumps(available_card_info) if available_card_info else None
# 更新数据库中的集群资源
cursor.execute('''
UPDATE cluster_resources
SET available_cpu = %s, available_memory = %s, available_card_json = %s
WHERE cluster_id = %s
''', (new_cpu, new_memory, updated_card_json, cluster_id))
# 提交事务
connection.commit()
logger.info(f"成功更新集群 {cluster_id} 的资源: {resource_changes}")
return True
except Error as e:
# 回滚事务
connection.rollback()
logger.error(f"更新集群 {cluster_id} 的资源时出错: {e}")
return False
finally:
cursor.close()
connection.close()
def init_database():
"""
初始化数据库，创建所需的表
"""
connection = get_db_connection()
if not connection:
return
cursor = connection.cursor()
# 创建集群资源表
create_cluster_resources_table = """
CREATE TABLE IF NOT EXISTS cluster_resources (
id INT AUTO_INCREMENT PRIMARY KEY,
cluster_id VARCHAR(50) NOT NULL UNIQUE COMMENT '集群ID',
cluster_type VARCHAR(20) NULL COMMENT '集群类型（Cloud/Ai/Hpc）',
total_cpu INT NULL COMMENT 'CPU总量',
total_memory INT NULL COMMENT '内存总量',
total_card_json JSON NULL COMMENT '加速卡总量信息',
available_cpu INT NULL COMMENT 'CPU可用量',
available_memory INT NULL COMMENT '内存可用量',
available_card_json JSON NULL COMMENT '加速卡可用量信息',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
upload_cluster_id INT NULL COMMENT '上传集群id',
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
"""
# 创建API配置表
create_api_config_table = """
CREATE TABLE IF NOT EXISTS api_config (
id INT AUTO_INCREMENT PRIMARY KEY,
config_key VARCHAR(50) NOT NULL COMMENT 'api名称',
url TEXT NULL COMMENT 'api地址',
timeout INT NULL COMMENT '超时时长',
`desc` VARCHAR(255) NULL COMMENT '描述',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE KEY config_key (config_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
"""
# 创建集群价格表
create_cluster_prices_table = """
CREATE TABLE IF NOT EXISTS cluster_prices (
id INT AUTO_INCREMENT PRIMARY KEY,
cluster_id VARCHAR(50) NOT NULL COMMENT '集群ID',
resource_type VARCHAR(20) NOT NULL COMMENT '资源类型',
price DECIMAL(10, 4) NULL COMMENT '资源价格',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
"""
# 创建文件映射表
create_file_mapping_table = """
CREATE TABLE IF NOT EXISTS file_mapping (
id INT AUTO_INCREMENT PRIMARY KEY COMMENT '数据集ID',
code_id INT NULL COMMENT '父算法ID',
son_code_id INT NULL COMMENT '子算法ID',
cluster_id VARCHAR(20) NULL COMMENT '集群ID',
file_location VARCHAR(512) NOT NULL COMMENT '本地文件路径',
file_type VARCHAR(255) NULL COMMENT 'algorithm为算法，dataset为数据集',
file_name VARCHAR(255) NULL COMMENT '数据集名称',
file_hash VARCHAR(255) NULL COMMENT '文件hash',
is_uploaded TINYINT(1) NULL COMMENT '是否已经上传，1为已上传，0为未上传',
size BIGINT NULL COMMENT '文件大小',
package_id BIGINT NULL COMMENT 'packageID',
object_id BIGINT NULL COMMENT 'objectID',
upload_time TIMESTAMP NULL COMMENT '上传时间',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE KEY file_location (file_location)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
"""
# 创建任务模板表
create_task_templates_table = """
CREATE TABLE IF NOT EXISTS task_templates (
id INT AUTO_INCREMENT PRIMARY KEY COMMENT '模拟器任务ID',
task_type VARCHAR(50) NOT NULL COMMENT '任务类型',
task_name VARCHAR(255) NULL COMMENT '任务名称',
dataset_id VARCHAR(255) NULL COMMENT '数据集ID',
strategy VARCHAR(50) NULL COMMENT '调度策略（resource对应资源优先策略，price对应价格优先策略，data对应数据优先策略)',
cpu INT NULL COMMENT '申请CPU数量',
mem INT NULL COMMENT '申请内存数量',
card_type VARCHAR(50) NULL COMMENT '卡类型',
card_num INT NULL COMMENT '卡数量',
status VARCHAR(255) NULL COMMENT '任务状态',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
"""
try:
# 使用事务确保初始化的原子性
connection.start_transaction()
# 创建表（不再删除旧表）
cursor.execute(create_cluster_resources_table)
cursor.execute(create_api_config_table)
cursor.execute(create_cluster_prices_table)
cursor.execute(create_file_mapping_table)
cursor.execute(create_task_templates_table)
connection.commit()
logger.info("数据库表初始化成功")
except Error as e:
connection.rollback()
logger.error(f"初始化数据库时出错: {e}")
# 重新抛出异常，以便调用者知道初始化失败
raise e
finally:
cursor.close()
connection.close()
def load_dataset_info_from_db() -> List[Dict[str, Any]]:
"""
从数据库加载数据集信息
:return: 数据集信息列表
"""
connection = get_db_connection()
if not connection:
return []
cursor = connection.cursor()
dataset_info_list = []
try:
# 从file_mapping表加载数据集信息
cursor.execute("""
SELECT file_location, file_name, id, is_uploaded, size, upload_time
FROM file_mapping
""")
rows = cursor.fetchall()
for row in rows:
file_location, file_name, file_id, is_uploaded, size, upload_time = row
dataset_info = {
"file_location": file_location,
"dataset_name": file_name,
"id": file_id,
"is_uploaded": is_uploaded,
"size": size,
"upload_time": upload_time.isoformat() if upload_time else ''
}
dataset_info_list.append(dataset_info)
logger.info(f"成功从数据库加载数据集信息 | 数量: {len(dataset_info_list)}")
except Error as e:
logger.error(f"从数据库加载数据集信息时出错: {e}")
finally:
cursor.close()
connection.close()
return dataset_info_list
def load_config_from_db() -> Dict[str, Any]:
"""
从数据库加载配置信息
:return: 包含集群资源、API配置、集群价格、文件映射和任务模板的配置字典
"""
connection = get_db_connection()
if not connection:
return {}
cursor = connection.cursor()
config = {
"cluster_resources": {},
"api_config": {},
"cluster_prices": {},
"file_mapping": {},
"task_templates": []
}
try:
# 加载集群资源信息
cursor.execute('''
SELECT cluster_id, cluster_type, total_cpu, total_memory, total_card_json,
available_cpu, available_memory, available_card_json
FROM cluster_resources
''')
resource_rows = cursor.fetchall()
for row in resource_rows:
cluster_id, cluster_type, total_cpu, total_memory, total_card_json, available_cpu, available_memory, available_card_json = row
total_card_info = json.loads(total_card_json) if total_card_json else {}
available_card_info = json.loads(available_card_json) if available_card_json else {}
config["cluster_resources"][cluster_id] = {
"cluster_type": cluster_type,
"total": {
"CPU": total_cpu,
"MEMORY": total_memory,
"card": total_card_info
},
"available": {
"CPU": available_cpu,
"MEMORY": available_memory,
"card": available_card_info
}
}
# 加载API配置信息
cursor.execute("SELECT config_key, url, timeout, `desc` FROM api_config")
api_rows = cursor.fetchall()
for config_key, url, timeout, desc in api_rows:
config["api_config"][config_key] = {
"url": url,
"timeout": timeout,
"desc": desc
}
# 加载集群价格信息
cursor.execute("SELECT cluster_id, resource_type, price FROM cluster_prices")
price_rows = cursor.fetchall()
cluster_price_mapping = {}
for cluster_id, resource_type, price in price_rows:
if cluster_id not in cluster_price_mapping:
cluster_price_mapping[cluster_id] = {}
cluster_price_mapping[cluster_id][resource_type] = float(price) if price is not None else 0.0
config["cluster_prices"] = cluster_price_mapping
# 加载文件映射信息（仅数据集）
cursor.execute(
"SELECT id, card_type,code_id, son_code_id, image_id, cluster_name, cluster_id, file_location, file_type, file_name, dataset_target_id, is_uploaded, size, upload_time FROM file_mapping")
file_mapping_rows = cursor.fetchall()
for row in file_mapping_rows:
file_id, card_type,code_id, son_code_id, image_id, cluster_name, cluster_id, file_location, file_type, file_name, dataset_target_id, is_uploaded, size, upload_time = row
config["file_mapping"][file_id] = {
"dataset_id": file_id,
"card_type": card_type,
"dataset_name": file_name,
"file_location": file_location,
"file_type": file_type,
"code_id": code_id,
"son_code_id": son_code_id,
"image_id":image_id,
"cluster_name": cluster_name,
"cluster_id": cluster_id,
"dataset_target_id": dataset_target_id,
"is_uploaded": is_uploaded,
"size": size,
"upload_time": upload_time.isoformat() if upload_time else ''
}
# task.setdefault("fail_count", 0)
# task.setdefault("max_fail_threshold", 3)
# task.setdefault("third_party_task_id", None)
# 加载任务模板信息
cursor.execute('''
SELECT id, task_type, task_name, dataset_id, strategy, cpu, mem, card_type, card_num, status,fail_count,max_fail_threshold,third_party_task_id,commit_time,end_time,upload_cluster_id,hpc_type,hpc_file,nodes,ntasks,package_name,hpc_partition,adapter_name,cloud_strategy,port,replicas
FROM task_templates
''')
task_template_rows = cursor.fetchall()
for row in task_template_rows:
id, task_type, task_name, dataset_id, strategy, cpu, mem, card_type, card_num, status, fail_count, max_fail_threshold, third_party_task_id, commit_time, end_time, upload_cluster_id,hpc_type,hpc_file,nodes,ntasks,package_name,hpc_partition,adapter_name,cloud_strategy,port,replicas = row
# 构建资源字典
resource = {}
if cpu is not None:
resource["CPU"] = cpu
if mem is not None:
resource["MEMORY"] = mem
if card_type and card_num is not None:
resource[card_type] = card_num
task_template = {
"id": id,
"type": task_type,
"task_name": task_name,
"dataset_id": dataset_id,
"card_type": card_type,
"strategy": strategy,
"resource": resource,
"status": status,
"fail_count": fail_count,
"max_fail_threshold": max_fail_threshold,
"third_party_task_id": third_party_task_id,
"commit_time": commit_time,
"end_time": end_time,
"upload_cluster_id": upload_cluster_id,
"hpc_type": hpc_type,
"hpc_file": hpc_file,
"nodes": nodes,
"ntasks": ntasks,
"package_name": package_name,
"partition": hpc_partition,
"adapter_name": adapter_name,
"cloud_strategy": cloud_strategy,
"port": port,
"replicas": replicas
}
config["task_templates"].append(task_template)
except Error as e:
logger.error(f"从数据库加载配置时出错: {e}")
finally:
cursor.close()
connection.close()
return config
def update_task_status(task_name: str, status: str, third_party_task_id: str = None) -> bool:
"""
更新任务状态
:param task_name: 任务名称
:param status: 新的状态
:param third_party_task_id: 第三方任务ID（可选）
:return: 是否更新成功
"""
connection = get_db_connection()
if not connection:
return False
cursor = connection.cursor()
try:
# 更新任务状态
if status == "SUBMITTING" and third_party_task_id:
cursor.execute('''
UPDATE task_templates
SET status = %s, third_party_task_id = %s, commit_time = NOW()
WHERE task_name = %s
''', (status, third_party_task_id, task_name))
elif third_party_task_id:
cursor.execute('''
UPDATE task_templates
SET status = %s, third_party_task_id = %s
WHERE task_name = %s
''', (status, third_party_task_id, task_name))
else:
cursor.execute('''
UPDATE task_templates
SET status = %s
WHERE task_name = %s
''', (status, task_name))
connection.commit()
logger.info(f"成功更新任务 {task_name} 的状态为 {status}")
return True
except Error as e:
connection.rollback()
logger.error(f"更新任务 {task_name} 状态时出错: {e}")
return False
finally:
cursor.close()
connection.close()
def update_task_all_fields(task_name: str, **kwargs) -> bool:
"""
更新任务的所有字段信息
:param task_name: 任务名称
:param kwargs: 要更新的字段和值
:return: 是否更新成功
"""
connection = get_db_connection()
if not connection:
return False
cursor = connection.cursor()
try:
# 构建动态更新语句
if not kwargs:
logger.warning("没有提供要更新的字段")
return False
# 过滤掉None值
update_fields = {k: v for k, v in kwargs.items() if v is not None}
if not update_fields:
logger.warning("所有字段值都为None，无需更新")
return False
# 构建SET子句
set_clause = ", ".join([f"{field} = %s" for field in update_fields.keys()])
values = list(update_fields.values())
values.append(task_name) # WHERE条件中的task_name
sql = f'''
UPDATE task_templates
SET {set_clause}
WHERE task_name = %s
'''
cursor.execute(sql, values)
connection.commit()
logger.info(f"成功更新任务 {task_name} 的字段: {list(update_fields.keys())}")
return True
except Error as e:
connection.rollback()
logger.error(f"更新任务 {task_name} 字段时出错: {e}")
return False
finally:
cursor.close()
connection.close()
def update_file_all_fields(dataset_id: str, **kwargs) -> bool:
"""
更新数据集文件表的所有字段信息
:param dataset_id: 数据集ID
:param kwargs: 要更新的字段和值
:return: 是否更新成功
"""
connection = get_db_connection()
if not connection:
return False
cursor = connection.cursor()
try:
# 构建动态更新语句
if not kwargs:
logger.warning("没有提供要更新的字段")
return False
# 过滤掉None值
update_fields = {k: v for k, v in kwargs.items() if v is not None}
if not update_fields:
logger.warning("所有字段值都为None，无需更新")
return False
# 构建SET子句
set_clause = ", ".join([f"{field} = %s" for field in update_fields.keys()])
values = list(update_fields.values())
values.append(dataset_id) # WHERE条件中的dataset_id
sql = f'''
UPDATE file_mapping
SET {set_clause}
WHERE id = %s
'''
cursor.execute(sql, values)
connection.commit()
logger.info(f"成功更新文件 {dataset_id} 的字段: {list(update_fields.keys())}")
return True
except Error as e:
connection.rollback()
logger.error(f"更新文件 {dataset_id} 字段时出错: {e}")
return False
finally:
cursor.close()
connection.close()

83
config/models.py
View File

@ -1,83 +0,0 @@
import time
import uuid
from typing import Dict, Any, Optional
from config.config import TASK_STATUS
class TaskInfo(dict):
"""任务信息结构 - 支持多种任务类型的通用结构"""
def __init__(self,
task_name: str,
task_type: str,
dataset_name: Optional[str] = None,
son_code_id: Optional[int] = None,
resource: Dict[str, Any] = None,
strategy: str = None, # 策略
**kwargs):
super().__init__()
# 通用基础字段
self["target_id"] = kwargs.get("target_id", str(uuid.uuid4())) # 任务唯一ID
self["task_name"] = task_name # 任务名称
self["package_name"] = kwargs.get("package_name", f"{task_name.lower()}-pkg") # 文件夹名称
self["type"] = task_type # 任务类型
self["status"] = kwargs.get("status", TASK_STATUS["SUBMITTED"]) # 任务状态
self["submit_time"] = kwargs.get("submit_time", time.strftime("%Y-%m-%d %H:%M:%S")) # 提交时间
self["success_time"] = kwargs.get("success_time", None) # 成功时间
self["third_party_task_id"] = kwargs.get("third_party_task_id", "") # 云际任务ID
self["error_msg"] = kwargs.get("error_msg", "") # 错误信息
self["fail_count"] = kwargs.get("fail_count", 0) # 失败次数
self["max_fail_threshold"] = kwargs.get("max_fail_threshold", 3) # 最大失败阈值
self["cluster_id"] = kwargs.get("cluster_id", "") # 提交的集群ID
self["strategy"] = strategy # 策略
# 根据任务类型设置特定字段
if task_type == "Ai":
# AI任务需要数据集名称（若不存在则设为空）
self["dataset_name"] = dataset_name if dataset_name is not None else ""
self["son_code_id"] = son_code_id # AI任务需要子算法ID
self["resource"] = resource or {"CPU": 1, "MEMORY": 4, "NPU": 0} # AI资源配置
self["file_location"] = kwargs.get("file_location", "") # 本地文件路径
elif task_type == "Hpc":
self["partition"] = kwargs.get("partition", "default") # HPC分区
self["ntasks"] = kwargs.get("ntasks", "1") # HPC任务数
self["nodes"] = kwargs.get("nodes", "1") # HPC节点数
self["packageName"] = kwargs.get("packageName", f"{task_name.lower()}-hpc") # HPC包名
self["hpcFile"] = kwargs.get("hpcFile", "") # HPC文件名
# HPC任务可选数据集名称
self["dataset_name"] = dataset_name if dataset_name is not None else ""
elif task_type == "Cloud":
self["replicas"] = kwargs.get("replicas", 1) # 副本数
self["strategy"] = kwargs.get("strategy", "resource") # 调度策略（对应原strategy字段）
self["cloud_strategy"] = kwargs.get("cloud_strategy", "replication") # 部署策略（新增）
self["adapter_name"] = kwargs.get("adapter_name", "") # 云服务适配器名称（新增）
self["resource"] = resource or {"CPU": 0.1, "MEMORY": 6, "port": 80} # 调整默认内存为6G（与YAML一致）
# 显式保留port字段（从资源配置中提取）
self["port"] = self["resource"].get("port", 80)
# 云任务可选数据集名称
self["dataset_name"] = dataset_name if dataset_name is not None else ""
else:
# 未知任务类型，数据集名称设为可选
self["dataset_name"] = dataset_name if dataset_name is not None else ""
self["son_code_id"] = son_code_id
self["resource"] = resource or {"CPU": 1, "MEMORY": 4}
def update_status(self, status: str, error_msg: str = "") -> None:
"""更新任务状态"""
self["status"] = status
if error_msg:
self["error_msg"] = error_msg
if status == TASK_STATUS["SUCCEED"]:
self["success_time"] = time.strftime("%Y-%m-%d %H:%M:%S")
def increment_fail_count(self) -> None:
"""增加失败次数"""
self["fail_count"] += 1
def get_resource_requirement(self) -> Dict[str, Any]:
"""获取资源需求"""
return self.get("resource", {})

203
mysql/simulator.sql
View File

@ -1,203 +0,0 @@
/*
Navicat Premium Data Transfer
Source Server : 119.45.255.234
Source Server Type : MySQL
Source Server Version : 80041 (8.0.41-0ubuntu0.22.04.1)
Source Host : 119.45.255.234:3306
Source Schema : scheduling_simulator
Target Server Type : MySQL
Target Server Version : 80041 (8.0.41-0ubuntu0.22.04.1)
File Encoding : 65001
Date: 08/09/2025 19:22:55
*/
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
-- ----------------------------
-- Table structure for algorithm_mapping
-- ----------------------------
DROP TABLE IF EXISTS `algorithm_mapping`;
CREATE TABLE `algorithm_mapping` (
`id` int NOT NULL AUTO_INCREMENT,
`code_id` int NULL DEFAULT NULL COMMENT '父算法ID',
`son_code_id` int NOT NULL COMMENT '子算法ID',
`file_location` text CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT '数据集对应本地路径',
`cluster_id` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '集群ID',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `unique_son_code`(`son_code_id` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 12 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of algorithm_mapping
-- ----------------------------
INSERT INTO `algorithm_mapping` VALUES (1, NULL, 1791, 'D:/数据集/cnn数据集/data2/', '1790300942428540928', '2025-09-08 17:02:07');
INSERT INTO `algorithm_mapping` VALUES (2, NULL, 1796, 'D:/数据集/cnn数据集/data3/', '1790300942428540928', '2025-09-08 17:02:07');
INSERT INTO `algorithm_mapping` VALUES (3, NULL, 1813, 'D:/数据集/transfomer数据集/BoolQ/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (4, NULL, 1816, 'D:/数据集/transfomer数据集/CEval/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (5, NULL, 1798, 'D:/数据集/transfomer数据集/GLUE(imdb)/imdb/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (6, NULL, 1818, 'D:/数据集/transfomer数据集/CMMLU/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (7, NULL, 1820, 'D:/数据集/transfomer数据集/GSM8K/GSM8K/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (8, NULL, 1822, 'D:/数据集/transfomer数据集/HumanEval/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (9, NULL, 1824, 'D:/数据集/transfomer数据集/HumanEval_X/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (10, NULL, 1794, 'D:/数据集/cnn数据集/data1/', '1790300942428540928', '2025-09-08 17:02:08');
INSERT INTO `algorithm_mapping` VALUES (11, NULL, 1367, 'D:/数据集/cnn数据集/data1/', '1865927992266461184', '2025-09-08 17:02:08');
-- ----------------------------
-- Table structure for api_config
-- ----------------------------
DROP TABLE IF EXISTS `api_config`;
CREATE TABLE `api_config` (
`id` int NOT NULL AUTO_INCREMENT,
`config_key` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT 'api名称',
`url` text CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT 'api地址',
`timeout` int NULL DEFAULT NULL COMMENT '超时时长',
`desc` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '描述',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `config_key`(`config_key` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 9 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of api_config
-- ----------------------------
INSERT INTO `api_config` VALUES (1, 'bind_cluster', 'http://jcc.jointcloud.net/jsm/jobSet/binding', 15, '绑定数据集', '2025-09-08 17:02:11');
INSERT INTO `api_config` VALUES (2, 'create_package', 'http://jcc.jointcloud.net/jsm/jobSet/createPackage', 15, '创建package', '2025-09-08 17:02:12');
INSERT INTO `api_config` VALUES (3, 'login', 'http://jcc.jointcloud.net/jcc-admin/admin/login', 10, '登录', '2025-09-08 17:02:12');
INSERT INTO `api_config` VALUES (4, 'notify_upload', 'http://jcc.jointcloud.net/jsm/jobSet/notifyUploaded', 15, '通知上传成功', '2025-09-08 17:02:14');
INSERT INTO `api_config` VALUES (5, 'query_binding', 'http://jcc.jointcloud.net/jsm/jobSet/queryBinding', 15, '查询绑定结果', '2025-09-08 17:02:15');
INSERT INTO `api_config` VALUES (6, 'submit_task', 'http://jcc.jointcloud.net/jsm/jobSet/submit', 100, '提交任务', '2025-09-08 17:02:16');
INSERT INTO `api_config` VALUES (7, 'task_detail', 'http://jcc.jointcloud.net/jsm/jobMgr/detail', 15, '任务详情', '2025-09-08 17:02:17');
INSERT INTO `api_config` VALUES (8, 'upload_file', 'http://localhost:32010/object/upload', 3000, '上传文件', '2025-09-08 17:02:18');
-- ----------------------------
-- Table structure for cluster_prices
-- ----------------------------
DROP TABLE IF EXISTS `cluster_prices`;
CREATE TABLE `cluster_prices` (
`id` int NOT NULL AUTO_INCREMENT,
`cluster_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '集群ID',
`resource_type` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '资源类型',
`price` decimal(10, 4) NULL DEFAULT NULL COMMENT '资源价格',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 7 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of cluster_prices
-- ----------------------------
INSERT INTO `cluster_prices` VALUES (1, '1865927992266461184', 'CPU', 0.1200, '2025-09-08 17:02:19');
INSERT INTO `cluster_prices` VALUES (2, '1865927992266461184', 'MEMORY', 0.0600, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (3, '1865927992266461184', 'NPU', 1.2000, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (4, '1790300942428540928', 'CPU', 0.1200, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (5, '1790300942428540928', 'MEMORY', 0.0600, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (6, '1790300942428540928', 'NPU', 1.2000, '2025-09-08 17:02:20');
-- ----------------------------
-- Table structure for cluster_resources
-- ----------------------------
DROP TABLE IF EXISTS `cluster_resources`;
CREATE TABLE `cluster_resources` (
`id` int NOT NULL AUTO_INCREMENT,
`cluster_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '集群ID',
`cluster_type` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '集群类型（Cloud/Ai/Hpc）',
`total_cpu` int NULL DEFAULT NULL COMMENT 'CPU总量',
`total_memory` int NULL DEFAULT NULL COMMENT '内存总量',
`total_card_json` json NULL COMMENT '加速卡总量信息',
`available_cpu` int NULL DEFAULT NULL COMMENT 'CPU可用量',
`available_memory` int NULL DEFAULT NULL COMMENT '内存可用量',
`available_card_json` json NULL COMMENT '加速卡可用量信息',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `cluster_id`(`cluster_id` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 11 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of cluster_resources
-- ----------------------------
INSERT INTO `cluster_resources` VALUES (1, '1763132837495574528', 'Cloud', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:08');
INSERT INTO `cluster_resources` VALUES (2, '1770703902472146944', 'Cloud', 1024, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:09');
INSERT INTO `cluster_resources` VALUES (3, '1790300942428540928', 'Ai', 1024, 2048, '{\"Num\": 56, \"Type\": \"NPU\"}', 2024, 4048, '{\"Num\": 56, \"Type\": \"NPU\"}', '2025-09-08 17:02:09');
INSERT INTO `cluster_resources` VALUES (4, '1830873578531228942', 'Hpc', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:09');
INSERT INTO `cluster_resources` VALUES (5, '1830873903296155648', 'Hpc', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:10');
INSERT INTO `cluster_resources` VALUES (6, '1830873903296155649', 'Hpc', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:10');
INSERT INTO `cluster_resources` VALUES (7, '1865927992266461184', 'Ai', 102, 409, '[{\"Num\": 72, \"Type\": \"NPU\"}, {\"Num\": 72, \"Type\": \"DCU\"}]', 102, 409, '[{\"Num\": 72, \"Type\": \"NPU\"}, {\"Num\": 72, \"Type\": \"DCU\"}]', '2025-09-08 17:02:11');
INSERT INTO `cluster_resources` VALUES (8, '1865927992266462180', 'Ai', 256, 512, '{\"Num\": 56, \"Type\": \"DCU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"DCU\"}', '2025-09-08 17:02:11');
INSERT INTO `cluster_resources` VALUES (9, '1865927992266462181', 'Ai', 50, 50, '{\"Num\": 56, \"Type\": \"NPU\"}', 50, 50, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:11');
INSERT INTO `cluster_resources` VALUES (10, '1865927992266462182', 'Ai', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:11');
-- ----------------------------
-- Table structure for file_mapping
-- ----------------------------
DROP TABLE IF EXISTS `file_mapping`;
CREATE TABLE `file_mapping` (
`id` int NOT NULL AUTO_INCREMENT COMMENT '数据集ID',
`file_location` varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '本地文件路径',
`file_type` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT 'algorithm为算法，dataset为数据集',
`file_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '数据集名称',
`dataset_target_id` bigint NULL DEFAULT NULL COMMENT '数据集ID',
`is_uploaded` tinyint(1) NULL DEFAULT NULL COMMENT '是否已经上传，1为已上传，0为未上传',
`size` bigint NULL DEFAULT NULL COMMENT '文件大小',
`package_id` bigint NULL DEFAULT NULL COMMENT 'packageID',
`object_id` bigint NULL DEFAULT NULL COMMENT 'objectID',
`upload_time` timestamp NULL DEFAULT NULL COMMENT '上传时间',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `file_location`(`file_location` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 11 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of file_mapping
-- ----------------------------
INSERT INTO `file_mapping` VALUES (1, 'D:/数据集/cnn数据集/data1/', NULL, 'data1.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (2, 'D:/数据集/cnn数据集/data2/', NULL, 'cifar-10-python.tar.gz', NULL, 0, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (3, 'D:/数据集/cnn数据集/data3/', NULL, 'cifar-100-python.tar.gz', NULL, 0, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (4, 'D:/数据集/transfomer数据集/BoolQ/', NULL, 'dev.jsonl', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (5, 'D:/数据集/transfomer数据集/CEval/', NULL, 'ceval.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (6, 'D:/数据集/transfomer数据集/CMMLU/', NULL, 'CMMLU.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (7, 'D:/数据集/transfomer数据集/GLUE(imdb)/imdb/', NULL, 'mental_health.csv', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `file_mapping` VALUES (8, 'D:/数据集/transfomer数据集/GSM8K/GSM8K/', NULL, 'GSM8K.jsonl', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `file_mapping` VALUES (9, 'D:/数据集/transfomer数据集/HumanEval/', NULL, 'HumanEval.zip', NULL, 0, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `file_mapping` VALUES (10, 'D:/数据集/transfomer数据集/HumanEval_X/', NULL, 'HumanEval_X.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
-- ----------------------------
-- Table structure for task_templates
-- ----------------------------
DROP TABLE IF EXISTS `task_templates`;
CREATE TABLE `task_templates` (
`id` int NOT NULL AUTO_INCREMENT COMMENT '模拟器任务ID',
`task_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '任务类型',
`task_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '任务名称',
`dataset_id` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '数据集ID',
`code_id` int NULL DEFAULT NULL COMMENT '算法ID',
`cpu` int NULL DEFAULT NULL COMMENT '申请CPU数量',
`strategy` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '调度策略（resource对应资源优先策略，price对应价格优先策略，data对应数据优先策略)',
`mem` int NULL DEFAULT NULL COMMENT '申请内存数量',
`card_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '卡类型',
`card_num` int NULL DEFAULT NULL COMMENT '卡数量',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 14 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of task_templates
-- ----------------------------
INSERT INTO `task_templates` VALUES (1, 'Ai', NULL, 'data1.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (2, 'Ai', NULL, 'data1.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (3, 'Ai', NULL, 'cifar-10-python.tar.gz', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (4, 'Ai', NULL, 'cifar-100-python.tar.gz', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (5, 'Ai', NULL, 'dev.jsonl', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (6, 'Ai', NULL, 'ceval.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (7, 'Ai', NULL, 'CMMLU.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (8, 'Ai', NULL, 'CMMLU.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (9, 'Ai', NULL, 'mental_health.csv', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (10, 'Ai', NULL, 'GSM8K.jsonl', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (11, 'Ai', NULL, 'GSM8K.jsonl', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (12, 'Ai', NULL, 'HumanEval_X.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `task_templates` VALUES (13, 'Ai', NULL, 'data1.zip', NULL, 0, 'resource', NULL, NULL, NULL, '2025-09-08 17:02:21');
SET FOREIGN_KEY_CHECKS = 1;

165
mysql/simulator_new.sql
View File

@ -1,165 +0,0 @@
/*
Navicat Premium Data Transfer
Source Server : 119.45.255.234
Source Server Type : MySQL
Source Server Version : 80041 (8.0.41-0ubuntu0.22.04.1)
Source Host : 119.45.255.234:3306
Source Schema : scheduling_simulator
Target Server Type : MySQL
Target Server Version : 80041 (8.0.41-0ubuntu0.22.04.1)
File Encoding : 65001
Date: 09/09/2025 17:09:21
*/
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
-- ----------------------------
-- Table structure for api_config
-- ----------------------------
DROP TABLE IF EXISTS `api_config`;
CREATE TABLE `api_config` (
`id` int NOT NULL AUTO_INCREMENT,
`config_key` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT 'api名称',
`url` text CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT 'api地址',
`timeout` int NULL DEFAULT NULL COMMENT '超时时长',
`desc` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '描述',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `config_key`(`config_key` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 9 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of api_config
-- ----------------------------
INSERT INTO `api_config` VALUES (1, 'bind_cluster', 'http://jcc.jointcloud.net/jsm/jobSet/binding', 15, '绑定数据集', '2025-09-08 17:02:11');
INSERT INTO `api_config` VALUES (2, 'create_package', 'http://jcc.jointcloud.net/jsm/jobSet/createPackage', 15, '创建package', '2025-09-08 17:02:12');
INSERT INTO `api_config` VALUES (3, 'login', 'http://jcc.jointcloud.net/jcc-admin/admin/login', 10, '登录', '2025-09-08 17:02:12');
INSERT INTO `api_config` VALUES (4, 'notify_upload', 'http://jcc.jointcloud.net/jsm/jobSet/notifyUploaded', 15, '通知上传成功', '2025-09-08 17:02:14');
INSERT INTO `api_config` VALUES (5, 'query_binding', 'http://jcc.jointcloud.net/jsm/jobSet/queryBinding', 15, '查询绑定结果', '2025-09-08 17:02:15');
INSERT INTO `api_config` VALUES (6, 'submit_task', 'http://jcc.jointcloud.net/jsm/jobSet/submit', 100, '提交任务', '2025-09-08 17:02:16');
INSERT INTO `api_config` VALUES (7, 'task_detail', 'http://jcc.jointcloud.net/jsm/jobMgr/detail', 15, '任务详情', '2025-09-08 17:02:17');
INSERT INTO `api_config` VALUES (8, 'upload_file', 'http://localhost:32010/object/upload', 3000, '上传文件', '2025-09-08 17:02:18');
-- ----------------------------
-- Table structure for cluster_prices
-- ----------------------------
DROP TABLE IF EXISTS `cluster_prices`;
CREATE TABLE `cluster_prices` (
`id` int NOT NULL AUTO_INCREMENT,
`cluster_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '集群ID',
`resource_type` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '资源类型',
`price` decimal(10, 4) NULL DEFAULT NULL COMMENT '资源价格',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 7 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of cluster_prices
-- ----------------------------
INSERT INTO `cluster_prices` VALUES (1, '1865927992266461184', 'CPU', 0.1200, '2025-09-08 17:02:19');
INSERT INTO `cluster_prices` VALUES (2, '1865927992266461184', 'MEMORY', 0.0600, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (3, '1865927992266461184', 'NPU', 1.2000, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (4, '1790300942428540928', 'CPU', 0.1200, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (5, '1790300942428540928', 'MEMORY', 0.0600, '2025-09-08 17:02:20');
INSERT INTO `cluster_prices` VALUES (6, '1790300942428540928', 'NPU', 1.2000, '2025-09-08 17:02:20');
-- ----------------------------
-- Table structure for cluster_resources
-- ----------------------------
DROP TABLE IF EXISTS `cluster_resources`;
CREATE TABLE `cluster_resources` (
`id` int NOT NULL AUTO_INCREMENT,
`cluster_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '集群ID',
`cluster_type` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '集群类型（Cloud/Ai/Hpc）',
`total_cpu` int NULL DEFAULT NULL COMMENT 'CPU总量',
`total_memory` int NULL DEFAULT NULL COMMENT '内存总量',
`total_card_json` json NULL COMMENT '加速卡总量信息',
`available_cpu` int NULL DEFAULT NULL COMMENT 'CPU可用量',
`available_memory` int NULL DEFAULT NULL COMMENT '内存可用量',
`available_card_json` json NULL COMMENT '加速卡可用量信息',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `cluster_id`(`cluster_id` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 11 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of cluster_resources
-- ----------------------------
INSERT INTO `cluster_resources` VALUES (1, '1763132837495574528', 'Cloud', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:08');
INSERT INTO `cluster_resources` VALUES (2, '1770703902472146944', 'Cloud', 1024, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:09');
INSERT INTO `cluster_resources` VALUES (3, '1790300942428540928', 'Ai', 1024, 2048, '{\"Num\": 56, \"Type\": \"NPU\"}', 2024, 4048, '{\"Num\": 56, \"Type\": \"NPU\"}', '2025-09-08 17:02:09');
INSERT INTO `cluster_resources` VALUES (4, '1830873578531228942', 'Hpc', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:09');
INSERT INTO `cluster_resources` VALUES (5, '1830873903296155648', 'Hpc', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:10');
INSERT INTO `cluster_resources` VALUES (6, '1830873903296155649', 'Hpc', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:10');
INSERT INTO `cluster_resources` VALUES (7, '1865927992266461184', 'Ai', 102, 409, '[{\"Num\": 72, \"Type\": \"NPU\"}, {\"Num\": 72, \"Type\": \"DCU\"}]', 102, 409, '[{\"Num\": 72, \"Type\": \"NPU\"}, {\"Num\": 72, \"Type\": \"DCU\"}]', '2025-09-08 17:02:11');
INSERT INTO `cluster_resources` VALUES (8, '1865927992266462180', 'Ai', 256, 512, '{\"Num\": 56, \"Type\": \"DCU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"DCU\"}', '2025-09-08 17:02:11');
INSERT INTO `cluster_resources` VALUES (9, '1865927992266462181', 'Ai', 50, 50, '{\"Num\": 56, \"Type\": \"NPU\"}', 54, 58, '{\"Num\": 51, \"Type\": \"NPU\"}', '2025-09-08 17:02:11');
INSERT INTO `cluster_resources` VALUES (10, '1865927992266462182', 'Ai', 512, 1024, '{\"Num\": 56, \"Type\": \"NPU\"}', 256, 512, '{\"Num\": 50, \"Type\": \"NPU\"}', '2025-09-08 17:02:11');
-- ----------------------------
-- Table structure for file_mapping
-- ----------------------------
DROP TABLE IF EXISTS `file_mapping`;
CREATE TABLE `file_mapping` (
`id` int NOT NULL AUTO_INCREMENT COMMENT '数据集ID',
`code_id` int NULL DEFAULT NULL COMMENT '父算法ID',
`son_code_id` int NULL DEFAULT NULL COMMENT '子算法ID',
`cluster_id` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '集群ID',
`file_location` varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '本地文件路径',
`file_type` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT 'algorithm为算法，dataset为数据集',
`file_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '数据集名称',
`dataset_target_id` bigint NULL DEFAULT NULL COMMENT '数据集ID',
`is_uploaded` tinyint(1) NULL DEFAULT NULL COMMENT '是否已经上传，1为已上传，0为未上传',
`size` bigint NULL DEFAULT NULL COMMENT '文件大小',
`package_id` bigint NULL DEFAULT NULL COMMENT 'packageID',
`object_id` bigint NULL DEFAULT NULL COMMENT 'objectID',
`upload_time` timestamp NULL DEFAULT NULL COMMENT '上传时间',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `file_location`(`file_location` ASC) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 11 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of file_mapping
-- ----------------------------
INSERT INTO `file_mapping` VALUES (1, NULL, NULL, NULL, 'D:/数据集/cnn数据集/data1/', NULL, 'data1.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (2, NULL, NULL, NULL, 'D:/数据集/cnn数据集/data2/', NULL, 'cifar-10-python.tar.gz', NULL, 0, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (3, NULL, NULL, NULL, 'D:/数据集/cnn数据集/data3/', NULL, 'cifar-100-python.tar.gz', NULL, 0, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (4, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/BoolQ/', NULL, 'dev.jsonl', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (5, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/CEval/', NULL, 'ceval.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (6, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/CMMLU/', NULL, 'CMMLU.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:20');
INSERT INTO `file_mapping` VALUES (7, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/GLUE(imdb)/imdb/', NULL, 'mental_health.csv', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `file_mapping` VALUES (8, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/GSM8K/GSM8K/', NULL, 'GSM8K.jsonl', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `file_mapping` VALUES (9, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/HumanEval/', NULL, 'HumanEval.zip', NULL, 0, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
INSERT INTO `file_mapping` VALUES (10, NULL, NULL, NULL, 'D:/数据集/transfomer数据集/HumanEval_X/', NULL, 'HumanEval_X.zip', NULL, 1, 0, NULL, NULL, NULL, '2025-09-08 17:02:21');
-- ----------------------------
-- Table structure for task_templates
-- ----------------------------
DROP TABLE IF EXISTS `task_templates`;
CREATE TABLE `task_templates` (
`id` int NOT NULL AUTO_INCREMENT COMMENT '模拟器任务ID',
`task_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '任务类型',
`task_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '任务名称',
`dataset_id` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '数据集ID',
`code_id` int NULL DEFAULT NULL COMMENT '算法ID',
`strategy` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '调度策略（resource对应资源优先策略，price对应价格优先策略，data对应数据优先策略)',
`cpu` int NULL DEFAULT NULL COMMENT '申请CPU数量',
`mem` int NULL DEFAULT NULL COMMENT '申请内存数量',
`card_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '卡类型',
`card_num` int NULL DEFAULT NULL COMMENT '卡数量',
`status` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '任务状态',
`created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 14 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of task_templates
-- ----------------------------
INSERT INTO `task_templates` VALUES (1, 'Ai', NULL, '1', NULL, 'resource', 4, 8, 'NPU', 1, NULL, '2025-09-08 17:02:21');
SET FOREIGN_KEY_CHECKS = 1;

525
mysql/sonCode_cluster_mapping.yaml
View File

@ -1,525 +0,0 @@
#algorithm_mapping:
# 1791:
# clusters: '1790300942428540928'
# file_location: D:/数据集/cnn数据集/data2/
# 1796:
# clusters: '1790300942428540928'
# file_location: D:/数据集/cnn数据集/data3/
# 1813:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/BoolQ/
# 1816:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/CEval/
# 1798:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/GLUE(imdb)/imdb/
# 1818:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/CMMLU/
# 1820:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/GSM8K/GSM8K/
# 1822:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/HumanEval/
# 1824:
# clusters: '1790300942428540928'
# file_location: D:/数据集/transfomer数据集/HumanEval_X/
# 1794:
# clusters: '1790300942428540928'
# file_location: D:/数据集/cnn数据集/data1/
# 1367:
# clusters: '1865927992266461184'
# file_location: D:/数据集/cnn数据集/data1/
#
#api_config:
# bind_cluster:
# timeout: 15
# url: http://jcc.jointcloud.net/jsm/jobSet/binding
# create_package:
# timeout: 15
# url: http://jcc.jointcloud.net/jsm/jobSet/createPackage
# login:
# timeout: 10
# url: http://jcc.jointcloud.net/jcc-admin/admin/login
# notify_upload:
# timeout: 15
# url: http://jcc.jointcloud.net/jsm/jobSet/notifyUploaded
# query_binding:
# timeout: 15
# url: http://jcc.jointcloud.net/jsm/jobSet/queryBinding
# submit_task:
# timeout: 100
# url: http://jcc.jointcloud.net/jsm/jobSet/submit
# task_detail:
# timeout: 15
# url: http://jcc.jointcloud.net/jsm/jobMgr/detail
# upload_file:
# timeout: 3000
# url: http://localhost:32010/object/upload
#cluster_prices:
# '1865927992266461184':
# CPU: 0.12
# MEMORY: 0.06
# NPU: 1.2
# '1790300942428540928':
# CPU: 0.12
# MEMORY: 0.06
# NPU: 1.2
#
#cluster_resources:
# '1763132837495574528':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: NPU
# cluster_type: Cloud
# total:
# CPU: 512
# MEMORY: 1024
# card:
# Num: 56
# Type: NPU
# '1770703902472146944':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: NPU
# cluster_type: Cloud
# total:
# CPU: 1024
# MEMORY: 1024
# card:
# Num: 56
# Type: NPU
# '1790300942428540928':
# available:
# CPU: 2024
# MEMORY: 4048
# card:
# Num: 56
# Type: NPU
# cluster_type: Ai
# total:
# CPU: 1024
# MEMORY: 2048
# card:
# Num: 56
# Type: NPU
# '1830873578531228942':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: NPU
# cluster_type: Hpc
# total:
# CPU: 512
# MEMORY: 1024
# card:
# Num: 56
# Type: NPU
# '1830873903296155648':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: NPU
# cluster_type: Hpc
# total:
# CPU: 512
# MEMORY: 1024
# card:
# Num: 56
# Type: NPU
# '1830873903296155649':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: NPU
# cluster_type: Hpc
# total:
# CPU: 512
# MEMORY: 1024
# card:
# Num: 56
# Type: NPU
# '1865927992266461184':
# available:
# CPU: 102
# MEMORY: 409
# card:
# - Num: 72
# Type: NPU
# - Num: 72
# Type: DCU
# cluster_type: Ai
# total:
# CPU: 102
# MEMORY: 409
# card:
# - Num: 72
# Type: NPU
# - Num: 72
# Type: DCU
# '1865927992266462180':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: DCU
# cluster_type: Ai
# total:
# CPU: 256
# MEMORY: 512
# card:
# Num: 56
# Type: DCU
# '1865927992266462181':
# available:
# CPU: 50
# MEMORY: 50
# card:
# Num: 50
# Type: NPU
# cluster_type: Ai
# total:
# CPU: 50
# MEMORY: 50
# card:
# Num: 56
# Type: NPU
# '1865927992266462182':
# available:
# CPU: 256
# MEMORY: 512
# card:
# Num: 50
# Type: NPU
# cluster_type: Ai
# total:
# CPU: 512
# MEMORY: 1024
# card:
# Num: 56
# Type: NPU
#
#
#file_mapping:
# D:/数据集/cnn数据集/data1/:
# dataset_name: data1.zip
# id: 1
# is_uploaded: true
# size: 0
# upload_time: ''
# D:/数据集/cnn数据集/data2/:
# dataset_name: cifar-10-python.tar.gz
# id: 2
# is_uploaded: false
# size: 0
# upload_time: ''
# D:/数据集/cnn数据集/data3/:
# dataset_name: cifar-100-python.tar.gz
# id: 3
# is_uploaded: false
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/BoolQ/:
# dataset_name: dev.jsonl
# id: 4
# is_uploaded: true
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/CEval/:
# dataset_name: ceval.zip
# id: 5
# is_uploaded: true
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/CMMLU/:
# dataset_name: CMMLU.zip
# id: 6
# is_uploaded: true
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/GLUE(imdb)/imdb/:
# dataset_name: mental_health.csv
# id: 7
# is_uploaded: true
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/GSM8K/GSM8K/:
# dataset_name: GSM8K.jsonl
# id: 8
# is_uploaded: true
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/HumanEval/:
# dataset_name: HumanEval.zip
# id: 9
# is_uploaded: false
# size: 0
# upload_time: ''
# D:/数据集/transfomer数据集/HumanEval_X/:
# dataset_name: HumanEval_X.zip
# id: 10
# is_uploaded: true
# size: 0
# upload_time: ''
#
#task_templates:
#- dataset_name: data1.zip
# file_location: D:/数据集/cnn数据集/data1/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# DCU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_ABtask_{{timestamp}}'
# type: Ai
#- dataset_name: data1.zip
# file_location: D:/数据集/cnn数据集/data1/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_ACtask_{{timestamp}}'
# type: Ai
#- dataset_name: cifar-10-python.tar.gz
# file_location: D:/数据集/cnn数据集/data2/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_ADtask_{{timestamp}}'
# type: Ai
#- dataset_name: cifar-100-python.tar.gz
# file_location: D:/数据集/cnn数据集/data3/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Aetask_{{timestamp}}'
# type: Ai
#- dataset_name: dev.jsonl
# file_location: D:/数据集/transfomer数据集/BoolQ/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Aftask_{{timestamp}}'
# type: Ai
#- dataset_name: ceval.zip
# file_location: D:/数据集/transfomer数据集/CEval/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Agtask_{{timestamp}}'
# type: Ai
#- dataset_name: CMMLU.zip
# file_location: D:/数据集/transfomer数据集/CMMLU/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Ahtask_{{timestamp}}'
# type: Ai
#- dataset_name: CMMLU.zip
# file_location: D:/数据集/transfomer数据集/CMMLU/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Aitask_{{timestamp}}'
# type: Ai
#- dataset_name: mental_health.csv
# file_location: D:/数据集/transfomer数据集/GLUE(imdb)/imdb/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Ajtask_{{timestamp}}'
# type: Ai
#- dataset_name: GSM8K.jsonl
# file_location: D:/数据集/transfomer数据集/GSM8K/GSM8K/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Aktask_{{timestamp}}'
# type: Ai
#- dataset_name: GSM8K.jsonl
# file_location: D:/数据集/transfomer数据集/GSM8K/GSM8K/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Altask_{{timestamp}}'
# type: Ai
#- dataset_name: HumanEval_X.zip
# file_location: D:/数据集/transfomer数据集/HumanEval_X/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Amtask_{{timestamp}}'
# type: Ai
#- dataset_name: data1.zip
# file_location: D:/数据集/cnn数据集/data1/
# prefix: Ai
# resource:
# CPU: 8
# MEMORY: 16
# NPU: 1
# son_code_id: null
# strategy: resource
# task_name_template: '{prefix}_Antask_{{timestamp}}'
# type: Ai
#- hpcFile: reference.fasta
# nodes: '1'
# ntasks: '1'
# packageName: bwa_data
# partition: ft_test
# prefix: hpc
# resource:
# CPU: 1
# MEMORY: 16
# strategy: resource
# task_name_template: '{prefix}_bwa_{{timestamp}}'
# type: Hpc
#- hpcFile: reads.fq
# nodes: '1'
# ntasks: '1'
# packageName: bwa_data
# partition: ft_test
# prefix: hpc
# resource:
# CPU: 1
# MEMORY: 16
# task_name_template: '{prefix}_bwa2_{{timestamp}}'
# type: Hpc
#- hpcFile: ''
# nodes: ''
# ntasks: ''
# packageName: ''
# partition: ''
# prefix: hpc
# resource:
# CPU: 1
# MEMORY: 16
# strategy: resource
# task_name_template: '{prefix}_hashcat_{{timestamp}}'
# type: Hpc
#- hpcFile: ''
# nodes: '1'
# ntasks: '1'
# packageName: lammps_data
# partition: ft_test
# prefix: hpc
# resource:
# CPU: 1
# MEMORY: 16
# strategy: resource
# task_name_template: '{prefix}_lammps_{{timestamp}}'
# type: Hpc
#- adapter_name: 阿里云数算
# cloud_strategy: replication
# port: 80
# prefix: contAinertask
# replicas: 1
# resource:
# CPU: 0.1
# MEMORY: 6
# strategy: resource
# task_name_template: '{prefix}_Acloud_{{timestamp}}'
# type: Cloud
#- adapter_name: 阿里云数算
# cloud_strategy: replication
# port: 80
# prefix: contAinertask
# replicas: 1
# resource:
# CPU: 0.1
# MEMORY: 6
# strategy: resource
# task_name_template: '{prefix}_Bcloud_{{timestamp}}'
# type: Cloud
#- adapter_name: 阿里云数算
# cloud_strategy: staticWeight
# port: 80
# prefix: contAinertask
# replicas: 1
# resource:
# CPU: 0.1
# MEMORY: 6
# strategy: resource
# task_name_template: '{prefix}_Ccloud_{{timestamp}}'
# type: Cloud
#- adapter_name: 腾讯云数算
# cloud_strategy: replication
# port: 80
# prefix: contAinertask
# replicas: 1
# resource:
# CPU: 0.1
# MEMORY: 6
# strategy: resource
# task_name_template: '{prefix}_Dcloud_{{timestamp}}'
# type: Cloud
#- adapter_name: 腾讯云数算
# cloud_strategy: staticWeight
# port: 80
# prefix: contAinertask
# replicas: 1
# resource:
# CPU: 0.1
# MEMORY: 6
# strategy: resource
# task_name_template: '{prefix}_Ecloud_{{timestamp}}'
# type: Cloud

196
mysql/upload_and_binding.py
View File

@ -1,196 +0,0 @@
# -*- coding: utf-8 -*-
import json
import os
from datetime import datetime
import pandas as pd
import requests
# =============================
# 基础配置
# =============================
TOKEN = "Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJsb2dpblR5cGUiOiJsb2dpbiIsImxvZ2luSWQiOjMsInJuU3RyIjoiTzZETElkYzFuZnhIVU5ZZ3lsczFIR0M5aTJRSHBoMTYiLCJ1c2VyX25hbWUiOiJhZG1pbiIsImlkIjozfQ.kzTxixrwZ7IvsdO8yrv50_huuFWWlspAyAA2OhEE8Q8"
HEADERS = {"Authorization": TOKEN}
USER_ID = 5
BUCKET_ID = 56
COPY_TO = [13]
APP_INSTANCE_ID = 2
STORAGE_PRESIGN_URL = "http://101.201.215.196:7891/storage/presign"
APP_SUBMIT_URL = "http://101.201.215.196:7891/app/submit"
# 输入文件夹路径（存放所有待上传文件）
LOCAL_FILES_DIR = r"C:\Users\Administrator\Desktop\算法重新上传"
# 输入的Excel文件路径
INPUT_EXCEL = r"C:\Users\Administrator\Desktop\算法重新上传\code.xlsx"
# 输出的结果Excel文件路径
OUTPUT_EXCEL = r"C:\Users\Administrator\Desktop\算法重新上传\output_result.xlsx"
# =============================
# 功能函数定义
# =============================
def create_presign(name: str):
"""
调用 /storage/presign 接口创建上传签名
"""
copy_path = f"/code/{name}"
payload = {
"userID": USER_ID,
"info": {
"type": "createUpload",
"params": {
"bucketID": BUCKET_ID,
"copyTo": COPY_TO,
"copyToPath": [copy_path],
"name": name
}
}
}
resp = requests.post(STORAGE_PRESIGN_URL, json=payload, headers=HEADERS, timeout=60)
if resp.status_code != 200:
raise Exception(f"Presign请求失败: {resp.status_code} - {resp.text}")
data = resp.json()
return data["data"]["presignUrl"]
def upload_file(presign_url: str, folder_name: str, local_file_path: str):
"""
上传文件到预签名URL
"""
if not os.path.exists(local_file_path):
raise FileNotFoundError(f"文件不存在: {local_file_path}")
with open(local_file_path, "rb") as f:
files = {
"files": (folder_name, f, "application/octet-stream")
}
resp = requests.post(presign_url, files=files, headers=HEADERS, timeout=120)
if resp.status_code != 200:
raise Exception(f"文件上传失败: {resp.status_code} - {resp.text}")
data = resp.json()
if data["code"] != "OK":
raise Exception(f"文件上传失败: {data['message']}")
package_id = data["data"]["package"]["packageID"]
object_id = data["data"]["objects"][0]["objectID"]
copied_to_paths = data["data"]["copiedToFullPaths"]
return package_id, object_id, copied_to_paths
def submit_app_binding(name, image_id, cluster_id, package_id, object_id, copied_to_paths):
"""
调用 /app/submit 接口完成绑定
"""
payload = {
"userID": USER_ID,
"appInstanceID": APP_INSTANCE_ID,
"appSetInfo": {
"apps": [
{
"type": "sceneAPI",
"name": name,
"description": "算法绑定",
"info": {
"type": "binding",
"objectIDs": [object_id],
"copiedTo": COPY_TO,
"copiedToFullRoots": copied_to_paths,
"bindingInfo": {
"type": "code",
"name": name,
"imageID": image_id,
"bootstrapObjectID": object_id,
"packageID": package_id,
"clusterID": str(cluster_id)
}
}
}
]
}
}
resp = requests.post(APP_SUBMIT_URL, json=payload, headers=HEADERS, timeout=60)
print("submit_app_binding: " + resp.text)
if resp.status_code != 200:
raise Exception(f"Submit失败: {resp.status_code} - {resp.text}")
data = resp.json()
try:
binding_info = json.loads(data["message"])
binding_id = binding_info.get("bindingID")
except Exception:
binding_id = None
return binding_id
def find_file_by_name(root_dir, target_name):
"""
在 root_dir 下递归查找与 target_name 同名的文件，返回完整路径。
若未找到则返回 None。
"""
for dirpath, _, filenames in os.walk(root_dir):
for filename in filenames:
if filename == target_name:
return os.path.join(dirpath, filename)
return None
# =============================
# 主逻辑
# =============================
def main():
df = pd.read_excel(INPUT_EXCEL)
results = []
for idx, row in df.iterrows():
name = row["name"]
image_id = int(row["imageID"])
cluster_id = str(row["clusterID"])
# 调用递归搜索函数
local_file_path = find_file_by_name(LOCAL_FILES_DIR, name)
if not local_file_path:
print(f"未找到与 {name} 匹配的文件，请检查目录 {LOCAL_FILES_DIR}")
continue
print(f"\n[{idx + 1}/{len(df)}] 处理文件: {name}")
try:
timestamp = int(datetime.timestamp(datetime.now()))
pkg_name = f"{timestamp}_{name}"
presign_url = create_presign(pkg_name)
print(f"获取 presignUrl 成功")
package_id, object_id, copied_to_paths = upload_file(presign_url, name, local_file_path)
print(f"上传成功 packageID={package_id}, objectID={object_id}")
binding_id = submit_app_binding(pkg_name, image_id, cluster_id, package_id, object_id, copied_to_paths)
print(f"提交成功 bindingID={binding_id}")
results.append({"name": pkg_name, "bindingID": binding_id})
except Exception as e:
print(f"处理失败: {e}")
results.append({"name": name, "bindingID": None})
continue
# 写入结果Excel
result_df = pd.DataFrame(results)
result_df.to_excel(OUTPUT_EXCEL, index=False)
print(f"\n全部完成, 结果已保存到: {OUTPUT_EXCEL}")
if __name__ == "__main__":
main()

349
scheduler/data_scheduler.py
View File

@ -1,349 +0,0 @@
import yaml
import os
import random
from typing import Dict, Any, List, Optional
from config.cache_manager import get_cached_config
from config.config import logger
def _select_by_data(
candidate_clusters: List[str],
cluster_resources: Dict[str, Dict],
task_resource: Dict,
dataset_id: str
) -> Optional[str]:
"""
基于数据位置的调度策略
:param candidate_clusters: 候选集群列表
:param cluster_resources: 集群资源信息
:param dataset_map: 数据集映射
:param dataset_id: 数据集ID
:return: 选中的集群ID
"""
try:
# 获取缓存的配置信息
config_data = get_cached_config()
file_mapping = config_data.get('file_mapping', {})
# 查找指定数据集的上传记录
dataset_upload_records = []
dataset_info = None
# 根据dataset_id查找数据集信息
for file_id, info in file_mapping.items():
if str(file_id) == str(dataset_id):
dataset_info = info
break
if not dataset_info:
logger.warning(f"未找到数据集ID {dataset_id} 的信息")
# 回退到随机选择
return random.choice(candidate_clusters) if candidate_clusters else None
dataset_name = dataset_info.get("dataset_name", "")
# 查找所有包含该数据集的集群上传记录
for file_id, info in file_mapping.items():
if info.get("dataset_name") == dataset_name and info.get("is_uploaded", 0) == 1:
dataset_upload_records.append({
"cluster_id": info.get("cluster_id"),
"file_location": info.get("file_location")
})
logger.info(f"数据集 {dataset_name} 在 {len(dataset_upload_records)} 个集群中存在上传记录")
# 情况一：file_mapping表中没有上传记录
if len(dataset_upload_records) == 0:
logger.info("情况一：没有集群上传记录")
# 1. 先把文件上传到JCS
# (上传逻辑应该在任务提交阶段处理，这里只需要选择集群)
# 选择资源最富裕的集群
selected_cluster = get_scheduled_cluster_id(candidate_clusters, task_resource, cluster_resources)
logger.info(f"选择资源最富裕的集群: {selected_cluster}")
return selected_cluster
# 情况二：file_mapping表中有1个集群的上传记录
elif len(dataset_upload_records) == 1:
cluster_id = dataset_upload_records[0]["cluster_id"]
logger.info(f"情况二：只有1个集群({cluster_id})有上传记录")
# 选择资源最富裕的集群
selected_cluster = get_scheduled_cluster_id(candidate_clusters, task_resource, cluster_resources)
logger.info(f"选择资源最富裕的集群: {selected_cluster}")
return selected_cluster
# 情况四：file_mapping表中有多个集群的上传记录
else:
logger.info(f"情况四：有{len(dataset_upload_records)}个集群有上传记录")
# 选择资源最富裕的集群
selected_cluster = get_scheduled_cluster_id(candidate_clusters, task_resource, cluster_resources)
logger.info(f"从多个有上传记录的集群中选择资源最富裕的集群: {selected_cluster}")
return selected_cluster
except Exception as e:
logger.error(f"基于数据的调度策略执行异常: {str(e)}", exc_info=True)
# 出错时回退到随机选择
return random.choice(candidate_clusters) if candidate_clusters else None
#
# def read_cluster_info(file_path: str) -> Dict[str, Dict[str, Any]]:
# """
# 读取集群资源信息的 YAML 文件。
# :param file_path: YAML 文件的路径。
# :return: 包含集群信息的字典，如果读取失败则返回空字典。
# """
# try:
# if not os.path.exists(file_path):
# raise FileNotFoundError(f"集群配置文件不存在: {file_path}")
# with open(file_path, 'r', encoding='utf-8') as f:
# return yaml.safe_load(f)
# except FileNotFoundError as fnf_error:
# print(fnf_error)
# except yaml.YAMLError as yaml_error:
# print(f"解析 YAML 文件时出错: {yaml_error}")
# return {}
def get_scheduled_cluster_id(candidate_clusters: List[str], task_resource: Dict, cluster_resources: Dict[str, Dict]) -> Optional[str]:
"""
根据任务资源需求和子算法ID选择合适的集群（优先从映射中选择）
:param task_resource: 任务的资源需求，例如 {"CPU": 8, "MEMORY": 16, "NPU": 1}
:param son_code_id: 子算法ID
:param cluster_resources: 集群资源配置，结构同YAML定义
:return: 选择的集群ID，如果没有合适的集群则返回None
"""
valid_clusters = []
for cluster_id in candidate_clusters:
cluster = cluster_resources.get(cluster_id)
if cluster:
available = cluster["available"]
# 检查CPU和MEMORY资源
cpu_available = available["CPU"] >= task_resource.get("CPU", 0)
memory_available = available["MEMORY"] >= task_resource.get("MEMORY", 0)
# 检查加速器资源（如NPU）
required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
accelerator_available = True
if required_accelerator:
# 处理 card 可能是列表或字典的情况
card_info = available["card"]
available_accelerator_num = 0
if isinstance(card_info, list):
# 如果 card 是列表，查找匹配类型的加速器
for card in card_info:
if card.get("Type") == required_accelerator:
available_accelerator_num = card.get("Num", 0)
break
elif isinstance(card_info, dict):
# 如果 card 是字典，直接获取
if card_info.get("Type") == required_accelerator:
available_accelerator_num = card_info.get("Num", 0)
required_accelerator_num = task_resource.get(required_accelerator, 0)
accelerator_available = available_accelerator_num >= required_accelerator_num
if cpu_available and memory_available and accelerator_available:
valid_clusters.append(cluster_id)
# 如果没有满足条件的集群，返回 None
if not valid_clusters:
return None
# 计算每个候选集群的资源富裕度分数
scores = {}
for cluster_id in valid_clusters:
cluster = cluster_resources.get(cluster_id)
if cluster:
available = cluster["available"]
score = 0
for resource_type, required in task_resource.items():
available_amount = 0
if resource_type in ["CPU", "MEMORY"]:
available_amount = available.get(resource_type, 0)
else:
# 处理加速器资源
card_info = available.get("card", {})
if isinstance(card_info, list):
# 如果 card 是列表，查找匹配类型的加速器
for card in card_info:
if card.get("Type") == resource_type:
available_amount = card.get("Num", 0)
break
elif isinstance(card_info, dict):
# 如果 card 是字典，直接获取
if card_info.get("Type") == resource_type:
available_amount = card_info.get("Num", 0)
# 计算资源富裕度分数
score += available_amount / required if required > 0 else 0
scores[cluster_id] = score / len(task_resource) if task_resource else 0
# 选择分数最高的集群
if scores:
return max(scores, key=scores.get)
# 如果无法计算分数，返回第一个有效集群
return valid_clusters[0] if valid_clusters else None
#
# def extract_resources(info: Dict[str, Any]) -> Dict[str, float]:
# """
# 从资源信息中提取 CPU、MEMORY 和 NPU 资源。
# :param info: 包含资源信息的字典。
# :return: 包含提取后资源的字典。
# """
# resources = {
# 'CPU': info.get('CPU', 0),
# 'MEMORY': info.get('MEMORY', 0),
# }
# npu_info = info.get('card')
# if npu_info and npu_info.get('Type') == 'NPU':
# resources['NPU'] = npu_info.get('Num', 0)
# return resources
#
# def parse_resource_request(task_resource: Dict[str, Any]) -> Dict[str, float]:
# """
# 解析任务的资源请求，将其转换为统一的资源字典格式。
#
# :param task_resource: 任务的资源请求字典。
# :return: 统一格式的资源字典。
# """
# return extract_resources(task_resource)
#
# def parse_cluster_resources(cluster_info: Dict[str, Any]) -> Dict[str, Dict[str, float]]:
# """
# 解析集群资源信息，将其转换为统一的资源字典格式。
#
# :param cluster_info: 包含集群信息的字典。
# :return: 统一格式的集群资源字典。
# """
# return {
# cluster_id: extract_resources(info.get('available', {}))
# for cluster_id, info in cluster_info.items()
# }
#
def select_best_cluster(task_resources: Dict[str, float],
cluster_resources: Dict[str, Dict[str, float]]) -> Optional[str]:
"""
选择最合适的集群。
策略：
1. 首先过滤掉不满足任务最低资源要求的集群。
2. 计算每个候选集群的资源富裕度分数。
3. 选择资源富裕度分数最高的集群。
:param task_resources: 任务的资源请求。
:param cluster_resources: 集群的可用资源。
:return: 最佳集群的 ID，如果没有找到则返回 None。
"""
# 过滤掉不满足最低资源要求的集群
candidate_clusters = {
cluster_id: resources
for cluster_id, resources in cluster_resources.items()
if all(resources.get(resource_type, 0) >= required for resource_type, required in task_resources.items())
}
# 如果没有候选集群，返回 None
if not candidate_clusters:
return None
# 计算每个候选集群的资源富裕度分数
scores = {
cluster_id: sum(resources.get(resource_type, 0) / required for resource_type, required in task_resources.items()) / len(task_resources)
for cluster_id, resources in candidate_clusters.items()
}
# 选择分数最高的集群
return max(scores, key=scores.get)
# def get_dataset_cluster_mapping(algorithm_mapping: Dict[int, Dict]) -> Dict[str, str]:
# """
# 获取数据集到集群的映射。
# :param algorithm_mapping: 子算法-集群-数据集映射。
# :return: 数据集到集群的映射。
# """
# dataset_cluster_mapping = {}
# for son_code_id, mapping in algorithm_mapping.items():
# file_location = mapping.get("file_location")
# clusters = mapping.get("clusters", [])
# if file_location and clusters:
# dataset_cluster_mapping[file_location] = clusters[0]
# return dataset_cluster_mapping
# def data_aware_scheduler(task_resource: Dict, son_code_id: int, algorithm_mapping: Dict[int, Dict], cluster_resources: Dict[str, Dict], dataset_cluster_mapping: Dict[str, str], first_time: bool) -> Optional[str]:
# """
# 数据感知调度策略。
# :param task_resource: 任务的资源需求。
# :param son_code_id: 子算法ID。
# :param algorithm_mapping: 子算法-集群-数据集映射。
# :param cluster_resources: 集群资源配置。
# :param dataset_cluster_mapping: 数据集到集群的映射。
# :param first_time: 是否是第一次提交任务。
# :return: 选择的集群ID，如果没有合适的集群则返回None。
# """
# if first_time:
# # 第一次提交任务，使用资源感知调度
# return get_scheduled_cluster_id(task_resource, son_code_id, algorithm_mapping, cluster_resources)
# else:
# # 后续提交任务，使用数据感知调度
# mapping = algorithm_mapping.get(son_code_id)
# if not mapping:
# return None
# file_location = mapping.get("file_location")
# if file_location:
# cluster_id = dataset_cluster_mapping.get(file_location)
# cluster = cluster_resources.get(cluster_id)
# if cluster:
# available = cluster["available"]
# # 检查CPU和MEMORY资源
# cpu_available = available["CPU"] >= task_resource.get("CPU", 0)
# memory_available = available["MEMORY"] >= task_resource.get("MEMORY", 0)
# # 检查加速器资源（如NPU）
# required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
# accelerator_available = True
# if required_accelerator:
# accelerator_available = available["card"]["Num"] >= task_resource.get(required_accelerator, 0)
#
# if cpu_available and memory_available and accelerator_available:
# return cluster_id
# return None
def main():
print("调度开始")
# # 任务资源请求
# task_resource = {"CPU": 8, "MEMORY": 16, "NPU": 1}
# son_code_id = 1127 # 示例子算法ID
#
# # 集群资源信息文件路径
# file_path = "../mysql/sonCode_cluster_mapping.yaml"
#
# # 读取集群信息
# cluster_info = read_cluster_info(file_path)
# if not cluster_info:
# print("无法获取集群信息，调度失败")
# return
#
# algorithm_mapping = cluster_info.get('algorithm_mapping', {})
# cluster_resources = cluster_info.get('cluster_resources', {})
#
# # 获取数据集到集群的映射
# dataset_cluster_mapping = get_dataset_cluster_mapping(algorithm_mapping)
#
# # 第一次提交任务
# first_time = True
# selected_cluster = data_aware_scheduler(task_resource, son_code_id, algorithm_mapping, cluster_resources, dataset_cluster_mapping, first_time)
# if selected_cluster:
# print(f"第一次提交任务，选择的集群: {selected_cluster}")
# else:
# print("第一次提交任务，没有找到满足条件的集群")
#
# # 后续提交任务
# first_time = False
# selected_cluster = data_aware_scheduler(task_resource, son_code_id, algorithm_mapping, cluster_resources, dataset_cluster_mapping, first_time)
# if selected_cluster:
# print(f"后续提交任务，选择的集群: {selected_cluster}")
# else:
# print("后续提交任务，没有找到满足条件的集群")
if __name__ == "__main__":
main()

100
scheduler/price_scheduler.py
View File

@ -1,100 +0,0 @@
from typing import Dict, Any, List, Optional
from typing import List, Dict, Optional
from config.config import logger
from typing import Dict, List, Optional
from config.config import logger
def _select_by_price(
candidate_clusters: List[str],
cluster_resources: Dict[str, Dict],
cluster_prices: Dict[str, Dict],
task_resource: Dict,
task_type: str
) -> Optional[str]:
"""
基于价格的调度策略（优化版）
:param candidate_clusters: 候选集群列表
:param cluster_resources: 集群资源信息
:param cluster_prices: 集群价格信息
:param task_resource: 任务资源需求
:param task_type: 任务类型
:return: 选中的集群ID
"""
try:
valid_clusters = []
# 1. 筛选满足资源需求的候选集群
for cluster_id in candidate_clusters:
if cluster_id not in cluster_resources or cluster_id not in cluster_prices:
continue
resources = cluster_resources[cluster_id]
# 检查集群类型是否匹配
if resources.get("cluster_type") != task_type:
continue
available = resources.get("available", {})
# 检查CPU和内存
cpu_available = available.get("CPU", 0) >= task_resource.get("CPU", 0)
memory_available = available.get("MEMORY", 0) >= task_resource.get("MEMORY", 0)
# 检查加速器
accelerator_available = True
required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
if required_accelerator:
card = available.get("card", {})
if isinstance(card, list):
found = False
for c in card:
if c.get("Type") == required_accelerator and c.get("Num", 0) >= task_resource.get(
required_accelerator, 0):
found = True
break
accelerator_available = found
else:
accelerator_available = (card.get("Type") == required_accelerator and
card.get("Num", 0) >= task_resource.get(required_accelerator, 0))
if cpu_available and memory_available and accelerator_available:
valid_clusters.append(cluster_id)
if not valid_clusters:
logger.warning("没有满足资源需求的集群")
return None
# 2. 计算价格和资源富裕度
cluster_info = []
for cluster_id in valid_clusters:
price = cluster_prices.get(cluster_id, {})
cost = (task_resource.get("CPU", 0) * price.get("CPU", 0) +
task_resource.get("MEMORY", 0) * price.get("MEMORY", 0))
required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
if required_accelerator:
cost += task_resource.get(required_accelerator, 0) * price.get(required_accelerator, 0)
# 计算资源富裕度
available = cluster_resources[cluster_id].get("available", {})
cpu_surplus = available.get("CPU", 0) - task_resource.get("CPU", 0)
memory_surplus = available.get("MEMORY", 0) - task_resource.get("MEMORY", 0)
surplus_score = cpu_surplus + memory_surplus # 简单相加作为富裕度分数
cluster_info.append((cluster_id, cost, surplus_score))
# 3. 按价格排序，如果价格相同则按资源富裕度排序
cluster_info.sort(key=lambda x: (x[1], -x[2]))
selected_cluster = cluster_info[0][0]
logger.info(f"基于价格策略选择集群: {selected_cluster}")
return selected_cluster
except Exception as e:
logger.error(f"基于价格的调度策略执行异常: {str(e)}", exc_info=True)
return None

27
scheduler/random_scheduler.py
View File

@ -1,27 +0,0 @@
import yaml
import os
import random
from typing import Dict, Any, List, Optional
from config.cache_manager import get_cached_config
from config.config import logger
def _select_by_random(
candidate_clusters: List[str],
) -> Optional[str]:
"""
基于数据位置的调度策略
:param candidate_clusters: 候选集群列表
:param cluster_resources: 集群资源信息
:param dataset_map: 数据集映射
:param dataset_id: 数据集ID
:return: 选中的集群ID
"""
try:
#遍历candidate_clusters列表，随机选一个集群返回
return random.choice(candidate_clusters)
except Exception as e:
logger.error(f"集群选择过程中发生异常: {str(e)}", exc_info=True)
return None

293
scheduler/resource_scheduler.py
View File

@ -1,293 +0,0 @@
import yaml
import os
import random
from typing import Dict, Any, List, Optional
from config.cache_manager import get_cached_config
from config.config import logger
def _select_by_resource(
candidate_clusters: List[str],
cluster_resources: Dict[str, Dict],
task_resource: Dict,
) -> Optional[str]:
"""
基于资源最大的调度策略
:param candidate_clusters: 候选集群列表
:param cluster_resources: 集群资源信息
:param dataset_map: 数据集映射
:param dataset_id: 数据集ID
:return: 选中的集群ID
"""
try:
# 不检查算法、镜像等可用性直接选择资源最富裕的集群，很有可能出现任务运行失败的情况
selected_cluster = get_scheduled_cluster_id(candidate_clusters, task_resource, cluster_resources)
logger.info(f"选择资源最富裕的集群: {selected_cluster}")
return selected_cluster
except Exception as e:
logger.error(f"基于数据的调度策略执行异常: {str(e)}", exc_info=True)
# 出错时回退到随机选择
return random.choice(candidate_clusters) if candidate_clusters else None
#
# def read_cluster_info(file_path: str) -> Dict[str, Dict[str, Any]]:
# """
# 读取集群资源信息的 YAML 文件。
# :param file_path: YAML 文件的路径。
# :return: 包含集群信息的字典，如果读取失败则返回空字典。
# """
# try:
# if not os.path.exists(file_path):
# raise FileNotFoundError(f"集群配置文件不存在: {file_path}")
# with open(file_path, 'r', encoding='utf-8') as f:
# return yaml.safe_load(f)
# except FileNotFoundError as fnf_error:
# print(fnf_error)
# except yaml.YAMLError as yaml_error:
# print(f"解析 YAML 文件时出错: {yaml_error}")
# return {}
def get_scheduled_cluster_id(candidate_clusters: List[str], task_resource: Dict, cluster_resources: Dict[str, Dict]) -> Optional[str]:
"""
根据任务资源需求和子算法ID选择合适的集群（优先从映射中选择）
:param task_resource: 任务的资源需求，例如 {"CPU": 8, "MEMORY": 16, "NPU": 1}
:param son_code_id: 子算法ID
:param cluster_resources: 集群资源配置，结构同YAML定义
:return: 选择的集群ID，如果没有合适的集群则返回None
"""
valid_clusters = []
for cluster_id in candidate_clusters:
cluster = cluster_resources.get(cluster_id)
if cluster:
available = cluster["available"]
# 检查CPU和MEMORY资源
cpu_available = available["CPU"] >= task_resource.get("CPU", 0)
memory_available = available["MEMORY"] >= task_resource.get("MEMORY", 0)
# 检查加速器资源（如NPU）
required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
accelerator_available = True
if required_accelerator:
# 处理 card 可能是列表或字典的情况
card_info = available["card"]
available_accelerator_num = 0
if isinstance(card_info, list):
# 如果 card 是列表，查找匹配类型的加速器
for card in card_info:
if card.get("Type") == required_accelerator:
available_accelerator_num = card.get("Num", 0)
break
elif isinstance(card_info, dict):
# 如果 card 是字典，直接获取
if card_info.get("Type") == required_accelerator:
available_accelerator_num = card_info.get("Num", 0)
required_accelerator_num = task_resource.get(required_accelerator, 0)
accelerator_available = available_accelerator_num >= required_accelerator_num
if cpu_available and memory_available and accelerator_available:
valid_clusters.append(cluster_id)
# 如果没有满足条件的集群，返回 None
if not valid_clusters:
return None
# 计算每个候选集群的资源富裕度分数
scores = {}
for cluster_id in valid_clusters:
cluster = cluster_resources.get(cluster_id)
if cluster:
available = cluster["available"]
score = 0
for resource_type, required in task_resource.items():
available_amount = 0
if resource_type in ["CPU", "MEMORY"]:
available_amount = available.get(resource_type, 0)
else:
# 处理加速器资源
card_info = available.get("card", {})
if isinstance(card_info, list):
# 如果 card 是列表，查找匹配类型的加速器
for card in card_info:
if card.get("Type") == resource_type:
available_amount = card.get("Num", 0)
break
elif isinstance(card_info, dict):
# 如果 card 是字典，直接获取
if card_info.get("Type") == resource_type:
available_amount = card_info.get("Num", 0)
# 计算资源富裕度分数
score += available_amount / required if required > 0 else 0
scores[cluster_id] = score / len(task_resource) if task_resource else 0
# 选择分数最高的集群
if scores:
return max(scores, key=scores.get)
# 如果无法计算分数，返回第一个有效集群
return valid_clusters[0] if valid_clusters else None
#
# def extract_resources(info: Dict[str, Any]) -> Dict[str, float]:
# """
# 从资源信息中提取 CPU、MEMORY 和 NPU 资源。
# :param info: 包含资源信息的字典。
# :return: 包含提取后资源的字典。
# """
# resources = {
# 'CPU': info.get('CPU', 0),
# 'MEMORY': info.get('MEMORY', 0),
# }
# npu_info = info.get('card')
# if npu_info and npu_info.get('Type') == 'NPU':
# resources['NPU'] = npu_info.get('Num', 0)
# return resources
#
# def parse_resource_request(task_resource: Dict[str, Any]) -> Dict[str, float]:
# """
# 解析任务的资源请求，将其转换为统一的资源字典格式。
#
# :param task_resource: 任务的资源请求字典。
# :return: 统一格式的资源字典。
# """
# return extract_resources(task_resource)
#
# def parse_cluster_resources(cluster_info: Dict[str, Any]) -> Dict[str, Dict[str, float]]:
# """
# 解析集群资源信息，将其转换为统一的资源字典格式。
#
# :param cluster_info: 包含集群信息的字典。
# :return: 统一格式的集群资源字典。
# """
# return {
# cluster_id: extract_resources(info.get('available', {}))
# for cluster_id, info in cluster_info.items()
# }
#
def select_best_cluster(task_resources: Dict[str, float],
cluster_resources: Dict[str, Dict[str, float]]) -> Optional[str]:
"""
选择最合适的集群。
策略：
1. 首先过滤掉不满足任务最低资源要求的集群。
2. 计算每个候选集群的资源富裕度分数。
3. 选择资源富裕度分数最高的集群。
:param task_resources: 任务的资源请求。
:param cluster_resources: 集群的可用资源。
:return: 最佳集群的 ID，如果没有找到则返回 None。
"""
# 过滤掉不满足最低资源要求的集群
candidate_clusters = {
cluster_id: resources
for cluster_id, resources in cluster_resources.items()
if all(resources.get(resource_type, 0) >= required for resource_type, required in task_resources.items())
}
# 如果没有候选集群，返回 None
if not candidate_clusters:
return None
# 计算每个候选集群的资源富裕度分数
scores = {
cluster_id: sum(resources.get(resource_type, 0) / required for resource_type, required in task_resources.items()) / len(task_resources)
for cluster_id, resources in candidate_clusters.items()
}
# 选择分数最高的集群
return max(scores, key=scores.get)
# def get_dataset_cluster_mapping(algorithm_mapping: Dict[int, Dict]) -> Dict[str, str]:
# """
# 获取数据集到集群的映射。
# :param algorithm_mapping: 子算法-集群-数据集映射。
# :return: 数据集到集群的映射。
# """
# dataset_cluster_mapping = {}
# for son_code_id, mapping in algorithm_mapping.items():
# file_location = mapping.get("file_location")
# clusters = mapping.get("clusters", [])
# if file_location and clusters:
# dataset_cluster_mapping[file_location] = clusters[0]
# return dataset_cluster_mapping
# def data_aware_scheduler(task_resource: Dict, son_code_id: int, algorithm_mapping: Dict[int, Dict], cluster_resources: Dict[str, Dict], dataset_cluster_mapping: Dict[str, str], first_time: bool) -> Optional[str]:
# """
# 数据感知调度策略。
# :param task_resource: 任务的资源需求。
# :param son_code_id: 子算法ID。
# :param algorithm_mapping: 子算法-集群-数据集映射。
# :param cluster_resources: 集群资源配置。
# :param dataset_cluster_mapping: 数据集到集群的映射。
# :param first_time: 是否是第一次提交任务。
# :return: 选择的集群ID，如果没有合适的集群则返回None。
# """
# if first_time:
# # 第一次提交任务，使用资源感知调度
# return get_scheduled_cluster_id(task_resource, son_code_id, algorithm_mapping, cluster_resources)
# else:
# # 后续提交任务，使用数据感知调度
# mapping = algorithm_mapping.get(son_code_id)
# if not mapping:
# return None
# file_location = mapping.get("file_location")
# if file_location:
# cluster_id = dataset_cluster_mapping.get(file_location)
# cluster = cluster_resources.get(cluster_id)
# if cluster:
# available = cluster["available"]
# # 检查CPU和MEMORY资源
# cpu_available = available["CPU"] >= task_resource.get("CPU", 0)
# memory_available = available["MEMORY"] >= task_resource.get("MEMORY", 0)
# # 检查加速器资源（如NPU）
# required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
# accelerator_available = True
# if required_accelerator:
# accelerator_available = available["card"]["Num"] >= task_resource.get(required_accelerator, 0)
#
# if cpu_available and memory_available and accelerator_available:
# return cluster_id
# return None
def main():
print("调度开始")
# # 任务资源请求
# task_resource = {"CPU": 8, "MEMORY": 16, "NPU": 1}
# son_code_id = 1127 # 示例子算法ID
#
# # 集群资源信息文件路径
# file_path = "../mysql/sonCode_cluster_mapping.yaml"
#
# # 读取集群信息
# cluster_info = read_cluster_info(file_path)
# if not cluster_info:
# print("无法获取集群信息，调度失败")
# return
#
# algorithm_mapping = cluster_info.get('algorithm_mapping', {})
# cluster_resources = cluster_info.get('cluster_resources', {})
#
# # 获取数据集到集群的映射
# dataset_cluster_mapping = get_dataset_cluster_mapping(algorithm_mapping)
#
# # 第一次提交任务
# first_time = True
# selected_cluster = data_aware_scheduler(task_resource, son_code_id, algorithm_mapping, cluster_resources, dataset_cluster_mapping, first_time)
# if selected_cluster:
# print(f"第一次提交任务，选择的集群: {selected_cluster}")
# else:
# print("第一次提交任务，没有找到满足条件的集群")
#
# # 后续提交任务
# first_time = False
# selected_cluster = data_aware_scheduler(task_resource, son_code_id, algorithm_mapping, cluster_resources, dataset_cluster_mapping, first_time)
# if selected_cluster:
# print(f"后续提交任务，选择的集群: {selected_cluster}")
# else:
# print("后续提交任务，没有找到满足条件的集群")
if __name__ == "__main__":
main()

119
scheduler/scheduler.py
View File

@ -1,119 +0,0 @@
import random
from typing import Dict, List, Optional
from config.config import logger
from config.cache_manager import get_cached_config, get_cached_dataset_info
from scheduler.data_scheduler import _select_by_data
from scheduler.price_scheduler import _select_by_price
from scheduler.random_scheduler import _select_by_random
from scheduler.resource_scheduler import _select_by_resource
from scheduler.storage_computing_scheduler import _select_by_storage_compute
def select_cluster(task: Dict) -> Optional[str]:
"""
根据任务信息和调度策略选择合适的集群
:param task: 任务信息
:return: 选中的集群ID，如果未找到合适的集群则返回None
"""
global file_name
try:
if task.get("type") == "Hpc":
return "1830873578531228942"
if task.get("type") == "Cloud" and "ali" in task.get("task_name"):
return "1770703902472146944"
if task.get("type") == "Cloud" and "tct" in task.get("task_name"):
return "1865927992266462782"
# 使用缓存的配置而不是每次都重新加载
config_data = get_cached_config()
if not config_data:
logger.error("无法获取配置数据，缓存为空")
return None
dataset_map = get_cached_dataset_info()
# 获取全局配置
strategy = task.get("strategy")
task_type = task["type"]
code_id = task.get("code_id", "")
dataset_id = task.get("dataset_id", "")
task_resource = task.get("resource", {})
file_location = task.get("file_location", "")
file_type = task.get("file_type", "")
file_mapping = config_data.get('file_mapping', {})
# 获取算法映射关系
if not file_mapping:
logger.warning("未找到算法映射关系")
return None
#获取该任务数据集ID对应的本地文件路径及名称
for file_id, info in file_mapping.items():
if file_id == dataset_id:
file_name = info.get("dataset_name", "")
file_location = info.get("file_location", "")
card_type = info.get("card_type", "")
break
logger.info(f"开始集群选择 | 任务类型: {task_type} | 调度策略: {strategy}")
logger.info(f"任务资源需求: {task_resource}")
cluster_resources = config_data.get('cluster_resources', {})
cluster_prices = config_data.get('cluster_prices', {})
# 优化候选集群筛选逻辑，增加数据校验
candidate_clusters = []
for file_id, info in file_mapping.items():
if not isinstance(info, dict):
logger.warning(f"算法映射格式错误: {info}")
continue
# 找到所有数据集名称和路径匹配的集群ID
if file_name == info.get("dataset_name", "") and file_location == info.get("file_location", "") and card_type == info.get("card_type", "") and dataset_id == info.get("dataset_id") :
cluster_id = info.get('cluster_id', "")
candidate_clusters.append(cluster_id)
continue
# if code_id == info.get("code_id", []):
# cluster_id = info.get('cluster_id', "")
# if cluster_id:
# candidate_clusters.append(cluster_id)
# else:
# logger.warning(f"找到匹配算法但未指定集群: {file_id}")
logger.info(f"候选集群列表: {candidate_clusters}")
#对candidate_clusters进行去重
candidate_clusters = list(dict.fromkeys(candidate_clusters))
if not candidate_clusters:
logger.warning("未找到有匹配算法的候选集群")
return None
# 根据调度策略选择集群
if strategy == "data":
selected_cluster = _select_by_data(
candidate_clusters, cluster_resources, task_resource, dataset_id)
elif strategy == "price":
selected_cluster = _select_by_price(
candidate_clusters, cluster_resources, cluster_prices, task_resource, task_type)
elif strategy == "resource":
selected_cluster = _select_by_resource(
candidate_clusters, cluster_resources, task_resource)
elif strategy == "random":
selected_cluster = _select_by_random(
candidate_clusters)
else: # 默认使用resource策略
selected_cluster = _select_by_storage_compute(
candidate_clusters, cluster_resources, task_resource, task_type)
if selected_cluster:
logger.info(f"集群选择成功 | 选中集群: {selected_cluster}")
else:
logger.warning("集群选择失败，未找到满足条件的集群")
logger.info(f"任务类型: {task_type}, 调度策略: {strategy}")
logger.info(f"候选集群: {candidate_clusters}")
logger.info(f"任务资源: {task_resource}")
return selected_cluster
except Exception as e:
logger.error(f"集群选择过程中发生异常: {str(e)}", exc_info=True)
return None

290
scheduler/storage_computing_scheduler.py
View File

@ -1,290 +0,0 @@
import os
import yaml
from typing import Dict, Optional, Any, List
from config.config import logger
def _select_by_storage_compute(
candidate_clusters: List[str],
cluster_resources: Dict[str, Dict],
task_resource: Dict,
task_type: str
) -> Optional[str]:
"""
基于资源的调度策略
:param candidate_clusters: 候选集群列表
:param cluster_resources: 集群资源信息
:param task_resource: 任务资源需求
:param task_type: 任务类型
:return: 选中的集群ID
"""
try:
valid_clusters = []
for cluster_id in candidate_clusters:
if cluster_id not in cluster_resources:
continue
cluster = cluster_resources[cluster_id]
# 检查任务类型是否匹配
if cluster["cluster_type"] != task_type:
continue
# 检查集群是否有任务所需要父算法对应的子算法
# if not _check_sub_algorithm(cluster, task_type):
# continue
# 检查资源是否满足需求
available = cluster["available"]
is_valid = True
for resource_type, required_amount in task_resource.items():
# 特殊处理card资源
if resource_type not in ["CPU", "MEMORY"]:
# 检查card资源中是否包含对应类型的资源
available_card = available.get("card", {})
card_type = resource_type
required_num = required_amount
# 处理字典形式的card信息
if isinstance(available_card, dict) and not isinstance(available_card, list):
if available_card.get("Type") == card_type:
if available_card.get("Num", 0) < required_num:
is_valid = False
break
else:
# 类型不匹配
is_valid = False
break
# 处理列表形式的card信息
elif isinstance(available_card, list):
total_available = 0
for card in available_card:
if card.get("Type") == card_type:
total_available += card.get("Num", 0)
if total_available < required_num:
is_valid = False
break
else:
# card信息格式不正确或不存在
is_valid = False
break
else:
# 处理普通资源(CPU、MEMORY等)
if available.get(resource_type, 0) < required_amount:
is_valid = False
break
if is_valid:
# 计算资源富余度分数（越小越好）
score = 0
for resource_type, required_amount in task_resource.items():
if resource_type not in ["CPU", "MEMORY"]:
# 处理card资源
available_card = available.get("card", {})
if isinstance(available_card, list):
available_amount = sum(card.get("Num", 0) for card in available_card)
else:
available_amount = available_card.get("Num", 0)
score += (available_amount - required_amount) / (required_amount or 1)
else:
# 处理普通资源
available_amount = available.get(resource_type, 0)
score += (available_amount - required_amount) / (required_amount or 1)
valid_clusters.append((cluster_id, score))
if not valid_clusters:
logger.warning("没有满足资源需求的集群")
return None
# 选择资源最接近的集群（分数最低）
selected_cluster, _ = max(valid_clusters, key=lambda x: x[1])
logger.info(f"基于资源策略选择集群: {selected_cluster}")
return selected_cluster
except Exception as e:
logger.error(f"基于资源的调度策略执行异常: {str(e)}", exc_info=True)
return None
def read_cluster_info(file_path: str) -> Dict[str, Dict[str, Any]]:
"""
读取集群资源信息的 YAML 文件。
:param file_path: YAML 文件的路径。
:return: 包含集群信息的字典，如果读取失败则返回空字典。
"""
try:
if not os.path.exists(file_path):
raise FileNotFoundError(f"集群配置文件不存在: {file_path}")
with open(file_path, 'r', encoding='utf-8') as f:
return yaml.safe_load(f)
except FileNotFoundError as fnf_error:
print(fnf_error)
except yaml.YAMLError as yaml_error:
print(f"解析 YAML 文件时出错: {yaml_error}")
return {}
def extract_resources(info: Dict[str, Any]) -> Dict[str, float]:
"""
从资源信息中提取 CPU、MEMORY 和 NPU 资源。
:param info: 包含资源信息的字典。
:return: 包含提取后资源的字典。
"""
resources = {
'CPU': info.get('CPU', 0),
'MEMORY': info.get('MEMORY', 0),
}
# 处理card可能是列表或字典的情况
card_info = info.get('card', [])
if isinstance(card_info, list):
# 列表形式：查找NPU类型的加速器
for card in card_info:
if card.get('Type') == 'NPU':
resources['NPU'] = card.get('Num', 0)
break
else:
# 字典形式：直接检查
if card_info.get('Type') == 'NPU':
resources['NPU'] = card_info.get('Num', 0)
return resources
def parse_resource_request(task_resource: Dict[str, Any]) -> Dict[str, float]:
"""
解析任务的资源请求，将其转换为统一的资源字典格式。
:param task_resource: 任务的资源请求字典。
:return: 统一格式的资源字典。
"""
return extract_resources(task_resource)
def parse_cluster_resources(cluster_info: Dict[str, Any]) -> Dict[str, Dict[str, float]]:
"""
解析集群资源信息，将其转换为统一的资源字典格式。
:param cluster_info: 包含集群信息的字典。
:return: 统一格式的集群资源字典。
"""
return {
cluster_id: extract_resources(info.get('available', {}))
for cluster_id, info in cluster_info.items()
}
def select_best_cluster(task_resources: Dict[str, float],
cluster_resources: Dict[str, Dict[str, float]],
task_type: str,
cluster_types: Dict[str, str]) -> Optional[str]:
"""
选择最合适的集群。
策略：
1. 首先过滤掉不满足任务最低资源要求和类型不匹配的集群。
2. 计算每个候选集群的资源富裕度分数。
3. 选择资源富裕度分数最高的集群。
:param task_resources: 任务的资源请求。
:param cluster_resources: 集群的可用资源。
:param task_type: 任务的类型。
:param cluster_types: 集群的类型映射。
:return: 最佳集群的 ID，如果没有找到则返回 None。
"""
# 过滤掉不满足最低资源要求和类型不匹配的集群
candidate_clusters = {
cluster_id: resources
for cluster_id, resources in cluster_resources.items()
if all(resources.get(resource_type, 0) >= required for resource_type, required in task_resources.items())
and cluster_types.get(cluster_id) == task_type
}
# 如果没有候选集群，返回 None
if not candidate_clusters:
return None
# 计算每个候选集群的资源富裕度分数
scores = {
cluster_id: sum(resources.get(resource_type, 0) / (required or 1) for resource_type, required in
task_resources.items()) / len(task_resources)
for cluster_id, resources in candidate_clusters.items()
}
# 选择分数最高的集群
return max(scores, key=scores.get)
def get_scheduled_cluster_id(task_resource: Dict, cluster_resources: Dict[str, Dict], task_type: str) -> Optional[str]:
"""
根据任务资源需求选择合适的集群，选择资源最富裕的集群
:param task_resource: 任务的资源需求，例如 {"CPU": 8, "MEMORY": 16, "NPU": 1}
:param cluster_resources: 集群资源配置，结构同YAML定义
:param task_type: 任务的类型
:return: 选择的集群ID，如果没有合适的集群则返回None
"""
candidate_clusters = {}
cluster_types = {cluster_id: cluster["cluster_type"] for cluster_id, cluster in cluster_resources.items()}
for cluster_id, cluster in cluster_resources.items():
if cluster_types.get(cluster_id) != task_type:
continue
total = cluster["total"]
# 检查CPU和MEMORY资源
cpu_available = total["CPU"] >= task_resource.get("CPU", 0)
memory_available = total["MEMORY"] >= task_resource.get("MEMORY", 0)
# 检查加速器资源（如NPU）
required_accelerator = next((key for key in task_resource if key not in ["CPU", "MEMORY"]), None)
accelerator_available = True
if required_accelerator:
# 处理card可能是列表或字典的情况
card_info = total.get("card", [])
if isinstance(card_info, list):
# 列表形式：查找匹配类型的加速器
accelerator_available = any(
card.get("Type") == required_accelerator and card.get("Num", 0) >= task_resource.get(
required_accelerator, 0)
for card in card_info
)
else:
# 字典形式：直接检查
accelerator_available = (
card_info.get("Type") == required_accelerator and
card_info.get("Num", 0) >= task_resource.get(required_accelerator, 0)
)
if cpu_available and memory_available and accelerator_available:
# 计算资源富裕度分数
score = sum((total.get(resource_type, 0) - task_resource.get(resource_type, 0)) / (
task_resource.get(resource_type, 1) or 1)
for resource_type in task_resource.keys())
candidate_clusters[cluster_id] = score
if not candidate_clusters:
return None
# 选择分数最高的集群
return max(candidate_clusters, key=candidate_clusters.get)
def main():
# 任务资源请求
task_resource = {"CPU": 8, "MEMORY": 16, "NPU": 1}
task_type = "Ai"
# 集群资源信息文件路径
file_path = "../mysql/sonCode_cluster_mapping.yaml"
# 读取集群信息
cluster_info = read_cluster_info(file_path)
if not cluster_info:
print("无法获取集群信息，调度失败")
return
cluster_resources = cluster_info.get('cluster_resources', {})
# 选择最佳集群
best_cluster = get_scheduled_cluster_id(task_resource, cluster_resources, task_type)
if best_cluster:
print(f"最佳集群: {best_cluster}")
else:
print("没有找到满足条件的集群")
if __name__ == "__main__":
main()

107
sonCode_cluster__mapping.yaml Normal file
View File

@ -0,0 +1,107 @@
cluster_mapping:
"1790300942428540928":
- son_code_id: 1217
file_location: "D:/数据集/cnn数据集/data1/"
- son_code_id: 1167
file_location: "D:/数据集/cnn数据集/data2/"
- son_code_id: 1171
file_location: "D:/数据集/transfomer数据集/BoolQ/"
- son_code_id: 1173
file_location: "D:/数据集/transfomer数据集/CEval/"
- son_code_id: 1178
file_location: "D:/数据集/transfomer数据集/CMMLU/"
- son_code_id: 1182
file_location: "D:/数据集/transfomer数据集/HumanEval/"
"1790300942428540929":
- son_code_id: 1169
file_location: "D:/数据集/cnn数据集/data3/"
- son_code_id: 1171
file_location: "D:/数据集/transfomer数据集/BoolQ/"
- son_code_id: 1175
file_location: "D:/数据集/transfomer数据集/GLUE(imdb)/imdb/"
- son_code_id: 1180
file_location: "D:/数据集/transfomer数据集/GSM8K/GSM8K/"
- son_code_id: 1184
file_location: "D:/数据集/transfomer数据集/HumanEval_X/"
tasks_config:
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AA"
dataset_name: "data1.zip"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AB"
dataset_name: "cifar-10-python.tar.gz"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AC"
dataset_name: "cifar-100-python.tar.gz"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AD"
dataset_name: "dev.jsonl"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AE"
dataset_name: "ceval.zip"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AF"
dataset_name: "CMMLU.zip"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AH"
dataset_name: "mental_health.csv"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AI"
dataset_name: "GSM8K.jsonl"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AJ"
dataset_name: "human-eval.jsonl"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1
- task_name_template: "{prefix}-jointCloudAi-trainingtask"
prefix: "AK"
dataset_name: "HumanEval_X.zip"
son_code_id:
resource:
CPU: 12
MEMORY: 24
NPU: 1

689
task/ai_task_submit.py
View File

@ -1,689 +0,0 @@
import os
import json
import time
import uuid
import requests
from typing import Dict, Optional, List
from threading import Lock
import logging # 导入 logging 模块
from config.config import TASK_STATUS as TASK_STATUS_COMMON
from config.db_manager import update_file_all_fields, update_cluster_resources
# 任务状态常量
TASK_STATUS = {
"PENDING": "pending",
"RUNNING": "running",
"SUCCEED": "Succeeded",
"FAILED": "failed",
"RETRY_EXHAUSTED": "retry_exhausted"
}
# 全局变量与锁
# 初始化日志记录器
logger = logging.getLogger(__name__)
dataset_map = {} # 存储数据集信息的映射
dataset_lock = Lock() # 数据集操作锁
task_map_lock = Lock() # 任务映射操作锁
cluster_lock = Lock() # 集群资源操作锁
cluster_resources = {} # 集群资源信息
API_CONFIG = {} # API配置
# 移除 YAML 配置文件路径，不再需要
# sonCode_cluster_mapping_path = "sonCode_cluster_mapping.yaml" # YAML配置文件路径
class DatasetInfo:
"""数据集信息类"""
def __init__(self, file_location: str, name: str, size: int, is_uploaded: bool,
upload_cluster: List[str], upload_time: str, dataset_target_id: str):
self.file_location = file_location
self.name = name
self.size = size
self.is_uploaded = is_uploaded
self.upload_cluster = upload_cluster
self.upload_time = upload_time
self.dataset_target_id = dataset_target_id
def __dict__(self):
return {
"file_location": self.file_location,
"name": self.name,
"size": self.size,
"is_uploaded": self.is_uploaded,
"upload_cluster": self.upload_cluster,
"upload_time": self.upload_time,
"dataset_target_id": self.dataset_target_id
}
def get_config_from_db() -> Dict:
"""从数据库获取配置"""
try:
from config.cache_manager import get_cached_config
return get_cached_config()
except Exception as e:
logger.error(f"从缓存加载配置失败: {str(e)}")
# 回退到直接从数据库加载
try:
from config.db_manager import load_config_from_db
return load_config_from_db()
except Exception as e:
logger.error(f"从数据库加载配置失败: {str(e)}")
return {}
UPLOAD_CLUSTER_MAPPING = {
"1865927992266463180": 1,
"1865927992266461184": 23,
"1790300942428540928": 3,
"1830873903296155648": 8,
"1830873578531228942": 9,
"1865927992266462181": 10,
"1830873903296155649": 11,
"1865927992266462182": 13,
"1865927992266462180": 14,
"1770703902472146944": 12
}
def upload_dataset(task: Dict, token) -> Optional[str]:
"""上传数据集并返回dataset_target_id"""
global task_name, copiedToFullRoots
try:
# 获取必要参数
cluster_id = task.get("cluster_id")
dataset_id = task.get("dataset_id")
card_type = task.get("card_type")
dataset_info = get_fileinfo_by_dataset_id(dataset_id)
is_uploaded = dataset_info.get("is_uploaded")
#如果文件已经上传完成并存在dataset_target_id，则直接返回
dataset_target_id = dataset_info.get("dataset_target_id")
if dataset_target_id and is_uploaded:
logger.info(f"数据集已经上传，数据集id为：{dataset_target_id}")
return dataset_target_id
file_location = dataset_info.get("file_location")
file_name = dataset_info.get("dataset_name")
file_path = f"{file_location}{file_name}"
package_name = task.get("package_name")
if not package_name:
package_name = f"dataset_{uuid.uuid4().hex[:8]}" # 生成默认包名
task_name = task.get("task_name", "unknown_task")
# 检查文件是否存在
if not os.path.exists(file_path):
print(f"[{task_name}] 数据集文件不存在: {file_path}")
return None
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {token}'
}
# 从数据库获取API配置
config_data = get_config_from_db()
api_config = config_data.get("api_config", {})
# 1. 创建数据集文件夹
create_config = api_config.get("create_package", {})
create_url = create_config.get("url", "http://jcc.jointcloud.net/jsm/jobSet/createPackage")
create_timeout = create_config.get("timeout", 10)
create_payload = {
"userID": 5,
"name": package_name,
"dataType": "dataset",
"packageID": 0,
"uploadPriority": {"type": "specify", "clusters": [cluster_id]},
"bindingInfo": {
"clusterIDs": [cluster_id],
"name": package_name,
"category": "image",
"type": "dataset",
"imageID": "",
"bias": [],
"region": [],
"chip": [card_type],
"selectedCluster": [],
"modelType": "",
"env": "",
"version": "",
"packageID": 0,
"points": 0
}
}
create_resp = requests.post(
create_url,
json=create_payload,
headers=headers,
timeout=create_timeout
)
create_resp.raise_for_status()
create_result = create_resp.json()
if create_result.get("code") != "OK":
raise ValueError(f"创建文件夹失败 | 响应: {create_result}")
packageID = create_result["data"]["newPackage"]["packageID"]
print(f"[{task_name}] 已创建数据集文件夹 | packageID: {packageID}")
# 2. 查询文件夹内容（验证创建结果）
query_url = "http://jcc.jointcloud.net/jsm/jobSet/queryUploaded"
query_payload = {
"queryParams": {
"dataType": "dataset",
"userID": 5,
"packageID": packageID,
"path": "",
"CurrentPage": 1,
"pageSize": 10,
"orderBy": "name"
}
}
query_resp = requests.post(query_url, json=query_payload, headers=headers, timeout=15)
query_resp.raise_for_status()
query_result = query_resp.json()
if query_result.get("code") != "OK":
raise ValueError(f"查询文件夹内容失败 | 响应: {query_result}")
print(f"[{task_name}] 文件夹内容查询成功")
# 3. 获取预签名URL用于上传文件到目标文件夹
#获取任务的调度类型
strategy = task.get("strategy")
# 根据调度策略选择集群
if strategy == "data":
info_data = {
"userID": 5,
"info": {
"type": "upload",
"params": {
"packageID": packageID,
"copyTo": [],
"copyToPath": [],
"path": file_name
}
}
}
else:
info_data = {
"userID": 5,
"info": {
"type": "upload",
"params": {
"packageID": packageID,
"copyTo": [UPLOAD_CLUSTER_MAPPING[cluster_id]],
"copyToPath": [f"/dataset/{package_name}/"],
"path": file_name
}
}
}
upload_url = "http://101.201.215.196:7891/storage/presign"
presign_resp = requests.post(
upload_url,
json=info_data,
headers={'Authorization': f'Bearer {token}'},
timeout=300
)
presign_resp.raise_for_status()
presign_result = presign_resp.json()
if presign_result.get("code") != "OK":
raise ValueError(f"获取预签名URL失败 | 响应: {presign_result}")
# 获取预签名URL
presign_url = presign_result["data"]["presignUrl"]
with open(file_path, "rb") as f:
file_data = f.read()
try:
upload_resp = requests.post(
url=presign_url,
data=file_data
)
# 打印响应结果
upload_resp.raise_for_status()
print(f"[{task_name}] 文件通过预签名URL上传成功")
upload_result = upload_resp.json()
if upload_result.get("code") != "OK":
raise ValueError(f"文件上传失败 | 响应: {upload_result}")
object_id = upload_result["data"]["object"]["objectID"]
copiedToFullRoots = upload_result["data"]["copiedToFullRoots"]
print(f"[{task_name}] 文件上传成功 | objectID: {object_id}")
except Exception as e:
print(f"上传失败，错误信息：{str(e)}")
# 4. 通知上传完成
notify_config = api_config.get("notify_upload", {})
notify_url = notify_config.get("url", "http://jcc.jointcloud.net/jsm/jobSet/notifyUploaded")
notify_timeout = notify_config.get("timeout", 15)
notify_payload = {
"userID": 5,
"packageID": packageID,
"uploadParams": {
"copiedTo": [UPLOAD_CLUSTER_MAPPING[cluster_id]],
"copiedToFullRoots": copiedToFullRoots,
"dataType": "dataset",
"uploadInfo": {
"type": "local",
"localPath": file_name,
"objectIDs": [object_id] # 补充objectID
}
}
}
notify_resp = requests.post(
notify_url,
json=notify_payload,
headers=headers,
timeout=notify_timeout
)
notify_resp.raise_for_status()
notify_result = notify_resp.json()
if notify_result.get("code") != "OK":
raise ValueError(f"通知上传完成失败 | 响应: {notify_result}")
print(f"[{task_name}] 已通知上传完成")
# 5. 二次查询上传内容
query_resp2 = requests.post(query_url, json=query_payload, headers=headers, timeout=15)
query_resp2.raise_for_status()
query_result2 = query_resp2.json()
if query_result2.get("code") != "OK":
raise ValueError(f"二次查询失败 | 响应: {query_result2}")
print(f"[{task_name}] 二次查询上传内容成功")
# 6. 查询上传状态
status_url = "http://jcc.jointcloud.net/jsm/jobSet/uploadStatus"
status_payload = {
"userID": 5,
"operate": "query",
"packageID": packageID,
"dataType": "code" # 修正为dataset类型
}
status_resp = requests.post(status_url, json=status_payload, headers=headers, timeout=15)
status_resp.raise_for_status()
status_result = status_resp.json()
# if status_result.get("code") != "OK":
# raise ValueError(f"查询上传状态失败 | 响应: {status_result}")
if status_resp.status_code == 200:
upload_status_result = status_resp.json()
print(f"查询上传状态: {upload_status_result}")
else:
print(f"查询上传状态，状态码: {status_resp.status_code}")
return
# print(f"[{task_name}] 上传状态: {status_result['data'].get('status')}")
# 7. 绑定数据集到集群
bind_config = api_config.get("bind_cluster", {})
bind_url = bind_config.get("url", "http://jcc.jointcloud.net/jsm/jobSet/binding")
bind_timeout = bind_config.get("timeout", 300)
bind_payload = {
"userID": 5,
"info": {"type": "dataset", "packageID": packageID, "clusterIDs": [cluster_id]}
}
bind_resp = requests.post(
bind_url,
json=bind_payload,
headers=headers,
timeout=300
)
bind_resp.raise_for_status()
bind_result = bind_resp.json()
if bind_result.get("code") != "OK":
raise ValueError(f"绑定集群失败 | 响应: {bind_result}")
print(f"[{task_name}] 已绑定集群 {cluster_id}")
# 8. 获取dataset_target_id
query_binding_url = "http://jcc.jointcloud.net/jsm/jobSet/queryBinding"
query_binding_payload = {
"dataType": "dataset",
"param": {"userID": 5, "bindingID": -1, "type": "private"}
}
binding_resp = requests.post(query_binding_url, json=query_binding_payload, headers=headers, timeout=15)
binding_resp.raise_for_status()
binding_result = binding_resp.json()
if binding_result.get("code") != "OK":
raise ValueError(f"查询绑定信息失败 | 响应: {binding_result}")
# 提取目标ID
dataset_target_id = None
for data in binding_result["data"]["datas"]:
if data["info"]["name"] == package_name:
dataset_target_id = data["ID"]
break
if not dataset_target_id:
raise ValueError(f"未找到 {package_name} 对应的绑定ID")
print(f"[{task_name}] 成功获取dataset_target_id: {dataset_target_id}")
# 保存数据集信息
with dataset_lock:
dataset_map[file_location] = DatasetInfo(
file_location=file_location,
name=file_name,
size=os.path.getsize(file_path),
is_uploaded=True,
upload_cluster=[cluster_id],
upload_time=time.strftime("%Y-%m-%d %H:%M:%S"),
dataset_target_id=dataset_target_id
)
#更新数据库中file_mapping表的数据
if update_file_all_fields(dataset_id, dataset_target_id=dataset_target_id, is_uploaded=1):
logger.info(
f"成功更新文件 {dataset_id} 在数据库中的dataset_target_id为 {dataset_target_id}")
return dataset_target_id
except Exception as e:
print(f"[{task_name}] 数据集上传失败: {str(e)}")
return None
def get_file_info(cluster_id, file_location, file_name,card_type):
""""#根据code_id和cluster_id获取son_code_id"""
try:
# 从数据库获取算法映射配置
config_data = get_config_from_db()
file_mapping = config_data.get('file_mapping', {})
for file_id, info in file_mapping.items():
if cluster_id == str(info["cluster_id"]) and file_location == info["file_location"] and file_name == info["dataset_name"] and card_type == info["card_type"]:
return info
except Exception as e:
logger.error(f"从数据库查询 文件信息 失败: {str(e)}")
return None
def get_fileinfo_by_dataset_id(dataset_id):
"""根据 dataset_id 从数据库中查询 code_id"""
try:
# 从数据库获取算法映射配置
config_data = get_config_from_db()
file_mapping = config_data.get('file_mapping', {})
for file_id, info in file_mapping.items():
if dataset_id == file_id:
return info
except Exception as e:
logger.error(f"从数据库查询 文件信息 失败: {str(e)}")
return None
def query_resource_info(card_type, cluster_id, token):
"""调用接口查询资源信息"""
try:
# 接口URL
API_URL = "https://jcc.jointcloud.net/jsm/jobSet/queryResource"
# 请求参数
payload = {
"queryResource": {
"cpu": {"min": 0, "max": 0},
"memory": {"min": 0, "max": 0},
"gpu": {"min": 0, "max": 0},
"storage": {"min": 0, "max": 0},
"type": card_type
},
"calcNetworkCost": False,
"resourceType": "Train",
"clusterIDs": [cluster_id]
}
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {token}'
}
# 发送请求并设置超时
response = requests.post(
url=API_URL,
json=payload,
headers=headers,
timeout=10 # 10秒超时设置
)
response.raise_for_status() # 触发HTTP错误状态码的异常
# 解析响应
result = response.json()
# 验证响应状态
if result.get("code") != "OK":
logger.error(f"接口返回非成功状态: {result.get('message', '未知错误')}")
return None
# 提取资源列表
resources = result.get("data", {}).get("resource", [])
if not resources:
logger.info("未获取到任何资源数据")
return None
# 筛选符合条件的资源
for resource in resources:
# 严格验证类型和集群ID
if resource.get("type") == card_type and resource.get("clusterId") in cluster_id:
# 特殊处理 章鱼部分规格不可用
if card_type == "ILUVATAR-GPGPU" and cluster_id == "1865927992266462182":
return "BI-V100"
resource_name = resource.get("name")
if resource_name:
logger.info(f"找到符合条件的资源: {resource_name}")
return resource_name
logger.warning("找到符合条件的资源，但名称为空")
# 未找到符合条件的资源
logger.info(f"没有找到类型为{card_type}且属于集群{cluster_id}的资源")
return None
except requests.exceptions.Timeout:
logger.error("请求接口超时")
return None
except requests.exceptions.RequestException as e:
logger.error(f"请求接口失败: {str(e)}")
return None
except ValueError as e:
logger.error(f"解析响应JSON失败: {str(e)}")
return None
# 提交单个AI任务到集群
def submit_ai_task(task: Dict, token) -> Optional[str]:
# 加载API配置
global API_CONFIG, TASK_STATUS
config_data = get_config_from_db()
API_CONFIG = config_data.get('api_config', {})
# 获取选择的集群ID
cluster_id = task.get("cluster_id")
resource_name = query_resource_info(task.get("card_type"), cluster_id, token)
if not cluster_id:
with task_map_lock:
task["status"] = TASK_STATUS_COMMON["FAILED"]
task["error_msg"] = "未指定集群ID"
print(f"[{task['task_name']}] 提交失败：未指定集群ID")
return None
# 数据集处理（获取dataset_target_id）
dataset_target_id = upload_dataset(task, token)
if task.get("type") == "Ai" and task.get("dataset_name") and not dataset_target_id:
# 构造资源变化字典
task_resource = task.get("resource", {})
resource_changes = {}
# 计算资源变化量（负值表示扣减）
for res_type in ["CPU", "MEMORY"]:
required = task_resource.get(res_type, 0)
if required > 0:
resource_changes[res_type] = +required
# 处理加速器资源
for acc_type in [key for key in task_resource if key not in ["CPU", "MEMORY"]]:
required_count = task_resource.get(acc_type, 0)
if required_count > 0:
resource_changes[acc_type] = +required_count
# 使用批量操作更新数据库
if update_cluster_resources(cluster_id, resource_changes):
logger.info(f"成功从数据库恢复集群 {cluster_id} 的资源: {resource_changes}")
else:
logger.error(f"从数据库恢复集群 {cluster_id} 的资源失败: {resource_changes}")
print(f"[{task['task_name']}] 提交失败：数据集处理失败")
return None
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {token}'
}
task_name = task["task_name"]
## code_id = task["code_id"]
card_type = task["card_type"]
# 从数据库中查询 son_code_id
fileInfo = get_fileinfo_by_dataset_id(task["dataset_id"])
file_location = fileInfo.get("file_location")
file_name = fileInfo.get("dataset_name")
file_info = get_file_info(cluster_id,file_location,file_name,card_type)
son_code_id = file_info.get("son_code_id")
image_id = file_info.get("image_id")
# 检查son_code_id是否有效
if not son_code_id:
error_msg = "无法获取有效的son_code_id，任务提交失败"
print(f"[{task_name}] {error_msg}")
# 构造资源变化字典
task_resource = task.get("resource", {})
resource_changes = {}
# 计算资源变化量（负值表示扣减）
for res_type in ["CPU", "MEMORY"]:
required = task_resource.get(res_type, 0)
if required > 0:
resource_changes[res_type] = +required
# 处理加速器资源
for acc_type in [key for key in task_resource if key not in ["CPU", "MEMORY"]]:
required_count = task_resource.get(acc_type, 0)
if required_count > 0:
resource_changes[acc_type] = +required_count
# 使用批量操作更新数据库
if update_cluster_resources(cluster_id, resource_changes):
logger.info(f"成功从数据库恢复集群 {cluster_id} 的资源: {resource_changes}")
else:
logger.error(f"从数据库恢复集群 {cluster_id} 的资源失败: {resource_changes}")
return None
try:
# 1. 提交训练任务
submit_config = API_CONFIG.get("submit_task", {})
config = submit_config.get("url", "http://jcc.jointcloud.net/jsm/jobSet/submit")
submit_timeout = submit_config.get("timeout", 100)
task_res = task["resource"]
# 获取加速器类型（如NPU）
acc_type = next((key for key in task_res if key not in ["CPU", "MEMORY"]), None)
submit_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"name": task_name,
"description": "",
"type": "AI",
"files": {
"dataset": {"type": "Binding", "bindingID": dataset_target_id},
"model": {"type": "Binding", "bindingID": None},
"image": {"type": "Image", "imageID": image_id}
},
"jobResources": {
"scheduleStrategy": "dataLocality",
"clusters": [
{
"code": {"type": "Binding", "bindingID": son_code_id},
"clusterID": cluster_id,
"runtime": {"envs": {}, "params": {}},
"resources": [
{"type": "CPU", "name": "CPU", "number": task_res["CPU"]},
{"type": "MEMORY", "name": "RAM", "number": task_res["MEMORY"]},
{"type": acc_type, "name": resource_name, "number": task_res.get(acc_type, 0)}
]
}
]
}
}
]
}
}
response = requests.post(
config,
json=submit_payload,
headers=headers,
timeout=submit_timeout
)
response.raise_for_status()
submit_resp = response.json()
if submit_resp.get("code") != "OK":
raise ValueError(f"任务提交失败 | API返回: {submit_resp}")
third_party_task_id = submit_resp.get('data', {}).get('jobSetID')
print(f"[{task_name}] 任务提交至集群 {cluster_id} 成功 | 云际任务ID: {third_party_task_id}")
# 更新任务状态为已提交到数据库
from config.db_manager import update_task_status
from config.config import TASK_STATUS
if not update_task_status(task_name, TASK_STATUS["SUBMITTING"], third_party_task_id):
logger.error(f"更新任务 {task_name} 状态到数据库失败")
return third_party_task_id
except Exception as e:
error_msg = f"提交失败: {str(e)}"
# 构造资源变化字典
task_resource = task.get("resource", {})
resource_changes = {}
# 计算资源变化量（负值表示扣减）
for res_type in ["CPU", "MEMORY"]:
required = task_resource.get(res_type, 0)
if required > 0:
resource_changes[res_type] = +required
# 处理加速器资源
for acc_type in [key for key in task_resource if key not in ["CPU", "MEMORY"]]:
required_count = task_resource.get(acc_type, 0)
if required_count > 0:
resource_changes[acc_type] = +required_count
# 使用批量操作更新数据库
if update_cluster_resources(cluster_id, resource_changes):
logger.info(f"成功从数据库恢复集群 {cluster_id} 的资源: {resource_changes}")
else:
logger.error(f"从数据库恢复集群 {cluster_id} 的资源失败: {resource_changes}")
task["fail_count"] = task.get("fail_count", 0) + 1
max_fail = task.get("max_fail_threshold", 3)
if task["fail_count"] >= max_fail:
task["status"] = TASK_STATUS["RETRY_EXHAUSTED"]
else:
task["status"] = TASK_STATUS["FAILED"]
task["error_msg"] = error_msg
print(f"[{task_name}] {error_msg}")
return None

243
task/cloud_task_submit.py
View File

@ -1,243 +0,0 @@
import random
import string
from typing import Optional, Dict, Any, List
import requests
import json
from config.db_manager import load_config_from_db
from config.config import logger
from config.auth import get_token
class CloudTaskSubmitter:
"""云算任务提交器，提供完整的云算任务管理功能"""
def __init__(self):
"""初始化云算任务提交器"""
self.config = load_config_from_db()
self.token = None
def authenticate(self) -> bool:
"""获取认证Token"""
self.token = get_token()
return self.token is not None
def query_adapter_list(self) -> Optional[List[Dict]]:
"""查询适配器列表"""
if not self.token:
logger.error("未获取到认证Token，请先调用authenticate()")
return None
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {self.token}'}
# 检查配置中是否有adapter_list配置
if "adapter_list" not in self.config["api_config"]:
logger.error("API配置中缺少adapter_list配置")
return None
api_url = self.config["api_config"]["adapter_list"]["url"]
try:
response = requests.get(api_url, headers=headers)
response.raise_for_status()
result = response.json()
if result.get("code") == "OK" and "data" in result:
logger.info("查询适配器列表成功")
return result["data"]
else:
logger.error(f"查询适配器列表失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"查询适配器列表时发生异常: {str(e)}", exc_info=True)
return None
def query_cluster_list(self, adapter_id: str) -> Optional[List[Dict]]:
"""查询集群列表"""
if not self.token:
logger.error("未获取到认证Token，请先调用authenticate()")
return None
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {self.token}'}
# 检查配置中是否有cluster_list配置
if "cluster_list" not in self.config["api_config"]:
logger.error("API配置中缺少cluster_list配置")
return None
api_url = self.config["api_config"]["cluster_list"]["url"].format(adapter_id=adapter_id)
try:
response = requests.get(api_url, headers=headers)
response.raise_for_status()
result = response.json()
if result.get("code") == "OK" and "data" in result:
logger.info("查询集群列表成功")
return result["data"]
else:
logger.error(f"查询集群列表失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"查询集群列表时发生异常: {str(e)}", exc_info=True)
return None
def generate_random_string(self, length: int = 7) -> str:
"""生成随机字符串作为任务名称后缀"""
characters = string.ascii_lowercase # 只使用小写字母
random_string = ''.join(random.choice(characters) for _ in range(length))
return random_string
def create_task(self, cluster_id: str, task_name: str, replicas: int = 1, cpu: int = 1, memory: int = 1, port: int = 80, image: str = "nginx:latest", strategy: str = "replication") -> Optional[str]:
"""创建云算任务，返回 jobSetID 或 None"""
if not self.token:
logger.error("未获取到认证Token，请先调用authenticate()")
return None
# 生成带随机后缀的任务名称
task_name = f"{task_name}-{self.generate_random_string()}"
# 根据策略创建不同的任务负载
if strategy == "replication":
# 使用标准Deployment方式创建任务
payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [{
"localJobID": "1",
"name": task_name,
"type": "PCM_Container",
"clusterId": cluster_id,
"containerGroupName": task_name,
"image": image,
"cpu": str(cpu),
"memory": str(memory),
"port": port,
"capacity": replicas,
"mountPath": "/models"
}]
}
}
else:
# 其他策略可以在此扩展
payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [{
"localJobID": "1",
"name": task_name,
"type": "PCM_Container",
"clusterId": cluster_id,
"containerGroupName": task_name,
"image": image,
"cpu": str(cpu),
"memory": str(memory),
"port": port,
"capacity": replicas,
"mountPath": "/models"
}]
}
}
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {self.token}'
}
# 尝试从配置中获取create_task的URL，如果不存在则使用默认URL
if "submit_task" not in self.config["api_config"]:
# 尝试使用其他可能的键名
possible_keys = ["create_task", "task_create", "submit_task", "create"]
api_url = None
for key in possible_keys:
if key in self.config["api_config"]:
api_url = self.config["api_config"][key]["url"]
break
# 如果仍然没有找到，使用默认URL
if api_url is None:
logger.warning("API配置中缺少create_task配置，使用默认URL")
api_url = "https://jcc.jointcloud.net/jsm/jobSet/submit" # 默认URL
else:
api_url = self.config["api_config"]["submit_task"]["url"]
try:
response = requests.post(api_url, headers=headers, json=payload)
response.raise_for_status()
result = response.json()
# 提取并返回 jobSetID
if result.get("code") == "OK" and "data" in result:
job_set_id = result["data"].get("jobSetID")
logger.info(f"任务创建成功 | 任务名称: {task_name} | JobSetID: {job_set_id}")
return job_set_id
else:
logger.error(f"任务创建失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"创建任务时发生异常: {str(e)}", exc_info=True)
return None
def submit_cloud_task(self, task_name: str, cluster_id: str, replicas: int = 1, cpu: int = 1, memory: int = 1, port: int = 80, image: str = "nginx:latest", strategy: str = "replication") -> Optional[str]:
"""提交云算任务的主方法"""
# 首先进行身份验证
if not self.authenticate():
logger.error("认证失败，无法提交任务")
return None
# 创建任务并返回JobSetID
return self.create_task(cluster_id, task_name, replicas, cpu, memory, port, image, strategy)
def get_available_clusters(self) -> List[Dict]:
"""获取所有可用的云集群"""
available_clusters = []
for cluster_id, cluster_info in self.config["cluster_resources"].items():
if cluster_info["cluster_type"] == "Cloud":
available_clusters.append({
"cluster_id": cluster_id,
"cluster_info": cluster_info
})
return available_clusters
def get_cluster_resources(self, cluster_id: str) -> Optional[Dict]:
"""获取指定集群的资源信息"""
return self.config["cluster_resources"].get(cluster_id)
def query_task_status(self, job_set_id: str) -> Optional[Dict]:
"""查询任务状态"""
if not self.token:
logger.error("未获取到认证Token，请先调用authenticate()")
return None
headers = {'Authorization': f'Bearer {self.token}'}
# 检查配置中是否有task_status配置
if "task_status" not in self.config["api_config"]:
logger.error("API配置中缺少task_status配置")
return None
api_url = self.config["api_config"]["task_status"]["url"]
try:
params = {
"jobSetID": job_set_id,
"localJobID": 1
}
response = requests.get(api_url, headers=headers, params=params)
response.raise_for_status()
result = response.json()
if result.get("code") == "OK" and "data" in result:
logger.info(f"任务状态查询成功 | JobSetID: {job_set_id}")
return result["data"]
else:
logger.error(f"任务状态查询失败 | 响应: {result}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"查询任务状态时发生异常: {str(e)}", exc_info=True)
return None
# 便捷函数，兼容原有接口
def submit_cloud_task(task_name: str, cluster_id: str, replicas: int = 1, cpu: int = 1, memory: int = 1, port: int = 80) -> Optional[str]:
"""便捷函数：提交云算任务"""
submitter = CloudTaskSubmitter()
return submitter.submit_cloud_task(task_name, cluster_id, replicas, cpu, memory, port)

441
task/hpc_task_submit.py
View File

@ -1,441 +0,0 @@
import requests
import json
from datetime import datetime
class HPCTaskSubmitter:
"""HPC任务提交器，提供完整的HPC任务管理功能"""
def __init__(self):
"""初始化HPC任务提交器"""
self.token = None
def get_token(self):
"""
调用登录接口获取 Token
:return: 返回获取到的 Token，如果获取失败则返回 None
"""
login_url = "http://jcc.jointcloud.net/jcc-admin/admin/login"
login_payload = {"username": "admin", "password": "Nudt@123"}
try:
# 发送 POST 请求进行登录
login_response = requests.post(login_url, json=login_payload)
# 检查响应状态码
if login_response.status_code == 200:
try:
# 解析响应内容为 JSON 格式
login_result = login_response.json()
# 提取 Token
token = login_result.get("data", {}).get("token")
if token:
self.token = token
return token
else:
print("登录响应中未找到有效的 Token")
except json.JSONDecodeError:
print("登录响应数据解析失败，不是有效的 JSON 格式")
else:
print(f"登录失败，状态码: {login_response.status_code}")
except requests.RequestException as e:
print(f"登录请求过程中发生异常: {e}")
return None
def query_uploaded_files(self):
"""
查询文件夹接口
:return: 返回查询结果，如果查询失败则返回 None
"""
if not self.token:
print("未获取到认证Token，请先调用get_token()")
return None
query_url = "http://jcc.jointcloud.net/jsm/jobSet/queryUploaded"
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {self.token}'}
query_payload = {
"queryParams": {
"dataType": "HPCSlurm",
"userID": 5,
"packageID": -1,
"path": "",
"CurrentPage": 1,
"pageSize": 99999,
"orderBy": "time"
}
}
try:
# 发送 POST 请求
query_response = requests.post(query_url, headers=headers, json=query_payload)
# 检查响应状态码
if query_response.status_code == 200:
try:
# 解析响应内容为 JSON 格式
query_result = query_response.json()
return query_result
except json.JSONDecodeError:
print("查询响应数据解析失败，不是有效的 JSON 格式")
else:
print(f"查询失败，状态码: {query_response.status_code}")
except requests.RequestException as e:
print(f"查询请求过程中发生异常: {e}")
return None
def get_package_id_by_name(self, packageName):
"""
根据 packageName 查询 packageID
:param packageName: 包名称
:return: 返回查询到的 packageID，如果未找到则返回 None
"""
query_result = self.query_uploaded_files()
if query_result:
uploaded_datas = query_result.get("data", {}).get("uploadedDatas", [])
for data in uploaded_datas:
if data.get("packageName") == packageName:
return data.get("packageID")
return None
def get_object_id_by_package_id(self, packageID, hpcFile, hpctype):
"""
根据 packageID 和 hpcFile 查询 objectID
:param packageID: 包 ID
:param hpcFile: 文件名
:param hpctype: HPC任务类型
:return: 返回查询到的 objectID，如果未找到则返回 None
"""
if not self.token:
print("未获取到认证Token，请先调用get_token()")
return None
query_url = "http://jcc.jointcloud.net/jsm/jobSet/queryUploaded"
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {self.token}'}
query_payload = {
"queryParams": {
"dataType": "HPCSlurm",
"userID": 5,
"packageID": packageID,
"path": "",
"CurrentPage": 1,
"pageSize": 99999,
"orderBy": "time"
}
}
try:
# 发送 POST 请求
query_response = requests.post(query_url, headers=headers, json=query_payload)
# 检查响应状态码
if query_response.status_code == 200:
try:
# 解析响应内容为 JSON 格式
query_result = query_response.json()
uploaded_datas = query_result.get("data", {}).get("uploadedDatas", [])
if uploaded_datas:
objects = uploaded_datas[0].get("objects", [])
if hpctype == "bwa":
for obj in objects:
if obj.get("path").strip('/') == hpcFile:
return obj.get("objectID")
elif hpctype == "lammps":
if objects:
return objects[0].get("objectID")
else:
# 对于其他类型，如hashcat，返回第一个objectID
if objects:
return objects[0].get("objectID")
except json.JSONDecodeError:
print("查询响应数据解析失败，不是有效的 JSON 格式")
else:
print(f"查询失败，状态码: {query_response.status_code}")
except requests.RequestException as e:
print(f"查询请求过程中发生异常: {e}")
return None
def submit_hpc_task(self, task_name, partition, ntasks, nodes, objectID, hpctype="bwa", hpcFile=""):
"""
提交任务到指定接口
:param task_name: 任务名称
:param partition: 分区名称
:param ntasks: 任务数量
:param nodes: 节点数量
:param objectID: 对象 ID
:param hpctype: HPC任务类型 (bwa, hashcat, lammps)
:param hpcFile: HPC文件名
:return: 提交任务后的响应结果，格式为字典。如果提交失败则返回 None
"""
if not self.token:
print("未获取到认证Token，请先调用get_token()")
return None
# 任务提交接口的 URL
request_url = "http://jcc.jointcloud.net/jsm/jobSet/submit"
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {self.token}'}
# 根据任务类型构造请求负载
if hpctype.lower() == "bwa":
create_package_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"type": "HPC",
"name": task_name,
"clusterId": "1830873578531228942",
"backend": "slurm",
"app": "bwa",
"operateType": "index",
"parameters": {
"hpcBindingFiles": [
{
"paramName": "indexFile",
"resource": {
"type": "object",
"objectID": objectID
}
}
],
"partition": partition,
"ntasks": ntasks,
"nodes": nodes
}
},
{
"localJobID": "4",
"type": "DataReturn",
"targetJob": [
{
"targetJobID": "1",
"inputParams": {}
}
]
}
]
}
}
elif hpctype.lower() == "hashcat":
create_package_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"type": "HPC",
"name": task_name,
"clusterId": "1830873578531228942",
"backend": "slurm",
"app": "hashcat",
"operateType": "",
"parameters": {
"partition": partition,
"ntasks": ntasks,
"nodes": nodes,
"hashType": "0",
"attackMode": "3",
"hashInput": "e2d0272ec941c0fb4021fb1b7104cfd7",
"mask": "?d?d?d?d?d?d?d?d"
}
},
{
"localJobID": "4",
"type": "DataReturn",
"targetLocalJobID": "1"
}
]
}
}
elif hpctype.lower() == "lammps":
create_package_payload = {
"userID": 5,
"jobSetInfo": {
"jobs": [
{
"localJobID": "1",
"type": "HPC",
"description": "qq",
"name": task_name,
"clusterId": "1830873578531228942",
"backend": "slurm",
"app": "lammps",
"operateType": "",
"parameters": {
"inputFile": "in.min",
"hpcBindingFiles": [
{
"paramName": "inputFile",
"resource": {
"type": "object",
"objectID": objectID
}
}
],
"partition": partition,
"ntasks": ntasks,
"nodes": nodes
}
},
{
"localJobID": "4",
"type": "DataReturn",
"targetJob": [
{
"targetJobID": "1",
"inputParams": {}
}
]
}
]
}
}
else:
print(f"未知任务类型: {hpctype}")
return None
try:
# 发送 POST 请求
submit_response = requests.post(request_url, headers=headers, json=create_package_payload)
# 检查响应状态码
if submit_response.status_code == 200:
try:
# 解析响应内容为 JSON 格式
submit_result = submit_response.json()
print(f"提交超算结果: {submit_result}")
return submit_result
except json.JSONDecodeError:
print("响应数据解析失败，不是有效的 JSON 格式")
else:
print(f"提交超算结果失败，状态码: {submit_response.status_code}")
except requests.RequestException as e:
print(f"请求过程中发生异常: {e}")
return None
def submit_single_task(self, task_name, hpctype, partition, ntasks, nodes, packageName, hpcFile):
"""
统一的HPC任务提交方法
:param task_name: 任务名称
:param hpctype: HPC任务类型 (bwa, hashcat, lammps)
:param partition: 分区名称
:param ntasks: 任务数量
:param nodes: 节点数量
:param packageName: 包名称
:param hpcFile: 文件名
:return: 任务提交结果
"""
# 获取 Token
if not self.token:
self.get_token()
if not self.token:
print("获取token失败")
return None
# 如果是hashcat任务，不需要查询文件ID，直接提交
if hpctype.lower() == "hashcat":
result = self.submit_hpc_task(task_name, partition, ntasks, nodes, None, hpctype)
return result
# 其他类型需要查询文件ID
if not packageName or not hpcFile:
print(f"{hpctype}任务需要packageName和hpcFile参数")
return None
# 根据 packageName 获取 packageID
packageID = self.get_package_id_by_name(packageName)
if not packageID:
print(f"未找到名为 {packageName} 的包")
return None
# 根据 packageID 和 hpcFile 获取 objectID
objectID = self.get_object_id_by_package_id(packageID, hpcFile, hpctype)
if not objectID:
print(f"未找到对应的 objectID for package {packageName} and file {hpcFile}")
return None
# 提交任务
result = self.submit_hpc_task(task_name, partition, ntasks, nodes, objectID, hpctype, hpcFile)
return result
def submit_multiple_tasks(self, tasks_config):
"""
批量提交HPC任务
:param tasks_config: 任务配置列表，每个元素包含任务参数的字典
:return: 提交结果列表
"""
results = []
for task_config in tasks_config:
task_name = task_config.get("task_name")
hpctype = task_config.get("hpctype")
partition = task_config.get("partition", "ft_test")
ntasks = task_config.get("ntasks", "1")
nodes = task_config.get("nodes", "1")
packageName = task_config.get("packageName", "")
hpcFile = task_config.get("hpcFile", "")
print(f"正在提交任务: {task_name}, 类型: {hpctype}")
result = self.submit_single_task(task_name, hpctype, partition, ntasks, nodes, packageName, hpcFile)
results.append({
"task_name": task_name,
"result": result
})
return results
def submit_task_with_timestamp(self, task_name, hpctype, partition="ft_test", ntasks="1", nodes="1", packageName="", hpcFile=""):
"""
提交带时间戳的任务
:param task_name: 任务名称
:param hpctype: HPC任务类型 (bwa, hashcat, lammps)
:param partition: 分区名称
:param ntasks: 任务数量
:param nodes: 节点数量
:param packageName: 包名称
:param hpcFile: 文件名
:return: 任务提交结果
"""
# 添加时间戳到任务名称
timestamped_task_name = task_name + "_" + datetime.now().strftime("%Y%m%d%H%M%S")
return self.submit_single_task(timestamped_task_name, hpctype, partition, ntasks, nodes, packageName, hpcFile)
def main(task_name, hpctype, partition="ft_test", ntasks="1", nodes="1", packageName="", hpcFile=""):
"""
主函数，统一的任务提交入口
:param task_name: 任务名称
:param hpctype: HPC任务类型 (bwa, hashcat, lammps)
:param partition: 分区名称
:param ntasks: 任务数量
:param nodes: 节点数量
:param packageName: 包名称
:param hpcFile: 文件名
:return: 任务提交结果
"""
submitter = HPCTaskSubmitter()
result = submitter.submit_task_with_timestamp(task_name, hpctype, partition, ntasks, nodes, packageName, hpcFile)
return result
if __name__ == "__main__":
# 示例：提交BWA任务
print("提交BWA任务示例:")
bwa_result = main("hpctask_bwa_test", "bwa", "ft_test", "1", "1", "bwa_data", "reads.fq")
if bwa_result:
print("BWA任务提交成功:", bwa_result)
print("\n提交Hashcat任务示例:")
hashcat_result = main("hashcat_test", "hashcat", "ft_test", "64", "4")
if hashcat_result:
print("Hashcat任务提交成功:", hashcat_result)
print("\n提交LAMMPS任务示例:")
lammps_result = main("hpctask_lammps_test", "lammps", "ft_test", "1", "1", "lammps_data", "in.min")
if lammps_result:
print("LAMMPS任务提交成功:", lammps_result)
# 批量提交示例
print("\n批量提交任务示例:")
tasks = [
{"task_name": "batch_bwa_task", "hpctype": "bwa", "partition": "ft_test", "ntasks": "1", "nodes": "1", "packageName": "bwa_data", "hpcFile": "reads.fq"},
{"task_name": "batch_hashcat_task", "hpctype": "hashcat", "partition": "ft_test", "ntasks": "64", "nodes": "4"},
{"task_name": "batch_lammps_task", "hpctype": "lammps", "partition": "ft_test", "ntasks": "1", "nodes": "1", "packageName": "lammps_data", "hpcFile": "in.min"}
]
submitter = HPCTaskSubmitter()
batch_results = submitter.submit_multiple_tasks(tasks)
for res in batch_results:
print(f"任务 {res['task_name']} 提交结果: {res['result']}")

279
task/task_manager.py
View File

@ -1,279 +0,0 @@
import time
import uuid
from datetime import datetime
from typing import Dict, List, Optional
import requests
import sys
import os
import importlib
from task.ai_task_submit import submit_ai_task
from config.db_manager import load_config_from_db, update_task_all_fields
# 添加TaskSubmissionScript到Python路径
sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'TaskSubmissionScript'))
from task.hpc_task_submit import HPCTaskSubmitter
from config.database import load_db_config
from config.db_manager import update_cluster_resources
from config.config import logger, TASK_STATUS, task_map, task_map_lock
from config.cache_manager import get_cached_config
from task.cloud_task_submit import submit_cloud_task
def generate_task_templates() -> List[Dict]:
"""从数据库加载并返回任务模板"""
# 使用缓存的配置而不是每次都从数据库加载
config = get_cached_config()
return config.get("task_templates", [])
def load_tasks_to_queue(templates: List[Dict]) -> None:
with task_map_lock:
task_map.clear()
for idx, template in enumerate(templates):
try:
# 检查必填字段
required_fields = ["type", "task_name", "resource"]
missing_fields = [f for f in required_fields if f not in template]
if missing_fields:
logger.warning(f"跳过无效任务模板（索引{idx}）：缺少字段 {missing_fields}")
continue
# 生成任务名称（使用数据库中的task_name）
task_name = template["task_name"] or f"task-{int(time.time())}"
task_type = template["type"]
resource = template["resource"]
strategy = template.get("strategy")
# 数据集ID设为可选
dataset_id = template.get("dataset_id", "")
code_id = template.get("code_id", "")
# 创建任务实例
task = {
"target_id": str(uuid.uuid4()), # 任务唯一ID
"task_name": task_name,
"package_name": f"{task_name.lower()}-pkg" if task_name else f"task-{int(time.time())}-pkg", # 文件夹名称
"type": task_type,
"status": TASK_STATUS["SUBMITTED"],
"submit_time": time.strftime("%Y-%m-%d %H:%M:%S"),
"success_time": None,
"third_party_task_id": "",
"dataset_id": dataset_id, # 数据集ID
"code_id":code_id,
"error_msg": "",
"fail_count": 0,
"max_fail_threshold": 3,
"cluster_id": "",
"strategy": strategy,
"resource": resource
}
# 根据任务类型设置特定字段
if task_type == "Ai":
# AI任务
task["dataset_id"] = dataset_id
task["resource"] = resource or {"CPU": 1, "MEMORY": 4, "NPU": 0}
# AI任务需要file_location字段，但数据库中没有直接提供，需要通过其他方式获取
elif task_type == "Hpc":
task["partition"] = "default" # 默认分区
task["ntasks"] = "1"
task["nodes"] = "1"
task["packageName"] = f"{task_name.lower()}-hpc" if task_name else f"task-{int(time.time())}-hpc"
task["hpcFile"] = "" # 数据库中没有提供
task["dataset_id"] = dataset_id
elif task_type == "Cloud":
task["replicas"] = 1
task["strategy"] = strategy or "resource" # 调度策略
task["cloud_strategy"] = "replication" # 部署策略
task["adapter_name"] = "" # 适配器名称
task["resource"] = resource or {"CPU": 0.1, "MEMORY": 6, "port": 80}
task["port"] = task["resource"].get("port", 80)
task["dataset_id"] = dataset_id
# 将任务添加到全局任务映射中
task_map[task["target_id"]] = task
logger.info(
f"任务入队 | task_name: {task_name} | 任务类型: {task_type} | "
f"{'含数据集' if dataset_id else '无数据集'}")
except Exception as e:
logger.error(f"处理任务模板时发生异常（索引{idx}）: {e}", exc_info=True)
continue
logger.info(f"任务队列加载完成 | 总任务数: {len(task_map)}")
def get_task_submit_function(module_name):
"""动态导入任务提交函数"""
try:
module = importlib.import_module(module_name)
return getattr(module, 'main')
except (ImportError, AttributeError) as e:
logger.error(f"无法导入模块 {module_name} 或其 main 函数: {e}")
return None
def submit_single_task(task: Dict,token) -> bool:
"""提交单个任务到集群"""
# 获取选择的集群ID
global resource_changes
cluster_id = task.get("cluster_id")
if not cluster_id:
logger.error(f"[{task['task_name']}] 提交失败：未指定集群ID")
return False
task_type = task["type"]
try:
if task_type == "Ai":
third_party_task_id = submit_ai_task(task,token)
elif task_type == "Hpc":
task_name = task["task_name"]
partition = task["partition"]
ntasks = task["ntasks"]
nodes = task["nodes"]
packageName = task["package_name"]
hpcFile = task.get("hpc_file", "") # 修正字段名，使用hpcFile而非hpc_file
# 确定HPC任务类型（根据任务名称推断）
hpc_type = "bwa" # 默认类型
if "_bwa_" in task_name or "_bwa2_" in task_name:
hpc_type = "bwa"
elif "_hashcat_" in task_name:
hpc_type = "hashcat"
elif "_lammps_" in task_name:
hpc_type = "lammps"
# 创建HPC任务提交器实例
hpc_submitter = HPCTaskSubmitter()
# 提交HPC任务
third_party_task_id = hpc_submitter.submit_single_task(
task_name, hpc_type, partition, ntasks, nodes, packageName, hpcFile
)
# 如果返回的是字典格式，提取jobSetID
if isinstance(third_party_task_id, dict) and "data" in third_party_task_id:
third_party_task_id = third_party_task_id["data"].get("jobSetID")
elif task_type == "Cloud":
# 提取云任务所需的参数
replicas = task.get("replicas")
task_res = task.get("resource")
cpu = task_res["CPU"]
memory = task_res["MEMORY"]
port = task.get("port")
task_name = task["task_name"]
cluster_id = task.get("cluster_id")
# 传递所有必要的参数给云任务提交函数
third_party_task_id = submit_cloud_task(
task_name,
cluster_id,
replicas=replicas,
cpu=cpu,
memory=memory,
port=port,
)
else:
raise ValueError(f"未知任务类型: {task_type}")
# 构造资源变化字典
task_resource = task.get("resource", {})
resource_changes = {}
# 计算资源变化量（负值表示扣减）
for res_type in ["CPU", "MEMORY"]:
required = task_resource.get(res_type, 0)
if required > 0:
resource_changes[res_type] = -required
# 处理加速器资源
for acc_type in [key for key in task_resource if key not in ["CPU", "MEMORY"]]:
required_count = task_resource.get(acc_type, 0)
if required_count > 0:
resource_changes[acc_type] = -required_count
if third_party_task_id:
# 使用批量操作更新数据库
if update_cluster_resources(cluster_id, resource_changes):
logger.info(f"成功从数据库扣减集群 {cluster_id} 的资源: {resource_changes}")
else:
logger.error(f"从数据库扣减集群 {cluster_id} 的资源失败: {resource_changes}")
# 更新任务状态为成功
logger.info(f"[{task['task_name']}] 任务提交至集群 {cluster_id} 成功 | 云际任务ID: {third_party_task_id}")
# 任务提交成功，更新数据库中的状态为SUBMITTING
if update_task_all_fields(task['task_name'], status=TASK_STATUS['SUBMITTING'],third_party_task_id=third_party_task_id,commit_time= datetime.now()):
logger.info(f"成功更新任务 {task['task_name']} 在数据库中的状态为 {TASK_STATUS['SUBMITTING']}")
else:
logger.error(f"更新任务 {task['task_name']} 在数据库中的状态失败")
return False
return True
else:
# 如果任务提交失败且资源已更新，需要恢复资源
# 这里使用原子操作避免多次数据库访问
if resource_changes:
recovery_changes = {res: -change for res, change in resource_changes.items()}
update_cluster_resources(cluster_id, recovery_changes)
logger.info(f"任务失败，已从数据库恢复集群 {cluster_id} 的资源: {recovery_changes}")
raise ValueError("任务提交返回ID为空")
except Exception as e:
# 使用更高效的日志记录方式
logger.error(f"[{task['task_name']}] 提交失败: {str(e)}", exc_info=True, extra={
'cluster_id': cluster_id,
'task_type': task_type,
'resource_changes': resource_changes if 'resource_changes' in locals() else {}
})
return False
def query_third_party_task_status(third_party_task_id: str,token) -> Optional[Dict]:
"""查询云际平台任务状态"""
if not third_party_task_id:
logger.warning("云际任务ID为空，无法查询状态")
return None
try:
config = load_config_from_db()
url = config["api_config"]["task_detail"]["url"]
params = {
"jobSetID": third_party_task_id, # 使用传入的任务ID作为参数
"localJobID": 1
}
headers = {"Authorization": f"Bearer {token}"} if token else {}
response = requests.get(
url,
params=params,
headers=headers,
timeout=config["api_config"]["task_detail"]["timeout"]
)
response.raise_for_status()
result = response.json()
# 适配新的接口返回结构，检查 code 是否为 "OK"
if result.get("code") != "OK":
logger.error(f"查询任务状态接口返回异常，响应码非 OK | 响应: {result}")
return None
# 检查 data 字段是否存在
if "data" not in result:
logger.error(f"查询任务状态接口返回异常，无 data 字段 | 响应: {result}")
return None
data = result["data"]
# 从数据库重新加载集群资源信息，确保获取最新的资源状态
load_db_config()
return data
except Exception as e:
logger.error(f"查询任务状态时发生异常: {str(e)}", exc_info=True)
return None