.env.example.yml

@@ -1,14 +0,0 @@
-# GitHub Configuration
-github_tokens:
-  - ghp_your_github_token_here
-  - ghp_your_second_github_token_here
-
-# GitHub API Configuration
-base_url: https://api.github.com
-min_stars: 20
-request_interval: 0.8
-max_retry_5xx: 3
-
-# Clone Configuration
-clone_retry: 2
-clone_timeout: 180

.gitignore

@@ -1,29 +0,0 @@
-# Environment files
-.env.yml
-.env
-
-# Python
-__pycache__/
-*.pyc
-*.pyo
-*.pyd
-
-# Virtual environment
-.venv/
-
-# Logs
-logs/
-*.log
-
-# Data directories
-data/
-
-# IDE
-.vscode/
-.idea/
-*.swp
-*.swo
-
-# OS
-.DS_Store
-Thumbs.db

README.md

@@ -1,117 +1,2 @@
 # benchmark
 
-## 开发模式
-
-### 1. 复制环境变量文件
-
-```bash
-cp .env.example.yml .env.yml
-```
-
-### 2. 编辑环境变量文件
-
-打开 `.env.yml` 文件，添加你的 GitHub Token：
-
-```yaml
-# GitHub Configuration
-github_tokens:
-  - ghp_your_first_github_token_here
-  - ghp_your_second_github_token_here
-```
-
-### 3. 创建虚拟环境
-
-使用 uv 工具创建虚拟环境，并指定 Python 版本为 3.13.9：
-
-```bash
-uv venv -p 3.13.9
-```
-
-### 4. 激活虚拟环境
-
-- **Windows**：
-  ```bash
-  .venv\Scripts\activate
-  ```
-
-- **Linux/macOS**：
-  ```bash
-  source .venv/bin/activate
-  ```
-
-### 5. 安装依赖
-
-使用 uv 安装项目依赖：
-
-```bash
-uv pip install -r requirements.txt
-```
-
-### 6. 运行脚本
-
-```bash
-爬取项目
-python springf_project.py
-```
-```bash
-项目三方包集合
-python third_library_set.py
-```
-```bash
-获取并存储methods表数据
-mvn clean compile
-mvn dependency:build-classpath | Out-File -Encoding UTF8 classpath.txt
-Get-Content classpath.txt | Select-String "mysql-connector-j"
-java -cp "$cp;target/classes" JavaParser "E:\repo_examole" "..\src\main\java\db_config.properties"
-```
-```bash
-获取并存储method_call表数据
-mvn -q exec:java -Dexec.mainClass="UnifiedCallParser" -Dexec.args="..\db_config.properties"
-```
-
-```bash
-获取并存储api表数据
-mvn exec:java -Dexec.mainClass="ApiParser" -Dexec.args="<project-path> ..\config\db_config.properties"
-```
-
-```bash
-获取并存储api_call表数据
-运行qpi_call.sql内的内容
-```
-
-```bash
-获取并存储method_api_depth表数据
-运行method_api_depth.sql内的内容
-```
-
-##流程图
-![系统流程图](./image/流程图.png)
-
-## 配置说明
-
-- `github_tokens`：GitHub Token 列表，用于轮训提高请求速率
-- `base_url`：GitHub API 基础 URL
-- `min_stars`：最小星星数筛选条件
-- `request_interval`：请求间隔时间（秒）
-- `max_retry_5xx`：5xx 错误最大重试次数
-- `clone_retry`：克隆仓库最大重试次数
-- `clone_timeout`：克隆仓库超时时间（秒）
-
-## 功能说明
-
-该脚本用于：
-1. 按日期搜索 GitHub 上的 Java 仓库
-2. 筛选出使用 Spring Framework 和测试框架（JUnit/TestNG）的仓库
-3. 克隆符合条件的仓库到本地
-4. 生成包含仓库信息的 JSON 文件
-
-## 目录结构
-
-- `data/`：存放收集的数据和克隆的仓库
-- `logs/`：存放日志文件
-- `.env.yml`：本地配置文件（不提交到 git）
--`.db_config.properties`：数据库配置文件（不提交到 git）
--`.llm_config.properties`：模型调用配置文件（不提交到 git）
-- `.env.example.yml`：示例配置文件
-- `requirements.txt`：项目依赖列表
-- `springf_preject.py`：主脚本文件

api_example.json

@@ -1,112 +0,0 @@
-"""
-1、整体结构说明
-{
-  "project": "project_name",
-  "file": "file_path",
-  "functions": [
-    {
-      "id": "function_id",
-      "methodname": "function_name",
-      "description": "function_description",
-      "methodbody": "function_body",
-      "deepth": overall_depth,
-      "difficulty": "function_difficulty",
-      "apis": [ list_of_apis_used_in_function ]
-    }
-  ]
-}
-2、API结构说明
-{
-  "apiname": "api_name",
-  "library": "library_name",
-  "id": "api_id",
-  "apisource": "api_source_type", // "custome" or "third_party"
-  "parameters": [
-    {
-      "name": "parameter_name",
-      "type": "parameter_type"
-    }
-  ],
-  "returns": {
-    "name": "return_value_name",
-    "type": "return_value_type"
-  },
-  "deepth": api_depth,
-  "para_depend": [ list_of_parameter_dependencies ],
-  "chain_depend": [ list_of_chain_dependencies ],
-  "apis": [ list_of_nested_apis ]
-}
-3、思考
-a.id应该从深度只有1的自定义函数开始编号，三方包是否需要id待定
-b.deepth的定义：对于整体deepth，个人认为基于max deepth来定义更合理；对于api的deepth，就是这个api调用链的深度
-4、方法
-a、仍然使用jdt解析java代码
-b、筛选：无对应的测试用例
-"""
-{
-  "project": "projectA",
-  "file": "src/com/example/Example.java",
-  "functions": [
-    {
-      "id": "*****1",
-      "methodname": "FunctionA",
-      "description": "This function does A",
-      "methodbody": "public void FunctionA() { /* ... */ }",
-      "deepth": 3,    //对于apic来说是3 对于apib是2，maxdeepth来定义整体deepth更合理
-      "difficulty": "medium",
-      "apis": [
-        {
-          "apiname": "apiA",
-          "library": "",
-          "id": "*****2",
-          "apisource": "custome",
-          "parameters": [
-            {
-              "name": "param1",
-              "type": "String"
-            }
-          ],
-          "returns": {
-            "name": "returnValue1",
-            "type": ""
-          },
-          "deepth": 2,
-          "para_depend": [],
-          "chain_depend": [],
-          "apis": [
-            {
-              "apiname": "apiC",
-              "library": "org.example",
-              "id": "*****4",   //三方包是否需要id
-              "apisource": "third_party",
-              "parameters": [],
-              "returns": {
-                "name": "returnValue2",
-                "type": ""
-              },
-              "deepth": 1,
-              "para_depend": [],
-              "chain_depend": [],
-              "apis": []
-            }
-          ]
-        },
-        {
-          "apiname": "apiB",
-          "library": "org.example",
-          "id": "*****3",  //三方包是否需要id
-          "apisource": "third_party",
-          "parameters": [],
-          "returns": {
-            "name": "returnValue3",
-            "type": ""
-          },
-          "deepth": 1,
-          "para_depend": [],
-          "chain_depend": [],
-          "apis": []
-        }
-      ]
-    }
-  ]
-}

pom.xml

@@ -1,92 +0,0 @@
-<project xmlns="http://maven.apache.org/POM/4.0.0"
-         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
-         http://maven.apache.org/xsd/maven-4.0.0.xsd">
-
-    <modelVersion>4.0.0</modelVersion>
-
-    <groupId>api.extractor</groupId>
-    <artifactId>jdt-extractor</artifactId>
-    <version>1.0</version>
-
-    <!-- ========= 关键：统一编码 & JDK 版本 ========= -->
-    <properties>
-        <!-- 源码 / 资源统一使用 UTF-8（解决 GBK 不可映射字符） -->
-        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
-
-        <!-- 与你当前 JDK 15/17 兼容 -->
-        <maven.compiler.source>17</maven.compiler.source>
-        <maven.compiler.target>17</maven.compiler.target>
-    </properties>
-
-    <dependencies>
-
-        <!-- Eclipse JDT Core（包含 ASTParser / DOM / Binding 能力） -->
-        <dependency>
-            <groupId>org.eclipse.jdt</groupId>
-            <artifactId>org.eclipse.jdt.core</artifactId>
-            <version>3.36.0</version>
-        </dependency>
-
-        <dependency>
-            <groupId>com.mysql</groupId>
-            <artifactId>mysql-connector-j</artifactId>
-            <version>8.0.33</version>
-        </dependency>
-
-        <!-- JSON 序列化 -->
-        <dependency>
-            <groupId>com.google.code.gson</groupId>
-            <artifactId>gson</artifactId>
-            <version>2.10.1</version>
-        </dependency>
-
-        <!-- org.json 库（解决 JSONArray 和 JSONObject 问题） -->
-        <dependency>
-            <groupId>org.json</groupId>
-            <artifactId>json</artifactId>
-            <version>20210307</version>
-        </dependency>
-
-    </dependencies>
-
-    <build>
-        <plugins>
-            <!-- 编译器插件 -->
-            <plugin>
-                <groupId>org.apache.maven.plugins</groupId>
-                <artifactId>maven-compiler-plugin</artifactId>
-                <version>3.13.0</version>
-                <configuration>
-                    <encoding>UTF-8</encoding>
-                    <source>17</source>
-                    <target>17</target>
-                </configuration>
-            </plugin>
-
-            <!-- exec-maven-plugin -->
-            <plugin>
-                <groupId>org.codehaus.mojo</groupId>
-                <artifactId>exec-maven-plugin</artifactId>
-                <version>3.1.0</version>
-            </plugin>
-
-            <!-- maven-dependency-plugin -->
-            <plugin>
-                <groupId>org.apache.maven.plugins</groupId>
-                <artifactId>maven-dependency-plugin</artifactId>
-                <version>3.5.0</version>
-                <executions>
-                    <execution>
-                        <id>build-classpath</id>
-                        <goals>
-                            <goal>build-classpath</goal>
-                        </goals>
-                    </execution>
-                </executions>
-            </plugin>
-        </plugins>
-    </build>
-
-</project>

requirements.txt

@@ -1,2 +0,0 @@
-requests==2.31.0
-PyYAML==6.0.1

springf_preject.py

@@ -0,0 +1,287 @@
+import requests
+import os
+import time
+import datetime
+import base64
+import json
+import xml.etree.ElementTree as ET
+import subprocess
+import re
+
+# 路径配置
+SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.dirname(SCRIPT_DIR)
+
+DATA_DIR = os.path.join(PROJECT_ROOT, "data")
+REPO_DIR = os.path.join(DATA_DIR, "repos1")
+LOG_DIR = os.path.join(PROJECT_ROOT, "logs")
+
+OUTPUT_JSON = os.path.join(DATA_DIR, "collected_repos.json")
+CLONE_FAIL_LOG = os.path.join(LOG_DIR, "clone_fail.txt")
+RATE_LIMIT_LOG = os.path.join(LOG_DIR, "rate_limit.log")
+MISSING_REPO_LOG = os.path.join(LOG_DIR, "missing_repo.log")
+
+for d in [DATA_DIR, REPO_DIR, LOG_DIR]:
+    os.makedirs(d, exist_ok=True)
+
+# GitHub 配置
+GITHUB_TOKEN = os.getenv("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")
+HEADERS = {"Accept": "application/vnd.github+json"}
+if GITHUB_TOKEN:
+    HEADERS["Authorization"] = f"token {GITHUB_TOKEN}"
+
+BASE_URL = "https://api.github.com"
+MIN_STARS = 20
+REQUEST_INTERVAL = 0.8
+MAX_RETRY_5XX = 3
+
+# clone 控制参数 
+CLONE_RETRY = 2
+CLONE_TIMEOUT = 180  
+
+#  日志工具
+def log_to_file(path, msg):
+    with open(path, "a", encoding="utf-8") as f:
+        f.write(msg + "\n")
+
+#  GitHub GET
+def github_get(url, params=None):
+    retry_5xx = 0
+    while True:
+        r = requests.get(url, headers=HEADERS, params=params)
+        if r.status_code == 200:
+            time.sleep(REQUEST_INTERVAL)
+            return r
+        elif r.status_code == 403:
+            reset = r.headers.get("X-RateLimit-Reset")
+            if reset:
+                sleep_time = int(reset) - int(time.time()) + 1
+                log_to_file(RATE_LIMIT_LOG, f"403 rate limit → sleep {sleep_time}s, url={url}")
+                time.sleep(max(sleep_time, 1))
+                continue
+            return None
+        elif r.status_code == 404:
+            log_to_file(MISSING_REPO_LOG, f"404 missing {url}")
+            return None
+        elif 500 <= r.status_code < 600:
+            retry_5xx += 1
+            if retry_5xx <= MAX_RETRY_5XX:
+                time.sleep(2)
+                continue
+            return None
+        else:
+            return None
+
+# 搜索仓库（按日期） 
+def search_repositories_by_stars(start_date, end_date):
+    query = f"language:Java stars:>={MIN_STARS} created:{start_date}..{end_date}"
+    repos = []
+
+    for page in range(1, 11):
+        r = github_get(
+            f"{BASE_URL}/search/repositories",
+            params={
+                "q": query,
+                "sort": "stars",
+                "order": "desc",
+                "per_page": 100,
+                "page": page
+            },
+        )
+        if not r:
+            break
+        items = r.json().get("items", [])
+        if not items:
+            break
+        repos.extend(items)
+    return repos
+
+#  文件 
+def get_file_content(repo, path):
+    r = github_get(f"{BASE_URL}/repos/{repo}/contents/{path}")
+    if not r:
+        return None
+    c = r.json().get("content")
+    if not c:
+        return None
+    return base64.b64decode(c).decode("utf-8", errors="ignore")
+
+#  Maven  解析
+def analyze_pom(pom_text):
+    uses_test = False
+    uses_spring = False
+
+    try:
+        root = ET.fromstring(pom_text)
+    except ET.ParseError:
+        return False, False
+
+    for dep in root.findall(".//dependencies/dependency"):
+        gid = dep.findtext("groupId")
+        if not gid:
+            continue
+        gid = gid.strip()
+
+        # 测试框架
+        if gid in {"junit", "org.junit.jupiter", "org.testng"}:
+            uses_test = True
+
+        # Spring Framework
+        if gid == "org.springframework":
+            uses_spring = True
+
+    return uses_test, uses_spring
+
+# Gradle 解析（正则）
+SPRING_RE = re.compile(r'["\']org\.springframework:([^"\']+)["\']')
+JUNIT_RE = re.compile(r'["\'](junit:junit|org\.junit\.jupiter:[^"\']+)["\']')
+
+def analyze_gradle(gradle_text):
+    uses_spring = bool(SPRING_RE.search(gradle_text))
+    uses_test = bool(JUNIT_RE.search(gradle_text))
+    return uses_test, uses_spring
+
+# 克隆仓库
+def clone_repo(repo_full_name):
+    local_name = repo_full_name.replace("/", "_")
+    target = os.path.join(REPO_DIR, local_name)
+
+    if os.path.exists(target):
+        print(f"  [SKIP] {repo_full_name} already cloned")
+        return target
+
+    for attempt in range(1, CLONE_RETRY + 1):
+        print(f"  [CLONE] {repo_full_name} (attempt {attempt}/{CLONE_RETRY})")
+
+        try:
+            result = subprocess.run(
+                ["git", "clone", "--depth", "1",
+                 f"https://github.com/{repo_full_name}.git", target],
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                text=True,
+                timeout=CLONE_TIMEOUT
+            )
+        except subprocess.TimeoutExpired:
+            msg = f"timeout after {CLONE_TIMEOUT}s"
+            print(f"    ↳ {msg}")
+            if attempt == CLONE_RETRY:
+                log_to_file(CLONE_FAIL_LOG, f"{repo_full_name} → {msg}")
+            continue
+
+        if result.returncode == 0:
+            return target
+
+        error_msg = result.stderr.strip() or "unknown error"
+        print(f"    ↳ git error: {error_msg}")
+
+        if attempt == CLONE_RETRY:
+            log_to_file(CLONE_FAIL_LOG, f"{repo_full_name} → {error_msg}")
+
+        time.sleep(2)
+
+    return None
+
+#  遍历多模块 Maven 项目 
+def find_all_poms(local_repo_path):
+    pom_paths = []
+    for root, dirs, files in os.walk(local_repo_path):
+        if "pom.xml" in files:
+            pom_paths.append(os.path.join(root, "pom.xml"))
+    return pom_paths
+
+def analyze_maven_multimodule(local_repo_path):
+    uses_test = False
+    uses_spring = False
+    poms = find_all_poms(local_repo_path)
+    for pom_file in poms:
+        try:
+            with open(pom_file, "r", encoding="utf-8") as f:
+                text = f.read()
+        except Exception:
+            continue
+        t, s = analyze_pom(text)
+        uses_test |= t
+        uses_spring |= s
+        if uses_test and uses_spring:
+            break
+    return uses_test, uses_spring
+
+# 分析仓库 
+def analyze_repository(repo):
+    name = repo["full_name"]
+    stars = repo.get("stargazers_count", 0)
+
+    gradle = get_file_content(name, "build.gradle")
+    gradle_kts = get_file_content(name, "build.gradle.kts")
+
+    # 克隆仓库
+    local_path = clone_repo(name)
+    if not local_path:
+        return None
+
+    # Maven 分析（多模块）
+    pom_paths = find_all_poms(local_path)
+    uses_test, uses_spring = False, False
+    if pom_paths:
+        uses_test, uses_spring = analyze_maven_multimodule(local_path)
+        build_tool = "maven"
+    else:
+        # Gradle 分析
+        gradle_files = [f for f in [gradle, gradle_kts] if f]
+        for g in gradle_files:
+            t, s = analyze_gradle(g)
+            uses_test |= t
+            uses_spring |= s
+        build_tool = "gradle"
+
+    if not uses_test:
+        print(f"  [SKIP] {name} no junit/testng detected")
+        return None
+    if not uses_spring:
+        print(f"  [SKIP] {name} no Spring Framework dependency")
+        return None
+
+    print(f"  [OK] {name} passed all filters")
+    return {
+        "repo": name,
+        "stars": stars,
+        "build_tool": build_tool,
+        "local_path": local_path
+    }
+
+def main():
+    results = []
+
+    start_date = datetime.date(2025, 2, 8)
+    end_date = datetime.date(2025, 6, 30)
+
+    delta = datetime.timedelta(days=1)
+    current_date = start_date
+
+    while current_date <= end_date:
+        next_date = current_date + delta
+        print(f"\n Searching date: {current_date}")
+
+        repos = search_repositories_by_stars(
+            current_date.isoformat(),
+            (next_date - datetime.timedelta(seconds=1)).isoformat()
+        )
+        print(f"  found {len(repos)} repos")
+
+        for repo in repos:
+            info = analyze_repository(repo)
+            if info:
+                results.append(info)
+
+        current_date = next_date
+
+    with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
+        json.dump(results, f, indent=2)
+
+    print(f"\n Total collected: {len(results)} repositories")
+    print(f" Saved to {OUTPUT_JSON}")
+    print(f" Clone failed repos saved to {CLONE_FAIL_LOG}")
+
+if __name__ == "__main__":
+    main()

springf_project.py

@@ -1,478 +0,0 @@
-import requests
-import os
-import time
-import datetime
-import base64
-import json
-import xml.etree.ElementTree as ET
-import subprocess
-import re
-import yaml
-
-# 路径配置
-SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
-
-DATA_DIR = os.path.join(SCRIPT_DIR, "data")
-REPO_DIR = os.path.join(DATA_DIR, "repos1")
-LOG_DIR = os.path.join(SCRIPT_DIR, "logs")
-
-OUTPUT_JSON = os.path.join(DATA_DIR, "collected_repos.json")
-CLONE_FAIL_FILE = os.path.join(LOG_DIR, "clone_fail.json")
-RATE_LIMIT_LOG = os.path.join(LOG_DIR, "rate_limit.log")
-MISSING_REPO_LOG = os.path.join(LOG_DIR, "missing_repo.log")
-SKIPPED_REPOS_FILE = os.path.join(LOG_DIR, "skipped_repos.json")
-COLLECTED_REPOS_FILE = os.path.join(LOG_DIR, "collected_repos.json")
-
-for d in [DATA_DIR, REPO_DIR, LOG_DIR]:
-    os.makedirs(d, exist_ok=True)
-
-# 读取配置文件
-def load_config():
-    config_path = os.path.join(SCRIPT_DIR, ".env.yml")
-    if not os.path.exists(config_path):
-        print(f"Config file not found: {config_path}")
-        print(f"Please create it based on .env.example.yml")
-        exit(1)
-    
-    with open(config_path, "r") as f:
-        return yaml.safe_load(f)
-
-# 加载配置
-config = load_config()
-
-# GitHub 配置
-GITHUB_TOKENS = config.get("github_tokens", [])
-# 兼容旧版配置
-if not GITHUB_TOKENS and config.get("github_token"):
-    GITHUB_TOKENS = [config.get("github_token")]
-
-# Token轮训索引
-current_token_index = 0
-
-# 获取当前Token并轮换
-def get_next_token():
-    global current_token_index
-    if not GITHUB_TOKENS:
-        return None
-    token = GITHUB_TOKENS[current_token_index]
-    # 轮换到下一个Token
-    current_token_index = (current_token_index + 1) % len(GITHUB_TOKENS)
-    return token
-
-BASE_URL = config.get("base_url", "https://api.github.com")
-MIN_STARS = config.get("min_stars", 20)
-REQUEST_INTERVAL = config.get("request_interval", 0.8)
-MAX_RETRY_5XX = config.get("max_retry_5xx", 3)
-
-# clone 控制参数 
-CLONE_RETRY = config.get("clone_retry", 2)
-CLONE_TIMEOUT = config.get("clone_timeout", 180)  
-
-#  日志工具
-def log_to_file(path, msg):
-    with open(path, "a", encoding="utf-8") as f:
-        f.write(msg + "\n")
-
-
-# 加载已处理的仓库列表（包括已收集和已跳过的）
-def load_processed_repos():
-    """加载所有已处理的仓库列表，包括已收集和已跳过的，用于后续运行时跳过这些仓库"""
-    processed = set()
-    
-    # 加载已跳过的仓库
-    if os.path.exists(SKIPPED_REPOS_FILE):
-        try:
-            with open(SKIPPED_REPOS_FILE, "r", encoding="utf-8") as f:
-                skipped = json.load(f)
-                processed.update(skipped)
-        except Exception as e:
-            print(f"Failed to load skipped repos: {e}")
-    
-    # 加载已收集的仓库
-    if os.path.exists(OUTPUT_JSON):
-        try:
-            with open(OUTPUT_JSON, "r", encoding="utf-8") as f:
-                collected = json.load(f)
-                processed.update(repo["repo"] for repo in collected)
-        except Exception as e:
-            print(f"Failed to load collected repos: {e}")
-    
-    return processed
-
-
-# 保存已跳过的仓库列表
-def save_skipped_repos(skipped_repos):
-    """保存已跳过的仓库列表，以便后续运行时跳过这些仓库"""
-    try:
-        with open(SKIPPED_REPOS_FILE, "w", encoding="utf-8") as f:
-            json.dump(list(skipped_repos), f, indent=2, ensure_ascii=False)
-    except Exception as e:
-        print(f"Failed to save skipped repos: {e}")
-
-
-# 保存已收集的仓库列表
-def save_collected_repos(collected_repos):
-    """保存已收集的仓库列表，以便后续运行时跳过这些仓库"""
-    try:
-        with open(COLLECTED_REPOS_FILE, "w", encoding="utf-8") as f:
-            json.dump(list(collected_repos), f, indent=2, ensure_ascii=False)
-    except Exception as e:
-        print(f"Failed to save collected repos: {e}")
-
-
-# 加载克隆失败的仓库列表
-def load_clone_failures():
-    """加载克隆失败的仓库列表"""
-    if not os.path.exists(CLONE_FAIL_FILE):
-        return []
-    
-    try:
-        with open(CLONE_FAIL_FILE, "r", encoding="utf-8") as f:
-            return json.load(f)
-    except Exception as e:
-        print(f"Failed to load clone failures: {e}")
-        return []
-
-
-# 保存克隆失败的仓库列表
-def save_clone_failures(clone_failures):
-    """保存克隆失败的仓库列表"""
-    try:
-        with open(CLONE_FAIL_FILE, "w", encoding="utf-8") as f:
-            json.dump(clone_failures, f, indent=2, ensure_ascii=False)
-    except Exception as e:
-        print(f"Failed to save clone failures: {e}")
-
-#  GitHub GET
-def github_get(url, params=None):
-    retry_5xx = 0
-    while True:
-        # 获取当前请求的Token和Headers
-        token = get_next_token()
-        headers = {"Accept": "application/vnd.github+json"}
-        if token:
-            headers["Authorization"] = f"token {token}"
-        
-        r = requests.get(url, headers=headers, params=params)
-        if r.status_code == 200:
-            time.sleep(REQUEST_INTERVAL)
-            return r
-        elif r.status_code == 403:
-            reset = r.headers.get("X-RateLimit-Reset")
-            if reset:
-                sleep_time = int(reset) - int(time.time()) + 1
-                log_to_file(RATE_LIMIT_LOG, f"403 rate limit → sleep {sleep_time}s, url={url}")
-                time.sleep(max(sleep_time, 1))
-                continue
-            return None
-        elif r.status_code == 404:
-            log_to_file(MISSING_REPO_LOG, f"404 missing {url}")
-            return None
-        elif 500 <= r.status_code < 600:
-            retry_5xx += 1
-            if retry_5xx <= MAX_RETRY_5XX:
-                time.sleep(2)
-                continue
-            return None
-        else:
-            return None
-
-# 搜索仓库（按日期） 
-def search_repositories_by_stars(start_date, end_date):
-    query = f"language:Java stars:>={MIN_STARS} created:{start_date}..{end_date}"
-    repos = []
-
-    for page in range(1, 11):
-        r = github_get(
-            f"{BASE_URL}/search/repositories",
-            params={
-                "q": query,
-                "sort": "stars",
-                "order": "desc",
-                "per_page": 100,
-                "page": page
-            },
-        )
-        if not r:
-            break
-        items = r.json().get("items", [])
-        if not items:
-            break
-        repos.extend(items)
-    return repos
-
-# 检查文件是否存在于仓库中并返回相对路径（支持任意路径）
-def check_file_exists(repo, filename):
-    """
-    使用 GitHub 搜索 API 检查文件是否存在于仓库的任何位置
-    使用模式：repo:{repo} path:{filename}
-    返回：如果存在返回文件的相对路径，否则返回 None
-    """
-    search_url = f"{BASE_URL}/search/code"
-    params = {
-        "q": f"repo:{repo} path:{filename}",
-        "per_page": 1  # 只需要知道第一个匹配结果
-    }
-    r = github_get(search_url, params=params)
-    if not r:
-        return None
-    try:
-        result = r.json()
-        if result.get("total_count", 0) > 0:
-            # 提取第一个匹配结果的相对路径
-            item = result.get("items", [])[0]
-            return item.get("path", None)
-        return None
-    except Exception:
-        return None
-
-
-# 检查仓库中是否包含@Test注解（支持任意路径）
-def check_has_test_annotations(repo):
-    """
-    使用 GitHub 搜索 API 检查仓库中是否包含@Test注解
-    使用模式：repo:{repo} @Test
-    """
-    search_url = f"{BASE_URL}/search/code"
-    params = {
-        "q": f"repo:{repo} @Test",
-        "per_page": 1  # 只需要知道是否存在，不需要所有结果
-    }
-    r = github_get(search_url, params=params)
-    if not r:
-        return False
-    try:
-        result = r.json()
-        return result.get("total_count", 0) > 0
-    except Exception:
-        return False
-
-
-# 检查仓库中是否包含 Spring Framework 导入语句（支持任意路径）
-def check_has_spring_imports(repo):
-    """
-    使用 GitHub 搜索 API 检查仓库中是否包含 Spring Framework 导入语句
-    使用模式：repo:{repo} import org.springframework
-    """
-    search_url = f"{BASE_URL}/search/code"
-    params = {
-        "q": f'repo:{repo} "import org.springframework"',
-        "per_page": 1  # 只需要知道是否存在，不需要所有结果
-    }
-    r = github_get(search_url, params=params)
-    if not r:
-        return False
-    try:
-        result = r.json()
-        return result.get("total_count", 0) > 0
-    except Exception:
-        return False
-
-# 获取文件内容（用于获取构建文件内容进行分析，支持任意路径）
-def get_file_content(repo, path):
-    r = github_get(f"{BASE_URL}/repos/{repo}/contents/{path}")
-    if not r:
-        return None
-    c = r.json().get("content")
-    if not c:
-        return None
-    return base64.b64decode(c).decode("utf-8", errors="ignore")
-
-#  Maven  解析
-def analyze_pom(pom_text):
-    uses_test = False
-    uses_spring = False
-
-    try:
-        root = ET.fromstring(pom_text)
-    except ET.ParseError:
-        return False, False
-
-    for dep in root.findall(".//dependencies/dependency"):
-        gid = dep.findtext("groupId")
-        if not gid:
-            continue
-        gid = gid.strip()
-
-        # 测试框架
-        if gid in {"junit", "org.junit.jupiter", "org.testng"}:
-            uses_test = True
-
-        # Spring Framework
-        if gid == "org.springframework":
-            uses_spring = True
-
-    return uses_test, uses_spring
-
-# Gradle 解析（正则）
-SPRING_RE = re.compile(r'["\']org\.springframework:([^"\']+)["\']')
-JUNIT_RE = re.compile(r'["\'](junit:junit|org\.junit\.jupiter:[^"\']+)["\']')
-
-def analyze_gradle(gradle_text):
-    uses_spring = bool(SPRING_RE.search(gradle_text))
-    uses_test = bool(JUNIT_RE.search(gradle_text))
-    return uses_test, uses_spring
-
-# 克隆仓库
-def clone_repo(repo_full_name, clone_failures):
-    local_name = repo_full_name.replace("/", "_")
-    target = os.path.join(REPO_DIR, local_name)
-
-    if os.path.exists(target):
-        print(f"  [SKIP] {repo_full_name} already cloned")
-        return target
-
-    for attempt in range(1, CLONE_RETRY + 1):
-        print(f"  [CLONE] {repo_full_name} (attempt {attempt}/{CLONE_RETRY})")
-
-        try:
-            result = subprocess.run(
-                ["git", "clone", "--depth", "1",
-                 f"git@github.com:{repo_full_name}.git", target],
-                stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE,
-                text=True,
-                timeout=CLONE_TIMEOUT
-            )
-        except subprocess.TimeoutExpired:
-            msg = f"timeout after {CLONE_TIMEOUT}s"
-            print(f"    ↳ {msg}")
-            if attempt == CLONE_RETRY:
-                # 添加到克隆失败列表
-                clone_failures.append({
-                    "repo": repo_full_name,
-                    "error": msg,
-                    "timestamp": datetime.datetime.now().isoformat()
-                })
-                # 立即保存
-                save_clone_failures(clone_failures)
-            continue
-
-        if result.returncode == 0:
-            return target
-
-        error_msg = result.stderr.strip() or "unknown error"
-        print(f"    ↳ git error: {error_msg}")
-
-        if attempt == CLONE_RETRY:
-            # 添加到克隆失败列表
-            clone_failures.append({
-                "repo": repo_full_name,
-                "error": error_msg,
-                "timestamp": datetime.datetime.now().isoformat()
-            })
-            # 立即保存
-            save_clone_failures(clone_failures)
-
-        time.sleep(2)
-
-    return None
-
-# 分析仓库 
-def analyze_repository(repo):
-    name = repo["full_name"]
-    stars = repo.get("stargazers_count", 0)
-
-    # 使用 GitHub 搜索 API 检查是否包含 Spring Framework 导入语句
-    uses_spring = check_has_spring_imports(name)
-    if not uses_spring:
-        print(f"  [SKIP] {name} no Spring Framework imports detected")
-        return None
-    
-    # 使用 GitHub 搜索 API 检查是否包含 @Test 注解
-    uses_test = check_has_test_annotations(name)
-
-    if not uses_test:
-        print(f"  [SKIP] {name} no junit/testng detected")
-        return None
-
-    print(f"  [OK] {name} passed all filters")
-    return {
-        "repo": name,
-        "stars": stars
-    }
-
-def main():
-    results = []
-    
-    # 加载已跳过的仓库列表
-    skipped_repos = set()
-    if os.path.exists(SKIPPED_REPOS_FILE):
-        try:
-            with open(SKIPPED_REPOS_FILE, "r", encoding="utf-8") as f:
-                skipped_repos = set(json.load(f))
-        except Exception as e:
-            print(f"Failed to load skipped repos: {e}")
-    
-    # 加载已收集的仓库列表
-    collected_repos = set()
-    if os.path.exists(OUTPUT_JSON):
-        try:
-            with open(OUTPUT_JSON, "r", encoding="utf-8") as f:
-                collected = json.load(f)
-                collected_repos = set(repo["repo"] for repo in collected)
-        except Exception as e:
-            print(f"Failed to load collected repos: {e}")
-    
-    # 加载克隆失败的仓库列表
-    clone_failures = load_clone_failures()
-    # 提取克隆失败的仓库名集合，用于跳过已失败的仓库
-    failed_repos = set(failure["repo"] for failure in clone_failures)
-    
-    # 合并为所有已处理的仓库（包括克隆失败的）
-    processed_repos = skipped_repos.union(collected_repos).union(failed_repos)
-    print(f"Loaded {len(processed_repos)} processed repositories")
-    print(f"Loaded {len(clone_failures)} clone failures")
-
-    start_date = datetime.date(2025, 2, 8)
-    end_date = datetime.date(2025, 6, 30)
-
-    delta = datetime.timedelta(days=1)
-    current_date = start_date
-
-    while current_date <= end_date:
-        next_date = current_date + delta
-        print(f"\n Searching date: {current_date}")
-
-        repos = search_repositories_by_stars(
-            current_date.isoformat(),
-            (next_date - datetime.timedelta(seconds=1)).isoformat()
-        )
-        print(f"  found {len(repos)} repos")
-
-        for repo in repos:
-            repo_full_name = repo["full_name"]
-            
-            # 检查是否已处理此仓库（无论是收集还是跳过）
-            if repo_full_name in processed_repos:
-                print(f"  [SKIP] {repo_full_name} already processed in previous run")
-                continue
-            
-            info = analyze_repository(repo)
-            if info:
-                results.append(info)
-                # 每次成功添加结果后立即保存
-                with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
-                    json.dump(results, f, indent=2)
-                # 更新已处理和已收集的仓库列表
-                processed_repos.add(repo_full_name)
-                collected_repos.add(repo_full_name)
-                # 立即保存已收集的仓库列表，确保数据不丢失
-                save_collected_repos(collected_repos)
-            else:
-                # 更新已处理和已跳过的仓库列表
-                processed_repos.add(repo_full_name)
-                skipped_repos.add(repo_full_name)
-                # 立即保存跳过的仓库列表，确保数据不丢失
-                save_skipped_repos(skipped_repos)
-
-        current_date = next_date
-
-    print(f"\n Total collected: {len(results)} repositories")
-    print(f" Saved to {OUTPUT_JSON}")
-    print(f" Clone failed repos saved to {CLONE_FAIL_FILE}")
-    print(f" Skipped {len(skipped_repos)} repositories")
-    print(f" Skipped repos saved to {SKIPPED_REPOS_FILE}")
-
-if __name__ == "__main__":
-    main()

sql/api_call.sql

@@ -1,14 +0,0 @@
-INSERT INTO api_call (caller_method_id, api_id, api_name)
-SELECT 
-    m.id AS caller_method_id,
-    a.id AS api_id,
-    a.api_name
-FROM 
-    methods m
-JOIN 
-    api a
-ON 
-    m.user_name = a.user_name
-    AND m.project_name = a.project_name
-    AND m.file_path = a.file_path
-    AND a.start_line BETWEEN m.start_line AND m.end_line;

sql/method_api_depth.sql

@@ -1,66 +0,0 @@
-USE qare67;
-DELIMITER $$
-
-DROP PROCEDURE IF EXISTS CalculateMethodApiDepth$$
-
-CREATE PROCEDURE CalculateMethodApiDepth()
-BEGIN
-    DECLARE current_depth INT DEFAULT 2;
-    DECLARE rows_affected INT DEFAULT 0;
-    DECLARE max_depth INT DEFAULT 20;  -- 安全阀
-
-    /* 1. 清空结果表 */
-    TRUNCATE TABLE method_api_depth;
-
-    /* 2. 初始层：直接调用 API 的方法
-          C → api  ==> depth = 2 */
-    INSERT INTO method_api_depth (method_id, api_id, depth, PATH)
-    SELECT
-        caller_method_id,
-        api_id,
-        2 AS depth,
-        CAST(caller_method_id AS CHAR) AS PATH
-    FROM api_call;
-
-    SELECT ROW_COUNT() INTO rows_affected;
-
-    /* 3. 逐层向上 BFS */
-    WHILE rows_affected > 0 AND current_depth < max_depth DO
-
-        SET current_depth = current_depth + 1;
-
-        INSERT IGNORE INTO method_api_depth (method_id, api_id, depth, PATH)
-        SELECT
-            mc.caller_method_id,
-            prev.api_id,
-            current_depth,
-            CONCAT(mc.caller_method_id, '->', prev.path)
-        FROM method_call mc
-        JOIN method_api_depth PREV
-            ON mc.callee_method_id = prev.method_id
-        WHERE prev.depth = current_depth - 1
-          -- 防环：A->A / A->B->A
-          AND FIND_IN_SET(
-                mc.caller_method_id,
-                REPLACE(prev.path, '->', ',')
-              ) = 0;
-
-        SELECT ROW_COUNT() INTO rows_affected;
-
-    END WHILE;
-
-    SELECT CONCAT('Calculation completed. Max depth = ', current_depth - 1) AS STATUS;
-
-END$$
-
-DELIMITER ;
-
-
-
-
-
-
-
-
-
-CALL CalculateMethodApiDepth();

src/main/java/JavaParser.java

@@ -1,383 +0,0 @@
-import org.eclipse.jdt.core.dom.*;
-import org.json.JSONArray;
-import org.json.JSONObject;
-
-import java.io.*;
-import java.nio.file.*;
-import java.sql.*;
-import java.util.*;
-import java.util.regex.Pattern;
-
-public class JavaParser {
-
-    private static String dbUrl;
-    private static String dbUser;
-    private static String dbPassword;
-
-    private static final String PROCESSED_FILES_FILE = "processed_files.json";
-    private static Set<String> processedFiles = new HashSet<>();
-
-    public static void main(String[] args) {
-        if (args.length != 2) {
-            System.out.println("Usage: java JavaParser <directory> <config-file>");
-            return;
-        }
-
-        String directoryPath = args[0];
-        String configFilePath = args[1];
-
-        File directory = new File(directoryPath);
-        if (!directory.isDirectory()) {
-            System.out.println("Error: " + directoryPath + " is not a valid directory.");
-            return;
-        }
-
-        // 加载配置文件
-        if (!loadConfig(configFilePath)) {
-            System.out.println("Error: Failed to load configuration file.");
-            return;
-        }
-
-        // 加载已处理的文件记录
-        loadProcessedFiles();
-
-        try (Connection connection = DriverManager.getConnection(dbUrl, dbUser, dbPassword)) {
-            connection.setAutoCommit(false); // 开启事务
-
-            try {
-                parseDirectory(directory, connection, directoryPath);
-                connection.commit(); // 提交事务
-                System.out.println("Data successfully stored in the database.");
-            } catch (Exception e) {
-                connection.rollback(); // 回滚事务
-                System.err.println("Transaction rolled back due to error: " + e.getMessage());
-                e.printStackTrace();
-            }
-        } catch (Exception e) {
-            e.printStackTrace();
-        }
-    }
-
-    // 加载配置文件
-    private static boolean loadConfig(String configFilePath) {
-        try (InputStream input = new FileInputStream(configFilePath)) {
-            Properties prop = new Properties();
-            prop.load(input);
-
-            dbUrl = prop.getProperty("db.url");
-            dbUser = prop.getProperty("db.user");
-            dbPassword = prop.getProperty("db.password");
-
-            return dbUrl != null && dbUser != null && dbPassword != null;
-        } catch (IOException e) {
-            System.err.println("Error reading configuration file: " + e.getMessage());
-            return false;
-        }
-    }
-
-    // 加载已处理的文件记录
-    private static void loadProcessedFiles() {
-        File file = new File(PROCESSED_FILES_FILE);
-        if (!file.exists()) {
-            return; // 文件不存在，直接返回
-        }
-
-        try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
-            String content = reader.lines().reduce("", (acc, line) -> acc + line).trim();
-
-            // 如果文件内容为空，初始化为空的 JSONArray
-            if (content.isEmpty()) {
-                System.out.println("Processed files list is empty. Initializing as empty.");
-                return;
-            }
-
-            // 尝试解析 JSON 数组
-            try {
-                JSONArray jsonArray = new JSONArray(content);
-                for (Object obj : jsonArray) {
-                    processedFiles.add(obj.toString());
-                }
-            } catch (org.json.JSONException e) {
-                System.err.println("Invalid JSON content in processed_files.json. Initializing as empty.");
-            }
-        } catch (IOException e) {
-            System.err.println("Error loading processed files: " + e.getMessage());
-        }
-    }
-
-    // 保存已处理的文件记录
-    private static void saveProcessedFiles() {
-        try (BufferedWriter writer = new BufferedWriter(new FileWriter(PROCESSED_FILES_FILE))) {
-            JSONArray jsonArray = new JSONArray(processedFiles);
-            writer.write(jsonArray.toString(2));
-        } catch (IOException e) {
-            System.err.println("Error saving processed files: " + e.getMessage());
-        }
-    }
-
-    // 批量插入方法
-    private static void insertMethodsBatch(Connection connection, List<Object[]> methods) throws SQLException {
-        String sql = "INSERT INTO methods (method_name, para_type, description, method_body, parameters, return_info, user_name, project_name, file_path, start_line, end_line, line_count) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
-        try (PreparedStatement stmt = connection.prepareStatement(sql)) {
-            for (Object[] method : methods) {
-                stmt.setString(1, (String) method[0]); // methodName
-                stmt.setString(2, (String) method[1]); // paraType
-                stmt.setString(3, (String) method[2]); // description
-                stmt.setString(4, (String) method[3]); // methodBody
-                stmt.setString(5, (String) method[4]); // parameters
-                stmt.setString(6, (String) method[5]); // returnInfo
-                stmt.setString(7, (String) method[6]); // userName
-                stmt.setString(8, (String) method[7]); // projectName
-                stmt.setString(9, (String) method[8]); // filePath
-                stmt.setInt(10, (int) method[9]); // startLine
-                stmt.setInt(11, (int) method[10]); // endLine
-                stmt.setInt(12, (int) method[11]); // lineCount
-                stmt.addBatch();
-            }
-            stmt.executeBatch();
-        } catch (SQLException e) {
-            System.err.println("Batch insert failed: " + e.getMessage());
-            throw e; // 抛出异常，交由调用方处理
-        }
-    }
-
-    // 解析目录
-    private static void parseDirectory(File directory, Connection connection, String repos1Path) throws Exception {
-        List<Object[]> methodsBatch = new ArrayList<>(); // 用于存储批量方法数据
-
-        for (File file : directory.listFiles()) {
-            if (file.isDirectory()) {
-                parseDirectory(file, connection, repos1Path);
-            } else if (file.getName().endsWith(".java")) {
-                // 检查文件是否已处理
-                String fileKey = file.getAbsolutePath();
-                if (processedFiles.contains(fileKey)) {
-                    System.out.println("File already processed: " + fileKey);
-                    continue;
-                }
-
-                parseJavaFile(file, methodsBatch, repos1Path);
-
-                // 记录已处理的文件
-                processedFiles.add(fileKey);
-                saveProcessedFiles();
-            }
-        }
-
-        // 批量插入方法数据
-        if (!methodsBatch.isEmpty()) {
-            insertMethodsBatch(connection, methodsBatch);
-        }
-    }
-
-    // 解析 Java 文件
-    private static void parseJavaFile(File file, List<Object[]> methodsBatch, String repos1Path) throws Exception {
-        String content = Files.readString(file.toPath());
-        ASTParser parser = ASTParser.newParser(AST.JLS17);
-        parser.setSource(content.toCharArray());
-        parser.setKind(ASTParser.K_COMPILATION_UNIT);
-
-        CompilationUnit cu = (CompilationUnit) parser.createAST(null);
-        cu.accept(new ASTVisitor() {
-            @Override
-            public boolean visit(MethodDeclaration node) {
-                try {
-                    // 提取方法名
-                    String methodName = node.getName().toString();
-
-                    // 提取参数类型签名
-                    StringBuilder paraTypeBuilder = new StringBuilder();
-                    JSONArray parametersArray = new JSONArray();
-                    for (Object param : node.parameters()) {
-                        SingleVariableDeclaration variable = (SingleVariableDeclaration) param;
-                        JSONObject paramObject = new JSONObject();
-                        paramObject.put("name", variable.getName().toString());
-                        paramObject.put("type", variable.getType().toString());
-                        parametersArray.put(paramObject);
-
-                        if (paraTypeBuilder.length() > 0) {
-                            paraTypeBuilder.append(",");
-                        }
-                        paraTypeBuilder.append(variable.getType().toString());
-                    }
-                    String paraType = paraTypeBuilder.toString();
-                    String parameters = parametersArray.toString();
-
-                    // 提取返回值类型
-                    String returnType = node.getReturnType2() != null ? node.getReturnType2().toString() : "void";
-
-                    // 提取方法签名
-                    StringBuilder methodSignatureBuilder = new StringBuilder();
-                    for (Object modifier : node.modifiers()) {
-                        methodSignatureBuilder.append(modifier.toString()).append(" ");
-                    }
-                    methodSignatureBuilder.append(returnType).append(" ");
-                    methodSignatureBuilder.append(methodName).append("(");
-                    methodSignatureBuilder.append(paraType);
-                    methodSignatureBuilder.append(")");
-
-                    IMethodBinding methodBinding = node.resolveBinding();
-                    if (methodBinding != null) {
-                        ITypeBinding[] exceptionTypes = methodBinding.getExceptionTypes();
-                        if (exceptionTypes.length > 0) {
-                            methodSignatureBuilder.append(" throws ");
-                            for (int i = 0; i < exceptionTypes.length; i++) {
-                                methodSignatureBuilder.append(exceptionTypes[i].getQualifiedName());
-                                if (i < exceptionTypes.length - 1) {
-                                    methodSignatureBuilder.append(", ");
-                                }
-                            }
-                        }
-                    }
-                    String methodSignature = methodSignatureBuilder.toString();
-
-                    // 提取方法描述（优先获取 Javadoc 注释）
-                    String description = extractCommentsForNode(node, content);
-
-                    // 检查方法体是否存在
-                    Block methodBodyBlock = node.getBody();
-                    if (methodBodyBlock == null) {
-                        System.out.println("Skipping method without a body: " + methodName);
-                        return super.visit(node);
-                    }
-
-                    // 判断是否为空方法
-                    if (methodBodyBlock.statements().isEmpty()) {
-                        System.out.println("Skipping empty method: " + methodName);
-                        return super.visit(node);
-                    }
-
-                    // 提取方法体的原始代码
-                    int bodyStart = methodBodyBlock.getStartPosition();
-                    int bodyEnd = bodyStart + methodBodyBlock.getLength();
-                    String methodBody = content.substring(bodyStart, bodyEnd).trim();
-
-                    // 提取 return 语句内容
-                    JSONArray returnInfoArray = new JSONArray();
-                    methodBodyBlock.accept(new ASTVisitor() {
-                        @Override
-                        public boolean visit(ReturnStatement returnNode) {
-                            JSONObject returnObject = new JSONObject();
-                            returnObject.put("name",
-                                    returnNode.getExpression() != null ? returnNode.getExpression().toString()
-                                            : "null");
-                            returnObject.put("type", returnType); // 返回值类型
-                            returnInfoArray.put(returnObject);
-                            return super.visit(returnNode);
-                        }
-                    });
-                    String returnInfo = returnInfoArray.toString();
-
-                    // 提取方法的起始和结束行号
-                    int startLine = cu.getLineNumber(node.getStartPosition());
-                    int endLine = cu.getLineNumber(node.getStartPosition() + node.getLength());
-
-                    // 计算方法的行数
-                    int lineCount = endLine - startLine + 1;
-
-                    // 提取用户名和项目名称
-                    String repoPath = file.getAbsolutePath(); // 获取文件的绝对路径
-                    int repos1Index = repoPath.indexOf(repos1Path);
-
-                    if (repos1Index != -1) {
-                        String relativePath = repoPath.substring(repos1Index + repos1Path.length() + 1); // 获取 repos1
-                        String[] repoParts = relativePath.split(Pattern.quote(File.separator), 2); // 分割路径，提取用户名和项目名
-                        String userName = repoParts[0].split("_")[0]; // 提取用户名
-                        String projectName = repoParts[0].split("_", 2)[1]; // 提取项目名
-
-                        // 添加到批量方法数据
-                        methodsBatch.add(new Object[] {
-                                methodName, paraType, description, methodBody, parameters, returnInfo,
-                                userName, projectName, file.getAbsolutePath(), startLine, endLine, lineCount
-                        });
-                    }
-                } catch (Exception e) {
-                    e.printStackTrace();
-                }
-                return super.visit(node);
-            }
-        });
-    }
-
-    // 提取注释的方法
-    // 提取方法对应的注释
-    private static String extractCommentsForNode(ASTNode node, String source) {
-        if (!(node instanceof MethodDeclaration)) {
-            return "";
-        }
-
-        MethodDeclaration method = (MethodDeclaration) node;
-        CompilationUnit cu = (CompilationUnit) node.getRoot();
-
-        // Javadoc 优先
-        Javadoc javadoc = method.getJavadoc();
-        if (javadoc != null) {
-            return javadoc.toString().trim();
-        }
-
-        // 找到方法开始行
-        int methodStartLine = cu.getLineNumber(method.getStartPosition());
-
-        // 获取所有注释
-        @SuppressWarnings("unchecked")
-        List<Comment> comments = cu.getCommentList();
-
-        Comment nearestComment = null;
-        int nearestEndLine = -1;
-
-        // 只找“结束行 < 方法开始行”的最近一个注释
-        for (Comment comment : comments) {
-            int commentEndLine = cu.getLineNumber(
-                    comment.getStartPosition() + comment.getLength());
-
-            if (commentEndLine < methodStartLine && commentEndLine > nearestEndLine) {
-                nearestComment = comment;
-                nearestEndLine = commentEndLine;
-            }
-        }
-
-        if (nearestComment == null) {
-            return "";
-        }
-
-        // 提取注释内容
-        return extractCommentContent(nearestComment, comments, source, cu, nearestEndLine);
-    }
-
-    // 提取注释内容
-    private static String extractCommentContent(Comment comment, List<Comment> comments, String source,
-            CompilationUnit cu, int nearestEndLine) {
-        // 如果是 BlockComment（/* ... */），直接返回
-        if (comment instanceof BlockComment) {
-            return source.substring(
-                    comment.getStartPosition(),
-                    comment.getStartPosition() + comment.getLength()).trim();
-        }
-
-        // 如果是 LineComment（//），向上合并连续的行注释
-        if (comment instanceof LineComment) {
-            StringBuilder sb = new StringBuilder();
-            int currentLine = nearestEndLine;
-
-            // 从后向前扫描，合并连续的 LineComment
-            for (int i = comments.size() - 1; i >= 0; i--) {
-                Comment c = comments.get(i);
-                if (c instanceof LineComment) {
-                    int endLine = cu.getLineNumber(
-                            c.getStartPosition() + c.getLength());
-
-                    if (endLine == currentLine) {
-                        sb.insert(0,
-                                source.substring(
-                                        c.getStartPosition(),
-                                        c.getStartPosition() + c.getLength()).trim() + "\n");
-                        currentLine--;
-                    }
-                }
-            }
-            return sb.toString().trim();
-        }
-
-        return "";
-    }
-}

src/main/java/UnifiedCallParser.java

@@ -1,334 +0,0 @@
-import org.eclipse.jdt.core.dom.*;
-import java.io.FileInputStream;
-import java.io.InputStream;
-import java.sql.Connection;
-import java.sql.DriverManager;
-import java.sql.PreparedStatement;
-import java.sql.ResultSet;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-public class UnifiedCallParser {
-
-    private static final int MIN_COMMENT_LENGTH = 20;
-
-    public static void parseCalls(Connection connection, int callerMethodId, Block methodBodyBlock, String userName,
-            String projectName) throws Exception {
-
-        List<Object[]> methodCallsBatch = new ArrayList<>();
-        final AtomicBoolean allCalleesFound = new AtomicBoolean(true);
-
-        methodBodyBlock.accept(new ASTVisitor() {
-            @Override
-            public boolean visit(MethodInvocation invocation) {
-                if (!allCalleesFound.get()) {
-                    return false;
-                }
-                try {
-                    String calledMethodName = invocation.getName().toString();
-                    String extractedParaType = extractParameterTypes(invocation);
-                    Integer calleeMethodId = resolveCalleeMethodId(connection, calledMethodName, extractedParaType,
-                            userName, projectName);
-
-                    if (calleeMethodId != null) {
-                        methodCallsBatch.add(new Object[] {
-                                callerMethodId,
-                                calleeMethodId,
-                                calledMethodName
-                        });
-                    } else {
-                        System.err.println("ERROR: A method (ID: " + callerMethodId
-                                + ") previously considered valid has an unresolvable call to '" + calledMethodName
-                                + "'.");
-                        allCalleesFound.set(false);
-                    }
-                } catch (Exception e) {
-                    e.printStackTrace();
-                    allCalleesFound.set(false);
-                }
-                return allCalleesFound.get();
-            }
-        });
-
-        if (allCalleesFound.get() && !methodCallsBatch.isEmpty()) {
-            insertMethodCallsBatch(connection, methodCallsBatch);
-            System.out.println("Inserted " + methodCallsBatch.size() + " calls for valid method ID: " + callerMethodId);
-        }
-    }
-
-    private static String extractParameterTypes(MethodInvocation invocation) {
-        List<?> arguments = invocation.arguments();
-        StringBuilder paraTypeBuilder = new StringBuilder();
-        for (Object arg : arguments) {
-            if (arg instanceof Expression) {
-                String type = resolveExpressionType((Expression) arg);
-                paraTypeBuilder.append(type).append(",");
-            }
-        }
-        if (paraTypeBuilder.length() > 0) {
-            paraTypeBuilder.setLength(paraTypeBuilder.length() - 1);
-        }
-        return paraTypeBuilder.toString();
-    }
-
-    private static String resolveExpressionType(Expression expression) {
-        if (expression instanceof StringLiteral) {
-            return "String";
-        } else if (expression instanceof NumberLiteral) {
-            return "int";
-        } else if (expression instanceof BooleanLiteral) {
-            return "boolean";
-        } else if (expression instanceof ClassInstanceCreation) {
-            return ((ClassInstanceCreation) expression).getType().toString();
-        } else if (expression instanceof MethodInvocation) {
-            return "Object";
-        } else if (expression instanceof SimpleName) {
-            return "Variable";
-        }
-        return "Object";
-    }
-
-    private static Integer resolveCalleeMethodId(Connection connection, String methodName, String extractedParaType,
-            String userName, String projectName) throws Exception {
-        String sql = "SELECT id, para_type FROM methods WHERE method_name = ? AND user_name = ? AND project_name = ?";
-        try (PreparedStatement stmt = connection.prepareStatement(sql)) {
-            stmt.setString(1, methodName);
-            stmt.setString(2, userName);
-            stmt.setString(3, projectName);
-
-            ResultSet rs = stmt.executeQuery();
-            while (rs.next()) {
-                String paraType = rs.getString("para_type");
-                if (compareParameterTypes(paraType, extractedParaType)) {
-                    return rs.getInt("id");
-                }
-            }
-        }
-        return null;
-    }
-
-    private static boolean compareParameterTypes(String paraType, String extractedParaType) {
-        if (paraType == null || extractedParaType == null) {
-            return paraType == null && (extractedParaType == null || extractedParaType.isEmpty());
-        }
-        String[] paraTypeArray = paraType.split(",");
-        String[] extractedParaTypeArray = extractedParaType.split(",");
-        if (paraType.isEmpty()) {
-            return extractedParaType.isEmpty();
-        }
-        if (paraTypeArray.length != extractedParaTypeArray.length) {
-            return false;
-        }
-        for (int i = 0; i < paraTypeArray.length; i++) {
-            if (!paraTypeArray[i].trim().equals(extractedParaTypeArray[i].trim())) {
-                return false;
-            }
-        }
-        return true;
-    }
-
-    private static void insertMethodCallsBatch(Connection connection, List<Object[]> methodCalls) throws Exception {
-        String sql = "INSERT INTO method_call (caller_method_id, callee_method_id, callee_method_name) VALUES (?, ?, ?)";
-        try (PreparedStatement stmt = connection.prepareStatement(sql)) {
-            for (Object[] call : methodCalls) {
-                stmt.setInt(1, (int) call[0]);
-                stmt.setObject(2, call[1]);
-                stmt.setString(3, (String) call[2]);
-                stmt.addBatch();
-            }
-            stmt.executeBatch();
-        }
-    }
-
-    private static class MethodInfo {
-        final int id;
-        final String body;
-        final String userName;
-        final String projectName;
-        final String description;
-
-        MethodInfo(int id, String body, String userName, String projectName, String description) {
-            this.id = id;
-            this.body = body;
-            this.userName = userName;
-            this.projectName = projectName;
-            this.description = description;
-        }
-    }
-
-    private static boolean isPrimarilyEnglish(String text) {
-        if (text == null || text.trim().isEmpty()) {
-            return true; // Let the length check handle this.
-        }
-        for (char c : text.toCharArray()) {
-            if (Character.UnicodeBlock.of(c) == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS ||
-                    Character.UnicodeBlock.of(c) == Character.UnicodeBlock.HANGUL_SYLLABLES ||
-                    Character.UnicodeBlock.of(c) == Character.UnicodeBlock.HIRAGANA ||
-                    Character.UnicodeBlock.of(c) == Character.UnicodeBlock.KATAKANA) {
-                return false; // Found a CJK character, assuming not English.
-            }
-        }
-        return true;
-    }
-
-    private static boolean isMethodInvalid(MethodInfo method, Connection connection, Set<Integer> knownInvalidIds) {
-        if (method.description == null || method.description.trim().length() < MIN_COMMENT_LENGTH) {
-            System.out.println("Invalidating method ID " + method.id + ": Comment is null or shorter than "
-                    + MIN_COMMENT_LENGTH + " chars.");
-            return true;
-        }
-        if (!isPrimarilyEnglish(method.description)) {
-            System.out.println("Invalidating method ID " + method.id + ": Comment appears to be non-English.");
-            return true;
-        }
-
-        if (method.body == null || method.body.trim().isEmpty()) {
-            return false;
-        }
-
-        AtomicBoolean invalidFlag = new AtomicBoolean(false);
-        String wrappedSource = "public class TempClass { public void tempMethod() " + method.body + " }";
-        ASTParser parser = ASTParser.newParser(AST.JLS17);
-        parser.setSource(wrappedSource.toCharArray());
-        parser.setKind(ASTParser.K_COMPILATION_UNIT);
-        CompilationUnit cu = (CompilationUnit) parser.createAST(null);
-
-        cu.accept(new ASTVisitor() {
-            @Override
-            public boolean visit(MethodInvocation invocation) {
-                if (invalidFlag.get()) {
-                    return false;
-                }
-                try {
-                    String calleeName = invocation.getName().toString();
-                    String calleeParams = extractParameterTypes(invocation);
-                    Integer calleeId = resolveCalleeMethodId(connection, calleeName, calleeParams, method.userName,
-                            method.projectName);
-
-                    if (calleeId == null) {
-                        invalidFlag.set(true);
-                        System.out.println("Invalidating method ID " + method.id + ": Calls non-existent method '"
-                                + calleeName + "'.");
-                    } else if (knownInvalidIds.contains(calleeId)) {
-                        invalidFlag.set(true);
-                        System.out.println("Invalidating method ID " + method.id + ": Calls an invalid method ID "
-                                + calleeId + ".");
-                    }
-                } catch (Exception e) {
-                    e.printStackTrace();
-                    invalidFlag.set(true);
-                }
-                return !invalidFlag.get();
-            }
-        });
-        return invalidFlag.get();
-    }
-
-    public static void main(String[] args) {
-        if (args.length < 1) {
-            System.err.println("Usage: java UnifiedCallParser <dbConfigFilePath>");
-            return;
-        }
-
-        String dbConfigFilePath = args[0];
-        System.out.println("Database config file path: " + dbConfigFilePath);
-
-        Properties dbProperties = new Properties();
-        try (InputStream input = new FileInputStream(dbConfigFilePath)) {
-            dbProperties.load(input);
-        } catch (Exception e) {
-            System.err.println("Failed to load database configuration: " + e.getMessage());
-            return;
-        }
-
-        String dbUrl = dbProperties.getProperty("db.url");
-        String dbUser = dbProperties.getProperty("db.user");
-        String dbPassword = dbProperties.getProperty("db.password");
-
-        try (Connection connection = DriverManager.getConnection(dbUrl, dbUser, dbPassword)) {
-            System.out.println("Database connected successfully!");
-
-            Map<Integer, MethodInfo> allMethods = new HashMap<>();
-            String selectAllSql = "SELECT id, method_body, user_name, project_name, description FROM methods";
-            try (PreparedStatement stmt = connection.prepareStatement(selectAllSql);
-                    ResultSet rs = stmt.executeQuery()) {
-                while (rs.next()) {
-                    allMethods.put(rs.getInt("id"), new MethodInfo(rs.getInt("id"), rs.getString("method_body"),
-                            rs.getString("user_name"), rs.getString("project_name"), rs.getString("description")));
-                }
-            }
-            System.out.println("Loaded " + allMethods.size() + " methods from database.");
-
-            Set<Integer> invalidMethodIds = new HashSet<>();
-            boolean newlyInvalidated;
-            int pass = 0;
-            do {
-                pass++;
-                newlyInvalidated = false;
-                for (MethodInfo method : allMethods.values()) {
-                    if (invalidMethodIds.contains(method.id)) {
-                        continue;
-                    }
-                    if (isMethodInvalid(method, connection, invalidMethodIds)) {
-                        invalidMethodIds.add(method.id);
-                        newlyInvalidated = true;
-                    }
-                }
-                System.out.println(
-                        "Invalidation Pass " + pass + ": Found " + invalidMethodIds.size() + " total invalid methods.");
-            } while (newlyInvalidated);
-
-            try (PreparedStatement stmt = connection.prepareStatement("DELETE FROM method_call")) {
-                int deletedRows = stmt.executeUpdate();
-                System.out.println("Cleared " + deletedRows + " existing rows from method_call table.");
-            }
-
-            for (MethodInfo method : allMethods.values()) {
-                if (invalidMethodIds.contains(method.id)) {
-                    continue;
-                }
-                if (method.body == null || method.body.trim().isEmpty()) {
-                    continue;
-                }
-
-                String wrappedSource = "public class TempClass { public void tempMethod() " + method.body + " }";
-                ASTParser parser = ASTParser.newParser(AST.JLS17);
-                parser.setSource(wrappedSource.toCharArray());
-                parser.setKind(ASTParser.K_COMPILATION_UNIT);
-                CompilationUnit cu = (CompilationUnit) parser.createAST(null);
-
-                cu.accept(new ASTVisitor() {
-                    @Override
-                    public boolean visit(MethodDeclaration methodDeclaration) {
-                        Block body = methodDeclaration.getBody();
-                        if (body != null) {
-                            try {
-                                parseCalls(connection, method.id, body, method.userName, method.projectName);
-                            } catch (Exception e) {
-                                e.printStackTrace();
-                            }
-                        }
-                        return false;
-                    }
-                });
-            }
-
-            System.out.println("解析完成！");
-        } catch (Exception e) {
-            e.printStackTrace();
-        } finally {
-            try {
-                com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkedShutdown();
-                System.out.println("MySQL cleanup thread shut down successfully.");
-            } catch (Exception e) {
-                System.err.println("Failed to shut down MySQL cleanup thread: " + e.getMessage());
-            }
-        }
-    }
-}

src/main/java/config.properties

@@ -1,4 +0,0 @@
-# MySQL 配置
-db.url=jdbc:mysql://localhost:3306/javaparser?useSSL=false&serverTimezone=UTC
-db.user=root
-db.password=******

third_library_set.py

@@ -1,148 +0,0 @@
-import os
-import re
-import json
-
-REPO_ROOT = r"E:\pipline\python\data\repos1"
-OUTPUT_JSON = "third_party_packages.json"
-PROCESSED_JSON = "processed_repos.json"  # 用于记录已处理项目
-
-# 提取 import / import static
-IMPORT_RE = re.compile(r'^\s*import\s+(static\s+)?([\w\.]+)\s*;')
-# 提取 package 声明
-PACKAGE_RE = re.compile(r'^\s*package\s+([\w\.]+)\s*;')
-
-# 排除 JDK 包
-JDK_PREFIXES = (
-    "java.",
-    "javax."
-)
-
-# 项目包前缀层级
-PROJECT_PREFIX_LEVELS = (2, 3)
-
-# ================= 工具函数 =================
-
-def is_jdk_package(pkg: str) -> bool:
-    return pkg.startswith(JDK_PREFIXES)
-
-def collect_project_prefixes(repo_path: str) -> set:
-    """
-    提取项目内 package 的多级前缀
-    """
-    prefixes = set()
-
-    for root, _, files in os.walk(repo_path):
-        for file in files:
-            if not file.endswith(".java"):
-                continue
-
-            file_path = os.path.join(root, file)
-            try:
-                with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
-                    for line in f:
-                        m = PACKAGE_RE.match(line)
-                        if m:
-                            full_pkg = m.group(1)
-                            parts = full_pkg.split(".")
-
-                            for level in PROJECT_PREFIX_LEVELS:
-                                if len(parts) >= level:
-                                    prefixes.add(".".join(parts[:level]))
-                            break
-            except Exception:
-                pass
-
-    return prefixes
-
-def is_project_package(pkg: str, project_prefixes: set) -> bool:
-    """
-    判断 import 是否属于项目内包
-    """
-    for prefix in project_prefixes:
-        if pkg == prefix or pkg.startswith(prefix + "."):
-            return True
-    return False
-
-def extract_third_party_packages(repo_path: str) -> list:
-    """
-    提取第三方 package（排除 JDK + 项目内包）
-    """
-    project_prefixes = collect_project_prefixes(repo_path)
-    third_party_packages = set()
-
-    for root, _, files in os.walk(repo_path):
-        for file in files:
-            if not file.endswith(".java"):
-                continue
-
-            file_path = os.path.join(root, file)
-            try:
-                with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
-                    for line in f:
-                        m = IMPORT_RE.match(line)
-                        if not m:
-                            continue
-
-                        full_import = m.group(2)  # 完整路径
-                        pkg = full_import  # 不再截断类名，直接使用完整 import
-
-                        if is_jdk_package(pkg):
-                            continue
-                        if is_project_package(pkg, project_prefixes):
-                            continue
-
-                        third_party_packages.add(pkg)
-            except Exception:
-                pass
-
-    return sorted(third_party_packages)
-
-# ================= 主流程 =================
-
-def main():
-    # 读取已处理项目
-    if os.path.exists(PROCESSED_JSON):
-        with open(PROCESSED_JSON, "r", encoding="utf-8") as f:
-            processed_repos = set(json.load(f))
-    else:
-        processed_repos = set()
-
-    # 读取已有结果（如果存在的话）
-    if os.path.exists(OUTPUT_JSON):
-        with open(OUTPUT_JSON, "r", encoding="utf-8") as f:
-            result = json.load(f)
-    else:
-        result = {}
-
-    new_processed = set()
-
-    for repo in os.listdir(REPO_ROOT):
-        repo_path = os.path.join(REPO_ROOT, repo)
-        if not os.path.isdir(repo_path):
-            continue
-
-        if repo in processed_repos:
-            print(f" 已处理，跳过 {repo}")
-            continue
-
-        print(f" 处理 {repo} ...")
-        pkgs = extract_third_party_packages(repo_path)
-        if pkgs:
-            result[repo] = pkgs
-
-        new_processed.add(repo)  # 标记为已处理
-
-    # 保存结果
-    with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
-        json.dump(result, f, indent=2, ensure_ascii=False)
-
-    # 更新已处理列表
-    processed_repos.update(new_processed)
-    with open(PROCESSED_JSON, "w", encoding="utf-8") as f:
-        json.dump(list(processed_repos), f, indent=2, ensure_ascii=False)
-
-    print(f" 完成，结果已写入 {OUTPUT_JSON}")
-    print(f" 已处理项目列表已更新到 {PROCESSED_JSON}")
-
-if __name__ == "__main__":
-    main()