feat: replace GitHub topic search with curated repos for skills aggregation
Replace blind GitHub Search API topic-based discovery with a curated list of 7 high-quality skill repositories. Fetch SKILL.md frontmatter for real names and descriptions, deduplicate by slug, and cap output at 300 skills. - Rewrite fetch-featured-skills.mjs: curated repos list, Repos API + Trees API + Contents API, .env auto-loading, SKILL.md name/description extraction - Add empty-result fallback in featured_skills.rs (skip to bundled JSON) - Add .env.example for GITHUB_TOKEN - Move requirements doc to docs/releases/v0.3.1/ API calls reduced from ~412 to ~314, output quality significantly improved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
--global
parent
3526d91da6
commit
441e3bd72c
|
|
@ -0,0 +1 @@
|
|||
GITHUB_TOKEN=ghp_your_github_token_here
|
||||
|
|
@ -0,0 +1,234 @@
|
|||
# 需求：Skills 聚合数据源升级 — 精选仓库列表方案
|
||||
|
||||
## 背景
|
||||
|
||||
Skills Hub 应用的"探索"功能依赖 `featured-skills.json` 提供精选技能列表。早期方案通过 GitHub Search API 搜索 `claude-code-skill` 等 topic 标签自动发现仓库，但属于盲目搜索，质量不可控，噪音多。
|
||||
|
||||
**新方案**：维护一份精选仓库列表（Curated Repos），直接从已知的高质量仓库中获取 skills。质量可控、API 调用极少、维护简单。
|
||||
|
||||
## 目标
|
||||
|
||||
重写 `scripts/fetch-featured-skills.mjs` 脚本，从精选仓库列表直接获取元数据并深入检测 skill 目录结构，为每个 skill 生成独立条目，最终输出 `featured-skills.json`（最多 300 条）。保持现有的 GitHub Actions 定时更新机制不变。
|
||||
|
||||
## 架构（保持不变）
|
||||
|
||||
```
|
||||
skills-desktop-app/
|
||||
├── featured-skills.json # 精选技能数据（应用内嵌 fallback + 在线更新源）
|
||||
├── scripts/
|
||||
│ └── fetch-featured-skills.mjs # 聚合脚本（需重写）
|
||||
└── .github/workflows/
|
||||
└── update-featured-skills.yml # 每日定时运行（保持不变）
|
||||
```
|
||||
|
||||
## 数据源 — 精选仓库列表
|
||||
|
||||
从 GitHub Topic 盲目搜索改为直接指定高质量仓库：
|
||||
|
||||
| 仓库 | Stars | 说明 |
|
||||
|------|-------|------|
|
||||
| `anthropics/skills` | ~96k | Anthropic 官方 Agent Skills |
|
||||
| `sickn33/antigravity-awesome-skills` | ~24.7k | 1000+ 社区 skills 集合 |
|
||||
| `K-Dense-AI/claude-scientific-skills` | ~15.3k | 170+ 科学研究 skills |
|
||||
| `travisvn/awesome-claude-skills` | ~9.1k | 精选 Claude skills 列表 |
|
||||
| `VoltAgent/awesome-agent-skills` | ~8.4k | 500+ Agent skills |
|
||||
| `anthropics/knowledge-work-plugins` | ~7.7k | 官方知识工作插件 |
|
||||
| `alirezarezvani/claude-skills` | ~5.2k | 192+ 社区 skills |
|
||||
|
||||
列表定义在脚本顶部 `CURATED_REPOS` 常量中，新增/移除仓库只需编辑此数组。
|
||||
|
||||
## 脚本重写逻辑
|
||||
|
||||
### 1. 数据采集（仓库元数据）
|
||||
|
||||
通过 GitHub Repos API 逐个获取精选仓库的元数据：
|
||||
|
||||
```
|
||||
GET /repos/{owner}/{repo}
|
||||
```
|
||||
|
||||
返回：`full_name`, `description`, `stargazers_count`, `topics[]`, `updated_at`, `html_url`, `default_branch`
|
||||
|
||||
认证：**必须**使用环境变量 `GITHUB_TOKEN`，脚本启动时校验。
|
||||
|
||||
失败处理：获取失败的仓库跳过（打印警告），不影响其余仓库。
|
||||
|
||||
### 2. Skill 检测（仓库内深入扫描）
|
||||
|
||||
使用 GitHub Git Trees API 获取仓库目录结构（单次请求，`recursive=1`）：
|
||||
|
||||
```
|
||||
GET /repos/{owner}/{repo}/git/trees/{default_branch}?recursive=1
|
||||
```
|
||||
|
||||
#### Skill 目录扫描规则（与应用端 `installer.rs` 保持一致）
|
||||
|
||||
扫描以下基础路径下的子目录：
|
||||
|
||||
```
|
||||
skills/
|
||||
skills/.curated/
|
||||
skills/.experimental/
|
||||
skills/.system/
|
||||
.claude/skills/
|
||||
```
|
||||
|
||||
以及**根目录的直接子目录**（排除 `skills/`、`.claude/`、`.git/` 等特殊目录）。
|
||||
|
||||
#### 新增：`.claude-plugin/plugin.json` 检测
|
||||
|
||||
为兼容 `anthropics/knowledge-work-plugins` 等使用插件格式的仓库，增加检测规则：
|
||||
|
||||
- 根级子目录包含 `.claude-plugin/plugin.json` 文件 → 视为有效 skill
|
||||
|
||||
#### Skill 判定条件
|
||||
|
||||
一个目录被视为有效 skill，需满足以下任一条件：
|
||||
- 目录内存在 `SKILL.md` 文件
|
||||
- 目录位于 `.claude/skills/` 路径下（即使没有 `SKILL.md`）
|
||||
- 目录内存在 `.claude-plugin/plugin.json` 文件（插件格式）
|
||||
|
||||
#### 单 skill 仓库
|
||||
|
||||
如果仓库根目录本身就是一个 skill（根目录有 `SKILL.md`），且没有检测到子目录 skill，则整个仓库视为单 skill，`source_url` 指向仓库根路径。
|
||||
|
||||
#### Skill 名称与描述
|
||||
|
||||
- **名称**：从 skill 目录名生成（kebab-case → Title Case）
|
||||
- **描述**：使用仓库的 `description` 字段（同仓库内所有 skill 共享）
|
||||
|
||||
> 注：不使用 Contents API 获取 SKILL.md 内容，避免大量 API 调用。目录名 + 仓库 description 已满足展示需求。
|
||||
|
||||
### 3. 自动分类
|
||||
|
||||
基于仓库 `topics[]` 和 `description` 关键词匹配（分类继承自仓库，同仓库内所有 skill 共享分类）：
|
||||
|
||||
| 关键词 | 分类 |
|
||||
|--------|------|
|
||||
| browser, automation, playwright, puppeteer | browser-automation |
|
||||
| security, audit, vulnerability, pentest | security |
|
||||
| devops, deploy, infra, docker, kubernetes | devops |
|
||||
| marketing, seo, ads, advertising | marketing |
|
||||
| database, sql, postgres, mongo | database |
|
||||
| git, github, pr, code-review | development |
|
||||
| ai, llm, agent, model | ai-assistant |
|
||||
| 以上都不匹配 | general |
|
||||
|
||||
### 4. 排序与截取
|
||||
|
||||
1. 按仓库 `stargazers_count` 降序，同星数按 skill 名称字母序
|
||||
2. **截取前 300 条**（`MAX_SKILLS = 300`），避免输出过大
|
||||
|
||||
### 5. 输出
|
||||
|
||||
生成 `featured-skills.json`，**向前兼容现有数据结构**：
|
||||
|
||||
```json
|
||||
{
|
||||
"updated_at": "2026-03-19T00:00:00Z",
|
||||
"total": 300,
|
||||
"categories": ["general", "browser-automation", "security", "devops", ...],
|
||||
"skills": [
|
||||
{
|
||||
"slug": "commit",
|
||||
"name": "Conventional Commit",
|
||||
"summary": "Generate conventional commit messages...",
|
||||
"downloads": 0,
|
||||
"stars": 96000,
|
||||
"category": "development",
|
||||
"tags": ["claude-code-skill", "git", "commit"],
|
||||
"source_url": "https://github.com/anthropics/skills/tree/main/skills/commit",
|
||||
"updated_at": "2026-03-17T15:10:09Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### 向前兼容策略
|
||||
|
||||
新格式**必须保留现有字段**，确保后端 `FeaturedSkillRaw` 和前端 `FeaturedSkillDto` 无需任何改动即可解析新数据：
|
||||
|
||||
| 字段 | 现有 | 新版 | 兼容处理 |
|
||||
|------|------|------|----------|
|
||||
| `slug` | ✅ | ✅ | 不变 |
|
||||
| `name` | ✅ | ✅ | 不变 |
|
||||
| `summary` | ✅ | ✅ | 不变 |
|
||||
| `downloads` | ✅ | ✅ | **保留，固定为 0**（无真实数据源） |
|
||||
| `stars` | ✅ | ✅ | 填入 GitHub 真实 star 数 |
|
||||
| `source_url` | ✅ | ✅ | 精确到 skill 目录的路径 |
|
||||
| `category` | ❌ | ✅ 新增 | 后端 `#[serde(default)]` 自动忽略未知字段 |
|
||||
| `tags` | ❌ | ✅ 新增 | 同上 |
|
||||
| `updated_at`（skill 级） | ❌ | ✅ 新增 | 同上 |
|
||||
|
||||
**结论**：脚本输出格式升级后，后端和前端代码**零改动**即可正常工作。
|
||||
|
||||
#### 字段说明
|
||||
|
||||
- `slug`：skill 目录名（单 skill 仓库则为仓库名）
|
||||
- `name`：基于 skill 目录名生成（kebab-case → Title Case）
|
||||
- `summary`：仓库 `description`（同仓库内所有 skill 共享）
|
||||
- `downloads`：固定为 `0`（向前兼容，无真实数据源）
|
||||
- `stars`：所属仓库的 GitHub star 数
|
||||
- `category`：基于仓库 topics/description 的自动分类结果
|
||||
- `tags`：仓库 `topics[]`（最多取 5 个）
|
||||
- `source_url`：**精确到 skill 目录的 GitHub URL**，格式为 `https://github.com/{owner}/{repo}/tree/{branch}/{skill-path}`；单 skill 仓库则为 `https://github.com/{owner}/{repo}`
|
||||
- `updated_at`：仓库最后更新时间
|
||||
|
||||
## API 用量估算
|
||||
|
||||
| 阶段 | API | 请求数 | 说明 |
|
||||
|------|-----|--------|------|
|
||||
| 获取仓库元数据 | Repos API | 7 | 每个精选仓库 1 次 |
|
||||
| 获取目录树 | Git Trees API | 7 | 每个仓库 1 次 |
|
||||
| **合计** | | **14** | |
|
||||
|
||||
对比旧方案（Topic 搜索）的 ~412 次请求，新方案仅需 14 次，**几乎不可能触发速率限制**。
|
||||
|
||||
GitHub API 限额（已认证）：5000 次/小时。即使未认证（60 次/小时）也完全够用，但仍建议使用 token 以确保稳定性。
|
||||
|
||||
## GitHub Actions 配置
|
||||
|
||||
保持现有 `.github/workflows/update-featured-skills.yml` 不变：
|
||||
|
||||
```yaml
|
||||
- name: Fetch featured skills
|
||||
run: node scripts/fetch-featured-skills.mjs
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
## Skills Hub 应用端适配
|
||||
|
||||
**后端和前端代码无需任何改动**（向前兼容）：
|
||||
|
||||
- `FeaturedSkillRaw` / `FeaturedSkillDto` 结构体不变
|
||||
- 新增的 `category`、`tags`、`updated_at` 等字段被 serde 自动忽略
|
||||
- 内嵌 fallback（`featured-skills.json`）随脚本运行自动更新
|
||||
|
||||
### 后续可选增强（独立需求）
|
||||
|
||||
1. 后端 DTO 新增 `category`、`tags` 字段以支持前端展示
|
||||
2. 探索页面增加按分类筛选
|
||||
3. 排序改为按星数排序
|
||||
|
||||
## 技术要点
|
||||
|
||||
- **精选仓库列表**：质量可控，新增仓库只需编辑 `CURATED_REPOS` 数组
|
||||
- **极低 API 用量**：14 次请求 vs 旧方案 412 次，无速率限制风险
|
||||
- **零外部依赖**：仅依赖 GitHub API（公开、稳定、有 SLA）
|
||||
- **Skill 粒度聚合**：每条记录精确到仓库内具体 skill 目录
|
||||
- **与应用检测逻辑一致**：skill 扫描规则与 `installer.rs` 保持同步
|
||||
- **新增插件格式支持**：兼容 `.claude-plugin/plugin.json` 结构
|
||||
- **数量上限**：最多 300 条，按星数排序取 top
|
||||
- **项目内维护**：脚本、数据、CI 全部在 skills-desktop-app 仓库内
|
||||
|
||||
## 变更清单
|
||||
|
||||
| 文件 | 操作 |
|
||||
|------|------|
|
||||
| `scripts/fetch-featured-skills.mjs` | 重写（精选仓库列表 + Repos API + Trees API） |
|
||||
| `featured-skills.json` | 内容更新（精选仓库来源，≤300 条） |
|
||||
| `docs/requirements/skills-aggregation-repo.md` | 更新需求文档 |
|
||||
| `.github/workflows/update-featured-skills.yml` | **无需改动** |
|
||||
| 后端代码 | **无需改动**（向前兼容） |
|
||||
| 前端代码 | **无需改动**（向前兼容） |
|
||||
5663
featured-skills.json
5663
featured-skills.json
File diff suppressed because it is too large
Load Diff
|
|
@ -1,94 +1,426 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Fetches popular skills from ClawHub API, resolves each skill's owner
|
||||
* via detail API, and produces featured-skills.json with GitHub source URLs.
|
||||
* Aggregates AI Agent Skills from a curated list of high-quality GitHub repositories.
|
||||
*
|
||||
* Fetches metadata and directory trees for each curated repo, detects individual skills
|
||||
* within each repo, and outputs featured-skills.json with one entry per skill.
|
||||
*
|
||||
* API budget: ~14 requests total (7 Repos API + 7 Trees API).
|
||||
*
|
||||
* Requires: GITHUB_TOKEN environment variable.
|
||||
*/
|
||||
|
||||
const CLAWHUB_LIST_API = 'https://clawhub.ai/api/v1/skills?sort=downloads&limit=100'
|
||||
const CLAWHUB_DETAIL_API = 'https://clawhub.ai/api/v1/skills'
|
||||
const GITHUB_REPO = 'openclaw/skills'
|
||||
import { existsSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { resolve } from 'node:path'
|
||||
|
||||
// Load .env file
|
||||
const envPath = resolve(import.meta.dirname, '..', '.env')
|
||||
if (existsSync(envPath)) {
|
||||
for (const line of readFileSync(envPath, 'utf-8').split('\n')) {
|
||||
const trimmed = line.trim()
|
||||
if (!trimmed || trimmed.startsWith('#')) continue
|
||||
const idx = trimmed.indexOf('=')
|
||||
if (idx === -1) continue
|
||||
const key = trimmed.slice(0, idx).trim()
|
||||
const value = trimmed.slice(idx + 1).trim()
|
||||
if (!process.env[key]) process.env[key] = value
|
||||
}
|
||||
}
|
||||
|
||||
const OUTPUT_FILE = 'featured-skills.json'
|
||||
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms))
|
||||
// Curated high-quality skill repositories (ordered by stars desc)
|
||||
const CURATED_REPOS = [
|
||||
'anthropics/skills',
|
||||
'sickn33/antigravity-awesome-skills',
|
||||
'K-Dense-AI/claude-scientific-skills',
|
||||
'travisvn/awesome-claude-skills',
|
||||
'VoltAgent/awesome-agent-skills',
|
||||
'anthropics/knowledge-work-plugins',
|
||||
'alirezarezvani/claude-skills',
|
||||
]
|
||||
|
||||
const MAX_SKILLS = 300
|
||||
const CONCURRENCY = 10
|
||||
const MAX_RATE_LIMIT_WAIT_SECS = 60
|
||||
|
||||
// Skill scan bases matching installer.rs SKILL_SCAN_BASES
|
||||
const SKILL_SCAN_BASES = [
|
||||
'skills',
|
||||
'skills/.curated',
|
||||
'skills/.experimental',
|
||||
'skills/.system',
|
||||
'.claude/skills',
|
||||
]
|
||||
|
||||
// Directories to skip when scanning root-level subdirs
|
||||
const ROOT_SKIP_DIRS = new Set([
|
||||
'skills', '.claude', '.git', '.github', '.vscode', 'node_modules',
|
||||
'.idea', '.DS_Store', 'dist', 'build', 'out', 'target',
|
||||
'docs', 'test', 'tests', '__tests__', 'examples', 'src', 'lib',
|
||||
])
|
||||
|
||||
// Category classification rules
|
||||
const CATEGORY_RULES = [
|
||||
{ keywords: ['browser', 'automation', 'playwright', 'puppeteer'], category: 'browser-automation' },
|
||||
{ keywords: ['security', 'audit', 'vulnerability', 'pentest'], category: 'security' },
|
||||
{ keywords: ['devops', 'deploy', 'infra', 'docker', 'kubernetes'], category: 'devops' },
|
||||
{ keywords: ['marketing', 'seo', 'ads', 'advertising'], category: 'marketing' },
|
||||
{ keywords: ['database', 'sql', 'postgres', 'mongo'], category: 'database' },
|
||||
{ keywords: ['git', 'github', 'pr', 'code-review'], category: 'development' },
|
||||
{ keywords: ['ai', 'llm', 'agent', 'model'], category: 'ai-assistant' },
|
||||
]
|
||||
|
||||
const GITHUB_TOKEN = process.env.GITHUB_TOKEN || ''
|
||||
|
||||
// ─── HTTP helpers ───
|
||||
|
||||
async function fetchJson(url, retries = 3) {
|
||||
const headers = {
|
||||
Accept: 'application/vnd.github+json',
|
||||
'User-Agent': 'skills-hub-aggregator',
|
||||
}
|
||||
if (GITHUB_TOKEN) {
|
||||
headers.Authorization = `Bearer ${GITHUB_TOKEN}`
|
||||
}
|
||||
|
||||
for (let attempt = 0; attempt <= retries; attempt++) {
|
||||
const res = await fetch(url, {
|
||||
headers: { 'User-Agent': 'skills-hub-ci' },
|
||||
})
|
||||
if (res.status === 429 && attempt < retries) {
|
||||
const delay = 2 ** (attempt + 1) * 1000 // 2s, 4s, 8s
|
||||
console.warn(`Rate limited (429), retrying in ${delay / 1000}s... (${attempt + 1}/${retries})`)
|
||||
await sleep(delay)
|
||||
const res = await fetch(url, { headers })
|
||||
|
||||
if (res.status === 403 || res.status === 429) {
|
||||
const resetHeader = res.headers.get('x-ratelimit-reset')
|
||||
let waitSecs = resetHeader
|
||||
? Math.max(Number(resetHeader) - Math.floor(Date.now() / 1000), 1)
|
||||
: Math.pow(2, attempt + 1)
|
||||
|
||||
if (waitSecs > MAX_RATE_LIMIT_WAIT_SECS) {
|
||||
console.warn(`Rate limited, reset in ${waitSecs}s (exceeds max ${MAX_RATE_LIMIT_WAIT_SECS}s) — skipping`)
|
||||
return null
|
||||
}
|
||||
console.warn(`Rate limited (${res.status}), waiting ${waitSecs}s (attempt ${attempt + 1}/${retries + 1})...`)
|
||||
await sleep(waitSecs * 1000)
|
||||
continue
|
||||
}
|
||||
if (!res.ok) throw new Error(`${url} returned ${res.status}`)
|
||||
|
||||
if (!res.ok) {
|
||||
if (attempt < retries) {
|
||||
await sleep(Math.pow(2, attempt) * 1000)
|
||||
continue
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
return res.json()
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
async function getOwnerHandle(slug) {
|
||||
try {
|
||||
const data = await fetchJson(`${CLAWHUB_DETAIL_API}/${encodeURIComponent(slug)}`)
|
||||
return data?.owner?.handle ?? ''
|
||||
} catch {
|
||||
return ''
|
||||
}
|
||||
function sleep(ms) {
|
||||
return new Promise((r) => setTimeout(r, ms))
|
||||
}
|
||||
|
||||
// Run async tasks with bounded concurrency
|
||||
async function pMap(items, fn, concurrency) {
|
||||
const results = new Array(items.length)
|
||||
let idx = 0
|
||||
|
||||
async function worker() {
|
||||
while (idx < items.length) {
|
||||
const i = idx++
|
||||
results[i] = await fn(items[i], i)
|
||||
}
|
||||
}
|
||||
|
||||
const workers = Array.from({ length: Math.min(concurrency, items.length) }, () => worker())
|
||||
await Promise.all(workers)
|
||||
return results
|
||||
}
|
||||
|
||||
// ─── Step 1: Fetch curated repo metadata ───
|
||||
|
||||
async function fetchRepoMetadata(fullName) {
|
||||
const url = `https://api.github.com/repos/${fullName}`
|
||||
const data = await fetchJson(url)
|
||||
if (!data || !data.full_name) {
|
||||
console.warn(` Skipping ${fullName}: unable to fetch metadata`)
|
||||
return null
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
async function fetchAllRepoMetadata() {
|
||||
console.log(`Fetching metadata for ${CURATED_REPOS.length} curated repos...`)
|
||||
const results = await pMap(
|
||||
CURATED_REPOS,
|
||||
async (fullName) => {
|
||||
console.log(` Fetching: ${fullName}`)
|
||||
return fetchRepoMetadata(fullName)
|
||||
},
|
||||
CONCURRENCY,
|
||||
)
|
||||
const repos = results.filter(Boolean)
|
||||
console.log(`Successfully fetched ${repos.length}/${CURATED_REPOS.length} repos`)
|
||||
return repos
|
||||
}
|
||||
|
||||
// ─── Step 2: Detect skills in each repo ───
|
||||
|
||||
function detectSkillsFromTree(treeItems) {
|
||||
const filePaths = new Set()
|
||||
const dirPaths = new Set()
|
||||
for (const item of treeItems) {
|
||||
if (item.type === 'blob') filePaths.add(item.path)
|
||||
else if (item.type === 'tree') dirPaths.add(item.path)
|
||||
}
|
||||
|
||||
const skills = [] // { dirPath }
|
||||
const foundDirs = new Set()
|
||||
|
||||
// Scan SKILL_SCAN_BASES
|
||||
for (const base of SKILL_SCAN_BASES) {
|
||||
for (const dir of dirPaths) {
|
||||
if (!dir.startsWith(base + '/')) continue
|
||||
const rest = dir.slice(base.length + 1)
|
||||
if (rest.includes('/')) continue // not a direct child
|
||||
|
||||
const hasSkillMd = filePaths.has(dir + '/SKILL.md')
|
||||
const isClaudeSkill = base === '.claude/skills'
|
||||
|
||||
if (hasSkillMd || isClaudeSkill) {
|
||||
if (!foundDirs.has(dir)) {
|
||||
foundDirs.add(dir)
|
||||
skills.push({ dirPath: dir })
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scan root-level subdirectories (must have SKILL.md or .claude-plugin/plugin.json)
|
||||
for (const dir of dirPaths) {
|
||||
if (dir.includes('/')) continue
|
||||
if (ROOT_SKIP_DIRS.has(dir) || dir.startsWith('.')) continue
|
||||
if (foundDirs.has(dir)) continue
|
||||
|
||||
if (filePaths.has(dir + '/SKILL.md') || filePaths.has(dir + '/.claude-plugin/plugin.json')) {
|
||||
foundDirs.add(dir)
|
||||
skills.push({ dirPath: dir })
|
||||
}
|
||||
}
|
||||
|
||||
// Single-skill repo: root has SKILL.md and no sub-skills found
|
||||
if (skills.length === 0 && filePaths.has('SKILL.md')) {
|
||||
skills.push({ dirPath: null })
|
||||
}
|
||||
|
||||
return skills
|
||||
}
|
||||
|
||||
async function getRepoTree(owner, repo, branch) {
|
||||
const url = `https://api.github.com/repos/${owner}/${repo}/git/trees/${encodeURIComponent(branch)}?recursive=1`
|
||||
const data = await fetchJson(url, 2)
|
||||
if (!data || !data.tree) return null
|
||||
if (data.truncated) {
|
||||
console.warn(` Warning: tree for ${owner}/${repo} was truncated, some skills may be missed`)
|
||||
}
|
||||
return data.tree
|
||||
}
|
||||
|
||||
// ─── Step 3: Classify ───
|
||||
|
||||
function classify(topics, description) {
|
||||
const text = [...(topics || []), description || ''].join(' ').toLowerCase()
|
||||
for (const rule of CATEGORY_RULES) {
|
||||
if (rule.keywords.some((kw) => text.includes(kw))) {
|
||||
return rule.category
|
||||
}
|
||||
}
|
||||
return 'general'
|
||||
}
|
||||
|
||||
// ─── SKILL.md helpers ───
|
||||
|
||||
function parseSkillMdFrontmatter(content) {
|
||||
const lines = content.split('\n')
|
||||
if (lines[0].trim() !== '---') return null
|
||||
let name = null
|
||||
let description = null
|
||||
for (let i = 1; i < lines.length; i++) {
|
||||
const l = lines[i].trim()
|
||||
if (l === '---') break
|
||||
if (l.startsWith('name:')) {
|
||||
name = l.slice(5).trim().replace(/^["']|["']$/g, '')
|
||||
} else if (l.startsWith('description:')) {
|
||||
description = l.slice(12).trim().replace(/^["']|["']$/g, '')
|
||||
}
|
||||
}
|
||||
return { name, description }
|
||||
}
|
||||
|
||||
async function fetchSkillMdContent(owner, repo, branch, dirPath) {
|
||||
const filePath = dirPath ? `${dirPath}/SKILL.md` : 'SKILL.md'
|
||||
const url = `https://api.github.com/repos/${owner}/${repo}/contents/${encodeURIComponent(filePath)}?ref=${encodeURIComponent(branch)}`
|
||||
const data = await fetchJson(url, 1)
|
||||
if (!data || !data.content) return null
|
||||
const content = Buffer.from(data.content, 'base64').toString('utf-8')
|
||||
return parseSkillMdFrontmatter(content)
|
||||
}
|
||||
|
||||
// ─── Name helpers ───
|
||||
|
||||
function kebabToTitle(name) {
|
||||
return name
|
||||
.split(/[-_]/)
|
||||
.map((w) => w.charAt(0).toUpperCase() + w.slice(1))
|
||||
.join(' ')
|
||||
}
|
||||
|
||||
function slugFromDirPath(dirPath, repoName) {
|
||||
if (!dirPath) return repoName
|
||||
const parts = dirPath.split('/')
|
||||
return parts[parts.length - 1]
|
||||
}
|
||||
|
||||
// ─── Main ───
|
||||
|
||||
async function main() {
|
||||
console.log('Fetching skills from ClawHub API...')
|
||||
let clawSkills
|
||||
try {
|
||||
const data = await fetchJson(CLAWHUB_LIST_API)
|
||||
clawSkills = Array.isArray(data) ? data : data.items ?? data.skills ?? data.data ?? []
|
||||
} catch (err) {
|
||||
console.error('Failed to fetch ClawHub API:', err.message)
|
||||
if (!GITHUB_TOKEN) {
|
||||
console.error('Error: GITHUB_TOKEN environment variable is required.')
|
||||
console.error('Set it via: export GITHUB_TOKEN=ghp_xxx')
|
||||
process.exit(1)
|
||||
}
|
||||
console.log(`Got ${clawSkills.length} skills from ClawHub`)
|
||||
|
||||
console.log('Resolving skill owners...')
|
||||
const skills = []
|
||||
// Process in batches of 10 for reasonable parallelism
|
||||
for (let i = 0; i < clawSkills.length; i += 10) {
|
||||
const batch = clawSkills.slice(i, i + 10)
|
||||
const results = await Promise.all(
|
||||
batch.map(async (s) => {
|
||||
const slug = s.slug ?? ''
|
||||
if (!slug) return null
|
||||
const owner = await getOwnerHandle(slug)
|
||||
const source_url = owner
|
||||
? `https://github.com/${GITHUB_REPO}/tree/main/skills/${owner}/${slug}`
|
||||
: ''
|
||||
const stats = s.stats ?? {}
|
||||
return {
|
||||
slug,
|
||||
name: s.displayName ?? s.name ?? slug,
|
||||
summary: s.summary ?? s.description ?? '',
|
||||
downloads: stats.downloads ?? s.downloads ?? 0,
|
||||
stars: stats.stars ?? s.stars ?? 0,
|
||||
source_url,
|
||||
}
|
||||
}),
|
||||
)
|
||||
skills.push(...results.filter(Boolean))
|
||||
process.stdout.write(` ${Math.min(i + 10, clawSkills.length)}/${clawSkills.length}\r`)
|
||||
// Step 1: Fetch curated repo metadata
|
||||
const repos = await fetchAllRepoMetadata()
|
||||
|
||||
// Step 2: Detect skills in each repo via Trees API
|
||||
console.log('Scanning repo trees for skills...')
|
||||
const skillEntries = [] // { repo, dirPath }
|
||||
let treeFailures = 0
|
||||
|
||||
await pMap(
|
||||
repos,
|
||||
async (repo) => {
|
||||
const [owner, repoName] = repo.full_name.split('/')
|
||||
const tree = await getRepoTree(owner, repoName, repo.default_branch)
|
||||
if (!tree) {
|
||||
treeFailures++
|
||||
return
|
||||
}
|
||||
|
||||
const detected = detectSkillsFromTree(tree)
|
||||
if (detected.length === 0) {
|
||||
// No detectable skill structure — treat whole repo as single skill
|
||||
skillEntries.push({ repo, dirPath: null })
|
||||
return
|
||||
}
|
||||
|
||||
for (const s of detected) {
|
||||
skillEntries.push({ repo, dirPath: s.dirPath })
|
||||
}
|
||||
},
|
||||
CONCURRENCY,
|
||||
)
|
||||
|
||||
console.log(`Detected ${skillEntries.length} skills across ${repos.length} repos (${treeFailures} tree fetch failures)`)
|
||||
|
||||
// Fallback: if no skills detected, keep existing local file
|
||||
if (skillEntries.length === 0) {
|
||||
if (existsSync(OUTPUT_FILE)) {
|
||||
console.warn('No skills fetched from GitHub — keeping existing local featured-skills.json')
|
||||
} else {
|
||||
console.error('No skills fetched and no local fallback exists.')
|
||||
process.exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
console.log('')
|
||||
|
||||
const matched = skills.filter((s) => s.source_url).length
|
||||
console.log(`Matched ${matched}/${skills.length} skills to GitHub paths`)
|
||||
// Step 3: Sort by stars desc, deduplicate, take top MAX_SKILLS
|
||||
skillEntries.sort((a, b) => b.repo.stargazers_count - a.repo.stargazers_count)
|
||||
|
||||
const seenSlugs = new Set()
|
||||
const dedupedEntries = skillEntries.filter((entry) => {
|
||||
const slug = slugFromDirPath(entry.dirPath, entry.repo.full_name.split('/')[1])
|
||||
if (seenSlugs.has(slug)) return false
|
||||
seenSlugs.add(slug)
|
||||
return true
|
||||
})
|
||||
console.log(`After dedup: ${dedupedEntries.length} unique skills (removed ${skillEntries.length - dedupedEntries.length} duplicates)`)
|
||||
|
||||
const topEntries = dedupedEntries.slice(0, MAX_SKILLS)
|
||||
|
||||
// Step 4: Fetch SKILL.md for top skills to get real names
|
||||
console.log(`Fetching SKILL.md for ${topEntries.length} skills...`)
|
||||
const skillMdMap = new Map() // index -> { name, description }
|
||||
await pMap(
|
||||
topEntries,
|
||||
async (entry, i) => {
|
||||
const [owner, repoName] = entry.repo.full_name.split('/')
|
||||
const md = await fetchSkillMdContent(owner, repoName, entry.repo.default_branch, entry.dirPath)
|
||||
if (md) skillMdMap.set(i, md)
|
||||
},
|
||||
CONCURRENCY,
|
||||
)
|
||||
console.log(`Fetched SKILL.md for ${skillMdMap.size}/${topEntries.length} skills`)
|
||||
|
||||
// Filter out entries without SKILL.md
|
||||
const validEntries = topEntries.filter((_, i) => skillMdMap.has(i))
|
||||
console.log(`${validEntries.length} skills have SKILL.md (filtered out ${topEntries.length - validEntries.length})`)
|
||||
|
||||
// Rebuild index mapping after filtering
|
||||
const validMdList = validEntries.map((entry, _i) => {
|
||||
const origIndex = topEntries.indexOf(entry)
|
||||
return { entry, md: skillMdMap.get(origIndex) }
|
||||
})
|
||||
|
||||
// Step 5: Build output
|
||||
const categorySet = new Set()
|
||||
const topSkills = validMdList.map(({ entry, md }) => {
|
||||
const { repo, dirPath } = entry
|
||||
const repoName = repo.full_name.split('/')[1]
|
||||
const slug = slugFromDirPath(dirPath, repoName)
|
||||
const name = md.name || slug
|
||||
const summary = (md && md.description) || repo.description || ''
|
||||
const category = classify(repo.topics, repo.description)
|
||||
categorySet.add(category)
|
||||
|
||||
let sourceUrl
|
||||
if (dirPath) {
|
||||
sourceUrl = `${repo.html_url}/tree/${repo.default_branch}/${dirPath}`
|
||||
} else {
|
||||
sourceUrl = repo.html_url
|
||||
}
|
||||
|
||||
return {
|
||||
slug,
|
||||
name,
|
||||
summary,
|
||||
downloads: 0,
|
||||
stars: repo.stargazers_count,
|
||||
category,
|
||||
tags: (repo.topics || []).slice(0, 5),
|
||||
source_url: sourceUrl,
|
||||
updated_at: repo.updated_at,
|
||||
}
|
||||
})
|
||||
|
||||
// Re-sort after name update (stars desc, then name asc)
|
||||
topSkills.sort((a, b) => b.stars - a.stars || a.name.localeCompare(b.name))
|
||||
|
||||
const categories = Array.from(categorySet).sort()
|
||||
|
||||
const output = {
|
||||
updated_at: new Date().toISOString(),
|
||||
skills,
|
||||
total: topSkills.length,
|
||||
categories,
|
||||
skills: topSkills,
|
||||
}
|
||||
|
||||
const { writeFileSync } = await import('node:fs')
|
||||
writeFileSync(OUTPUT_FILE, JSON.stringify(output, null, 2) + '\n')
|
||||
console.log(`Wrote ${skills.length} skills to ${OUTPUT_FILE}`)
|
||||
console.log(`Wrote ${topSkills.length} skills (of ${skillEntries.length} detected) to ${OUTPUT_FILE}`)
|
||||
}
|
||||
|
||||
main()
|
||||
main().catch((err) => {
|
||||
console.error('Fatal error:', err)
|
||||
process.exit(1)
|
||||
})
|
||||
|
|
|
|||
|
|
@ -48,14 +48,18 @@ pub fn fetch_featured_skills(store: &SkillStore) -> Result<Vec<FeaturedSkill>> {
|
|||
fn fetch_featured_skills_inner(url: &str, store: &SkillStore) -> Result<Vec<FeaturedSkill>> {
|
||||
if let Ok(json_str) = fetch_from_url(url) {
|
||||
if let Ok(skills) = parse_and_filter(&json_str) {
|
||||
let _ = store.set_setting(CACHE_KEY, &json_str);
|
||||
return Ok(skills);
|
||||
if !skills.is_empty() {
|
||||
let _ = store.set_setting(CACHE_KEY, &json_str);
|
||||
return Ok(skills);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Fallback to cache
|
||||
if let Ok(Some(cached)) = store.get_setting(CACHE_KEY) {
|
||||
if let Ok(skills) = parse_and_filter(&cached) {
|
||||
return Ok(skills);
|
||||
if !skills.is_empty() {
|
||||
return Ok(skills);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Fallback to bundled JSON
|
||||
|
|
|
|||
Loading…
Reference in New Issue