Knowledge2Model/ModelStatus

ModelStatus

Automated health checks for HuggingFace models across different tasks.

Overview

This folder contains scripts that test and validate models from the HuggingFace database. The database file (huggingface2.db) is first generated by the Hugging2KG pipeline, then processed here to add health status information.

Scope: Problem Types Tested

The HuggingFace database contains 50 total problem types, but only 14 major problem types account for ~29,160 models (~89% of all models). The remaining 36 problem types together contain only ~3,352 models and are out of scope for this testing effort. Health checks are run exclusively on the 14 major problem types below.

Problem Type	Total Models	Tested Models	Passed (OK)	Failed
text-generation	15,376	4,536	2,356	2,180
text-to-image	3,571	3,481	1,028	2,453
text-classification	2,386	2,304	1,946	358
feature-extraction	2,136	2,036	1,129	907
automatic-speech-recognition	1,250	1,010	747	263
image-classification	966	895	755	140
fill-mask	929	929	802	127
image-to-text	871	834	379	455
token-classification	759	733	502	231
summarization	292	284	219	65
image-to-image	286	0	—	—
object-detection	278	0	—	—
question-answering	274	0	—	—
image-segmentation	186	0	—	—

Summary: ~29,160 models across the 14 major problem types are in scope. The remaining ~3,352 models spread across 36 minor problem types are not being tested.

---

Pipeline

1. Generate Database: Run scripts in Hugging2KG/ folder to create huggingface2.db
2. Copy Database: Move huggingface2.db from Hugging2KG/ to ModelStatus/
3. Run Health Checks: Execute ModelStatus scripts to test models and add health columns

Scripts

Health Check Scripts

Script	Task
txt_clsf.py	Text classification (transformers, sentence-transformers)
txt_gen.py	Text generation (transformers)
txt2img.py	Text-to-image (diffusers)
fet_ext.py	Feature extraction (transformers, sentence-transformers)
asr.py	Automatic speech recognition (transformers)
fillmask.py	Fill-mask (transformers)
img2img.py	Image-to-image (diffusers, transformers)
img2txt.py	Image-to-text (transformers)
imgclsf.py	Image classification (transformers, timm)
q&A.py	Question answering (transformers)
summarization.py	Summarization (transformers)
tokenclsf.py	Token classification (transformers)
translation.py	Translation (transformers)

Reporting Scripts

Script	Purpose
summary.py	Database summary — counts and coverage across all problem types
healthreportbyproblemtype.py	Detailed health report filtered by a specific problem type

Setup

1. Ensure huggingface2.db exists in this folder (copied from Hugging2KG/).
2. Set your HuggingFace token in each script you plan to run:

   HF_TOKEN = "your_token_here"
   

3. Update the database path if needed. The default is ./huggingface2.db (same folder as the script).

Usage

Run any health check script directly from the ModelStatus/ folder:

cd ModelStatus
python txt_clsf.py
python txt_gen.py
python asr.py
# ...etc

For reporting:

python summary.py
python healthreportbyproblemtype.py

Note: Set DATABASE_PATH at the top of summary.py and healthreportbyproblemtype.py to point to your huggingface2.db file. Also set TARGET_PROBLEM in healthreportbyproblemtype.py to the problem type you want to inspect (e.g. 'token-classification').

Each Script

• Tests models from the database in batches
• Updates health status: OK, FAIL, OOM, NOT_FOUND, GATED, etc.
• Cleans up cache after each model
• Shows progress and final summary

Database Columns

Scripts add these columns to the Models table:

• health_status - Current model status
• health_error - Error message if failed
• last_checked - Timestamp of last check

---

Database Info

Problem Type	Libraries / Frameworks	Model Count
text-generation	transformers, sentence-transformers, mlx, PaddlePaddle, Model Optimizer, llama.cpp, gguf, vllm, transformers.js, litert-lm, peft, adapter-transformers, bitsandbytes, accelerate, datasets, deepspeed, trl, node-llama-cpp, llamacpp, nemo, grok, PyTorch, Unsloth, AiFlow, ggml, axolotl, fastai, optimum-executorch, pytorch, zeroshot_classifier, GGUF, gemma_torch, mlx-llm, fla, unity-sentis, diffusers, keras-hub, exllamav3, coreml, exllamav2, furiosa-llm, rwkv, reverb, keras	15376
text-to-image	diffusers, transformers, gguf, diffusion-single-file, diffusionkit, merlin, infinite-you, wan2.2, transformers.js, sana, cosmos, open_clip, stable-diffusion, keras, t5, sana-sprint, peft, tf-keras	3571
text-classification	transformers, sentence-transformers, pysentimiento, fasttext, peft, transformers.js, staticvectors, LogClassifier, setfit, bertopic, adaptive-classifier, generic, pytorch, keras, adapter-transformers, tf-keras, zeroshot_classifier, transformer	2386
feature-extraction	sentence-transformers, transformers, transformers.js, light-embed, pytorch, PyLate, generic, timm, model2vec, mlx, nemo, fasttext, diffusers, hierarchy-transformers, llamafile, hezar, mlx-llm, distiller, setfit, sentencepiece	2136
automatic-speech-recognition	pyannote-audio, transformers, ctranslate2, mlx, nemo, whisperkit, transformers.js, speechbrain, pytorch, espnet, faster-whisper, hezar, unity-sentis, peft, onnx	1250
image-classification	transformers, timm, transformers.js, ultralytics, wildlife-datasets, torchgeo, derm-foundation, keras, pytorch, cxr-foundation, mlx-image, coreml, tf-keras, peft, mindspore, configilm	966
fill-mask	transformers, transformers.js, multimolecule	929
image-to-text	transformers, dots_ocr, PaddleOCR, hezar, sentence-transformers, gguf, htrflow, open_clip, transformers.js, pytorch, diffusers, tf-keras	871
token-classification	transformers, flair, gliner, spacy, stanza, span-marker, transformers.js, edsnlp, cadence-punctuation, adapter-transformers	759
translation	transformers, comet, transformers.js, hibiki, peft, ctranslate2, litert, fastai, mlx	526
image-text-to-text	transformers, vllm, mlx, xtuner, LLaVA, PaddlePaddle, peft, describe-anything, nanovlm, monkeyocr, gguf, llama.cpp, huatuogpt_vision, transformers.js, pytorch, hpsv3, keras-hub	508
summarization	transformers, transformers.js, fastai, peft	292
image-to-image	diffusers, gguf, diffusion-single-file, transformers, medvae, pytorch, refiners, univa, tf-keras, transformers.js, timm	286
object-detection	ultralytics, transformers, yolov5, hezar, transformers.js, yolov10, pytorch, unity-sentis, coreml, tf-keras	278
question-answering	transformers, transformers.js, peft, diffusers, adapter-transformers, allennlp, sentence-transformers	274
text-to-speech	coqui, f5-tts, chatterbox, moshi, speechbrain, transformers, transformers.js, zonos, outetts, kimi-audio, pytorch, vui, mlx, nemo, txtai, metavoice, index-tts, TTS, espnet, voicecraft, mars5-tts, fairseq, onnx, dia, peft	270
zero-shot-image-classification	transformers, open_clip, transformers.js, perception-encoder, terratorch, generic, tic-clip	212
image-segmentation	transformers, birefnet, ben2, refiners, transformers.js, ultralytics, sapiens, pytorch, tf-keras, coreml, anime_segmentation, keras, segmentation-models-pytorch, torch, generic	186
audio-classification	transformers, speechbrain, nemo, pytorch, transformers.js, timm	145
image-feature-extraction	transformers, timm, transformers.js, diffusers, kronos, perception-encoder, diffusion-single-file, py-feat, open_clip, sapiens	140
text-to-video	gguf, diffusers, wan2.2, cosmos, transformers, open-sora, open_clip, mtvcraft	119
sentence-similarity	sentence-transformers, transformers, colbert-ai, staticvectors, RAGatouille, txtai, gguf, unity-sentis, peft	109
image-to-video	diffusers, gguf, wan2.2, liveportrait, comfyui, cosmos	103
reinforcement-learning	transformers, stable-baselines3, safe-rlhf, ml-agents, mlx, unity-sentis	97
audio-to-audio	PyTorch, speechbrain, transformers, asteroid, stable-audio-tools, pytorch, espnet, fairseq, soloaudio	80
depth-estimation	transformers, depth-anything-v2, diffusers, DepthCrafter, depth-pro, transformers.js, pytorch, coreml, sapiens, unity-sentis	71
text-to-audio	transformers, stable-audio-tools, tencent-song-generation, transformers.js, diffusers, audiocraft, chat_tts, custom	61
robotics	lerobot, timm, transformers, pytorch, crossformer	47
text-ranking	sentence-transformers, lightning-ir, transformers, treehop-rag	42
video-classification	transformers, videoprism, pytorch	41
time-series-forecasting	transformers, granite-tsfm, tirex, pytorch, timesfm, timer, YingLong, transformers.js, tf-keras	36
image-to-3d	trellis, mast3r, pytorch, diffusers, diffusion-single-file, dust3r, mesh-anything, fast3r, stream3r, transformers, align3r, craftsman-v1-5, hermes	35
any-to-any	transformers, diffusers, gguf, bagel-mot, ml-4m, mlx, mini-omni2	32
visual-document-retrieval	colpali, peft, Tevatron, transformers	30
table-question-answering	transformers	30
mask-generation	transformers, transformers.js, coreml, sam2	28
keypoint-detection	transformers, pytorch, sapiens	28
unconditional-image-generation	diffusers, pytorch, tf-keras, transformers	27
zero-shot-object-detection	transformers, transformers.js	21
video-to-video	diffusers, seedvr	21
zero-shot-classification	transformers, sentence-transformers, open_clip, zeroshot_classifier	19
text-to-3d	hunyuan3d-2, trellis, transformers, 3dtopia-xl, uni-3dar, diffusers	15
audio-text-to-text	transformers, vllm, transformers.js, peft	14
voice-activity-detection	pyannote-audio, nemo, coreml, transformers	10
tabular-classification	contexttab, tabpfn, sklearn, tf-keras	8
graph-ml	transformers, diffusers, anemoi, birder	8
document-question-answering	transformers	8
visual-question-answering	transformers, peft	7
video-text-to-text	peft, transformers	3
tabular-regression	tabpfn, tf-keras	2