# model-support-checker

Check whether a HuggingFace / ModelScope model is supported by **SGLang**,
**vLLM**, or **vLLM-Ascend**, and since which version.

## Setup (one-time)

The tool is **stateful**: your access mode is persisted in `.state/state.json`
(gitignored), so later runs need no flags or tokens.

On first run you choose between two access modes — **local clone is recommended**:

| | Local clone (recommended) | GitHub PAT |
|---|---|---|
| Result quality | Definitive (files grepped on disk) | Best-effort without token; code search 403s anonymously |
| Rate limits | None | Yes — can break batch/version checks |
| Repeat checks | Fast (disk reads) | Slower (GitHub API) |
| Works offline | Yes, after initial clone | No |
| Cost | ~1 GB+ disk (vLLM full history needed) | None |
| Freshness | Stale until `git pull` | Always current |

```bash
python3 main.py --setup local                                              # clone all repos into .state/repos/
python3 main.py --setup local --vllm-path P1 --sglang-path P2 --vllm-ascend-path P3  # reuse existing checkouts
python3 main.py --setup token                                              # GitHub API mode

python3 main.py --doctor        # check setup state and local checkouts
python3 main.py --reset-state   # forget setup (cloned repos are kept and reused)
```

Notes:

- Local clones are full clones (`--depth 1` breaks version detection).
- The GitHub token is NEVER written to `.state/`; in token mode export
  `GITHUB_TOKEN` per run.
- Re-run `--setup <mode>` anytime to switch modes; explicit `--vllm-path` /
  `--sglang-path` / `--vllm-ascend-path` / `--token` flags override the saved state for that run.
- Each run in local mode refreshes checkouts in the background (`git fetch`)
  and reports staleness or failures at the end — never blocking the main
  check.

## Methodology

For each framework the checker runs four framework-agnostic steps:

1. **Architecture name** — read `architectures` from the model's `config.json`
   (HuggingFace first, ModelScope API as fallback). Can be overridden with
   `--arch` to skip the network lookup entirely.
2. **Official docs** (supplementary) — grep the framework's supported-models page.
3. **GitHub source** (authoritative) — search the framework's models directory on
   GitHub. All frameworks register a model by mapping the HF architecture
   string to a class in `<models_dir>/<file>.py`, so the architecture string is
   always present in its implementation file.
   - For vLLM this step additionally parses `registry.py` to extract the model
     category, module path, class name, and checks
     `_PREVIOUSLY_SUPPORTED_MODELS` / `_OOT_SUPPORTED_MODELS`.
   - For vLLM-Ascend this step parses `__init__.py` for `ModelRegistry.register_model()`
     calls to extract module and class.
4. **Version** — determine the first framework release that contains the
   implementation (via the file's earliest commit date → nearest release).

The authoritative "is it supported" check uses the GitHub **code-search API**
with a path filter on the models directory. Anonymous code search returns 403,
so a `GITHUB_TOKEN` is required for a definitive answer. Without a token the
tool falls back to scraping the models directory tree + grepping candidate files
on the raw CDN; that fallback can be incomplete, so a NO without a token is not
definitive. Step 4 also needs a token on rate-limited IPs.

### Local checkouts (no GitHub API)

Local mode is the recommended setup (`--setup local` clones all repos into
`.state/repos/` automatically). All source reads (models directory,
`registry.py`, and version history) are served from disk via `git`, and **no
GitHub API requests are made** — only the architecture lookup (step 1) and the
docs check (step 2) still touch the network.

```bash
# Override saved paths for a single run (no GITHUB_TOKEN needed for source/registry/version)
python3 main.py --framework vllm         --vllm-path         /path/to/vllm         deepseek-ai/DeepSeek-V3
python3 main.py --framework sglang       --sglang-path       /path/to/sglang       Qwen/Qwen3.6-35B-A3B
python3 main.py --framework vllm-ascend  --vllm-ascend-path  /path/to/vllm-ascend  deepseek-ai/DeepSeek-V3
```

`--vllm-ref` is ignored when `--vllm-path` is given. The local path must be a
git checkout of the framework repo (so `git log`/`git tag` are available for
version detection). Keep checkouts fresh with an occasional `git pull`.

No third-party dependencies — standard library only.

## Usage

```bash
# After setup, just pass a model id
python3 main.py --framework sglang Qwen/Qwen3.6-35B-A3B
python3 main.py --framework vllm deepseek-ai/DeepSeek-V3
python3 main.py --framework vllm-ascend deepseek-ai/DeepSeek-V3

# Check all frameworks at once (default)
python3 main.py meta-llama/Llama-3.1-8B

# Token mode: GITHUB_TOKEN per run (never stored)
GITHUB_TOKEN=xxx python3 main.py meta-llama/Llama-3.1-8B

# Manually specify architecture name (skip config.json fetch)
python3 main.py --arch LlamaForCausalLM
python3 main.py --framework vllm --arch DeepseekV3ForCausalLM
python3 main.py --framework vllm --arch "DeepseekV3ForCausalLM,DeepseekV2ForCausalLM" some-org/some-model
```

### Options

| Flag                 | Description                                                                                       |
| -------------------- | ------------------------------------------------------------------------------------------------- |
| `model_id`           | HuggingFace or ModelScope model id (positional; optional when `--arch` is given).                 |
| `--setup`            | One-time setup: `local` (clone repos into `.state/repos/`, recommended) or `token`; persists to `.state/state.json`. |
| `--reset-state`      | Forget saved setup state (cloned repos kept).                                                     |
| `--doctor`           | Check setup state and local checkouts, then exit.                                                 |
| `--framework`        | `sglang`, `vllm`, `vllm-ascend`, or `all` (default: `all`).                                       |
| `--source`           | `auto`, `hf`, or `modelscope` for reading `config.json`.                                          |
| `--arch`             | Manually specify architecture name(s), comma-separated. Skips config.json fetch entirely.         |
| `--token`            | GitHub token (or set `GITHUB_TOKEN` env); overrides saved state for this run.                      |
| `--no-docs`          | Skip the docs check.                                                                              |
| `--vllm-ref`         | vLLM git ref (branch/tag) for the registry check (default: `main`).                               |
| `--vllm-path`        | Local vllm repo checkout; overrides saved path for this run.                                      |
| `--sglang-path`      | Local sglang repo checkout; overrides saved path for this run.                                    |
| `--vllm-ascend-path` | Local vllm-ascend repo checkout; overrides saved path for this run.                               |
| `-v, --verbose`      | Verbose output.                                                                                   |

## License

[The Unlicense](../LICENSE) — released into the public domain.