[teamai] Push 87 resource(s) from XingfenD
This commit is contained in:
@@ -0,0 +1 @@
|
||||
eval-report.html
|
||||
@@ -0,0 +1 @@
|
||||
XingfenD
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
name: media-use
|
||||
description: Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.
|
||||
---
|
||||
|
||||
# media-use
|
||||
|
||||
The media OS for HyperFrames: resolve · generate · operate · remember — every media type, one skill, zero context noise.
|
||||
|
||||
First run: install and sign in to the `heygen` CLI (the free-usage path), then verify with `node <SKILL_DIR>/scripts/resolve.mjs --doctor`. Setup and providers: `references/setup-providers.md`.
|
||||
|
||||
## Resolve — the one verb
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type <type> --intent "<description>" --project <dir>
|
||||
```
|
||||
|
||||
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`. All search noise stays on disk.
|
||||
|
||||
| Type | One-line intent |
|
||||
| ------- | ----------------------------------------------------------------------------------- |
|
||||
| `bgm` | background music (HeyGen catalog, 10k+ tracks) |
|
||||
| `sfx` | sound effects (bundled 19-file library + catalog) |
|
||||
| `image` | photos, backgrounds (HeyGen asset search, 75k+ vectors) |
|
||||
| `icon` | icons, symbols (transparent) |
|
||||
| `logo` | official brand marks (svgl → simple-icons → GitHub avatar → favicon; never redrawn) |
|
||||
| `voice` | TTS voiceover (HeyGen free-usage path; optional local Kokoro) |
|
||||
| `grade` | measured correction candidate; broad polish/stylization follows Media Treatments |
|
||||
| `lut` | user-provided or explicitly chosen reusable validated `.cube` file |
|
||||
|
||||
Before resolving fresh, list reusable candidates with `--candidates` and judge fit yourself — reuse rules, all flags, ingest (`--from`), and adopt are in `references/resolve.md`.
|
||||
|
||||
## Treat broad visual feedback as media intent
|
||||
|
||||
When a user explicitly asks to fix, polish, stylize, obscure, emphasize, or
|
||||
reveal photographic media, read `references/media-treatments.md` even if they
|
||||
do not name color grading or an effect. Inspect the real `<img>`/`<video>`,
|
||||
choose one primary intent, then use deterministic persistence and verification.
|
||||
Use a matching recipe as an optional tested seed, or inspect
|
||||
`hyperframes media-treatment --capabilities --json`, then request one relevant
|
||||
family/effect with `--capability <id>` and assemble a custom treatment from
|
||||
canonical controls. Never load `--all` for ordinary authoring. A treatment may
|
||||
compose correction, a preset, finishing, compatible shader effects, supported
|
||||
keyframes, and optional Registry overlays. Add only source-justified bounded
|
||||
tuning and compatible parts, never effects merely to make the result look more
|
||||
sophisticated. Persist the final combined payload with
|
||||
`hyperframes media-treatment`.
|
||||
|
||||
Use one progressively escalating workflow. For video, inspect one labeled
|
||||
early/middle/late contact sheet rather than reading frames separately. Apply one
|
||||
candidate and inspect one after-sheet for ordinary correction or polish.
|
||||
Escalate to individual frames or moving draft evidence only when the result is
|
||||
ambiguous, temporal, stylized, LUT-based, HDR/LOG-sensitive, private, or
|
||||
brand-critical.
|
||||
|
||||
For ordinary correction or polish, persist the final treatment's
|
||||
preset/adjustment JSON.
|
||||
Do not generate a `.cube` LUT merely to encode exposure, shadows, contrast, or
|
||||
warmth. Use a LUT only when the user supplies one or the selected treatment
|
||||
explicitly owns one. `resolve --type grade --for ... --analyze` is measurement
|
||||
evidence, not permission to replace the chosen treatment with a generated LUT.
|
||||
Do not recreate supported vignette, grain, blur, pixelate, color, or treatment
|
||||
effects with CSS/SVG overlays; that bypasses Studio controls and the canonical
|
||||
preview/render shader path.
|
||||
|
||||
## Be proactive — run a media opportunity pass
|
||||
|
||||
The human usually can't tell which media would lift the piece. You can. When you build or review a composition, do **one** grounded scan and then **ask once** — don't silently add, and don't nag per asset.
|
||||
|
||||
Surface an opportunity only when a concrete signal is present:
|
||||
|
||||
| Signal detected | Offer |
|
||||
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
|
||||
| On-screen text / a script with no voiceover | TTS voiceover (audio engine) |
|
||||
| Emoji or a `<div>` styled as an icon | resolve real `icon`s |
|
||||
| Image that is a placeholder, tiny, or upscaled-looking | a better `image` (and/or upscale — see `references/operations.md`) |
|
||||
| Hard scene cuts / transitions with no sound | transition `sfx` |
|
||||
| A piece over ~10s with no music bed | `bgm` |
|
||||
| Footage that reads under/over-exposed or color-cast | a corrective grade (inspect it with `hyperframes media-treatment --selector '#hero' --analyze --json`) |
|
||||
| Photographic media that feels visually flat or off-topic | one specific source-appropriate preset or custom treatment, with the intended target named |
|
||||
| A meaningful media entrance/reveal that feels static | one supported seek-safe treatment animation; preserve color unless the request also justifies a preset |
|
||||
|
||||
Rules that keep this a help, not nagware: **grounded, not generic** (no signal → no suggestion); **opinionated + concrete** (propose the specific fix with defaults chosen — the human approves **all / some / none**); **once per project** (one consolidated ask; respect "leave it"); **surface, never silently mutate** (color grades especially: propose and preview — a gray-world "correction" ruins an intentional sunset or neon look).
|
||||
|
||||
## Where to look — read only the file your task needs
|
||||
|
||||
| Task | Read |
|
||||
| ------------------------------------------------------------------------- | -------------------------------- |
|
||||
| resolve / reuse / adopt / ingest, flags, cascade, inventory | `references/resolve.md` |
|
||||
| color grading, LUTs, smart grade (`--for`), grade-compare | `references/grading.md` |
|
||||
| voiceover / TTS, music, SFX, captions, transcription (audio engine) | `references/audio.md` |
|
||||
| cut / reframe / transform existing media, exact error diffusion, HEVC | `references/operations.md` |
|
||||
| source-aware creative treatments, realtime effects, overlays, reveals | `references/media-treatments.md` |
|
||||
| install + auth, provider table, RAM ladders, `--local-only`, `--provider` | `references/setup-providers.md` |
|
||||
| remembered preferences + frozen recipes (user memory) | `references/memory.md` |
|
||||
| ownership matrix, usage stats, telemetry, privacy (maintainer-facing) | `references/meta.md` |
|
||||
@@ -0,0 +1,35 @@
|
||||
# SFX Credits
|
||||
|
||||
All sound effects in this directory are sourced from [Pixabay](https://pixabay.com/sound-effects/) and used under the [Pixabay Content License](https://pixabay.com/service/license-summary/).
|
||||
|
||||
The Pixabay license allows free use for commercial and non-commercial purposes without attribution, but attribution is appreciated and given here for transparency.
|
||||
|
||||
## Files
|
||||
|
||||
The following `.mp3` files are bundled with this skill:
|
||||
|
||||
- `chime.mp3`
|
||||
- `click.mp3` / `click-soft.mp3`
|
||||
- `error.mp3`
|
||||
- `glitch-1.mp3` / `glitch-2.mp3` / `glitch-3.mp3`
|
||||
- `impact-bass-1.mp3` / `impact-bass-2.mp3`
|
||||
- `key-press.mp3`
|
||||
- `notification.mp3`
|
||||
- `ping.mp3`
|
||||
- `pop.mp3`
|
||||
- `riser.mp3`
|
||||
- `sparkle.mp3`
|
||||
- `typing.mp3`
|
||||
- `whoosh.mp3` / `whoosh-short.mp3` / `whoosh-cinematic.mp3`
|
||||
|
||||
See `manifest.json` for per-file metadata (duration, energy character, recommended use).
|
||||
|
||||
## License
|
||||
|
||||
All files are distributed under the [Pixabay Content License](https://pixabay.com/service/license-summary/), which permits:
|
||||
|
||||
- Commercial and non-commercial use
|
||||
- Modification and remixing
|
||||
- Redistribution as part of derivative works (such as videos rendered with HyperFrames)
|
||||
|
||||
without any attribution requirement.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"chime": {
|
||||
"file": "chime.mp3",
|
||||
"duration": 2.5,
|
||||
"description": "Soft melodic chime — gentle positive beat: success/confirmation or a lighthearted transition. Sync to the visual moment."
|
||||
},
|
||||
"click-soft": {
|
||||
"file": "click-soft.mp3",
|
||||
"duration": 0.37,
|
||||
"description": "Quiet short click — low-key UI tap / soft selection. Short accent, sync exactly to the on-screen action."
|
||||
},
|
||||
"click": {
|
||||
"file": "click.mp3",
|
||||
"duration": 0.37,
|
||||
"description": "Crisp UI click — button press, toggle, selection. Short accent, sync exactly to the on-screen action."
|
||||
},
|
||||
"error": {
|
||||
"file": "error.mp3",
|
||||
"duration": 1.62,
|
||||
"description": "Negative / error tone — failure state, a 'wrong' beat, or a glitchy interruption."
|
||||
},
|
||||
"glitch-1": {
|
||||
"file": "glitch-1.mp3",
|
||||
"duration": 2.64,
|
||||
"description": "Punchy digital glitch — hard-cut accent or sudden reveal. Trigger on the hit; let the decay bleed into the next shot (J-cut)."
|
||||
},
|
||||
"glitch-2": {
|
||||
"file": "glitch-2.mp3",
|
||||
"duration": 3.5,
|
||||
"description": "Harsh, longer glitch — chaotic / jarring transition or a distorted reveal."
|
||||
},
|
||||
"glitch-3": {
|
||||
"file": "glitch-3.mp3",
|
||||
"duration": 3.1,
|
||||
"description": "Low-key glitch texture — subtle digital shift, minimal transition that sits under other audio."
|
||||
},
|
||||
"impact-bass-1": {
|
||||
"file": "impact-bass-1.mp3",
|
||||
"duration": 2.12,
|
||||
"description": "Bass impact hit — logo/hero snap, headline slam. Trigger on the visual landing; decay carries into the next shot (J-cut)."
|
||||
},
|
||||
"impact-bass-2": {
|
||||
"file": "impact-bass-2.mp3",
|
||||
"duration": 2.59,
|
||||
"description": "Bass impact with a short swell — brief anticipation then a deep hit. Place so the peak lands on the reveal."
|
||||
},
|
||||
"key-press": {
|
||||
"file": "key-press.mp3",
|
||||
"duration": 0.4,
|
||||
"description": "Single key press — one keystroke / terminal-input beat. Short accent, sync to the typed character."
|
||||
},
|
||||
"notification": {
|
||||
"file": "notification.mp3",
|
||||
"duration": 2.46,
|
||||
"description": "Notification chime — alert, message-in, toast/badge appears. Sync to the element entering."
|
||||
},
|
||||
"ping": {
|
||||
"file": "ping.mp3",
|
||||
"duration": 1.32,
|
||||
"description": "Sharp electronic ping — punchy accent on a key reveal or data point. Sync to the beat."
|
||||
},
|
||||
"pop": {
|
||||
"file": "pop.mp3",
|
||||
"duration": 0.72,
|
||||
"description": "Quick pop — element appear/spawn, chip/tag/badge in. Small precise accent, sync to the pop-in."
|
||||
},
|
||||
"riser": {
|
||||
"file": "riser.mp3",
|
||||
"duration": 10.03,
|
||||
"description": "Long cinematic riser (~10s build, peak at the end). Trigger at (climax_time − 10.03s) so it crests exactly on the reveal."
|
||||
},
|
||||
"sparkle": {
|
||||
"file": "sparkle.mp3",
|
||||
"duration": 1.8,
|
||||
"description": "Bright sparkle / shimmer — magical reveal or 'shine' highlight on a hero element. Sync to the highlight."
|
||||
},
|
||||
"typing": {
|
||||
"file": "typing.mp3",
|
||||
"duration": 1.5,
|
||||
"description": "Typing burst (~1.5s of keys) — keyboard / code typing reveal, text-being-typed beat. Start as the text begins typing."
|
||||
},
|
||||
"whoosh-cinematic": {
|
||||
"file": "whoosh-cinematic.mp3",
|
||||
"duration": 5.54,
|
||||
"description": "Cinematic whoosh build (~5.5s) — sweeping scene transition. Align so the swell peaks on the cut."
|
||||
},
|
||||
"whoosh-short": {
|
||||
"file": "whoosh-short.mp3",
|
||||
"duration": 0.57,
|
||||
"description": "Short whoosh — quick swipe/slide accent, fast element move, snappy transition. Sync to the motion."
|
||||
},
|
||||
"whoosh": {
|
||||
"file": "whoosh.mp3",
|
||||
"duration": 0.57,
|
||||
"description": "Punchy whoosh/impact — fast reveal or hard transition accent. Sync to the motion."
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,74 @@
|
||||
# Background music (BGM)
|
||||
|
||||
One music bed per composition, produced by the shared audio engine (`scripts/audio.mjs` → `scripts/lib/bgm.mjs`). Two routes, chosen by the engine's one switch — whether a HeyGen credential is present:
|
||||
|
||||
- **HeyGen retrieval — the default when credentialed.** Search HeyGen's music catalog by mood, download the top track. No generation; same `~/.heygen` / `$HEYGEN_API_KEY` credential as TTS.
|
||||
- **Local generation (Lyria → MusicGen) — the fallback when there is no credential** (or when asked for explicitly). Generate a WAV from a mood prompt. There is **no `npx hyperframes bgm` command**; the engine spawns `scripts/lyria-recipe.py` or an inline MusicGen script directly.
|
||||
|
||||
> **Run the Preflight first — no credential is not a green light to silently generate locally.** Before generating, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen's music library, or continue offline with local generation). This applies to a one-off "generate a BGM" request just as much as inside a full workflow.
|
||||
|
||||
## Driving it from the request
|
||||
|
||||
`audio_request.json` → `bgm: { mode?, query?, prompt? }`:
|
||||
|
||||
- **`mode`** — `retrieve | generate | none`. Omit for **auto** (retrieve when credentialed, else generate). An **explicit** `retrieve` is strict: no credential ⇒ skip, never a detached generate (so a caller with no `wait-bgm` step, e.g. product-launch, can't get a pending job it won't await).
|
||||
- **`query`** — the mood, used for retrieval and as a fallback prompt seed (e.g. a storyboard's `music:` field, falling back to `message` → `arc` → `"calm cinematic underscore"`).
|
||||
- **`prompt`** — an explicit full prompt for generation; omit and the engine infers one (see Mood inference). Optional `blob` / `archetype` / `arc` feed that inference.
|
||||
|
||||
## HeyGen retrieval (default)
|
||||
|
||||
`searchSounds(query, "music", { limit: 5 })` → `GET /audio/sounds?query=<mood>&type=music&limit=5`. Take the top result (ranked by `score`), download its presigned `audio_url` → `assets/bgm/track.mp3`. Synchronous. No match → skip (BGM is optional; never fail the render over it). Cue written to `audio_meta.json`:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"path": "assets/bgm/track.mp3",
|
||||
"volume": 0.12,
|
||||
"mode": "retrieve",
|
||||
"query": "calm cinematic underscore",
|
||||
"duration_s": 42.0,
|
||||
}
|
||||
```
|
||||
|
||||
`volume` comes from the engine's `bgmDefaultVolume()`: `BGM_BED_VOLUME` (currently `0.12` ≈ -18 dB — a bed under the voice) under narration, `BGM_SILENT_VOLUME` (currently `0.9`) for a silent film (no voice). Tune those constants in `scripts/lib/bgm.mjs`, not call sites. An explicit `volume` in `audio_meta.json` always overrides this default. `bgm_pending` is `false` — the file is on disk when the engine returns.
|
||||
|
||||
For short launch videos, do not assume the beginning of the retrieved file is the best edit point. Check the opening against later five-second sections. If the track starts with a quiet build but a later section has a stronger, clean musical entrance, trim from that section and apply a short fade-in and longer fade-out. Repeat this check whenever the composition duration changes; the final music file must cover the full cut without a silent tail.
|
||||
|
||||
## Local generation (fallback) — Lyria → MusicGen
|
||||
|
||||
Spawned **detached** so voice work isn't blocked; `audio_meta.bgm_pending: true` and `bgm_pid` / `bgm_log` are set until it finishes. **Run `scripts/wait-bgm.mjs` before assembling** — it polls the output file / process / log, detects crashes, and writes `bgm_status.json` (`status: ready | failed | timeout | disabled`). A failed/absent track is simply omitted; it never blocks voice/SFX.
|
||||
|
||||
| Order | Provider | Env / deps | Speed | Quality |
|
||||
| ----- | ------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------------- | --------------------------- |
|
||||
| 1 | Google Lyria RealTime | `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `google-genai` (auto-installed on demand) | Real-time stream (≈ requested duration) | Production-grade |
|
||||
| 2 | MusicGen (`facebook/musicgen-small`) | Python `transformers + torch + soundfile + numpy` (~300 MB first run; auto-installed) | Slow on CPU; fast on Apple MPS / CUDA | Decent; prompt-only control |
|
||||
|
||||
Output → `assets/bgm/track.wav`, target = total voice duration. MusicGen generates **one** seed clip (≤28–30s, under the decoder's positional limit) then crossfade-loops it up to the target (or trims down if shorter), avoiding per-segment seams. Backend selection is by what can actually **run**: Lyria only when `import google.genai` succeeds, else MusicGen; if neither can be made to run, BGM is skipped (voice + SFX still render).
|
||||
|
||||
## Mood inference (the generate prompt)
|
||||
|
||||
`inferBgmPrompt()` in `scripts/lib/bgm.mjs`: an explicit `prompt` wins; otherwise industry-keyword **base** → narrative-**archetype** shape → emotional-**arc** tiebreaker.
|
||||
|
||||
| Match in `blob` / `query` | Base prompt | BPM |
|
||||
| ------------------------------------------------------ | --------------------------------------------------------------------------- | --- |
|
||||
| `crypto / nft / web3 / defi / token / blockchain` | atmospheric electronic, deep bass, futuristic synths, restrained percussion | 100 |
|
||||
| `finance / fintech / bank / payment / invest / wealth` | calm cinematic, soft strings, subtle piano, restrained percussion | 92 |
|
||||
| `creative / agency / design / studio / art / brand` | playful electronic, warm pads, light percussion | 115 |
|
||||
| _(default: SaaS / tech / platform)_ | uplifting corporate tech, bright modern piano with synth pads | 108 |
|
||||
|
||||
Archetype then reshapes the arc — PAS → "MINOR to MAJOR" build; BAB / future-pacing → aspirational rising; feature-cascade → +10 BPM driving; demo-loop → −8 BPM minimal. The emotional arc breaks remaining ties (tension→relief, excitement, trust/reassurance).
|
||||
|
||||
## Lyria knobs (direct recipe use)
|
||||
|
||||
The engine bakes BPM / scale into the **prompt text** (via the inference above) and passes only `--output` / `--duration` / `--prompt` to the recipe. If you invoke `scripts/lyria-recipe.py` directly you can also set: `--bpm` (90–110 calm, 110–130 energetic), `--brightness` (0–1, ≥0.7 promotional), `--density` (0–1, higher = fuller), `--scale` (`MAJOR` / `MINOR` / `PENTATONIC` / …), `--negative-prompt` (styles to exclude). MusicGen ignores all of these — put the mood in the prompt.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Behavior |
|
||||
| --------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| No music match (retrieve) | `bgm: null`, anomaly logged. Render proceeds without BGM. |
|
||||
| Explicit `retrieve`, no credential | Skipped (no silent generate fallback). Use `mode: generate` or omit `mode` for auto. |
|
||||
| Neither Lyria nor MusicGen can run (generate) | `bgm` disabled with a `pip install …` hint. Voice + SFX still render. |
|
||||
| Generate still rendering at assemble time | `bgm_pending: true`; `wait-bgm.mjs` waits/checks and writes `bgm_status.json` first. |
|
||||
| Generate crashed | `wait-bgm.mjs` → `bgm_status.json { status: "failed" }`; the `<audio>` track is omitted. |
|
||||
|
||||
BGM failure never blocks a render.
|
||||
@@ -0,0 +1,159 @@
|
||||
# Captions
|
||||
|
||||
Before authoring: confirm the transcript came from the right Whisper model. CLI default `small.en` silently translates non-English audio — see [`../transcribe.md`](../transcribe.md) → "Language Rule" and [`transcript-handling.md`](transcript-handling.md) for the mandatory quality check.
|
||||
|
||||
Analyze spoken content to determine caption style. If user specifies a style, use that. Otherwise, detect tone from the transcript.
|
||||
|
||||
## Transcript Source
|
||||
|
||||
```json
|
||||
[
|
||||
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
|
||||
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
|
||||
]
|
||||
```
|
||||
|
||||
`id` (`w0`, `w1`, …) is the stable reference for per-word overrides and is added by `hyperframes transcribe`. It's optional for backwards compatibility with hand-authored transcripts. See [`../transcribe.md`](../transcribe.md) → "Output Shape" for how this is produced, and [`transcript-handling.md`](transcript-handling.md) for cleanup before consumption.
|
||||
|
||||
## Style Detection (When No Style Specified)
|
||||
|
||||
Read the full transcript before choosing. Four dimensions:
|
||||
|
||||
**1. Visual feel** — corporate→clean; energetic→bold; storytelling→elegant; technical→precise; social→playful.
|
||||
|
||||
**2. Color palette** — dark+bright for energy; muted for professional; high contrast for clarity; one accent color.
|
||||
|
||||
**3. Font mood** — heavy/condensed for impact; clean sans for modern; rounded for friendly; serif for elegance.
|
||||
|
||||
**4. Animation character** — scale-pop for punchy; gentle fade for calm; word-by-word for emphasis; typewriter for technical.
|
||||
|
||||
## Per-Word Styling
|
||||
|
||||
Scan for words deserving distinct treatment:
|
||||
|
||||
- **Brand/product names** — larger size, unique color
|
||||
- **ALL CAPS** — scale boost, flash, accent color
|
||||
- **Numbers/statistics** — bold weight, accent color
|
||||
- **Emotional keywords** — exaggerated animation (overshoot, bounce)
|
||||
- **Call-to-action** — highlight, underline, color pop
|
||||
- **Marker highlight** — for beyond-color emphasis (highlight sweep, circle, burst, scribble, sketchout), see `hyperframes-animation/rules/css-marker-patterns.md`.
|
||||
|
||||
## Script-to-Style Mapping
|
||||
|
||||
| Tone | Font mood | Animation | Color | Size |
|
||||
| ------------ | ------------------------ | ---------------------------------- | --------------------------- | ------- |
|
||||
| Hype/launch | Heavy condensed, 800-900 | Scale-pop, back.out(1.7), 0.1-0.2s | Bright on dark | 72-96px |
|
||||
| Corporate | Clean sans, 600-700 | Fade+slide, power3.out, 0.3s | White/neutral, muted accent | 56-72px |
|
||||
| Tutorial | Mono/clean sans, 500-600 | Typewriter/fade, 0.4-0.5s | High contrast, minimal | 48-64px |
|
||||
| Storytelling | Serif/elegant, 400-500 | Slow fade, power2.out, 0.5-0.6s | Warm muted tones | 44-56px |
|
||||
| Social | Rounded sans, 700-800 | Bounce, elastic.out, word-by-word | Playful, colored pills | 56-80px |
|
||||
|
||||
## Word Grouping
|
||||
|
||||
- **High energy:** 2-3 words. Quick turnover.
|
||||
- **Conversational:** 3-5 words. Natural phrases.
|
||||
- **Measured/calm:** 4-6 words. Longer groups.
|
||||
|
||||
Break on sentence boundaries, 150ms+ pauses, or max word count.
|
||||
|
||||
## Positioning
|
||||
|
||||
- **Landscape (1920x1080):** Bottom 80-120px, centered
|
||||
- **Portrait (1080x1920):** Lower middle ~600-700px from bottom, centered
|
||||
- Never cover the subject's face
|
||||
- `position: absolute` — never relative
|
||||
- One caption group visible at a time
|
||||
|
||||
## Text Overflow Prevention
|
||||
|
||||
Use `window.__hyperframes.fitTextFontSize()`:
|
||||
|
||||
```js
|
||||
var result = window.__hyperframes.fitTextFontSize(group.text.toUpperCase(), {
|
||||
fontFamily: "Outfit",
|
||||
fontWeight: 900,
|
||||
maxWidth: 1600,
|
||||
});
|
||||
el.style.fontSize = result.fontSize + "px";
|
||||
```
|
||||
|
||||
Options: `maxWidth` (1600 landscape, 900 portrait), `baseFontSize` (78), `minFontSize` (42), `fontWeight`, `fontFamily`, `step` (2).
|
||||
|
||||
CSS safety nets: `max-width` on container, `overflow: visible` (**not** `hidden` — hidden clips scaled emphasis words and glow effects), `position: absolute`, explicit `height`. When per-word styling uses `scale > 1.0`, compute `maxWidth = safeWidth / maxScale` to leave headroom.
|
||||
|
||||
**Container pattern:** Full-width absolute container, centered. Do **not** use `left: 50%; transform: translateX(-50%)` — causes clipping at composition edges.
|
||||
|
||||
## Caption Exit Guarantee
|
||||
|
||||
Every group **must** have a hard kill after exit animation:
|
||||
|
||||
```js
|
||||
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
|
||||
// `tl.set` is an instant flip, not a tween — safe to set `visibility` here (core's "no animating
|
||||
// visibility" rule applies to tweens, which can't smoothly interpolate non-numeric values anyway).
|
||||
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
|
||||
```
|
||||
|
||||
Self-lint after building timeline — place **before** `window.__timelines[id] = tl` so it runs at composition init:
|
||||
|
||||
```js
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var el = document.getElementById("cg-" + gi);
|
||||
if (!el) return;
|
||||
tl.seek(group.end + 0.01);
|
||||
var computed = window.getComputedStyle(el);
|
||||
if (computed.opacity !== "0" && computed.visibility !== "hidden") {
|
||||
console.warn(
|
||||
"[caption-lint] group " + gi + " still visible at t=" + (group.end + 0.01).toFixed(2) + "s",
|
||||
);
|
||||
}
|
||||
});
|
||||
tl.seek(0);
|
||||
```
|
||||
|
||||
## Pre-Built Caption Components
|
||||
|
||||
Before building caption styles from scratch, check the registry — 15 ready-to-use caption components cover the most common styles. Install with `npx hyperframes add <name>` and wire as a sub-composition via `data-composition-src` (see `hyperframes-registry`).
|
||||
|
||||
```bash
|
||||
npx hyperframes catalog --tag caption-style # list all caption components
|
||||
npx hyperframes add caption-highlight # install a specific one
|
||||
```
|
||||
|
||||
| Style | Component | Best for |
|
||||
| ------------------------- | ---------------------------- | ---------------------------- |
|
||||
| TikTok-style highlight | `caption-highlight` | Social, high-energy |
|
||||
| Karaoke pill | `caption-pill-karaoke` | Music, lyric videos |
|
||||
| Cinematic editorial | `caption-editorial-emphasis` | Documentary, storytelling |
|
||||
| Glitch / cyber | `caption-glitch-rgb` | Tech, gaming |
|
||||
| Full-screen slam | `caption-kinetic-slam` | Hype, announcements |
|
||||
| Neon glow | `caption-neon-glow` | Night, club, neon aesthetics |
|
||||
| Neon accent (multi-color) | `caption-neon-accent` | Colorful, playful |
|
||||
| Wipe reveal | `caption-clip-wipe` | Clean, modern |
|
||||
| Gradient fill | `caption-gradient-fill` | Vibrant, eye-catching |
|
||||
| Matrix decode | `caption-matrix-decode` | Sci-fi, tech reveals |
|
||||
| Emoji pop | `caption-emoji-pop` | Social, casual |
|
||||
| Parallax layers | `caption-parallax-layers` | Depth, cinematic |
|
||||
| Particle burst | `caption-particle-burst` | Celebration, impact keywords |
|
||||
| Lava texture | `caption-texture` | Bold, dramatic |
|
||||
| Weight shift | `caption-weight-shift` | Elegant, typographic |
|
||||
|
||||
Related: `caption-blend-difference` (tagged `text` / `blend-mode`, not `caption-style`, so it won't appear under the filter above) auto-inverts text against any background via `mix-blend-mode: difference` — useful when the background is busy or unpredictable.
|
||||
|
||||
Browse all with previews: [hyperframes.heygen.com/catalog](https://hyperframes.heygen.com/catalog)
|
||||
|
||||
Caption components ship with transparent backgrounds — they're pure overlays. If the underlying video is bright or busy, add a contrast layer (e.g. a semi-transparent dark div) in the host composition beneath the caption sub-composition, not inside the component itself.
|
||||
|
||||
## Further References
|
||||
|
||||
- [`motion.md`](motion.md) — karaoke, marker effects, audio-reactive modulation, scatter exits.
|
||||
- [`transcript-handling.md`](transcript-handling.md) — input formats, quality checks, cleaning, external API fallback.
|
||||
- `hyperframes-animation/rules/css-marker-patterns.md` — marker highlighting (deterministic, fully seekable).
|
||||
|
||||
## Constraints
|
||||
|
||||
- Deterministic. No `Math.random()`, no `Date.now()`.
|
||||
- Sync to transcript timestamps.
|
||||
- One group visible at a time.
|
||||
- Every group must have a hard `tl.set` kill at `group.end`.
|
||||
- Fonts: the compiler auto-embeds only its **built-in mapped set** (Inter, Roboto, Montserrat, …) — for those, just declare `font-family` in CSS. Any **other** font (a brand/custom font like `TT Norms Pro`, or a non-Latin CJK/Devanagari family) is **not** auto-supplied: it needs an `@font-face` pointing at a real `.woff2` shipped with the project, or the text silently falls back to a generic font in the render. Don't assume a `font-family` you can see locally will render — the render machine is a clean headless Chrome with no installed fonts.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Dynamic Caption Techniques
|
||||
|
||||
You are here because SKILL.md told you to read this file before writing animation code. Pick your technique combination from the table below based on the energy level you detected from the transcript, then implement using standard GSAP patterns.
|
||||
|
||||
## Technique Selection by Energy
|
||||
|
||||
| Energy level | Highlight | Exit | Cycle pattern |
|
||||
| ------------ | ------------------------------------- | ------------------- | ----------------------------------------- |
|
||||
| High | Karaoke with accent glow + scale pop | Scatter or drop | Alternate highlight styles every 2 groups |
|
||||
| Medium-high | Karaoke with color pop | Scatter or collapse | Alternate every 3 groups |
|
||||
| Medium | Karaoke (subtle, white only) | Fade + slide | Alternate every 3 groups |
|
||||
| Medium-low | Karaoke (minimal scale change) | Fade | Single style, vary ease per group |
|
||||
| Low | Karaoke (warm tones, slow transition) | Collapse | Alternate every 4 groups |
|
||||
|
||||
**All energy levels use karaoke highlight as the baseline.** The difference is intensity — high energy gets accent color + glow + 15% scale pop on active words, low energy gets a gentle white shift with 3% scale.
|
||||
|
||||
**Emphasis words always break the pattern.** When a word is flagged as emphasis (emotional keyword, ALL CAPS, brand name), give it a stronger animation than surrounding words (larger scale, accent color, overshoot ease). This creates contrast.
|
||||
|
||||
**Marker highlight modes add a visual layer on top of karaoke.** For emphasis words that need more than color/scale, add a marker-style effect: highlight sweep, circle, burst, scribble, or sketchout. See `hyperframes-animation/rules/css-marker-patterns.md` for implementation details. Match mode to energy: burst for hype, circle for key terms, highlight for standard, scribble for subtle.
|
||||
|
||||
## Audio-Reactive Captions (Mandatory for Music)
|
||||
|
||||
**If the source audio is music (vocals over instrumentation, beats, any musical content), you MUST extract audio data and add audio-reactive animations.** This is not optional — music without audio reactivity looks disconnected. Even low-energy ballads get subtle bass pulse and treble glow.
|
||||
|
||||
No special wiring is needed. The group loop already iterates over every caption group to build entrance, karaoke, and exit tweens. At that point, read the audio data for each group's time range and use it to modulate the group's animation intensity with regular GSAP tweens.
|
||||
|
||||
```js
|
||||
// Load audio data inline (same pattern as TRANSCRIPT)
|
||||
var AUDIO = JSON.parse(audioDataJson); // { fps, totalFrames, frames: [{ bands: [...] }] }
|
||||
|
||||
GROUPS.forEach(function (group, gi) {
|
||||
var groupEl = document.getElementById("cg-" + gi);
|
||||
if (!groupEl) return;
|
||||
|
||||
// Read peak energy for this group's time range
|
||||
var startFrame = Math.floor(group.start * AUDIO.fps);
|
||||
var endFrame = Math.min(Math.floor(group.end * AUDIO.fps), AUDIO.totalFrames - 1);
|
||||
var peakBass = 0;
|
||||
var peakTreble = 0;
|
||||
for (var f = startFrame; f <= endFrame; f++) {
|
||||
var frame = AUDIO.frames[f];
|
||||
if (!frame) continue;
|
||||
peakBass = Math.max(peakBass, frame.bands[0] || 0, frame.bands[1] || 0);
|
||||
peakTreble = Math.max(peakTreble, frame.bands[6] || 0, frame.bands[7] || 0);
|
||||
}
|
||||
|
||||
// Modulate entrance — louder groups enter bigger and glowier
|
||||
tl.to(
|
||||
groupEl,
|
||||
{
|
||||
scale: 1 + peakBass * 0.06,
|
||||
textShadow:
|
||||
"0 0 " + Math.round(peakTreble * 12) + "px rgba(255,255,255," + peakTreble * 0.4 + ")",
|
||||
duration: 0.3,
|
||||
ease: "power2.out",
|
||||
},
|
||||
group.start,
|
||||
);
|
||||
|
||||
// Reset at exit so audio-driven values don't persist
|
||||
tl.set(groupEl, { scale: 1, textShadow: "none" }, group.end - 0.15);
|
||||
});
|
||||
```
|
||||
|
||||
This shapes the animation at build time, not playback time — no per-frame callbacks, no `tl.call()` loops, no async fetch timing issues. Loud groups come in with more weight and glow; quiet groups come in soft. The audio data modulates _how much_, the content determines _what_.
|
||||
|
||||
Keep audio reactivity subtle — 3-6% scale variation and soft glow. Heavy pulsing makes text unreadable.
|
||||
|
||||
To generate the audio data file:
|
||||
|
||||
```bash
|
||||
python3 skills/hyperframes-creative/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json
|
||||
```
|
||||
|
||||
## Combining Techniques
|
||||
|
||||
Don't use the same highlight animation on every group — cycle through styles using the group index. Don't combine multiple competing animations on the same word at the same timestamp. Vary techniques across groups to match the content's pace changes.
|
||||
|
||||
**Marker highlight effects** layer well with karaoke — use karaoke for the word-by-word reveal, then add a marker effect on emphasis words only. For example: karaoke highlights each word in white, but brand names get a yellow highlight sweep and stats get a red circle. Cycle marker modes across groups for visual variety.
|
||||
|
||||
## Runtime Tools
|
||||
|
||||
Caption motion uses standard HyperFrames runtime APIs. Use the canonical sources:
|
||||
|
||||
- **GSAP timeline + tween syntax** — `hyperframes-animation/adapters/gsap.md` (eases, position parameter, performance)
|
||||
- **`window.__hyperframes.fitTextFontSize` / `pretext`** — `hyperframes-core/references/determinism-rules.md` → Layout Contract (overflow prevention, per-frame text measurement)
|
||||
- **Audio data extraction** — generate via `python3 skills/hyperframes-creative/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json`, then load inline as shown in "Audio-Reactive Captions" above
|
||||
@@ -0,0 +1,97 @@
|
||||
# Transcript Guide
|
||||
|
||||
For the `transcribe` CLI invocation, the `.en`-translates-non-English rule, and whisper model selection, see [`../transcribe.md`](../transcribe.md). This file covers what to do with the resulting transcript when authoring captions: input formats, mandatory quality checks, cleaning code, external-API fallbacks.
|
||||
|
||||
## Supported Input Formats
|
||||
|
||||
The CLI auto-detects and normalizes these formats:
|
||||
|
||||
| Format | Extension | Source | Word-level? |
|
||||
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
|
||||
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
|
||||
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
|
||||
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
|
||||
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
|
||||
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
|
||||
|
||||
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
|
||||
|
||||
## Transcript Quality Check (Mandatory)
|
||||
|
||||
After every transcription, **read the transcript and check for quality issues before proceeding.** Bad transcripts produce nonsensical captions. Never skip this step.
|
||||
|
||||
### What to look for
|
||||
|
||||
| Signal | Example | Cause |
|
||||
| ---------------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| Music note tokens (`♪`, `�`) | `{ "text": "♪" }` or `{ "text": "�" }` | Whisper detected music, not speech |
|
||||
| Garbled / nonsense words | "Do a chin", "Get so gay", "huh" | Model misheard lyrics or background noise |
|
||||
| Long gaps with no words | 20+ seconds of only `♪` tokens | Instrumental section — expected, but high ratio means speech is being missed |
|
||||
| Repeated filler | Many "huh", "uh", "oh" entries | Model is hallucinating on music |
|
||||
| Very short word spans | Words with `end - start < 0.05` | Unreliable timestamp alignment |
|
||||
|
||||
### Automatic retry rules
|
||||
|
||||
**If more than 20% of entries are `♪`/`�` tokens, or the transcript contains obvious nonsense words, the transcription failed.** Do not proceed with the bad transcript. Instead:
|
||||
|
||||
1. **Retry with `medium.en`** if the original used `small.en` or smaller:
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model medium.en
|
||||
```
|
||||
2. **If `medium.en` also fails** (still >20% music tokens or garbled), tell the user the audio is too noisy for local transcription and suggest:
|
||||
- Providing lyrics manually as an SRT/VTT file
|
||||
- Using an external API (OpenAI or Groq Whisper — see below)
|
||||
3. **Always clean the transcript** before building captions — filter out `♪`/`�` tokens and entries where `text` is a single non-word character. Only real words should reach the caption composition.
|
||||
|
||||
### Cleaning a transcript
|
||||
|
||||
After transcription (even with a good model), strip non-word entries:
|
||||
|
||||
```js
|
||||
var raw = JSON.parse(transcriptJson);
|
||||
var words = raw.filter(function (w) {
|
||||
if (!w.text || w.text.trim().length === 0) return false;
|
||||
if (/^[♪�\u266a\u266b\u266c\u266d\u266e\u266f]+$/.test(w.text)) return false;
|
||||
if (/^(huh|uh|um|ah|oh)$/i.test(w.text) && w.end - w.start < 0.1) return false;
|
||||
return true;
|
||||
});
|
||||
```
|
||||
|
||||
For model-selection guidance by content type, see [`../transcribe.md`](../transcribe.md) → "Picking a model by content type".
|
||||
|
||||
## Using External Transcription APIs
|
||||
|
||||
For the best accuracy, use an external API and import the result:
|
||||
|
||||
**OpenAI Whisper API** (recommended for quality):
|
||||
|
||||
```bash
|
||||
# Generate with word timestamps, then import
|
||||
curl https://api.openai.com/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-1 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-openai.json
|
||||
|
||||
npx hyperframes transcribe transcript-openai.json
|
||||
```
|
||||
|
||||
**Groq Whisper API** (fast, free tier available):
|
||||
|
||||
```bash
|
||||
curl https://api.groq.com/openai/v1/audio/transcriptions \
|
||||
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||
-F file=@audio.mp3 -F model=whisper-large-v3 \
|
||||
-F response_format=verbose_json \
|
||||
-F "timestamp_granularities[]=word" \
|
||||
-o transcript-groq.json
|
||||
|
||||
npx hyperframes transcribe transcript-groq.json
|
||||
```
|
||||
|
||||
## If No Transcript Exists
|
||||
|
||||
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files.
|
||||
2. If none found, run [`../transcribe.md`](../transcribe.md) — pick the starting model from "Picking a model by content type" there.
|
||||
3. Run the quality check above. If it fails, retry with a larger model or fall back to manual lyrics / external API.
|
||||
@@ -0,0 +1,143 @@
|
||||
# Background Removal
|
||||
|
||||
Make a transparent overlay (typical: a talking head over an arbitrary scene). Uses `u2net_human_seg` (MIT).
|
||||
|
||||
```bash
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm # default: VP9 + alpha
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.mov # ProRes 4444 (editing)
|
||||
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
|
||||
npx hyperframes remove-background subject.mp4 -o subject.webm \
|
||||
--background-output plate.webm # both layers, one pass
|
||||
npx hyperframes remove-background subject.mp4 -o transparent.webm --device cpu
|
||||
npx hyperframes remove-background --info # detected providers
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
- **`.webm` (VP9 alpha)** — default. Plug straight into `<video>` for Chrome-native transparent playback (~1 MB / 4s @ 1080p).
|
||||
- **`.mov` (ProRes 4444)** — round-trip in editors (Premiere / Resolve / DaVinci). ~50 MB / 4s.
|
||||
- **`.png`** — single-image cutout.
|
||||
|
||||
## Quality (`--quality`)
|
||||
|
||||
Controls VP9 encoder CRF only — segmentation quality is fixed. Higher quality keeps the cutout's RGB closer to the source MP4 (important when overlaying the cutout on its own source).
|
||||
|
||||
| Preset | CRF | When |
|
||||
| ---------- | --- | --------------------------------------------- |
|
||||
| `fast` | 30 | Iterating, smaller files, looser color match |
|
||||
| `balanced` | 18 | **Default**; visually identical for most uses |
|
||||
| `best` | 12 | Master / final delivery, tightest color match |
|
||||
|
||||
## Device (`--device`)
|
||||
|
||||
`auto` (default) picks CoreML on Apple Silicon, CUDA when available, otherwise CPU. Force with `--device cpu | coreml | cuda`. CUDA requires `HYPERFRAMES_CUDA=1` plus a GPU-enabled `onnxruntime-node` build. Use `--info` to inspect detected providers without rendering.
|
||||
|
||||
## Compositing patterns — pick the right one
|
||||
|
||||
The cutout WebM is a **re-encoded copy** of the source MP4's RGB. What sits behind it matters.
|
||||
|
||||
| Pattern | Behind the cutout | Result |
|
||||
| -------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
||||
| **Cutout over a different scene** (most common) | Static image, gradient, unrelated video | Looks great. Single RGB source for the subject. |
|
||||
| **Cutout over its own source mp4** (text-behind-subject) | Same mp4 the cutout came from | At `balanced` doubling is barely visible; at `fast` you'll see color shift / edge halo. Use `best` for masters. |
|
||||
| **Cutout over a different take of the same person** | Footage of the same subject | **Two overlapping people. Don't do this.** |
|
||||
|
||||
## Text-behind-subject pattern (two non-obvious rules)
|
||||
|
||||
Putting a headline behind a presenter cutout:
|
||||
|
||||
```html
|
||||
<video
|
||||
src="presenter.mp4"
|
||||
id="bg"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
|
||||
|
||||
<div class="cutout-wrap" style="position:absolute; inset:0; z-index:3; opacity:0">
|
||||
<video
|
||||
src="presenter.webm"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="1"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
```js
|
||||
// Flip the wrapper's opacity at the cut, NOT the video's
|
||||
tl.set(".cutout-wrap", { opacity: 1 }, 3.3);
|
||||
```
|
||||
|
||||
Two rules that are easy to miss:
|
||||
|
||||
1. **Wrap the cutout `<video>` in a non-timed `<div>` and animate the wrapper's opacity, not the video element's.** The framework forces `opacity: 1` on active clips (any element with `data-start` / `data-duration`), so animating the video's opacity directly is silently overridden. The wrapper has no `data-*` attributes, so it's owned by your CSS / GSAP.
|
||||
2. **Both videos use `data-start="0"` and `data-media-start="0"`** so the framework decodes them in sync from t=0. Late-mounting the cutout (`data-start=3.3`) introduces a seek + warm-up that lands a frame off the base mp4 — visible as one frame of misalignment at the cut.
|
||||
|
||||
## Layer separation (`--background-output`)
|
||||
|
||||
Emits a **second** transparent video alongside the cutout: same source RGB, alpha is `255 - mask` instead of `mask`. The cutout has the subject opaque; the plate has the surroundings opaque (with a transparent hole where the subject was). Use it when text / graphics need to live **between** the two layers.
|
||||
|
||||
| File | Alpha is… | Use it for |
|
||||
| -------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------- |
|
||||
| `-o subject.webm` | mask — subject opaque, background transparent | Foreground layer (top) |
|
||||
| `--background-output plate.webm` | inverse mask — surroundings opaque, subject transparent | Bottom layer; place text / graphics between this and the subject |
|
||||
|
||||
Both share the same `--quality` and run from a single inference pass — only encode cost roughly doubles. Only valid for video inputs with `.webm` / `.mov` outputs.
|
||||
|
||||
**Hole-cut, not inpainted.** The subject region in `plate.webm` is fully transparent — composite something opaque under it to fill the hole.
|
||||
|
||||
**Single test for whether `--background-output` is the right tool:** _will anything ever be visible through the subject's silhouette where the subject used to be?_ If no, you don't need the plate — `subject.webm` alone over a different background is enough.
|
||||
|
||||
### Use case → right tool
|
||||
|
||||
| Use case | Right tool |
|
||||
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
|
||||
| Text/graphics between the cutout and the plate (this command's reason for existing) | **Hole-cut** (`--background-output`) |
|
||||
| Subject onto an unrelated scene | Just `subject.webm`; ignore the plate |
|
||||
| Show the room _without_ the person, alone over no other content | **Clean plate** — needs an inpainter (LaMa, ProPainter, E2FGVI). Not this command. |
|
||||
| Replace the subject with a different subject | **Clean plate** — same as above |
|
||||
|
||||
### Canonical 3-layer template (plate + content + cutout)
|
||||
|
||||
Ship just the two transparent layers and let arbitrary content live between them — no original mp4 needed:
|
||||
|
||||
```html
|
||||
<!-- z=1 plate: surroundings opaque, subject silhouette transparent -->
|
||||
<video
|
||||
src="plate.webm"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="0"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
|
||||
<!-- z=2 your content lives between the layers -->
|
||||
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
|
||||
|
||||
<!-- z=3 cutout floats the subject back on top -->
|
||||
<div class="cutout-wrap" style="position:absolute; inset:0; z-index:3">
|
||||
<video
|
||||
src="subject.webm"
|
||||
data-start="0"
|
||||
data-duration="6"
|
||||
data-track-index="1"
|
||||
muted
|
||||
playsinline
|
||||
></video>
|
||||
</div>
|
||||
```
|
||||
|
||||
Functionally equivalent to the text-behind-subject pattern above, but doesn't require shipping the original mp4 — the plate replaces it. Use this when delivering just the two transparent layers as a reusable asset.
|
||||
|
||||
## When `remove-background` is NOT the right tool
|
||||
|
||||
If a user asks for "the room **without** the person, displayed standalone" (no subject anywhere, no compositing on top), `--background-output` is wrong — its plate has a transparent hole, not a filled-in clean plate. They need an **inpainter**: LaMa, ProPainter, or E2FGVI. Tell them this command can't do it.
|
||||
@@ -0,0 +1,29 @@
|
||||
# Requirements & Caches
|
||||
|
||||
## Credential & key priority
|
||||
|
||||
Run `npx hyperframes auth status` to see what's configured and which engines a workflow will use (see the skill's **Preflight** section). Keys resolve in this order — **first match wins**:
|
||||
|
||||
| Provider | Resolution order (first non-empty wins) | Local deps when used |
|
||||
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
|
||||
| **HeyGen** (TTS + BGM/SFX retrieval) | `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; `$HEYGEN_CONFIG_DIR` overrides the dir; written by `hyperframes auth login`) | none (REST) |
|
||||
| **ElevenLabs** (TTS fallback) | `$ELEVENLABS_API_KEY` | `pip install elevenlabs` |
|
||||
| **Lyria** (BGM fallback) | `$GEMINI_API_KEY` → `$GOOGLE_API_KEY` | `pip install google-genai` |
|
||||
| **Kokoro** (TTS, no key) | always — final voice fallback | `pip install kokoro-onnx soundfile` |
|
||||
| **MusicGen** (BGM, no key) | always — final music fallback | `pip install transformers torch soundfile numpy` |
|
||||
|
||||
`hyperframes auth login` (browser OAuth) is the recommended setup: one sign-in, every project, no per-repo `.env`. An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both are tagged with `X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance for HeyGen TTS (10 min/month); API keys follow the normal API billing path. With no HeyGen credential, voice/BGM run fully locally (Kokoro / MusicGen) — `hyperframes auth status` and `hyperframes doctor` both report whether those local deps are installed.
|
||||
|
||||
## Model caches & system dependencies
|
||||
|
||||
Each command downloads its own model on first run and caches it under `~/.cache/hyperframes/`:
|
||||
|
||||
- **TTS (HeyGen)** — no local deps; needs a HeyGen credential + `ffmpeg` on PATH (to transcode the mp3 response to `.wav`). Credential resolves like the CLI: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; run `npx hyperframes auth login`). An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both include `X-HeyGen-Source: cli` so the backend can apply CLI OAuth free usage.
|
||||
- **TTS (ElevenLabs)** — same as HeyGen: API key + `ffmpeg`.
|
||||
- **TTS (Kokoro)** — Kokoro-82M (~311 MB) + voices (~27 MB) in `tts/`. Requires Python 3.8+ with `kokoro-onnx` and `soundfile` (`pip install kokoro-onnx soundfile`). Non-English text also needs `espeak-ng` system-wide.
|
||||
- **BGM (Lyria)** — needs `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `pip install google-genai`. No local model cache.
|
||||
- **BGM (MusicGen)** — `pip install transformers torch soundfile`. `facebook/musicgen-small` (~300 MB) cached under `~/.cache/huggingface/` on first run.
|
||||
- **Transcribe** — Whisper model size depending on choice (75 MB – 3.1 GB) in `whisper/`, downloaded from HuggingFace on first use. `whisper.cpp` itself is NOT bundled: the CLI resolves it from PATH, installs via Homebrew (macOS), or builds it from source with git+cmake on first use (`$HYPERFRAMES_WHISPER_PATH` overrides).
|
||||
- **Remove-background** — `u2net_human_seg` (~168 MB ONNX) in `background-removal/models/`. Peak inference RAM ~1.5 GB.
|
||||
|
||||
Run `npx hyperframes doctor` if a command fails because of a missing dependency.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Sound effects (SFX)
|
||||
|
||||
Named sound effects, produced by the shared audio engine (`scripts/audio.mjs` → `scripts/lib/sfx.mjs`). **Provider-gated** by the engine's one switch — whether a HeyGen credential is present, decided once (not per cue):
|
||||
|
||||
- **HeyGen credential present → retrieve every cue** from HeyGen's audio library (`/v3/audio/sounds`, `type=sound_effects`, `min_score=0.4`). Search-and-download, **not** generation. The bundled library is NOT consulted.
|
||||
- **No credential → the bundled 21-file library** (`assets/sfx/` + `manifest.json`): match each cue name, copy the matched file into the project. Offline, deterministic, free.
|
||||
|
||||
There is no `npx hyperframes sfx` command. SFX is never generated — it is retrieved (online) or taken from the bundled library (offline).
|
||||
|
||||
## Cues — request → meta
|
||||
|
||||
Each line names the effects it wants: `lines[].sfx: ["whoosh", "ui click"]`. The engine flattens these into cues, resolves them per the switch, dedupes identical `(id, name)` pairs (the same effect named twice downloads/copies once), and writes `audio_meta.sfx[]`:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "3", // joins the cue to the caller's model (frame / scene / segment)
|
||||
"name": "whoosh",
|
||||
"file": "assets/sfx/whoosh.mp3", // downloaded or copied, relative to project root
|
||||
"source": "heygen" | "local", // which route resolved it
|
||||
"offset_s": 0, // delay from the line's start
|
||||
"duration_s": 0.57,
|
||||
"volume": 0.35 // SFX sit UNDER voice + BGM
|
||||
}
|
||||
```
|
||||
|
||||
A cue that matches nothing is **skipped** (recorded as an anomaly); SFX never blocks a render.
|
||||
|
||||
## HeyGen retrieval (credentialed)
|
||||
|
||||
`searchSounds(name, "sound_effects", { limit: 3, minScore: 0.4 })` → top hit → `assets/sfx/<slug>.mp3`. Results are ranked by `score` (each carries a presigned `audio_url`, `duration`, `description`). The floor is **0.4** because good SFX hits score ~0.5–0.67 — below the API's default `0.7`, which would silently drop most named cues (only whoosh/swoosh-family clears 0.7). `duration_s` comes from the result (else 1.0). Name effects concretely (`glass shatter`, not `dramatic sound`); a vague query returns a poor match.
|
||||
|
||||
## Bundled library (no credential)
|
||||
|
||||
21 curated files in `assets/sfx/`, indexed by `manifest.json` — `{ file, duration, description }` per key (e.g. `whoosh`, `pop`, `click`, `chime`, `riser`, `impact-bass-1`, `glitch-1`, `typing`, …). A cue name resolves by **manifest key, file basename, or slug**, so `whoosh`, `whoosh.mp3`, or `"ui click"` (→ slug) all match. Matched files are copied into the project's `assets/sfx/`; `duration_s` comes from the manifest, so timing is known **offline** — e.g. `riser` is 10.03s, so trigger it at `climax − 10.03s`. The manifest's `description` field carries placement hints per effect; read `assets/sfx/manifest.json` for the full set and usage.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Volume ~0.35.** SFX must sit under narration and BGM, not fight them.
|
||||
- **No match → skip, don't fail.** A missing effect logs an anomaly and moves on; never a render blocker.
|
||||
- **Retrieval (credentialed) or bundled library (offline) — never generation.** You search HeyGen by text, or match a name against the 21-file manifest.
|
||||
- **One asset per distinct name.** Reuse across lines is deduped to a single download/copy, many cues.
|
||||
- **The switch is global, not per cue.** With a credential, retrieval handles even the long tail (effects not in the 21); without one, only the 21 bundled names resolve.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Transcription
|
||||
|
||||
Create normalized word-level timestamps. **Always specify `--model` explicitly** — the CLI default is `small.en`, which silently translates non-English audio into English.
|
||||
|
||||
```bash
|
||||
npx hyperframes transcribe audio.mp3 --model small.en # known English
|
||||
npx hyperframes transcribe video.mp4 --model small --language es # known Spanish
|
||||
npx hyperframes transcribe audio.mp3 --model small # unknown language (auto-detect)
|
||||
npx hyperframes transcribe subtitles.srt # import existing
|
||||
npx hyperframes transcribe subtitles.vtt
|
||||
npx hyperframes transcribe openai-response.json
|
||||
```
|
||||
|
||||
## Language Rule (Non-Negotiable)
|
||||
|
||||
`.en` models (`tiny.en` / `base.en` / `small.en` / `medium.en`) **translate** non-English audio into English. This silently destroys the original language.
|
||||
|
||||
1. **Known English** → `--model small.en` (or `medium.en` for music / noisy audio)
|
||||
2. **Known non-English** → `--model small --language <iso-code>` (no `.en` suffix)
|
||||
3. **Unknown language** → `--model small` (whisper auto-detects)
|
||||
|
||||
**CLI default is `small.en`** — do not rely on it; always pass `--model` to make the choice explicit. `--language` also filters out non-target-language segments from mixed-language audio.
|
||||
|
||||
## Model Sizes
|
||||
|
||||
| Model | Size | Speed | When |
|
||||
| ---------- | ------ | -------- | ------------------------------------- |
|
||||
| `tiny` | 75 MB | Fastest | Quick previews, smoke tests |
|
||||
| `base` | 142 MB | Fast | Short clips, clear audio |
|
||||
| `small` | 466 MB | Moderate | Default for most multilingual content |
|
||||
| `medium` | 1.5 GB | Slow | Music with vocals, noisy audio |
|
||||
| `large-v3` | 3.1 GB | Slowest | Production quality |
|
||||
|
||||
### Picking a model by content type
|
||||
|
||||
1. Speech over silence / light background → `small.en`
|
||||
2. Speech over music, or music with vocals → start with `medium.en`
|
||||
3. Produced music track (vocals + full instrumentation) → start with `medium.en`; expect to need manual lyrics or an external API ([`captions/transcript-handling.md`](captions/transcript-handling.md) → "Using External Transcription APIs")
|
||||
4. Multilingual → `medium` or `large-v3` (no `.en` suffix), pair with `--language`
|
||||
|
||||
## Output Shape
|
||||
|
||||
Compositions consume a flat array of word objects. The `id` (`w0`, `w1`, …) is added during normalization for stable references in caption overrides; optional for backwards compatibility.
|
||||
|
||||
```json
|
||||
[
|
||||
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
|
||||
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
|
||||
]
|
||||
```
|
||||
|
||||
For mandatory caption-quality checks, retry rules, and the OpenAI/Groq Whisper API import path, see `captions/transcript-handling.md`.
|
||||
@@ -0,0 +1,26 @@
|
||||
# TTS → Captions
|
||||
|
||||
When no recorded voiceover exists, generate one and obtain word-level caption timing. Two paths depending on which TTS provider is in use:
|
||||
|
||||
## Path A — HeyGen (single call, no Whisper)
|
||||
|
||||
HeyGen returns word timestamps in the same response as the audio. Use the
|
||||
bundled REST helper (the `hyperframes tts` command is Kokoro-only):
|
||||
|
||||
```bash
|
||||
node skills/media-use/audio/scripts/heygen-tts.mjs \
|
||||
script.txt --output narration.wav --words narration.words.json
|
||||
```
|
||||
|
||||
`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.
|
||||
|
||||
## Path B — ElevenLabs / Kokoro (TTS → Whisper)
|
||||
|
||||
These providers don't return word data. Generate the audio, then transcribe:
|
||||
|
||||
```bash
|
||||
npx hyperframes tts script.txt --voice af_heart --output narration.wav
|
||||
npx hyperframes transcribe narration.wav --model small.en # voice af_heart is American English
|
||||
```
|
||||
|
||||
Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
|
||||
@@ -0,0 +1,165 @@
|
||||
# Text To Speech
|
||||
|
||||
`npx hyperframes tts` synthesizes locally with Kokoro. It does not accept a
|
||||
`--provider` or `--words` flag. For HeyGen audio plus word timestamps, use the
|
||||
bundled `heygen-tts.mjs` script below.
|
||||
|
||||
> **Run the Preflight first — no credential is not a green light to silently use the local voice.** Before generating a voiceover, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen voices, or continue offline with local Kokoro). This applies to a one-off "generate a voiceover" request just as much as inside a full workflow.
|
||||
|
||||
## Narrating a HyperFrames docs video
|
||||
|
||||
Anything that will sit on hyperframes.heygen.com uses one narrator, so the site
|
||||
does not sound like two products.
|
||||
|
||||
| | |
|
||||
| -------------- | ---------------------------------------------- |
|
||||
| Voice | **River** — "Relaxed, Neutral, Informative" |
|
||||
| Provider | ElevenLabs |
|
||||
| `voice_id` | `SAz9YHcvj6GT2YYXdXww` |
|
||||
| Model | `eleven_multilingual_v2` |
|
||||
| Pace | 145–155 words per minute, with room to breathe |
|
||||
| Music under it | about −31 LUFS, never masking the voice |
|
||||
|
||||
```bash
|
||||
curl -s -X POST "https://api.elevenlabs.io/v1/text-to-speech/SAz9YHcvj6GT2YYXdXww" \
|
||||
-H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
|
||||
-d '{"text":"...","model_id":"eleven_multilingual_v2"}' -o take.mp3
|
||||
```
|
||||
|
||||
This is the voice every user-journey film on the docs site already uses. Falling
|
||||
back to local Kokoro because a key was not to hand produces a film that sounds
|
||||
wrong beside the others — three docs videos were built that way and had to be
|
||||
re-voiced. If you cannot reach ElevenLabs, say so and stop rather than
|
||||
substituting a different voice.
|
||||
|
||||
Use another voice only for a documented reason, and write the reason down.
|
||||
|
||||
## Available routes
|
||||
|
||||
| Order | Provider | Env trigger | Voice IDs | Word timestamps | Audio format |
|
||||
| ----- | ----------------- | ------------------------------------------- | ------------------------------------------- | ----------------------------------------- | -------------------- |
|
||||
| 1 | HeyGen (Starfish) | `$HEYGEN_API_KEY` / `~/.heygen/credentials` | UUIDs from `GET /v3/voices?engine=starfish` | **Yes** (`word_timestamps[]` in response) | mp3 → wav via ffmpeg |
|
||||
| 2 | ElevenLabs | `$ELEVENLABS_API_KEY` | UUIDs from elevenlabs.io dashboard | No | mp3 → wav via ffmpeg |
|
||||
| 3 | Kokoro-82M | always (local fallback) | `am_michael`, `af_heart`, … (54 voices) | No | wav direct |
|
||||
|
||||
```bash
|
||||
# Local Kokoro CLI
|
||||
npx hyperframes tts "Welcome to HyperFrames" -o narration.wav
|
||||
```
|
||||
|
||||
## Self-contained HeyGen (no CLI) — `scripts/heygen-tts.mjs`
|
||||
|
||||
The published `hyperframes tts` CLI synthesizes locally with Kokoro only. When you
|
||||
want HeyGen specifically — best quality **plus** word timestamps in one call — use
|
||||
the skill's bundled script, which calls the HeyGen v3 REST API directly and needs
|
||||
no CLI provider plumbing:
|
||||
|
||||
The script resolves a HeyGen credential the same way the CLI does — first source
|
||||
wins: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → a project `.env` (auto-loaded,
|
||||
walks up ≤5 dirs) → `~/.heygen/credentials` (shared with heygen-cli;
|
||||
`$HEYGEN_CONFIG_DIR` overrides the dir). An OAuth login is sent as
|
||||
`Authorization: Bearer`; an API key as `X-Api-Key`; both include
|
||||
`X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance
|
||||
(10 min/month) before paid usage; API keys follow normal API billing. If the
|
||||
only credential is an expired OAuth token it stops with a hint to run
|
||||
`npx hyperframes auth refresh`.
|
||||
|
||||
```bash
|
||||
# Only needed if you haven't run `npx hyperframes auth login`:
|
||||
export HEYGEN_API_KEY=... # or put it in a project .env
|
||||
|
||||
# Synthesize + capture word timestamps in one call (skips a Whisper pass)
|
||||
node skills/media-use/audio/scripts/heygen-tts.mjs \
|
||||
"Welcome to HyperFrames." -o narration.wav --words narration.words.json
|
||||
|
||||
node skills/media-use/audio/scripts/heygen-tts.mjs ./script.txt -o narration.wav
|
||||
node skills/media-use/audio/scripts/heygen-tts.mjs --list # public starfish voices
|
||||
```
|
||||
|
||||
- **Voice:** `--voice <id>` must be a **starfish** voice_id (`--list`, or `GET /v3/voices?engine=starfish`). v2-catalog ids are rejected with HTTP 400. Omit `--voice` (English) and it defaults to **Marcia** (`05f19352e8f74b0392a8f411eba40de1`, a fixed default so the choice is deterministic). Non-English with no `--voice` falls back to the first matching catalog voice.
|
||||
- **Output:** `.wav` → transcoded to 44.1k mono via ffmpeg; `.mp3` → raw bytes (no ffmpeg needed).
|
||||
- **Words:** `--words <path>` writes the flat `[{id,text,start,end}]` shape below, drop-in for the captions pipeline. HeyGen's `<start>`/`<end>` boundary sentinels are filtered out and ids are re-contiguous.
|
||||
- **Non-English:** `--lang <code>` (anything but `en`) is sent as the request `language`.
|
||||
|
||||
## When to use which provider
|
||||
|
||||
| Goal | Use |
|
||||
| --------------------------------------------------------- | --------------------------------------------------- |
|
||||
| Best voice quality + word timestamps in one call | **HeyGen** |
|
||||
| Drop-in cloud TTS, big voice catalog | **ElevenLabs** |
|
||||
| Offline, no API key, fast iteration | **Kokoro** |
|
||||
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |
|
||||
|
||||
## ffmpeg requirement
|
||||
|
||||
HeyGen + ElevenLabs return mp3. The bundled HeyGen helper transcodes to wav
|
||||
when `--output` ends in `.wav` (the default and what downstream `ffprobe` +
|
||||
Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`.
|
||||
Without `ffmpeg` on PATH, wav output from cloud providers fails; the local
|
||||
Kokoro CLI writes wav directly.
|
||||
|
||||
## Voice selection (Kokoro)
|
||||
|
||||
Default `af_heart`. Curated picks:
|
||||
|
||||
| Content type | Voice |
|
||||
| ----------------- | ---------------------- |
|
||||
| Product demo | `af_heart`, `af_nova` |
|
||||
| Tutorial / how-to | `am_adam`, `bf_emma` |
|
||||
| Marketing / promo | `af_sky`, `am_michael` |
|
||||
| Documentation | `bf_emma`, `bm_george` |
|
||||
| Casual / social | `af_heart`, `af_sky` |
|
||||
|
||||
Run `npx hyperframes tts --list` for the bundled set.
|
||||
|
||||
## Multilingual (Kokoro voice prefix → language)
|
||||
|
||||
The first letter of a Kokoro voice ID picks the phonemizer language; `--lang` overrides auto-detection.
|
||||
|
||||
| Prefix | Language |
|
||||
| ------ | -------------------- |
|
||||
| `a` | American English |
|
||||
| `b` | British English |
|
||||
| `e` | Spanish |
|
||||
| `f` | French |
|
||||
| `h` | Hindi |
|
||||
| `i` | Italian |
|
||||
| `j` | Japanese |
|
||||
| `p` | Brazilian Portuguese |
|
||||
| `z` | Mandarin |
|
||||
|
||||
```bash
|
||||
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora
|
||||
npx hyperframes tts "Today is a nice day" --voice af_heart
|
||||
```
|
||||
|
||||
Valid `--lang` codes (only needed to override the voice's auto-detected language): `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`.
|
||||
|
||||
Non-English phonemization requires `espeak-ng` system-wide (`brew install espeak-ng` / `apt-get install espeak-ng`).
|
||||
|
||||
## Speed
|
||||
|
||||
- `0.7-0.8` — tutorial, complex content, accessibility
|
||||
- `1.0` — natural pace (default)
|
||||
- `1.1-1.2` — intros, transitions, upbeat content
|
||||
- `1.5+` — rarely appropriate, test carefully
|
||||
|
||||
The `hyperframes tts` command honors `--speed` for Kokoro. Provider-specific
|
||||
helpers document their own pacing controls.
|
||||
|
||||
## Long scripts
|
||||
|
||||
Past a few paragraphs, write the text to a `.txt` file and pass the path. Inputs over ~5 minutes of speech may benefit from splitting into segments.
|
||||
|
||||
## HeyGen word-timestamp shape
|
||||
|
||||
When `--words <path>` is passed to a HeyGen call, the file is written in the same flat shape `transcribe` produces — drop-in compatible with the captions pipeline:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "id": "w0", "text": "Hi", "start": 0.0, "end": 0.21 },
|
||||
{ "id": "w1", "text": "there", "start": 0.22, "end": 0.55 }
|
||||
]
|
||||
```
|
||||
|
||||
For ElevenLabs / Kokoro, run `npx hyperframes transcribe narration.wav --model small.en` to get the same shape.
|
||||
@@ -0,0 +1,295 @@
|
||||
#!/usr/bin/env node
|
||||
// audio.mjs — the shared HyperFrames audio engine. ONE implementation of TTS +
|
||||
// BGM + SFX for every video workflow (product-launch, general-video, pr-to-video,
|
||||
// …). Workflows do NOT vendor a copy: they write a neutral `audio_request.json`
|
||||
// (a tiny per-workflow adapter maps their storyboard/scenes into it) and call:
|
||||
//
|
||||
// node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json
|
||||
//
|
||||
// The three capabilities degrade on ONE switch — whether HeyGen is configured
|
||||
// (credential present, NOT the CLI). This mirrors the table in ../SKILL.md:
|
||||
//
|
||||
// TTS : HeyGen REST → ElevenLabs → Kokoro (CLI)
|
||||
// BGM : HeyGen retrieve → (no credential) Lyria/MusicGen generate
|
||||
// SFX : HeyGen retrieve → (no credential) bundled 19-file library
|
||||
//
|
||||
// ── audio_request.json (input) ────────────────────────────────────────────────
|
||||
// {
|
||||
// "provider": "auto", // auto|heygen|elevenlabs|kokoro (override: --provider)
|
||||
// "lang": "en", "speed": 1.0,
|
||||
// "lines": [ // one TTS unit each; id joins back to the caller's model
|
||||
// { "id": "01", "text": "...", "sfx": ["whoosh", "ui click"] }
|
||||
// ],
|
||||
// "bgm": { "mode": "retrieve", // retrieve|generate|none (override: --bgm-mode / --no-bgm)
|
||||
// "query": "calm cinematic underscore", // mood for retrieval
|
||||
// "prompt": null, // full prompt for generation (else inferred)
|
||||
// "blob": "...", "archetype": "...", "arc": "..." } // optional mood-inference hints
|
||||
// }
|
||||
//
|
||||
// ── audio_meta.json (output, id-keyed) ───────────────────────────────────────
|
||||
// { tts_provider, voice_id,
|
||||
// bgm: { path, volume, mode, query?, duration_s? } | null,
|
||||
// bgm_pending, bgm_provider, bgm_pid, bgm_log, bgm_mode, bgm_target_duration_s, …,
|
||||
// voices: [ { id, path, duration_s, words: [{id,text,start,end}] } ],
|
||||
// sfx: [ { id, name, file, source, offset_s, duration_s, volume } ],
|
||||
// total_duration_s }
|
||||
//
|
||||
// --only tts,bgm,sfx runs a subset and MERGES into an existing --out (so a
|
||||
// workflow can do TTS+BGM early, then SFX later once cues exist). When BGM uses
|
||||
// the generate path it is spawned detached (bgm_pending:true) — run wait-bgm.mjs
|
||||
// before assembling.
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { heygenAuthHeaders, heygenCredential, loadEnvFromDir } from "./lib/heygen.mjs";
|
||||
import {
|
||||
ffprobeDuration,
|
||||
pickProvider,
|
||||
resolveVoiceId,
|
||||
synthesizeOne,
|
||||
transcribeWav,
|
||||
withWordIds,
|
||||
} from "./lib/tts.mjs";
|
||||
import { generateBgmDetached, inferBgmPrompt, retrieveBgm } from "./lib/bgm.mjs";
|
||||
import { resolveSfx } from "./lib/sfx.mjs";
|
||||
import { mapWithConcurrency } from "./lib/concurrency.mjs";
|
||||
import { openAudioMeta } from "./lib/audio-meta.mjs";
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const argv = process.argv.slice(2);
|
||||
const flag = (name, def) => {
|
||||
const i = argv.indexOf(`--${name}`);
|
||||
return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
|
||||
};
|
||||
const has = (name) => argv.includes(`--${name}`);
|
||||
const die = (m) => {
|
||||
console.error(`✗ audio engine: ${m}`);
|
||||
process.exit(1);
|
||||
};
|
||||
const r3 = (x) => Number(x.toFixed(3));
|
||||
|
||||
// Two independent reports of an unbounded Promise.all over TTS lines
|
||||
// overwhelming a machine: one OOM'd 12/13 concurrent Kokoro TTS +
|
||||
// whisper-transcribe lines on a resource-constrained laptop, the other saw
|
||||
// 7/8 lines fail on first run (concurrent cold-start model loads) and pass on
|
||||
// retry once the model was cached. Kokoro/Whisper each load their own local
|
||||
// model per subprocess, so firing every line at once multiplies that cost by
|
||||
// the line count. mapWithConcurrency caps how many run at once — still
|
||||
// parallel, just bounded.
|
||||
const ttsConcurrency = Math.max(1, Number(process.env.HYPERFRAMES_TTS_CONCURRENCY) || 4);
|
||||
|
||||
const hyperframesDir = resolve(flag("hyperframes", "."));
|
||||
const requestPath = resolve(flag("request", join(hyperframesDir, "audio_request.json")));
|
||||
const outPath = resolve(flag("out", join(hyperframesDir, "audio_meta.json")));
|
||||
const sfxLibDir = resolve(flag("sfx-lib", join(HERE, "..", "assets", "sfx")));
|
||||
const lyriaRecipe = resolve(flag("lyria-recipe", join(HERE, "lyria-recipe.py")));
|
||||
const onlyArg = flag("only", "tts,bgm,sfx");
|
||||
const only = new Set(
|
||||
onlyArg
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean),
|
||||
);
|
||||
const providerOverride = flag("provider", null);
|
||||
const bgmModeOverride = flag("bgm-mode", null);
|
||||
const noBgm = has("no-bgm");
|
||||
const voiceOverride = flag("voice", null);
|
||||
const speedOverride = flag("speed", null);
|
||||
const langOverride = flag("lang", null);
|
||||
const seedSeconds = Number(flag("seed-seconds", "28")) || 28;
|
||||
|
||||
if (!existsSync(requestPath)) die(`audio_request.json not found at ${requestPath}`);
|
||||
let request;
|
||||
try {
|
||||
request = JSON.parse(readFileSync(requestPath, "utf8"));
|
||||
} catch (e) {
|
||||
die(`audio_request.json parse: ${e.message}`);
|
||||
}
|
||||
const lines = Array.isArray(request.lines) ? request.lines : [];
|
||||
const lang = langOverride || request.lang || "en";
|
||||
const speed = Number(speedOverride ?? request.speed ?? 1.0) || 1.0;
|
||||
|
||||
// ── env + HeyGen availability (the single switch) ─────────────────────────────
|
||||
loadEnvFromDir(hyperframesDir);
|
||||
const heygenOK = heygenCredential() !== null;
|
||||
const headers = heygenOK ? heygenAuthHeaders() : null;
|
||||
|
||||
// ── merge base: preserve sections not selected by --only ──────────────────────
|
||||
const audioMeta = openAudioMeta(outPath);
|
||||
const prev = audioMeta.value;
|
||||
const anomalies = [];
|
||||
|
||||
// ── TTS ───────────────────────────────────────────────────────────────────────
|
||||
let voices = prev.voices ?? [];
|
||||
let ttsProvider = prev.tts_provider ?? null;
|
||||
let voiceId = prev.voice_id ?? null;
|
||||
if (only.has("tts") && lines.length) {
|
||||
try {
|
||||
ttsProvider = pickProvider(
|
||||
providerOverride || (request.provider === "auto" ? null : request.provider),
|
||||
);
|
||||
} catch (e) {
|
||||
die(e.message);
|
||||
}
|
||||
voiceId = await resolveVoiceId({
|
||||
provider: ttsProvider,
|
||||
userVoice: voiceOverride || request.voice,
|
||||
lang,
|
||||
});
|
||||
console.error(`· tts: ${ttsProvider} · voice ${voiceId} · ${lines.length} line(s)`);
|
||||
const synthLine = async (line) => {
|
||||
const id = String(line.id);
|
||||
const text = String(line.text ?? "").trim();
|
||||
if (!text) {
|
||||
anomalies.push(`line ${id}: empty text — skipped`);
|
||||
return null;
|
||||
}
|
||||
const rel = `assets/voice/${id}.wav`;
|
||||
const abs = join(hyperframesDir, rel);
|
||||
const { ok, words, error } = await synthesizeOne({
|
||||
provider: ttsProvider,
|
||||
text,
|
||||
voiceId,
|
||||
lang,
|
||||
speed,
|
||||
wavAbs: abs,
|
||||
hyperframesDir,
|
||||
});
|
||||
if (!ok) {
|
||||
anomalies.push(`line ${id}: TTS failed — omitted${error ? ` (${error})` : ""}`);
|
||||
return null;
|
||||
}
|
||||
let wordArr = words; // heygen: native; else transcribe
|
||||
if (!wordArr) wordArr = await transcribeWav({ wavRel: rel, lang, hyperframesDir });
|
||||
const dur = ffprobeDuration(abs);
|
||||
if (!isFinite(dur) || dur <= 0) {
|
||||
anomalies.push(`line ${id}: bad voice duration — omitted`);
|
||||
return null;
|
||||
}
|
||||
return { id, path: rel, duration_s: r3(dur), words: withWordIds(wordArr) };
|
||||
};
|
||||
const results = await mapWithConcurrency(lines, ttsConcurrency, synthLine);
|
||||
voices = results.filter(Boolean);
|
||||
for (const v of voices)
|
||||
console.error(` voice ${v.id}: ${v.path} (${v.duration_s}s, ${v.words.length} words)`);
|
||||
}
|
||||
const hasVoice = voices.length > 0;
|
||||
const totalDuration = r3(voices.reduce((a, v) => a + (v.duration_s || 0), 0));
|
||||
|
||||
// ── BGM ─────────────────────────────────────────────────────────────────────
|
||||
let bgm = prev.bgm ?? null;
|
||||
const bgmFields = {
|
||||
bgm_pending: prev.bgm_pending ?? false,
|
||||
bgm_provider: prev.bgm_provider ?? null,
|
||||
bgm_pid: prev.bgm_pid ?? null,
|
||||
bgm_log: prev.bgm_log ?? null,
|
||||
bgm_mode: prev.bgm_mode ?? null,
|
||||
bgm_target_duration_s: prev.bgm_target_duration_s ?? null,
|
||||
bgm_seed_duration_s: prev.bgm_seed_duration_s ?? null,
|
||||
bgm_loop_count: prev.bgm_loop_count ?? null,
|
||||
};
|
||||
if (only.has("bgm")) {
|
||||
bgm = null;
|
||||
Object.keys(bgmFields).forEach((k) => (bgmFields[k] = k === "bgm_pending" ? false : null));
|
||||
// Mode resolution. An EXPLICIT mode (flag or request.bgm.mode) is strict:
|
||||
// "retrieve" means retrieve-or-nothing — it never silently starts a detached
|
||||
// generate (a caller with no wait-bgm step, e.g. product-launch, must not get
|
||||
// a pending job it can't await). Only the UNSET/auto default picks generate
|
||||
// when HeyGen is absent.
|
||||
const explicitMode = bgmModeOverride || request.bgm?.mode || null;
|
||||
let mode = noBgm ? "none" : explicitMode || (heygenOK ? "retrieve" : "generate");
|
||||
if (mode === "retrieve" && !heygenOK) {
|
||||
anomalies.push(
|
||||
"bgm: retrieve requires a HeyGen credential — skipped (no generate fallback for an explicit retrieve)",
|
||||
);
|
||||
mode = "none";
|
||||
}
|
||||
|
||||
if (mode === "none") {
|
||||
console.error(`· bgm: disabled`);
|
||||
} else if (mode === "retrieve") {
|
||||
try {
|
||||
bgm = await retrieveBgm({ query: request.bgm?.query, headers, hyperframesDir, hasVoice });
|
||||
if (bgm) {
|
||||
bgmFields.bgm_provider = "heygen";
|
||||
bgmFields.bgm_mode = "retrieve";
|
||||
console.error(` bgm: ${bgm.path} (retrieve "${bgm.query}")`);
|
||||
} else {
|
||||
anomalies.push(`bgm: no music match for "${request.bgm?.query ?? ""}" — skipped`);
|
||||
}
|
||||
} catch (e) {
|
||||
anomalies.push(`bgm retrieve failed: ${e.message} — skipped`);
|
||||
}
|
||||
} else {
|
||||
// generate
|
||||
const prompt = inferBgmPrompt({
|
||||
userPrompt: request.bgm?.prompt,
|
||||
blob: request.bgm?.blob || request.bgm?.query,
|
||||
archetype: request.bgm?.archetype,
|
||||
arc: request.bgm?.arc,
|
||||
});
|
||||
const gen = generateBgmDetached({
|
||||
prompt,
|
||||
durationS: totalDuration || 30,
|
||||
hyperframesDir,
|
||||
lyriaRecipe: existsSync(lyriaRecipe) ? lyriaRecipe : null,
|
||||
seedSeconds,
|
||||
hasVoice,
|
||||
});
|
||||
if (gen.disabled) {
|
||||
anomalies.push(`bgm: ${gen.reason}`);
|
||||
} else {
|
||||
bgm = { path: gen.path, volume: gen.volume, mode: gen.mode, duration_s: null };
|
||||
bgmFields.bgm_pending = true;
|
||||
bgmFields.bgm_provider = gen.provider;
|
||||
bgmFields.bgm_pid = gen.pid;
|
||||
bgmFields.bgm_log = gen.log;
|
||||
bgmFields.bgm_mode = gen.mode;
|
||||
bgmFields.bgm_target_duration_s = gen.target_duration_s ?? null;
|
||||
bgmFields.bgm_seed_duration_s = gen.seed_duration_s ?? null;
|
||||
bgmFields.bgm_loop_count = gen.loop_count ?? null;
|
||||
console.error(` bgm: launched ${gen.provider} (detached, pid ${gen.pid}) → ${gen.path}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── SFX ─────────────────────────────────────────────────────────────────────
|
||||
let sfx = prev.sfx ?? [];
|
||||
if (only.has("sfx")) {
|
||||
const cues = lines.flatMap((l) =>
|
||||
(Array.isArray(l.sfx) ? l.sfx : [])
|
||||
.map((name) => ({ id: String(l.id), name: String(name).trim() }))
|
||||
.filter((c) => c.name),
|
||||
);
|
||||
const res = await resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir });
|
||||
sfx = res.sfx;
|
||||
anomalies.push(...res.anomalies);
|
||||
console.error(
|
||||
`· sfx: ${sfx.length} cue(s) resolved (${heygenOK ? "heygen retrieval" : "bundled library"})`,
|
||||
);
|
||||
}
|
||||
|
||||
// ── write audio_meta.json ─────────────────────────────────────────────────────
|
||||
const meta = {
|
||||
tts_provider: ttsProvider,
|
||||
voice_id: voiceId,
|
||||
bgm,
|
||||
...bgmFields,
|
||||
voices,
|
||||
sfx,
|
||||
total_duration_s: totalDuration,
|
||||
};
|
||||
mkdirSync(dirname(outPath), { recursive: true });
|
||||
audioMeta.write(meta);
|
||||
|
||||
console.log(`✓ audio engine → ${outPath}`);
|
||||
console.log(` heygen: ${heygenOK ? "yes" : "no"} · ran: ${[...only].join(",")}`);
|
||||
console.log(
|
||||
` voices: ${voices.length} · bgm: ${bgm ? `${bgmFields.bgm_provider}${bgmFields.bgm_pending ? " (pending)" : ""}` : "none"} · sfx: ${sfx.length}`,
|
||||
);
|
||||
console.log(` total voice duration: ${totalDuration}s`);
|
||||
if (anomalies.length) {
|
||||
console.log(`\nanomalies (non-fatal):`);
|
||||
for (const a of anomalies) console.log(` - ${a}`);
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { test } from "node:test";
|
||||
import { mkdtempSync, rmSync, existsSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { resolveSfx } from "./lib/sfx.mjs";
|
||||
|
||||
// Proves the relocated engine (skills/media-use/audio/) still resolves its
|
||||
// bundled SFX library from the moved location — the path most likely to break
|
||||
// on a subtree move. Offline (heygenOK:false), no network.
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const sfxLibDir = join(HERE, "..", "assets", "sfx"); // same offset the engine uses
|
||||
|
||||
test("bundled SFX library resolves from the relocated path", async () => {
|
||||
assert.ok(existsSync(join(sfxLibDir, "manifest.json")), "moved manifest is present");
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-audio-"));
|
||||
try {
|
||||
const { sfx, anomalies } = await resolveSfx({
|
||||
cues: [{ id: "1", name: "whoosh" }],
|
||||
heygenOK: false,
|
||||
hyperframesDir: dir,
|
||||
sfxLibDir,
|
||||
});
|
||||
assert.equal(sfx.length, 1, `expected 1 resolved cue, got anomalies: ${anomalies.join("; ")}`);
|
||||
assert.equal(sfx[0].source, "local");
|
||||
assert.match(sfx[0].file, /assets\/sfx\//);
|
||||
assert.ok(existsSync(join(dir, sfx[0].file)), "matched SFX copied into the project");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("an unknown cue is reported, not fatal", async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-audio-"));
|
||||
try {
|
||||
const { sfx, anomalies } = await resolveSfx({
|
||||
cues: [{ id: "1", name: "definitely-not-a-real-sfx" }],
|
||||
heygenOK: false,
|
||||
hyperframesDir: dir,
|
||||
sfxLibDir,
|
||||
});
|
||||
assert.equal(sfx.length, 0);
|
||||
assert.ok(anomalies.some((a) => /not in bundled library/.test(a)));
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env node
|
||||
// Self-contained HeyGen TTS — single text in → one wav (+ optional words JSON)
|
||||
// out. A thin CLI over lib/tts.mjs (the same code the audio engine uses), so the
|
||||
// HeyGen REST call, starfish voice pick, mp3→wav transcode, and word-timestamp
|
||||
// filtering live in exactly one place. Bypasses the `hyperframes` CLI, which in
|
||||
// the published build is Kokoro-only.
|
||||
//
|
||||
// Usage:
|
||||
// node heygen-tts.mjs "Text to speak" -o narration.wav [--words narration.words.json]
|
||||
// node heygen-tts.mjs ./script.txt -o narration.wav --words narration.words.json
|
||||
// node heygen-tts.mjs "Bonjour" -o fr.wav --lang fr --voice <id>
|
||||
// node heygen-tts.mjs --list # list starfish voices and exit
|
||||
//
|
||||
// Flags: -o/--output (.wav → ffmpeg transcode; .mp3 → raw bytes), --words,
|
||||
// --voice (starfish id), --speed, --lang, --list.
|
||||
// Requires: $HEYGEN_API_KEY / OAuth ~/.heygen credentials and ffmpeg for .wav output.
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { heygenAuthHeaders, heygenJSON, loadEnvFromDir } from "./lib/heygen.mjs";
|
||||
import { ffprobeDuration, resolveVoiceId, synthesizeOne, withWordIds } from "./lib/tts.mjs";
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
function flag(name, def) {
|
||||
const i = argv.indexOf(`--${name}`);
|
||||
if (i < 0) return def;
|
||||
if (i + 1 >= argv.length) return true;
|
||||
const v = argv[i + 1];
|
||||
return v.startsWith("--") ? true : v;
|
||||
}
|
||||
const die = (m) => {
|
||||
console.error(`✗ heygen-tts: ${m}`);
|
||||
process.exit(1);
|
||||
};
|
||||
|
||||
// First arg that isn't a flag or the -o value is the text / .txt path.
|
||||
const positional = (() => {
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a.startsWith("--")) {
|
||||
const next = argv[i + 1];
|
||||
if (next && !next.startsWith("--")) i++;
|
||||
continue;
|
||||
}
|
||||
if (a === "-o") {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
return a;
|
||||
}
|
||||
return null;
|
||||
})();
|
||||
|
||||
const output = resolve(
|
||||
(typeof flag("output") === "string" && flag("output")) ||
|
||||
(argv.includes("-o") && argv[argv.indexOf("-o") + 1]) ||
|
||||
"narration.wav",
|
||||
);
|
||||
const wordsPath = typeof flag("words") === "string" ? resolve(flag("words")) : null;
|
||||
const userVoice = typeof flag("voice") === "string" ? flag("voice") : null;
|
||||
const speedRaw = typeof flag("speed") === "string" ? Number(flag("speed")) : 1.0;
|
||||
const speed = isFinite(speedRaw) && speedRaw > 0 ? speedRaw : 1.0;
|
||||
const lang = typeof flag("lang") === "string" ? flag("lang") : "en";
|
||||
const listOnly = flag("list") === true;
|
||||
|
||||
loadEnvFromDir(process.cwd());
|
||||
let authHeaders;
|
||||
try {
|
||||
authHeaders = heygenAuthHeaders();
|
||||
} catch (e) {
|
||||
die(e.message);
|
||||
}
|
||||
|
||||
// ---------- --list ----------
|
||||
if (listOnly) {
|
||||
const payload = await heygenJSON(`/voices?engine=starfish&type=public&limit=50`, {
|
||||
headers: authHeaders,
|
||||
});
|
||||
for (const v of payload.data ?? payload.voices ?? []) {
|
||||
console.log(`${v.voice_id}\t${v.name}\t${v.language ?? ""}`);
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
// ---------- resolve text + voice ----------
|
||||
if (!positional) die("no text given. Pass a string or a .txt path, or use --list.");
|
||||
const text =
|
||||
positional.endsWith(".txt") && existsSync(resolve(positional))
|
||||
? readFileSync(resolve(positional), "utf8").trim()
|
||||
: positional;
|
||||
if (!text) die("input text is empty");
|
||||
|
||||
const voiceId = await resolveVoiceId({ provider: "heygen", userVoice, lang });
|
||||
if (!userVoice) console.error(`· using voice ${voiceId}`);
|
||||
|
||||
// ---------- synthesize (shared engine code) ----------
|
||||
const { ok, words } = await synthesizeOne({
|
||||
provider: "heygen",
|
||||
text,
|
||||
voiceId,
|
||||
lang,
|
||||
speed,
|
||||
wavAbs: output,
|
||||
hyperframesDir: process.cwd(),
|
||||
});
|
||||
if (!ok) die("synthesis failed (HeyGen request/transcode error)");
|
||||
|
||||
let wordCount = 0;
|
||||
if (wordsPath) {
|
||||
if (words && words.length) {
|
||||
mkdirSync(dirname(wordsPath), { recursive: true });
|
||||
writeFileSync(wordsPath, JSON.stringify(withWordIds(words), null, 2));
|
||||
wordCount = words.length;
|
||||
} else {
|
||||
console.error("⚠ no word_timestamps in response — run `hyperframes transcribe` instead");
|
||||
}
|
||||
}
|
||||
|
||||
const dur = ffprobeDuration(output);
|
||||
const durStr = isFinite(dur) ? ` (${dur.toFixed(2)}s)` : "";
|
||||
console.log(`✓ ${output}${durStr}${wordCount ? ` · ${wordsPath} (${wordCount} words)` : ""}`);
|
||||
@@ -0,0 +1,39 @@
|
||||
import { closeSync, ftruncateSync, openSync, readFileSync, writeSync } from "node:fs";
|
||||
|
||||
// Keep the merge base and output on the same file even if its pathname changes
|
||||
// while audio generation runs. The CLI owns this handle until write or exit.
|
||||
export function openAudioMeta(path) {
|
||||
let fd;
|
||||
try {
|
||||
fd = openSync(path, "r+");
|
||||
} catch (error) {
|
||||
if (error.code !== "ENOENT") throw error;
|
||||
}
|
||||
let value = {};
|
||||
if (fd !== undefined) {
|
||||
try {
|
||||
value = JSON.parse(readFileSync(fd, "utf8"));
|
||||
} catch (error) {
|
||||
closeSync(fd);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
return {
|
||||
value,
|
||||
write(meta) {
|
||||
const bytes = Buffer.from(JSON.stringify(meta, null, 2));
|
||||
// Defer new-file creation until generation succeeds. Never overwrite a
|
||||
// file (or follow a link) that appeared since the missing merge base.
|
||||
if (fd === undefined) fd = openSync(path, "wx");
|
||||
try {
|
||||
let offset = 0;
|
||||
while (offset < bytes.length) {
|
||||
offset += writeSync(fd, bytes, offset, bytes.length - offset, offset);
|
||||
}
|
||||
ftruncateSync(fd, bytes.length);
|
||||
} finally {
|
||||
closeSync(fd);
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { openAudioMeta } from "./audio-meta.mjs";
|
||||
|
||||
function fixture(t) {
|
||||
const dir = mkdtempSync(join(tmpdir(), "audio-meta-"));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
return dir;
|
||||
}
|
||||
|
||||
test("updates the read file from byte zero, truncates, and retains its mode", (t) => {
|
||||
const path = join(fixture(t), "meta.json");
|
||||
writeFileSync(path, JSON.stringify({ voices: ["long existing voice metadata"] }), {
|
||||
mode: 0o600,
|
||||
});
|
||||
const originalMode = statSync(path).mode;
|
||||
const handle = openAudioMeta(path);
|
||||
assert.deepEqual(handle.value, { voices: ["long existing voice metadata"] });
|
||||
handle.write({ voices: ["声"] });
|
||||
assert.equal(readFileSync(path, "utf8"), JSON.stringify({ voices: ["声"] }, null, 2));
|
||||
assert.equal(statSync(path).mode, originalMode);
|
||||
});
|
||||
|
||||
test("pathname replacement cannot redirect the existing-file write", (t) => {
|
||||
const dir = fixture(t);
|
||||
const path = join(dir, "meta.json");
|
||||
const moved = join(dir, "original.json");
|
||||
writeFileSync(path, "{}");
|
||||
const handle = openAudioMeta(path);
|
||||
renameSync(path, moved);
|
||||
writeFileSync(path, "replacement must survive");
|
||||
handle.write({ voices: [] });
|
||||
assert.equal(readFileSync(path, "utf8"), "replacement must survive");
|
||||
assert.deepEqual(JSON.parse(readFileSync(moved, "utf8")), { voices: [] });
|
||||
});
|
||||
|
||||
test("existing symlink outputs retain their original target even after retargeting", (t) => {
|
||||
const dir = fixture(t);
|
||||
const path = join(dir, "meta.json");
|
||||
const target = join(dir, "target.json");
|
||||
const victim = join(dir, "victim.json");
|
||||
writeFileSync(target, "{}");
|
||||
writeFileSync(victim, "untouched");
|
||||
symlinkSync(target, path);
|
||||
const handle = openAudioMeta(path);
|
||||
rmSync(path);
|
||||
symlinkSync(victim, path);
|
||||
handle.write({ bgm: null });
|
||||
assert.equal(readFileSync(victim, "utf8"), "untouched");
|
||||
assert.deepEqual(JSON.parse(readFileSync(target, "utf8")), { bgm: null });
|
||||
});
|
||||
|
||||
test("missing output is created only when saved", (t) => {
|
||||
const path = join(fixture(t), "meta.json");
|
||||
const handle = openAudioMeta(path);
|
||||
assert.deepEqual(handle.value, {});
|
||||
assert.throws(() => statSync(path), { code: "ENOENT" });
|
||||
handle.write({ sfx: [] });
|
||||
assert.deepEqual(JSON.parse(readFileSync(path, "utf8")), { sfx: [] });
|
||||
});
|
||||
|
||||
for (const replacement of ["file", "symlink"]) {
|
||||
test(`new output does not overwrite a racing ${replacement}`, (t) => {
|
||||
const dir = fixture(t);
|
||||
const path = join(dir, "meta.json");
|
||||
const victim = join(dir, "victim.json");
|
||||
const handle = openAudioMeta(path);
|
||||
writeFileSync(victim, "untouched");
|
||||
if (replacement === "symlink") symlinkSync(victim, path);
|
||||
else writeFileSync(path, "untouched");
|
||||
assert.throws(() => handle.write({}), { code: "EEXIST" });
|
||||
assert.equal(readFileSync(path, "utf8"), "untouched");
|
||||
assert.equal(readFileSync(victim, "utf8"), "untouched");
|
||||
});
|
||||
}
|
||||
|
||||
test("malformed merge base remains an error without changing bytes", (t) => {
|
||||
const path = join(fixture(t), "meta.json");
|
||||
writeFileSync(path, "bad json");
|
||||
assert.throws(() => openAudioMeta(path), SyntaxError);
|
||||
assert.equal(readFileSync(path, "utf8"), "bad json");
|
||||
});
|
||||
|
||||
test("CLI partial run retains unselected voices, BGM and SFX", (t) => {
|
||||
const dir = fixture(t);
|
||||
const path = join(dir, "meta.json");
|
||||
const request = join(dir, "request.json");
|
||||
const previous = {
|
||||
voices: [{ id: "a", duration_s: 2 }],
|
||||
bgm: { path: "bgm.wav" },
|
||||
sfx: [{ name: "click" }],
|
||||
tts_provider: "kokoro",
|
||||
voice_id: "voice-a",
|
||||
};
|
||||
writeFileSync(path, JSON.stringify(previous));
|
||||
writeFileSync(request, "{}");
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
fileURLToPath(new URL("../audio.mjs", import.meta.url)),
|
||||
"--request",
|
||||
request,
|
||||
"--hyperframes",
|
||||
dir,
|
||||
"--out",
|
||||
path,
|
||||
"--only",
|
||||
"none",
|
||||
],
|
||||
{
|
||||
encoding: "utf8",
|
||||
env: {
|
||||
...process.env,
|
||||
HEYGEN_CONFIG_DIR: dir,
|
||||
HEYGEN_API_KEY: "",
|
||||
HYPERFRAMES_API_KEY: "",
|
||||
},
|
||||
},
|
||||
);
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
const actual = JSON.parse(readFileSync(path, "utf8"));
|
||||
for (const key of Object.keys(previous)) assert.deepEqual(actual[key], previous[key]);
|
||||
});
|
||||
@@ -0,0 +1,257 @@
|
||||
// bgm.mjs — background music for the media audio engine. Two routes, gated the
|
||||
// same way as TTS/SFX:
|
||||
//
|
||||
// retrieve (default when HeyGen is configured) — search HeyGen's music library
|
||||
// by mood, download the top track. Synchronous. assets/bgm/track.mp3.
|
||||
// generate (the alternative; the automatic choice when HeyGen is absent) —
|
||||
// Lyria (cloud, $GEMINI_API_KEY/$GOOGLE_API_KEY + google-genai) preferred,
|
||||
// else local MusicGen (facebook/musicgen-small via transformers). Spawned
|
||||
// DETACHED so the engine can return while audio renders; the caller marks
|
||||
// bgm_pending and runs wait-bgm.mjs before assembling. assets/bgm/track.wav.
|
||||
//
|
||||
// Missing/failed BGM never blocks a render.
|
||||
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync, openSync, closeSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { downloadTo, searchSounds } from "./heygen.mjs";
|
||||
import { pythonInvocation } from "./python.mjs";
|
||||
|
||||
const r3 = (x) => Number(x.toFixed(3));
|
||||
const lyriaKey = () => process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY || "";
|
||||
|
||||
// Default BGM level. Under narration music is a bed that must stay under the
|
||||
// voice — 0.12 linear ≈ -18 dB. A silent film (no voice) has no voice to duck
|
||||
// beneath, so BGM sits forward at 0.9. Callers may override per composition.
|
||||
export const BGM_BED_VOLUME = 0.12;
|
||||
export const BGM_SILENT_VOLUME = 0.9;
|
||||
export const bgmDefaultVolume = (hasVoice) => (hasVoice ? BGM_BED_VOLUME : BGM_SILENT_VOLUME);
|
||||
|
||||
const BGM_PY_DEPS = ["transformers", "torch", "soundfile", "numpy"];
|
||||
const BGM_PY_PROBE =
|
||||
"import transformers, soundfile, torch, numpy; from transformers import MusicgenForConditionalGeneration";
|
||||
const LYRIA_PY_DEPS = ["google-genai", "python-dotenv"];
|
||||
const LYRIA_PY_PROBE = "import google.genai";
|
||||
|
||||
function pyOk(probe) {
|
||||
const { cmd, args } = pythonInvocation(["-c", probe]);
|
||||
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
|
||||
}
|
||||
// `python -m pip`, not a bare `pip` binary: a Homebrew/system Python often
|
||||
// exposes only `python3`/`pip3` on PATH, so a plain `pip` spawn silently
|
||||
// no-ops (ENOENT) and the documented "auto-installed on demand" path never
|
||||
// actually installs. `-m pip` also guarantees the packages land in the SAME
|
||||
// interpreter pyOk() probes — a bare `pip`/`pip3` could resolve to a
|
||||
// different Python installation than `python3` if more than one is on PATH.
|
||||
function pipInstall(deps) {
|
||||
const { cmd, args } = pythonInvocation(["-m", "pip", "install", "-q", ...deps]);
|
||||
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
|
||||
}
|
||||
|
||||
// ── retrieval (HeyGen music library) ──────────────────────────────────────────
|
||||
export async function retrieveBgm({ query, headers, hyperframesDir, hasVoice }) {
|
||||
const q = query || "calm cinematic underscore";
|
||||
const results = await searchSounds(q, "music", headers, { limit: 5 });
|
||||
if (!results.length) return null;
|
||||
const top = results[0];
|
||||
const rel = "assets/bgm/track.mp3";
|
||||
await downloadTo(top.audio_url, join(hyperframesDir, rel));
|
||||
return {
|
||||
path: rel,
|
||||
volume: bgmDefaultVolume(hasVoice),
|
||||
query: q,
|
||||
mode: "retrieve",
|
||||
duration_s: typeof top.duration === "number" ? r3(top.duration) : null,
|
||||
};
|
||||
}
|
||||
|
||||
// ── mood inference (for the generate path's prompt) ──────────────────────────
|
||||
// Industry base → archetype shape → emotional-arc tiebreaker. Exported so a
|
||||
// workflow adapter can build a rich prompt from its own narrative metadata; the
|
||||
// engine also calls it when generate has only a plain mood query.
|
||||
export function inferBgmPrompt({ blob = "", archetype = "", arc = "", userPrompt = "" } = {}) {
|
||||
if (userPrompt) return userPrompt;
|
||||
const b = String(blob).toLowerCase();
|
||||
let base;
|
||||
let bpm;
|
||||
if (/\b(crypto|nft|web3|defi|token|blockchain|exchange|wallet|dao)\b/.test(b)) {
|
||||
base = "atmospheric electronic, deep bass, futuristic synths, restrained percussion";
|
||||
bpm = 100;
|
||||
} else if (/\b(finance|fintech|bank|payment|invest|wealth|insurance|treasury)\b/.test(b)) {
|
||||
base = "calm cinematic, soft strings, subtle piano, restrained percussion";
|
||||
bpm = 92;
|
||||
} else if (/\b(creative|agency|design|studio|art|brand|marketing|content)\b/.test(b)) {
|
||||
base = "playful electronic, warm pads, light percussion";
|
||||
bpm = 115;
|
||||
} else {
|
||||
base = "uplifting corporate tech, bright modern piano with synth pads";
|
||||
bpm = 108;
|
||||
}
|
||||
const at = String(archetype).toLowerCase();
|
||||
const ar = String(arc).toLowerCase();
|
||||
if (/\bpas\b|pain.agitate|pain.+solve/.test(at))
|
||||
return `${base}, starts with subtle tension then builds to resolution, BPM ${bpm}, transitions from MINOR to MAJOR`;
|
||||
if (/\bbab\b|before.after|future.pac|vision/.test(at))
|
||||
return `${base}, cinematic and aspirational, steady build with rising energy, BPM ${bpm}, MAJOR`;
|
||||
if (/cascade|feature.benefit/.test(at))
|
||||
return `${base}, energetic and driving, consistent momentum, BPM ${Math.min(bpm + 10, 128)}, MAJOR`;
|
||||
if (/demo.loop|question.+answer/.test(at))
|
||||
return `${base}, clean and focused, minimal arrangement, BPM ${Math.max(bpm - 8, 88)}`;
|
||||
if (/frustrat|anxiety|overwhelm|tension/.test(ar) && /relief|excite|triumph/.test(ar))
|
||||
return `${base}, builds from understated tension to uplifting resolution, BPM ${bpm}, MINOR to MAJOR`;
|
||||
if (/excit|awe|power|triumph/.test(ar))
|
||||
return `${base}, energetic and confident, BPM ${bpm}, MAJOR`;
|
||||
if (/trust|ease|clarity|reassur/.test(ar))
|
||||
return `${base}, warm and reassuring, BPM ${Math.max(bpm - 5, 85)}`;
|
||||
return `${base}, BPM ${bpm}, MAJOR`;
|
||||
}
|
||||
|
||||
// ── generation (Lyria → MusicGen, detached) ──────────────────────────────────
|
||||
// Returns a bgmMeta the caller folds into audio_meta:
|
||||
// { path, mode, volume, provider, pid, log, target_duration_s, seed_duration_s,
|
||||
// loop_count, pending:true } on success, or { disabled:true, reason }.
|
||||
export function generateBgmDetached({
|
||||
prompt,
|
||||
durationS,
|
||||
hyperframesDir,
|
||||
lyriaRecipe,
|
||||
seedSeconds = 28,
|
||||
hasVoice,
|
||||
}) {
|
||||
const rel = "assets/bgm/track.wav";
|
||||
const abs = join(hyperframesDir, rel);
|
||||
mkdirSync(join(hyperframesDir, "assets", "bgm"), { recursive: true });
|
||||
const log = join(hyperframesDir, "assets", "bgm", `bgm-${Date.now()}.log`);
|
||||
const targetS = Math.max(1, durationS);
|
||||
const baseMeta = { path: rel, mode: null, volume: bgmDefaultVolume(hasVoice), pending: true };
|
||||
|
||||
const lyriaConfigured = !!lyriaKey() && !!lyriaRecipe && existsSync(lyriaRecipe);
|
||||
|
||||
// Make a backend runnable: prefer Lyria when configured (install google-genai
|
||||
// on demand), else ensure local MusicGen deps. Installs are synchronous here —
|
||||
// generation itself is detached, so the engine still returns promptly.
|
||||
if (lyriaConfigured && !pyOk(LYRIA_PY_PROBE)) pipInstall(LYRIA_PY_DEPS);
|
||||
const useLyria = lyriaConfigured && pyOk(LYRIA_PY_PROBE);
|
||||
if (!useLyria && !pyOk(BGM_PY_PROBE)) pipInstall(BGM_PY_DEPS);
|
||||
|
||||
const fd = openSync(log, "w");
|
||||
if (useLyria) {
|
||||
const { cmd, args } = pythonInvocation([
|
||||
lyriaRecipe,
|
||||
"--output",
|
||||
abs,
|
||||
"--duration",
|
||||
String(targetS),
|
||||
"--prompt",
|
||||
prompt,
|
||||
]);
|
||||
const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
|
||||
proc.unref();
|
||||
closeSync(fd);
|
||||
return {
|
||||
...baseMeta,
|
||||
mode: "detached-single",
|
||||
provider: "lyria",
|
||||
pid: proc.pid,
|
||||
log,
|
||||
target_duration_s: r3(targetS),
|
||||
};
|
||||
}
|
||||
|
||||
if (pyOk(BGM_PY_PROBE)) {
|
||||
const seedS = Math.min(Math.max(seedSeconds, 10), 30);
|
||||
const loops = targetS > seedS ? Math.ceil(targetS / seedS) : 1;
|
||||
const script = musicgenScript({ prompt, abs, targetS, seedS });
|
||||
const { cmd, args } = pythonInvocation(["-c", script]);
|
||||
const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
|
||||
proc.unref();
|
||||
closeSync(fd);
|
||||
return {
|
||||
...baseMeta,
|
||||
mode: targetS > seedS ? "detached-seed-loop" : "detached-seed-trim",
|
||||
provider: "musicgen",
|
||||
pid: proc.pid,
|
||||
log,
|
||||
target_duration_s: r3(targetS),
|
||||
seed_duration_s: seedS,
|
||||
loop_count: loops,
|
||||
};
|
||||
}
|
||||
|
||||
closeSync(fd);
|
||||
return {
|
||||
disabled: true,
|
||||
reason: lyriaConfigured
|
||||
? `Lyria configured but google-genai uninstallable, and local MusicGen unavailable (pip install ${BGM_PY_DEPS.join(" ")})`
|
||||
: `no Lyria key/recipe and local MusicGen deps unavailable (pip install ${BGM_PY_DEPS.join(" ")})`,
|
||||
};
|
||||
}
|
||||
|
||||
// Inline MusicGen: generate ONE seed clip (≤30s to stay under the decoder's
|
||||
// positional limit), then trim it down or crossfade-loop it up to the target.
|
||||
function musicgenScript({ prompt, abs, targetS, seedS }) {
|
||||
return `
|
||||
import math, os, sys, traceback
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
from transformers import MusicgenForConditionalGeneration, AutoProcessor
|
||||
|
||||
prompt = ${JSON.stringify(prompt)}
|
||||
out_path = ${JSON.stringify(abs)}
|
||||
target_s = float(${targetS.toFixed(3)})
|
||||
seed_s = float(${seedS.toFixed(3)})
|
||||
token_rate = 50
|
||||
crossfade_s = 0.3
|
||||
|
||||
def apply_fade(arr, sr, fade_in_s=0.08, fade_out_s=0.5):
|
||||
n_in = min(int(round(fade_in_s * sr)), arr.shape[0] // 2)
|
||||
n_out = min(int(round(fade_out_s * sr)), arr.shape[0] // 2)
|
||||
if n_in > 1: arr[:n_in] *= np.linspace(0.0, 1.0, n_in, dtype="float32")
|
||||
if n_out > 1: arr[-n_out:] *= np.linspace(1.0, 0.0, n_out, dtype="float32")
|
||||
return arr
|
||||
|
||||
def loop_crossfade(seed, target_len, xf):
|
||||
if seed.shape[0] >= target_len: return seed[:target_len]
|
||||
xf = min(xf, seed.shape[0] // 2)
|
||||
if xf < 1:
|
||||
reps = int(math.ceil(target_len / seed.shape[0]))
|
||||
return np.tile(seed, reps)[:target_len]
|
||||
t = np.linspace(0.0, 1.0, xf, dtype="float32")
|
||||
fade_out = np.cos(t * (math.pi / 2)); fade_in = np.sin(t * (math.pi / 2))
|
||||
out = seed.copy()
|
||||
while out.shape[0] < target_len:
|
||||
tail = out[-xf:] * fade_out; head = seed[:xf] * fade_in
|
||||
out = np.concatenate([out[:-xf], tail + head, seed[xf:]])
|
||||
return out[:target_len]
|
||||
|
||||
try:
|
||||
Path(os.path.dirname(out_path)).mkdir(parents=True, exist_ok=True)
|
||||
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
|
||||
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
|
||||
model.eval()
|
||||
sr = int(model.config.audio_encoder.sampling_rate)
|
||||
gen_s = min(seed_s, target_s)
|
||||
tokens = max(1, int(math.ceil(gen_s * token_rate)))
|
||||
print(f"[musicgen] seed dur={gen_s:.2f}s tokens={tokens}", flush=True)
|
||||
inputs = processor(text=[prompt], padding=True, return_tensors="pt")
|
||||
audio = model.generate(**inputs, max_new_tokens=tokens)
|
||||
seed = audio[0, 0].detach().cpu().numpy().astype("float32")
|
||||
peak = float(np.max(np.abs(seed)))
|
||||
if peak > 1e-6: seed = seed * (0.89 / peak)
|
||||
want = max(1, int(round(target_s * sr)))
|
||||
if seed.shape[0] >= want:
|
||||
final = seed[:want].copy()
|
||||
else:
|
||||
final = loop_crossfade(seed, want, int(round(crossfade_s * sr)))
|
||||
if final.shape[0] < want: final = np.pad(final, (0, want - final.shape[0]))
|
||||
else: final = final[:want]
|
||||
final = apply_fade(final, sr)
|
||||
peak = float(np.max(np.abs(final)))
|
||||
if peak > 1.0: final = final / peak
|
||||
sf.write(out_path, final, sr)
|
||||
print(f"[musicgen] wrote {out_path} samples={final.shape[0]} sr={sr}", flush=True)
|
||||
except Exception:
|
||||
traceback.print_exc(); sys.exit(1)
|
||||
`;
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { BGM_BED_VOLUME, BGM_SILENT_VOLUME, bgmDefaultVolume } from "./bgm.mjs";
|
||||
|
||||
// Regression: narrated pipelines used to ship BGM at 0.8 (≈ -2 dB), ~16 dB
|
||||
// hotter than a music bed under a voice should be. The default under narration
|
||||
// must be a proper bed (≈ -18 dB); a silent film keeps the louder default.
|
||||
|
||||
const dbfs = (linear) => 20 * Math.log10(linear);
|
||||
|
||||
test("BGM under narration is a bed near -18 dB", () => {
|
||||
assert.equal(bgmDefaultVolume(true), BGM_BED_VOLUME);
|
||||
assert.equal(BGM_BED_VOLUME, 0.12);
|
||||
const db = dbfs(BGM_BED_VOLUME);
|
||||
assert.ok(db < -17 && db > -19, `bed should be ≈ -18 dB, got ${db.toFixed(1)} dB`);
|
||||
});
|
||||
|
||||
test("a silent film (no voice) keeps BGM forward", () => {
|
||||
assert.equal(bgmDefaultVolume(false), BGM_SILENT_VOLUME);
|
||||
assert.equal(BGM_SILENT_VOLUME, 0.9);
|
||||
});
|
||||
|
||||
test("the narrated default is well below the voice (≈ 0 dBFS)", () => {
|
||||
// Voice sits at data-volume="1" (0 dBFS); the bed must be ~16+ dB under it.
|
||||
const separation = dbfs(1) - dbfs(bgmDefaultVolume(true));
|
||||
assert.ok(
|
||||
separation >= 16,
|
||||
`bed should sit ≥16 dB under the voice, got ${separation.toFixed(1)} dB`,
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,14 @@
|
||||
// mapWithConcurrency — run `fn` over `items` with at most `limit` in flight at
|
||||
// once. Preserves input order in the result array regardless of completion order.
|
||||
export async function mapWithConcurrency(items, limit, fn) {
|
||||
const results = new Array(items.length);
|
||||
let next = 0;
|
||||
async function worker() {
|
||||
while (next < items.length) {
|
||||
const i = next++;
|
||||
results[i] = await fn(items[i], i);
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
|
||||
return results;
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mapWithConcurrency } from "./concurrency.mjs";
|
||||
|
||||
// Regression: audio.mjs used a bare Promise.all(lines.map(synthLine)) to
|
||||
// synthesize every TTS line at once, spawning one Kokoro/whisper model load
|
||||
// per line concurrently. Two independent reports of this overwhelming a
|
||||
// machine (OOM, and cold-start contention causing spurious failures).
|
||||
// mapWithConcurrency is the extracted cap; test it in isolation since
|
||||
// audio.mjs itself is a script (runs CLI/exit side effects on import).
|
||||
test("processes every item and preserves input order regardless of completion order", async () => {
|
||||
const order = [5, 1, 3, 2, 4];
|
||||
const results = await mapWithConcurrency(order, 2, async (n) => {
|
||||
await new Promise((r) => setTimeout(r, n));
|
||||
return n * 10;
|
||||
});
|
||||
assert.deepEqual(results, [50, 10, 30, 20, 40]);
|
||||
});
|
||||
|
||||
test("never runs more than `limit` at once", async () => {
|
||||
let inFlight = 0;
|
||||
let maxInFlight = 0;
|
||||
const items = Array.from({ length: 10 }, (_, i) => i);
|
||||
await mapWithConcurrency(items, 3, async () => {
|
||||
inFlight++;
|
||||
maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
inFlight--;
|
||||
});
|
||||
assert.equal(maxInFlight, 3);
|
||||
});
|
||||
|
||||
test("limit larger than the item count runs everything without hanging", async () => {
|
||||
const results = await mapWithConcurrency([1, 2], 10, async (n) => n * 2);
|
||||
assert.deepEqual(results, [2, 4]);
|
||||
});
|
||||
|
||||
test("empty input resolves to an empty array", async () => {
|
||||
const results = await mapWithConcurrency([], 4, async (n) => n);
|
||||
assert.deepEqual(results, []);
|
||||
});
|
||||
@@ -0,0 +1,155 @@
|
||||
// heygen.mjs — vendored HeyGen REST helpers (auth + transport) for the audio
|
||||
// pipeline. The credential resolver matches the hyperframes CLI auth: first
|
||||
// usable source wins — $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY → a nearby .env → ~/.heygen/
|
||||
// credentials (oauth → Bearer, else api_key → X-Api-Key; $HEYGEN_CONFIG_DIR
|
||||
// overrides the dir). Vendored so the skill ships standalone. Pure node.
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
|
||||
export const HEYGEN_BASE = "https://api.heygen.com/v3";
|
||||
export const HEYGEN_CLI_SOURCE_HEADERS = { "X-HeyGen-Source": "cli" };
|
||||
// Tool-attribution sent on EVERY media-use HeyGen call regardless of auth type, so
|
||||
// the backend can isolate media-use consumption from other free TTS / avatar video.
|
||||
// Unconditional — a paying user's media-use call is still media-use — unlike the
|
||||
// OAuth-only cli-source header above, which also gates the free allowance.
|
||||
export const HEYGEN_CLIENT_SOURCE_HEADERS = { "X-HeyGen-Client-Source": "media-use" };
|
||||
|
||||
// Walk up ≤5 dirs from startDir; load the first .env (shell env always wins).
|
||||
export function loadEnvFromDir(startDir) {
|
||||
let dir = resolve(startDir);
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const envPath = join(dir, ".env");
|
||||
if (existsSync(envPath)) {
|
||||
for (const raw of readFileSync(envPath, "utf8").split("\n")) {
|
||||
let line = raw.trim();
|
||||
if (!line || line.startsWith("#")) continue;
|
||||
if (line.startsWith("export ")) line = line.slice(7).trim();
|
||||
const eq = line.indexOf("=");
|
||||
if (eq < 1) continue;
|
||||
const key = line.slice(0, eq).trim();
|
||||
let val = line.slice(eq + 1).trim();
|
||||
if (val.startsWith('"') || val.startsWith("'")) {
|
||||
const q = val[0];
|
||||
const end = val.indexOf(q, 1);
|
||||
val = end > 0 ? val.slice(1, end) : val.slice(1);
|
||||
}
|
||||
if (!(key in process.env)) process.env[key] = val;
|
||||
}
|
||||
return;
|
||||
}
|
||||
const parent = dirname(dir);
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
}
|
||||
|
||||
// → { headers } | { expired: true } | null. Never throws.
|
||||
export function heygenCredential() {
|
||||
const envKey = process.env.HEYGEN_API_KEY || process.env.HYPERFRAMES_API_KEY;
|
||||
if (envKey) return { headers: { "X-Api-Key": envKey } };
|
||||
|
||||
const file = join(process.env.HEYGEN_CONFIG_DIR || join(homedir(), ".heygen"), "credentials");
|
||||
if (!existsSync(file)) return null;
|
||||
const raw = readFileSync(file, "utf8").trim();
|
||||
if (!raw) return null;
|
||||
if (!raw.startsWith("{")) return { headers: { "X-Api-Key": raw } };
|
||||
|
||||
// A malformed credentials file (partial write / wrong shape) must degrade to
|
||||
// "no credential", not crash the engine at startup — this function never throws.
|
||||
let cred;
|
||||
try {
|
||||
cred = JSON.parse(raw);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const oauth = cred.oauth;
|
||||
if (oauth?.access_token) {
|
||||
const expired = oauth.expires_at && new Date(oauth.expires_at).getTime() - 60_000 < Date.now();
|
||||
if (!expired) return { headers: { Authorization: `Bearer ${oauth.access_token}` } };
|
||||
if (!cred.api_key) return { expired: true };
|
||||
}
|
||||
if (cred.api_key) return { headers: { "X-Api-Key": cred.api_key } };
|
||||
return null;
|
||||
}
|
||||
|
||||
// → "oauth" | "api_key" | null. Same oauth-vs-api-key check heygenAuthHeaders()
|
||||
// makes internally, exposed on its own so callers that only need to *tag* the
|
||||
// auth path (telemetry) don't have to parse headers back apart. Never throws:
|
||||
// no credential (or an expired one) is just `null`, same as a fresh resolve
|
||||
// with nothing to tag.
|
||||
export function heygenAuthMethod() {
|
||||
const cred = heygenCredential();
|
||||
if (!cred?.headers) return null;
|
||||
return "Authorization" in cred.headers ? "oauth" : "api_key";
|
||||
}
|
||||
|
||||
// → auth headers object, or throw with a fix hint.
|
||||
export function heygenAuthHeaders() {
|
||||
const cred = heygenCredential();
|
||||
if (cred?.headers) {
|
||||
// Only tag OAuth (Bearer) traffic as cli-source — the backend uses it to
|
||||
// grant the free allowance for OAuth requests and ignores it for API-key
|
||||
// (X-Api-Key) traffic, where it's dead metadata.
|
||||
const isOauth = "Authorization" in cred.headers;
|
||||
return isOauth
|
||||
? { ...cred.headers, ...HEYGEN_CLI_SOURCE_HEADERS, ...HEYGEN_CLIENT_SOURCE_HEADERS }
|
||||
: { ...cred.headers, ...HEYGEN_CLIENT_SOURCE_HEADERS };
|
||||
}
|
||||
if (cred?.expired)
|
||||
throw new Error(
|
||||
"HeyGen OAuth token expired — run `npx hyperframes auth refresh` (or `npx hyperframes auth login`)",
|
||||
);
|
||||
throw new Error(
|
||||
"no HeyGen credentials — set $HEYGEN_API_KEY, or run `npx hyperframes auth login` (writes ~/.heygen/credentials)",
|
||||
);
|
||||
}
|
||||
|
||||
// Authed JSON request against the v3 API; throws on a non-OK status.
|
||||
export async function heygenJSON(path, { method = "GET", headers = {}, body } = {}) {
|
||||
const opts = { method, headers: { ...HEYGEN_CLIENT_SOURCE_HEADERS, ...headers } };
|
||||
if (body !== undefined) {
|
||||
opts.headers["Content-Type"] = "application/json";
|
||||
opts.body = JSON.stringify(body);
|
||||
}
|
||||
const res = await fetch(`${HEYGEN_BASE}${path}`, opts);
|
||||
if (!res.ok) {
|
||||
const detail = await res.text().catch(() => "");
|
||||
throw new Error(
|
||||
`HeyGen ${method} ${path} → HTTP ${res.status}${detail ? `\n${detail.slice(0, 300)}` : ""}`,
|
||||
);
|
||||
}
|
||||
return res.json();
|
||||
}
|
||||
|
||||
// Download a (presigned) URL to destPath; returns byte length.
|
||||
export async function downloadTo(url, destPath) {
|
||||
const res = await fetch(url);
|
||||
if (!res.ok) throw new Error(`download HTTP ${res.status}: ${String(url).slice(0, 80)}`);
|
||||
const bytes = Buffer.from(await res.arrayBuffer());
|
||||
mkdirSync(dirname(destPath), { recursive: true });
|
||||
writeFileSync(destPath, bytes);
|
||||
return bytes.length;
|
||||
}
|
||||
|
||||
// Retrieval search over HeyGen's audio catalog (NOT generation). type =
|
||||
// "music" | "sound_effects". Returns the ranked results array (best first); each
|
||||
// item has a presigned `audio_url` (+ `duration`, `description`, `name`, `score`).
|
||||
// `query` is required (≥1 char, empty → HTTP 400) and `limit` is capped at 50.
|
||||
// `minScore`: omit to use the server default (0.7). That default is TOO HIGH for
|
||||
// sound_effects — good SFX hits score ~0.5–0.67, so callers wanting SFX should
|
||||
// pass a lower floor (~0.4); music scores high and is fine at the default.
|
||||
export async function searchSounds(query, type, headers, { limit = 5, minScore } = {}) {
|
||||
const params = new URLSearchParams({ query, type, limit: String(limit) });
|
||||
if (minScore != null) params.set("min_score", String(minScore));
|
||||
const payload = await heygenJSON(`/audio/sounds?${params.toString()}`, { headers });
|
||||
// `data` comes back as a ranked array (best first). Older responses keyed it by
|
||||
// numeric index ("0","1",…); normalize both shapes to an array (empty → []).
|
||||
const data = payload?.data ?? payload;
|
||||
if (Array.isArray(data)) return data;
|
||||
if (data && typeof data === "object") return Object.values(data);
|
||||
throw new Error(
|
||||
`unexpected /audio/sounds shape — top keys: ${Object.keys(payload ?? {}).join(", ")}`,
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { heygenAuthHeaders, heygenAuthMethod } from "./heygen.mjs";
|
||||
|
||||
function withCleanHeygenEnv(fn) {
|
||||
const previousApiKey = process.env.HEYGEN_API_KEY;
|
||||
const previousHyperframesApiKey = process.env.HYPERFRAMES_API_KEY;
|
||||
const previousConfigDir = process.env.HEYGEN_CONFIG_DIR;
|
||||
try {
|
||||
delete process.env.HEYGEN_API_KEY;
|
||||
delete process.env.HYPERFRAMES_API_KEY;
|
||||
delete process.env.HEYGEN_CONFIG_DIR;
|
||||
return fn();
|
||||
} finally {
|
||||
if (previousApiKey === undefined) delete process.env.HEYGEN_API_KEY;
|
||||
else process.env.HEYGEN_API_KEY = previousApiKey;
|
||||
if (previousHyperframesApiKey === undefined) delete process.env.HYPERFRAMES_API_KEY;
|
||||
else process.env.HYPERFRAMES_API_KEY = previousHyperframesApiKey;
|
||||
if (previousConfigDir === undefined) delete process.env.HEYGEN_CONFIG_DIR;
|
||||
else process.env.HEYGEN_CONFIG_DIR = previousConfigDir;
|
||||
}
|
||||
}
|
||||
|
||||
test("heygenAuthHeaders does not tag API-key requests as CLI traffic, but still carries the media-use tool tag", () => {
|
||||
withCleanHeygenEnv(() => {
|
||||
process.env.HEYGEN_API_KEY = "hg_test";
|
||||
// API-key requests use normal billing; the backend ignores the cli-source
|
||||
// header for them, so it's not sent. The tool-attribution header IS sent on
|
||||
// every media-use call (any auth type) so the backend can isolate media-use.
|
||||
assert.deepEqual(heygenAuthHeaders(), {
|
||||
"X-Api-Key": "hg_test",
|
||||
"X-HeyGen-Client-Source": "media-use",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
test("heygenAuthHeaders tags OAuth requests as CLI traffic and with the media-use tool tag", () => {
|
||||
withCleanHeygenEnv(() => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
|
||||
try {
|
||||
process.env.HEYGEN_CONFIG_DIR = dir;
|
||||
writeFileSync(
|
||||
join(dir, "credentials"),
|
||||
JSON.stringify({
|
||||
oauth: {
|
||||
access_token: "at_test",
|
||||
expires_at: "2099-01-01T00:00:00Z",
|
||||
},
|
||||
}),
|
||||
);
|
||||
assert.deepEqual(heygenAuthHeaders(), {
|
||||
Authorization: "Bearer at_test",
|
||||
"X-HeyGen-Source": "cli",
|
||||
"X-HeyGen-Client-Source": "media-use",
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test("heygenAuthMethod returns api_key for an env API key, without tagging headers", () => {
|
||||
withCleanHeygenEnv(() => {
|
||||
process.env.HEYGEN_API_KEY = "hg_test";
|
||||
assert.equal(heygenAuthMethod(), "api_key");
|
||||
});
|
||||
});
|
||||
|
||||
test("heygenAuthMethod returns oauth for a live OAuth credential", () => {
|
||||
withCleanHeygenEnv(() => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
|
||||
try {
|
||||
process.env.HEYGEN_CONFIG_DIR = dir;
|
||||
writeFileSync(
|
||||
join(dir, "credentials"),
|
||||
JSON.stringify({
|
||||
oauth: {
|
||||
access_token: "at_test",
|
||||
expires_at: "2099-01-01T00:00:00Z",
|
||||
},
|
||||
}),
|
||||
);
|
||||
assert.equal(heygenAuthMethod(), "oauth");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test("heygenAuthMethod returns null with no credential at all", () => {
|
||||
withCleanHeygenEnv(() => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
|
||||
try {
|
||||
process.env.HEYGEN_CONFIG_DIR = dir; // no credentials file written
|
||||
assert.equal(heygenAuthMethod(), null);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,63 @@
|
||||
// python.mjs — resolve which Python 3 executable to spawn, per platform.
|
||||
//
|
||||
// The audio engine (tts.mjs, bgm.mjs) shells out to `python3` for ElevenLabs
|
||||
// TTS and the local Lyria/MusicGen BGM paths. `python3` is the right name on
|
||||
// macOS/Linux, but on Windows the python.org installer only creates
|
||||
// `python.exe` plus the `py` launcher — there is no `python3.exe` (only the
|
||||
// Microsoft Store build adds one). So a bare `spawn("python3", …)` ENOENTs on a
|
||||
// standard Windows Python install, silently disabling every Python-backed audio
|
||||
// feature until the user hand-creates a `python3.exe` shim (reported twice).
|
||||
//
|
||||
// Resolve once, per process: probe the platform's candidates in order and take
|
||||
// the first that actually runs. `py` is the launcher, so it needs a `-3` arg to
|
||||
// select Python 3 — hence candidates are argv PREFIXES, not bare names.
|
||||
|
||||
import { spawnSync } from "node:child_process";
|
||||
|
||||
function defaultProbe(cmd, args) {
|
||||
try {
|
||||
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the argv prefix that launches Python 3 on this platform.
|
||||
* Returns e.g. `["python3"]`, `["python"]`, or `["py", "-3"]`.
|
||||
*
|
||||
* Pure except for `probe` (which runs `<cmd> … --version`); both `platform`
|
||||
* and `probe` are injectable so every branch is unit-testable without spawning.
|
||||
* If nothing probes OK, falls back to the canonical name for the platform so
|
||||
* the eventual spawn fails loudly exactly as it did before — never worse.
|
||||
*/
|
||||
export function resolvePythonCommand(platform = process.platform, probe = defaultProbe) {
|
||||
const candidates =
|
||||
platform === "win32" ? [["python3"], ["python"], ["py", "-3"]] : [["python3"], ["python"]];
|
||||
for (const prefix of candidates) {
|
||||
if (probe(prefix[0], [...prefix.slice(1), "--version"])) return prefix;
|
||||
}
|
||||
return candidates[0];
|
||||
}
|
||||
|
||||
let cached = null;
|
||||
|
||||
/** Cached `resolvePythonCommand()` — probing spawns, so resolve at most once. */
|
||||
export function pythonCommand() {
|
||||
if (!cached) cached = resolvePythonCommand();
|
||||
return cached;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `{ cmd, args }` for running Python 3 with `extraArgs`, using the
|
||||
* resolved (or supplied) prefix. Keeps the launcher's `-3` (and any future
|
||||
* prefix args) ahead of the caller's own arguments.
|
||||
*/
|
||||
export function pythonInvocation(extraArgs, prefix = pythonCommand()) {
|
||||
return { cmd: prefix[0], args: [...prefix.slice(1), ...extraArgs] };
|
||||
}
|
||||
|
||||
/** Test-only: clear the cached resolution so a test can re-probe. */
|
||||
export function _resetPythonCommandCacheForTests() {
|
||||
cached = null;
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { resolvePythonCommand, pythonInvocation } from "./python.mjs";
|
||||
|
||||
// Regression: on Windows a standard python.org install has no `python3.exe`
|
||||
// (only `python.exe` + the `py` launcher), so `spawn("python3", …)` ENOENTs and
|
||||
// every Python-backed audio feature silently no-ops. resolvePythonCommand takes
|
||||
// injectable platform/probe params so all branches are testable without
|
||||
// spawning a real interpreter.
|
||||
|
||||
// probeFor(names): a probe that reports success only for the given argv-0 names.
|
||||
function probeFor(...names) {
|
||||
const ok = new Set(names);
|
||||
return (cmd) => ok.has(cmd);
|
||||
}
|
||||
|
||||
test("non-win32 uses python3 when it runs", () => {
|
||||
assert.deepEqual(resolvePythonCommand("linux", probeFor("python3")), ["python3"]);
|
||||
assert.deepEqual(resolvePythonCommand("darwin", probeFor("python3")), ["python3"]);
|
||||
});
|
||||
|
||||
test("win32 prefers python3 when the Microsoft Store build provides it", () => {
|
||||
assert.deepEqual(resolvePythonCommand("win32", probeFor("python3", "python", "py")), ["python3"]);
|
||||
});
|
||||
|
||||
test("win32 falls back to python.exe when python3 is absent (python.org install)", () => {
|
||||
// The exact reported scenario: no python3, but `python` exists.
|
||||
assert.deepEqual(resolvePythonCommand("win32", probeFor("python", "py")), ["python"]);
|
||||
});
|
||||
|
||||
test("win32 falls back to the py launcher with -3 when only py exists", () => {
|
||||
assert.deepEqual(resolvePythonCommand("win32", probeFor("py")), ["py", "-3"]);
|
||||
});
|
||||
|
||||
test("py launcher is probed as `py -3 --version`, not bare `py`", () => {
|
||||
const seen = [];
|
||||
const probe = (cmd, args) => {
|
||||
seen.push([cmd, ...args]);
|
||||
return cmd === "py";
|
||||
};
|
||||
resolvePythonCommand("win32", probe);
|
||||
assert.deepEqual(seen.at(-1), ["py", "-3", "--version"]);
|
||||
});
|
||||
|
||||
test("falls back to the canonical name (loud failure, unchanged) when nothing runs", () => {
|
||||
// No interpreter anywhere — must not throw, and must return python3 so the
|
||||
// eventual spawn fails exactly as it did before this fix, never worse.
|
||||
assert.deepEqual(
|
||||
resolvePythonCommand("win32", () => false),
|
||||
["python3"],
|
||||
);
|
||||
assert.deepEqual(
|
||||
resolvePythonCommand("linux", () => false),
|
||||
["python3"],
|
||||
);
|
||||
});
|
||||
|
||||
test("pythonInvocation prepends the resolved prefix ahead of caller args", () => {
|
||||
assert.deepEqual(pythonInvocation(["-c", "import x"], ["python"]), {
|
||||
cmd: "python",
|
||||
args: ["-c", "import x"],
|
||||
});
|
||||
// The py launcher's -3 must stay ahead of the caller's own arguments.
|
||||
assert.deepEqual(pythonInvocation(["-c", "import x"], ["py", "-3"]), {
|
||||
cmd: "py",
|
||||
args: ["-3", "-c", "import x"],
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,143 @@
|
||||
// sfx.mjs — sound effects for the media audio engine. Provider-gated (NOT a
|
||||
// per-cue merge): the decision is made once, by whether HeyGen is configured —
|
||||
// mirroring how TTS and BGM degrade.
|
||||
//
|
||||
// HeyGen credential present → retrieve EVERY cue from HeyGen's audio library
|
||||
// (/v3/audio/sounds, type=sound_effects, min_score=0.4). The bundled
|
||||
// library is NOT consulted.
|
||||
// HeyGen credential absent → resolve cues against the bundled 21-file
|
||||
// library (assets/sfx/manifest.json), copying matched files into the
|
||||
// project. Offline, deterministic, free.
|
||||
//
|
||||
// A cue that matches nothing is skipped (recorded as an anomaly); SFX never
|
||||
// blocks a render. Every cue sits at volume ~0.35, under voice + BGM.
|
||||
|
||||
import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { downloadTo, searchSounds } from "./heygen.mjs";
|
||||
|
||||
const SFX_VOLUME = 0.35;
|
||||
const slug = (s) =>
|
||||
s
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, "-")
|
||||
.replace(/^-+|-+$/g, "")
|
||||
.slice(0, 40) || "x";
|
||||
const r3 = (x) => Number(x.toFixed(3));
|
||||
|
||||
// cues: [{ id, name }] (id = the line/frame/scene the cue fires in). Returns
|
||||
// { sfx: [{ id, name, file, source, offset_s, duration_s, volume }], anomalies }.
|
||||
export async function resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir }) {
|
||||
const sfx = [];
|
||||
const anomalies = [];
|
||||
const destDir = join(hyperframesDir, "assets", "sfx");
|
||||
|
||||
// Dedupe identical (id,name) cues — the same effect named twice in one line
|
||||
// downloads/copies once.
|
||||
const seen = new Set();
|
||||
const uniq = cues.filter((c) => {
|
||||
const k = `${c.id}:${c.name}`;
|
||||
if (seen.has(k)) return false;
|
||||
seen.add(k);
|
||||
return true;
|
||||
});
|
||||
|
||||
if (heygenOK) {
|
||||
for (const { id, name } of uniq) {
|
||||
try {
|
||||
// SFX hits score low (~0.5–0.67), below the API's default 0.7 which
|
||||
// silently drops most named cues — floor to 0.4. (BGM/music score high
|
||||
// and keep the default.)
|
||||
const results = await searchSounds(name, "sound_effects", headers, {
|
||||
limit: 3,
|
||||
minScore: 0.4,
|
||||
});
|
||||
if (!results.length) {
|
||||
anomalies.push(`sfx "${name}" (id ${id}): no HeyGen match — skipped`);
|
||||
continue;
|
||||
}
|
||||
const top = results[0];
|
||||
const file = `assets/sfx/${slug(name)}.mp3`;
|
||||
await downloadTo(top.audio_url, join(hyperframesDir, file));
|
||||
sfx.push({
|
||||
id,
|
||||
name,
|
||||
file,
|
||||
source: "heygen",
|
||||
offset_s: 0,
|
||||
duration_s: typeof top.duration === "number" ? r3(top.duration) : 1.0,
|
||||
volume: SFX_VOLUME,
|
||||
});
|
||||
} catch (e) {
|
||||
anomalies.push(`sfx "${name}" (id ${id}): retrieval failed — ${e.message}`);
|
||||
}
|
||||
}
|
||||
return { sfx, anomalies };
|
||||
}
|
||||
|
||||
// ── offline: bundled library ──
|
||||
const manifestPath = join(sfxLibDir, "manifest.json");
|
||||
if (!existsSync(manifestPath)) {
|
||||
if (uniq.length)
|
||||
anomalies.push(`no HeyGen credential and no SFX library at ${sfxLibDir} — all cues dropped`);
|
||||
return { sfx, anomalies };
|
||||
}
|
||||
let manifest;
|
||||
try {
|
||||
manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
|
||||
} catch (e) {
|
||||
anomalies.push(`SFX manifest parse failed (${e.message}) — all cues dropped`);
|
||||
return { sfx, anomalies };
|
||||
}
|
||||
// Build lookups: by manifest key, by file basename, and by slug of either, so
|
||||
// a cue can name "whoosh", "whoosh.mp3", or "ui click" (→ slug match).
|
||||
const byKey = new Map();
|
||||
for (const [key, entry] of Object.entries(manifest)) {
|
||||
if (!entry?.file || !isFinite(entry.duration)) continue;
|
||||
const rec = { key, file: entry.file, duration: entry.duration };
|
||||
byKey.set(key, rec);
|
||||
byKey.set(entry.file, rec);
|
||||
byKey.set(slug(key), rec);
|
||||
byKey.set(slug(entry.file.replace(/\.\w+$/, "")), rec);
|
||||
}
|
||||
mkdirSync(destDir, { recursive: true });
|
||||
for (const { id, name } of uniq) {
|
||||
const hit = byKey.get(name) ?? byKey.get(slug(name));
|
||||
if (!hit) {
|
||||
const known = [...new Set([...byKey.values()].map((v) => v.key))].slice(0, 8).join(", ");
|
||||
anomalies.push(
|
||||
`sfx "${name}" (id ${id}): not in bundled library — skipped (have: ${known}…)`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
const src = join(sfxLibDir, hit.file);
|
||||
const destRel = `assets/sfx/${hit.file}`;
|
||||
const dest = join(hyperframesDir, destRel);
|
||||
// The bundled library may be incomplete: some installs of the skill ship
|
||||
// manifest.json without the actual mp3s. Pushing an sfx entry that points at
|
||||
// a file we never copied produces a dangling reference that silently drops
|
||||
// downstream ("not on disk"). Surface it as a loud anomaly and skip the cue
|
||||
// instead, so the audio_meta never references a missing file.
|
||||
if (!existsSync(dest)) {
|
||||
if (!existsSync(src)) {
|
||||
anomalies.push(
|
||||
`sfx "${name}" (id ${id}): bundled file ${hit.file} missing from the offline ` +
|
||||
`library (${sfxLibDir}) — skipped. Reinstall the media-use skill to ` +
|
||||
`restore assets/sfx/*.mp3, or configure a HeyGen credential for retrieval.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
copyFileSync(src, dest);
|
||||
}
|
||||
sfx.push({
|
||||
id,
|
||||
name,
|
||||
file: destRel,
|
||||
source: "local",
|
||||
offset_s: 0,
|
||||
duration_s: r3(hit.duration),
|
||||
volume: SFX_VOLUME,
|
||||
});
|
||||
}
|
||||
return { sfx, anomalies };
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, existsSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { resolveSfx } from "./sfx.mjs";
|
||||
|
||||
// Offline (no HeyGen) SFX resolution: the bundled library may ship manifest.json
|
||||
// without the actual mp3s. The old code copied only when the source existed but
|
||||
// pushed the sfx entry unconditionally — producing a dangling reference that
|
||||
// silently dropped downstream ("not on disk"). These tests lock in the loud
|
||||
// behavior: a present file is copied + referenced; a missing file yields an
|
||||
// anomaly and NO dangling entry.
|
||||
|
||||
async function withDirs(fn) {
|
||||
const root = mkdtempSync(join(tmpdir(), "hf-sfx-"));
|
||||
const libDir = join(root, "lib");
|
||||
const projDir = join(root, "proj");
|
||||
mkdirSync(libDir, { recursive: true });
|
||||
mkdirSync(projDir, { recursive: true });
|
||||
try {
|
||||
// `await` is load-bearing: without it the finally cleanup runs before the
|
||||
// async test body resolves, deleting the temp dir mid-assertion.
|
||||
return await fn({ libDir, projDir });
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
test("offline: copies and references a present bundled file", async () => {
|
||||
await withDirs(async ({ libDir, projDir }) => {
|
||||
writeFileSync(
|
||||
join(libDir, "manifest.json"),
|
||||
JSON.stringify({ whoosh: { file: "whoosh.mp3", duration: 0.8 } }),
|
||||
);
|
||||
writeFileSync(join(libDir, "whoosh.mp3"), "ID3-fake-bytes");
|
||||
const { sfx, anomalies } = await resolveSfx({
|
||||
cues: [{ id: "s1", name: "whoosh" }],
|
||||
heygenOK: false,
|
||||
hyperframesDir: projDir,
|
||||
sfxLibDir: libDir,
|
||||
});
|
||||
assert.equal(sfx.length, 1);
|
||||
assert.equal(sfx[0].file, "assets/sfx/whoosh.mp3");
|
||||
assert.equal(sfx[0].source, "local");
|
||||
assert.ok(existsSync(join(projDir, "assets/sfx/whoosh.mp3")), "mp3 copied into project");
|
||||
assert.equal(anomalies.length, 0);
|
||||
});
|
||||
});
|
||||
|
||||
test("offline: a matched-but-missing bundled file yields an anomaly and NO dangling entry", async () => {
|
||||
await withDirs(async ({ libDir, projDir }) => {
|
||||
// Manifest names whoosh.mp3, but the mp3 was never shipped (the reported bug).
|
||||
writeFileSync(
|
||||
join(libDir, "manifest.json"),
|
||||
JSON.stringify({ whoosh: { file: "whoosh.mp3", duration: 0.8 } }),
|
||||
);
|
||||
const { sfx, anomalies } = await resolveSfx({
|
||||
cues: [{ id: "s1", name: "whoosh" }],
|
||||
heygenOK: false,
|
||||
hyperframesDir: projDir,
|
||||
sfxLibDir: libDir,
|
||||
});
|
||||
assert.equal(sfx.length, 0, "no dangling entry for a file that was never copied");
|
||||
assert.equal(anomalies.length, 1);
|
||||
assert.match(anomalies[0], /missing from the offline library/);
|
||||
assert.ok(!existsSync(join(projDir, "assets/sfx/whoosh.mp3")), "nothing copied");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,378 @@
|
||||
// tts.mjs — multi-provider TTS for the media audio engine. The provider chain,
|
||||
// auto-detected from env, is the one documented in ../SKILL.md:
|
||||
//
|
||||
// 1. HeyGen (Starfish) — $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen.
|
||||
// Direct v3 REST (NOT `hyperframes tts`, which in the published build is
|
||||
// Kokoro-only and silently ignores a HeyGen key). Returns word_timestamps
|
||||
// in the same call, so no separate transcribe pass.
|
||||
// 2. ElevenLabs — $ELEVENLABS_API_KEY + `pip install elevenlabs`. No
|
||||
// word timings → caller chains transcribeWav().
|
||||
// 3. Kokoro-82M (local) — always available, via the published `hyperframes tts`
|
||||
// CLI. No word timings → caller chains transcribeWav().
|
||||
//
|
||||
// "HeyGen available" is decided by CREDENTIAL presence (heygenCredential), never
|
||||
// by the CLI — see the note above.
|
||||
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { heygenAuthHeaders, heygenCredential, heygenJSON } from "./heygen.mjs";
|
||||
import { pythonInvocation } from "./python.mjs";
|
||||
|
||||
// ── provider detection ────────────────────────────────────────────────────────
|
||||
export function heygenAvailable() {
|
||||
return heygenCredential() !== null;
|
||||
}
|
||||
export function elevenlabsAvailable() {
|
||||
if (!process.env.ELEVENLABS_API_KEY) return false;
|
||||
const { cmd, args } = pythonInvocation(["-c", "import elevenlabs"]);
|
||||
const r = spawnSync(cmd, args, {
|
||||
stdio: "ignore",
|
||||
});
|
||||
return r.status === 0;
|
||||
}
|
||||
|
||||
// First available provider wins; an explicit choice is honored (and validated).
|
||||
export function pickProvider(userProvider) {
|
||||
if (userProvider) {
|
||||
if (!["heygen", "elevenlabs", "kokoro"].includes(userProvider))
|
||||
throw new Error(`invalid provider "${userProvider}" (heygen | elevenlabs | kokoro)`);
|
||||
if (userProvider === "heygen" && !heygenAvailable())
|
||||
throw new Error(
|
||||
"provider=heygen but no HeyGen credentials (set $HEYGEN_API_KEY or run `npx hyperframes auth login`)",
|
||||
);
|
||||
if (userProvider === "elevenlabs" && !process.env.ELEVENLABS_API_KEY)
|
||||
throw new Error("provider=elevenlabs but $ELEVENLABS_API_KEY is not set");
|
||||
return userProvider;
|
||||
}
|
||||
return heygenAvailable() ? "heygen" : elevenlabsAvailable() ? "elevenlabs" : "kokoro";
|
||||
}
|
||||
|
||||
// ── voice resolution ──────────────────────────────────────────────────────────
|
||||
// HeyGen /v3/voices/speech only accepts STARFISH voice_ids; auto-pick the first
|
||||
// English public starfish voice when none is pinned. ElevenLabs/Kokoro have
|
||||
// their own defaults.
|
||||
export async function resolveVoiceId({ provider, userVoice, lang = "en" }) {
|
||||
if (userVoice) return userVoice;
|
||||
if (provider === "elevenlabs") return "21m00Tcm4TlvDq8ikWAM"; // Rachel
|
||||
if (provider === "kokoro") {
|
||||
if (lang === "en") return "am_michael";
|
||||
throw new Error("Kokoro non-English needs an explicit --voice (see references/tts.md)");
|
||||
}
|
||||
// heygen — pin a fixed English default so the choice is deterministic. The old
|
||||
// "first English voice the API returns" drifts whenever HeyGen re-sorts the
|
||||
// public catalog. Marcia (mature, low female). Override with --voice / request.voice.
|
||||
if (lang === "en") return "05f19352e8f74b0392a8f411eba40de1"; // Marcia · English · female
|
||||
// Non-English: no fixed default — fall back to the first matching catalog voice.
|
||||
const payload = await heygenJSON(`/voices?engine=starfish&type=public&limit=50`, {
|
||||
headers: heygenAuthHeaders(),
|
||||
});
|
||||
const voices = payload.data ?? payload.voices ?? [];
|
||||
const pick = voices.find((v) => v.language === "English") ?? voices[0];
|
||||
if (!pick) throw new Error("no public starfish voice to default to — pass --voice");
|
||||
return pick.voice_id;
|
||||
}
|
||||
|
||||
// ── helpers ─────────────────────────────────────────────────────────────────
|
||||
export function withWordIds(words) {
|
||||
return (words ?? []).map((w, i) => ({
|
||||
id: `w${i}`,
|
||||
text: w.text,
|
||||
start: w.start,
|
||||
end: w.end,
|
||||
}));
|
||||
}
|
||||
|
||||
// `ffmpeg -i <file>` prints a `Duration: HH:MM:SS.ms` line to stderr even
|
||||
// though it exits non-zero with no output requested. Parsing pulled out as
|
||||
// a pure function so the ENOENT fallback below can be tested without
|
||||
// depending on whether ffprobe/ffmpeg are actually installed on the
|
||||
// machine running the tests.
|
||||
export function parseFfmpegDurationBanner(stderrText) {
|
||||
const match = /Duration:\s*(\d+):(\d+):(\d+(?:\.\d+)?)/.exec(stderrText ?? "");
|
||||
if (!match) return NaN;
|
||||
const [, hours, minutes, seconds] = match;
|
||||
return Number(hours) * 3600 + Number(minutes) * 60 + Number(seconds);
|
||||
}
|
||||
|
||||
// Some "essentials"-style ffmpeg distributions (common on Windows) ship
|
||||
// ffmpeg.exe without ffprobe.exe. ffprobeDuration's caller (audio.mjs)
|
||||
// otherwise reads a spurious NaN as "the WAV file is corrupt" and drops an
|
||||
// already-successfully-synthesized TTS line, rather than "the tool for
|
||||
// measuring it is missing".
|
||||
function ffmpegDurationFallback(absPath) {
|
||||
const r = spawnSync("ffmpeg", ["-i", absPath], { encoding: "utf8" });
|
||||
return parseFfmpegDurationBanner(r.stderr);
|
||||
}
|
||||
|
||||
export function ffprobeDuration(absPath) {
|
||||
const r = spawnSync(
|
||||
"ffprobe",
|
||||
["-v", "error", "-show_entries", "format=duration", "-of", "default=nw=1:nk=1", "--", absPath],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
if (r.error?.code === "ENOENT") return ffmpegDurationFallback(absPath);
|
||||
if (r.status !== 0) return NaN;
|
||||
return parseFloat(String(r.stdout).trim());
|
||||
}
|
||||
|
||||
export function resolveNpxCliFromNpmExecPath(
|
||||
npmExecPath = process.env.npm_execpath,
|
||||
pathExists = existsSync,
|
||||
) {
|
||||
if (!npmExecPath) return null;
|
||||
const fileName = npmExecPath.replace(/\\/g, "/").split("/").pop()?.toLowerCase();
|
||||
const npxCliPath =
|
||||
fileName === "npx-cli.js" ? npmExecPath : join(dirname(npmExecPath), "npx-cli.js");
|
||||
return pathExists(npxCliPath) ? npxCliPath : null;
|
||||
}
|
||||
|
||||
export function resolveNpxCliPath(
|
||||
npmExecPath = process.env.npm_execpath,
|
||||
nodeExecPath = process.env.npm_node_execpath || process.execPath,
|
||||
pathExists = existsSync,
|
||||
) {
|
||||
const fromNpm = resolveNpxCliFromNpmExecPath(npmExecPath, pathExists);
|
||||
if (fromNpm) return fromNpm;
|
||||
const besideNode = join(dirname(nodeExecPath), "node_modules", "npm", "bin", "npx-cli.js");
|
||||
return pathExists(besideNode) ? besideNode : null;
|
||||
}
|
||||
|
||||
export function resolveSpawnCommand(
|
||||
cmd,
|
||||
args,
|
||||
opts = {},
|
||||
platform = process.platform,
|
||||
env = process.env,
|
||||
pathExists = existsSync,
|
||||
) {
|
||||
if (cmd !== "npx" || platform !== "win32") {
|
||||
return { cmd, args, opts: { stdio: "ignore", ...opts } };
|
||||
}
|
||||
|
||||
// On Windows, npx resolves to npx.cmd, which Node cannot execute directly.
|
||||
// Avoid `shell:true` and the .cmd shim entirely by invoking npm's JS CLI with
|
||||
// node, preserving request-provided values as argv data instead of shell text.
|
||||
const nodeExecPath = env.npm_node_execpath || process.execPath;
|
||||
const npxCliPath = resolveNpxCliPath(env.npm_execpath, nodeExecPath, pathExists);
|
||||
if (!npxCliPath) return null;
|
||||
return {
|
||||
cmd: nodeExecPath,
|
||||
args: [npxCliPath, ...args.map((arg) => String(arg))],
|
||||
opts: { stdio: "ignore", windowsHide: true, ...opts },
|
||||
};
|
||||
}
|
||||
|
||||
// `platform`/`spawnFn` params (default process.platform / the real spawn)
|
||||
// exist so tests can exercise the win32 branch without mocking node:child_process
|
||||
// (its ESM exports are non-configurable, so mock.method can't patch it).
|
||||
// One-shot so a whole batch of TTS lines doesn't repeat the same diagnostic.
|
||||
let _warnedNpxResolution = false;
|
||||
/** Test-only: reset the one-shot npx-resolution warning latch. */
|
||||
export function _resetNpxResolutionWarnForTests() {
|
||||
_warnedNpxResolution = false;
|
||||
}
|
||||
|
||||
export function spawnP(
|
||||
cmd,
|
||||
args,
|
||||
opts = {},
|
||||
platform = process.platform,
|
||||
spawnFn = spawn,
|
||||
env = process.env,
|
||||
pathExists = existsSync,
|
||||
) {
|
||||
const resolved = resolveSpawnCommand(cmd, args, opts, platform, env, pathExists);
|
||||
if (!resolved) {
|
||||
// resolveSpawnCommand only returns null for the npx-on-win32 case where
|
||||
// neither npm's configured CLI nor the beside-node fallback exists. Without
|
||||
// this, every call silently returns status:-1 and stdio:"ignore" hides why.
|
||||
if (!_warnedNpxResolution) {
|
||||
_warnedNpxResolution = true;
|
||||
const reason = env.npm_execpath
|
||||
? `npm_execpath (${env.npm_execpath}) and the beside-node npm fallback could not be found`
|
||||
: "npm_execpath is unset and the beside-node npm fallback could not be found";
|
||||
console.error(
|
||||
`[media-use] Cannot run "${cmd}" on Windows: ${reason}. ` +
|
||||
`Every "${cmd}" call is being skipped. Install npm with Node, or run via ` +
|
||||
`\`npx\`/\`npm run\` with a valid npm_execpath.`,
|
||||
);
|
||||
}
|
||||
return Promise.resolve({ status: -1 });
|
||||
}
|
||||
return new Promise((resolve) => {
|
||||
const p = spawnFn(resolved.cmd, resolved.args, resolved.opts);
|
||||
p.on("exit", (code) => resolve({ status: code ?? -1 }));
|
||||
p.on("error", () => resolve({ status: -1 }));
|
||||
});
|
||||
}
|
||||
|
||||
// mp3/whatever bytes → wav 44.1k mono at destWav (ffmpeg detects true format).
|
||||
function transcodeToWav(bytes, destWav) {
|
||||
const td = mkdtempSync(join(tmpdir(), "hf-tts-"));
|
||||
const tmp = join(td, "a.mp3");
|
||||
writeFileSync(tmp, bytes);
|
||||
mkdirSync(dirname(destWav), { recursive: true });
|
||||
const ff = spawnSync(
|
||||
"ffmpeg",
|
||||
["-y", "-loglevel", "error", "-i", tmp, "-ar", "44100", "-ac", "1", destWav],
|
||||
{ stdio: "ignore" },
|
||||
);
|
||||
rmSync(td, { recursive: true, force: true });
|
||||
return ff.status === 0 && existsSync(destWav);
|
||||
}
|
||||
|
||||
const ELEVENLABS_PY = `
|
||||
import os, sys
|
||||
from elevenlabs.client import ElevenLabs
|
||||
from elevenlabs import save
|
||||
client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])
|
||||
text = open(sys.argv[1]).read()
|
||||
audio = client.text_to_speech.convert(
|
||||
text=text, voice_id=sys.argv[2],
|
||||
model_id="eleven_multilingual_v2", output_format="mp3_44100_128",
|
||||
)
|
||||
save(audio, sys.argv[3])
|
||||
`;
|
||||
|
||||
// ── synthesize one line ───────────────────────────────────────────────────────
|
||||
// Writes wav at wavAbs. Returns { ok, words, error } — words is the raw
|
||||
// [{text,start,end}] array for HeyGen (native), or null for ElevenLabs/Kokoro
|
||||
// (caller must transcribeWav). Never throws; failures return { ok:false, error }
|
||||
// where `error` states WHY (so the caller can surface it, not a bare "TTS failed").
|
||||
export async function synthesizeOne({
|
||||
provider,
|
||||
text,
|
||||
voiceId,
|
||||
lang = "en",
|
||||
speed = 1.0,
|
||||
wavAbs,
|
||||
hyperframesDir,
|
||||
}) {
|
||||
if (provider === "heygen") return synthesizeHeygen({ text, voiceId, lang, speed, wavAbs });
|
||||
if (provider === "elevenlabs") {
|
||||
// The Python helper writes straight to wavAbs; unlike heygen (transcodeToWav)
|
||||
// and kokoro (the `hyperframes tts` CLI), it does NOT create the parent dir,
|
||||
// so on a fresh project (no assets/voice/ yet) the save fails and the line is
|
||||
// silently dropped as "TTS failed - omitted". Create it first, like the other
|
||||
// providers do. Guarded so a mkdir failure (EACCES/EROFS) returns
|
||||
// { ok:false } like the rest of this branch rather than throwing (the
|
||||
// function's contract is "never throws; failures return { ok:false }").
|
||||
try {
|
||||
mkdirSync(dirname(wavAbs), { recursive: true });
|
||||
} catch {
|
||||
return { ok: false, words: null };
|
||||
}
|
||||
const { cmd, args } = pythonInvocation([
|
||||
"-c",
|
||||
ELEVENLABS_PY,
|
||||
writeTmpText(text),
|
||||
voiceId,
|
||||
wavAbs,
|
||||
]);
|
||||
const r = await spawnP(cmd, args, {});
|
||||
return synthResult(r, wavAbs, "elevenlabs (python)");
|
||||
}
|
||||
// kokoro — via the published CLI; --output is relative to the project dir.
|
||||
const wavRel = relTo(hyperframesDir, wavAbs);
|
||||
const args = ["hyperframes", "tts", writeTmpText(text), "--voice", voiceId, "--output", wavRel];
|
||||
if (lang !== "en") args.push("--lang", lang);
|
||||
const r = await spawnP("npx", args, { cwd: hyperframesDir });
|
||||
return synthResult(r, wavAbs, "kokoro (npx hyperframes tts)");
|
||||
}
|
||||
|
||||
// Shape a spawn result into { ok, words, error }, naming why on failure so the
|
||||
// caller surfaces it instead of a bare "TTS failed".
|
||||
export function synthResult(r, wavAbs, label) {
|
||||
if (r.status === 0 && existsSync(wavAbs)) return { ok: true, words: null };
|
||||
const why =
|
||||
r.status !== 0 ? `${label} exited with status ${r.status}` : `${label} produced no wav file`;
|
||||
return { ok: false, words: null, error: why };
|
||||
}
|
||||
|
||||
// `deps` is injectable for tests; production uses the real network/ffmpeg impls.
|
||||
// Every failure path returns an `error` string so the caller can surface WHY a
|
||||
// line was dropped instead of the bare "TTS failed" that hid the real cause
|
||||
// (e.g. an HTTP 402 plan_upgrade_required thrown by heygenJSON was swallowed).
|
||||
export async function synthesizeHeygen({ text, voiceId, lang, speed, wavAbs }, deps = {}) {
|
||||
const requestJSON = deps.heygenJSON ?? heygenJSON;
|
||||
const authHeaders = deps.heygenAuthHeaders ?? heygenAuthHeaders;
|
||||
const fetchImpl = deps.fetch ?? fetch;
|
||||
const transcode = deps.transcodeToWav ?? transcodeToWav;
|
||||
try {
|
||||
const body = { text, voice_id: voiceId, speed };
|
||||
if (lang !== "en") body.language = lang;
|
||||
const payload = await requestJSON(`/voices/speech`, {
|
||||
method: "POST",
|
||||
headers: authHeaders(),
|
||||
body,
|
||||
});
|
||||
const inner = payload.data ?? payload;
|
||||
if (!inner.audio_url) {
|
||||
return { ok: false, words: null, error: "HeyGen /voices/speech returned no audio_url" };
|
||||
}
|
||||
const res = await fetchImpl(inner.audio_url);
|
||||
if (!res.ok) {
|
||||
return { ok: false, words: null, error: `audio_url fetch failed: HTTP ${res.status}` };
|
||||
}
|
||||
const bytes = Buffer.from(await res.arrayBuffer());
|
||||
// .wav output → transcode to 44.1k mono; .mp3 → raw bytes (no ffmpeg). The
|
||||
// engine always asks for .wav; the standalone heygen-tts CLI may ask for .mp3.
|
||||
if (wavAbs.endsWith(".wav")) {
|
||||
if (!transcode(bytes, wavAbs)) {
|
||||
return {
|
||||
ok: false,
|
||||
words: null,
|
||||
error: "wav transcode failed (ffmpeg)",
|
||||
};
|
||||
}
|
||||
} else {
|
||||
mkdirSync(dirname(wavAbs), { recursive: true });
|
||||
writeFileSync(wavAbs, bytes);
|
||||
}
|
||||
const words = Array.isArray(inner.word_timestamps)
|
||||
? inner.word_timestamps
|
||||
.filter((w) => w && typeof w.word === "string" && isFinite(w.start) && isFinite(w.end))
|
||||
.filter((w) => !/^<.*>$/.test(w.word.trim())) // drop <start>/<end> sentinels
|
||||
.map((w) => ({ text: w.word, start: w.start, end: w.end }))
|
||||
: [];
|
||||
return { ok: true, words };
|
||||
} catch (e) {
|
||||
return { ok: false, words: null, error: e?.message ? String(e.message) : String(e) };
|
||||
}
|
||||
}
|
||||
|
||||
// ElevenLabs/Kokoro have no word timings — run Whisper over the wav. Returns the
|
||||
// flat [{id,text,start,end}] word array, or null. Each call uses a throwaway
|
||||
// --dir so parallel scenes don't collide on transcript.json.
|
||||
export async function transcribeWav({ wavRel, lang = "en", hyperframesDir }) {
|
||||
const model = lang === "en" ? "small.en" : "small";
|
||||
const td = mkdtempSync(join(tmpdir(), "hf-trans-"));
|
||||
const args = ["hyperframes", "transcribe", wavRel, "--model", model, "--dir", td];
|
||||
if (lang !== "en") args.push("--language", lang);
|
||||
const r = await spawnP("npx", args, { cwd: hyperframesDir });
|
||||
let words = null;
|
||||
if (r.status === 0) {
|
||||
const src = join(td, "transcript.json");
|
||||
if (existsSync(src)) {
|
||||
try {
|
||||
const arr = JSON.parse(readFileSync(src, "utf8"));
|
||||
if (Array.isArray(arr) && arr.length) words = arr;
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
rmSync(td, { recursive: true, force: true });
|
||||
return words;
|
||||
}
|
||||
|
||||
// ── tiny local utils ──────────────────────────────────────────────────────────
|
||||
function writeTmpText(text) {
|
||||
const td = mkdtempSync(join(tmpdir(), "hf-txt-"));
|
||||
const p = join(td, "line.txt");
|
||||
writeFileSync(p, text);
|
||||
return p;
|
||||
}
|
||||
function relTo(base, abs) {
|
||||
return abs.startsWith(base + "/") ? abs.slice(base.length + 1) : abs;
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { EventEmitter } from "node:events";
|
||||
import {
|
||||
resolveNpxCliFromNpmExecPath,
|
||||
resolveNpxCliPath,
|
||||
resolveSpawnCommand,
|
||||
spawnP,
|
||||
_resetNpxResolutionWarnForTests,
|
||||
} from "./tts.mjs";
|
||||
|
||||
// Regression: on Windows, npx resolves to npx.cmd, which spawn() cannot exec
|
||||
// without shell:true — it fails ENOENT, silently swallowed as ok:false by the
|
||||
// caller. spawnP takes injectable platform/spawnFn params so this doesn't
|
||||
// need to touch the real process.platform or mock node:child_process (whose
|
||||
// ESM exports are non-configurable).
|
||||
function fakeSpawn(captured) {
|
||||
return (cmd, args, opts) => {
|
||||
captured.push({ cmd, args, opts });
|
||||
const p = new EventEmitter();
|
||||
setImmediate(() => p.emit("exit", 0));
|
||||
return p;
|
||||
};
|
||||
}
|
||||
|
||||
const envWithNpxCli = {
|
||||
npm_execpath: "/opt/node/lib/node_modules/npm/bin/npm-cli.js",
|
||||
npm_node_execpath: "/opt/node/bin/node",
|
||||
};
|
||||
const npxCliPath = "/opt/node/lib/node_modules/npm/bin/npx-cli.js";
|
||||
const pathExists = (path) => path === npxCliPath;
|
||||
|
||||
test("resolveNpxCliFromNpmExecPath finds npx-cli next to npm-cli", () => {
|
||||
assert.equal(resolveNpxCliFromNpmExecPath(envWithNpxCli.npm_execpath, pathExists), npxCliPath);
|
||||
});
|
||||
|
||||
test("resolveNpxCliPath finds npx-cli beside node when npm_execpath is unset", () => {
|
||||
const node = "C:/Program Files/nodejs/node.exe";
|
||||
const expected = "C:/Program Files/nodejs/node_modules/npm/bin/npx-cli.js";
|
||||
assert.equal(
|
||||
resolveNpxCliPath(undefined, node, (path) => path === expected),
|
||||
expected,
|
||||
);
|
||||
});
|
||||
|
||||
test("resolveSpawnCommand routes npx through node+npx-cli on win32 without shell:true", () => {
|
||||
const resolved = resolveSpawnCommand(
|
||||
"npx",
|
||||
["hyperframes", "tts", "C:\\Users\\Test User\\line.txt", "--voice", "am_michael"],
|
||||
{},
|
||||
"win32",
|
||||
envWithNpxCli,
|
||||
pathExists,
|
||||
);
|
||||
assert.ok(resolved);
|
||||
assert.equal(resolved.cmd, envWithNpxCli.npm_node_execpath);
|
||||
assert.deepEqual(resolved.args, [
|
||||
npxCliPath,
|
||||
"hyperframes",
|
||||
"tts",
|
||||
"C:\\Users\\Test User\\line.txt",
|
||||
"--voice",
|
||||
"am_michael",
|
||||
]);
|
||||
assert.equal(resolved.opts.shell, undefined);
|
||||
});
|
||||
|
||||
test("resolveSpawnCommand preserves Windows npx shell metacharacters as argv data", () => {
|
||||
const resolved = resolveSpawnCommand(
|
||||
"npx",
|
||||
["hyperframes", "tts", "hello & calc"],
|
||||
{},
|
||||
"win32",
|
||||
envWithNpxCli,
|
||||
pathExists,
|
||||
);
|
||||
assert.ok(resolved);
|
||||
assert.deepEqual(resolved.args, [npxCliPath, "hyperframes", "tts", "hello & calc"]);
|
||||
});
|
||||
|
||||
test("spawnP uses the resolved node+npx-cli command for npx on win32", async () => {
|
||||
const captured = [];
|
||||
await spawnP(
|
||||
"npx",
|
||||
["hyperframes", "tts"],
|
||||
{},
|
||||
"win32",
|
||||
fakeSpawn(captured),
|
||||
envWithNpxCli,
|
||||
pathExists,
|
||||
);
|
||||
assert.equal(captured.length, 1);
|
||||
assert.equal(captured[0].cmd, envWithNpxCli.npm_node_execpath);
|
||||
assert.deepEqual(captured[0].args, [npxCliPath, "hyperframes", "tts"]);
|
||||
assert.equal(captured[0].opts.shell, undefined);
|
||||
});
|
||||
|
||||
test("spawnP does not enable shell for npx on darwin/linux", async () => {
|
||||
const captured = [];
|
||||
await spawnP("npx", ["hyperframes", "tts"], {}, "darwin", fakeSpawn(captured));
|
||||
assert.equal(captured[0].cmd, "npx");
|
||||
assert.deepEqual(captured[0].args, ["hyperframes", "tts"]);
|
||||
assert.equal(captured[0].opts.shell, undefined);
|
||||
});
|
||||
|
||||
test("spawnP does not enable shell for non-npx commands even on win32", async () => {
|
||||
const captured = [];
|
||||
await spawnP("python3", ["-c", "pass"], {}, "win32", fakeSpawn(captured));
|
||||
assert.equal(captured[0].cmd, "python3");
|
||||
assert.deepEqual(captured[0].args, ["-c", "pass"]);
|
||||
assert.equal(captured[0].opts.shell, undefined);
|
||||
});
|
||||
|
||||
test("spawnP resolves npx beside node when npm_execpath is unset on win32", async () => {
|
||||
_resetNpxResolutionWarnForTests();
|
||||
const captured = [];
|
||||
const node = "C:/Program Files/nodejs/node.exe";
|
||||
const npxCli = "C:/Program Files/nodejs/node_modules/npm/bin/npx-cli.js";
|
||||
const result = await spawnP(
|
||||
"npx",
|
||||
["hyperframes", "tts"],
|
||||
{},
|
||||
"win32",
|
||||
fakeSpawn(captured),
|
||||
{ npm_node_execpath: node },
|
||||
(path) => path === npxCli,
|
||||
);
|
||||
assert.equal(result.status, 0);
|
||||
assert.equal(captured.length, 1);
|
||||
assert.equal(captured[0].cmd, node);
|
||||
assert.deepEqual(captured[0].args, [npxCli, "hyperframes", "tts"]);
|
||||
});
|
||||
|
||||
test("spawnP warns once with an accurate diagnostic when neither npx path exists", async () => {
|
||||
_resetNpxResolutionWarnForTests();
|
||||
const errors = [];
|
||||
const originalError = console.error;
|
||||
console.error = (message) => errors.push(String(message));
|
||||
try {
|
||||
const env = { npm_execpath: "C:/missing/npm-cli.js", npm_node_execpath: "C:/node/node.exe" };
|
||||
const missing = () => false;
|
||||
assert.equal(
|
||||
(await spawnP("npx", ["hyperframes", "tts"], {}, "win32", fakeSpawn([]), env, missing))
|
||||
.status,
|
||||
-1,
|
||||
);
|
||||
assert.equal(
|
||||
(await spawnP("npx", ["hyperframes", "tts"], {}, "win32", fakeSpawn([]), env, missing))
|
||||
.status,
|
||||
-1,
|
||||
);
|
||||
} finally {
|
||||
console.error = originalError;
|
||||
}
|
||||
assert.equal(errors.length, 1);
|
||||
assert.match(errors[0], /npm_execpath \(C:\/missing\/npm-cli\.js\)/);
|
||||
assert.doesNotMatch(errors[0], /npm_execpath is not set/);
|
||||
});
|
||||
@@ -0,0 +1,157 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, writeFileSync, chmodSync, rmSync, existsSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import {
|
||||
parseFfmpegDurationBanner,
|
||||
ffprobeDuration,
|
||||
synthesizeOne,
|
||||
synthesizeHeygen,
|
||||
synthResult,
|
||||
} from "./tts.mjs";
|
||||
|
||||
test("parseFfmpegDurationBanner reads ffmpeg's stderr Duration line", () => {
|
||||
const stderr = [
|
||||
"ffmpeg version 6.0",
|
||||
"Input #0, wav, from 'a.wav':",
|
||||
" Duration: 00:00:03.42, bitrate: 705 kb/s",
|
||||
"At least one output file must be specified",
|
||||
].join("\n");
|
||||
assert.equal(parseFfmpegDurationBanner(stderr), 3.42);
|
||||
});
|
||||
|
||||
test("parseFfmpegDurationBanner handles an hours component", () => {
|
||||
const stderr = " Duration: 01:02:03.50, start: 0.000000, bitrate: 128 kb/s";
|
||||
assert.equal(parseFfmpegDurationBanner(stderr), 3723.5);
|
||||
});
|
||||
|
||||
test("parseFfmpegDurationBanner returns NaN when there is no Duration line", () => {
|
||||
assert.ok(Number.isNaN(parseFfmpegDurationBanner("ffmpeg: command not found")));
|
||||
assert.ok(Number.isNaN(parseFfmpegDurationBanner("")));
|
||||
assert.ok(Number.isNaN(parseFfmpegDurationBanner(undefined)));
|
||||
});
|
||||
|
||||
// Regression for the actual bug: ffprobeDuration used to collapse "ffprobe
|
||||
// binary is missing" (ENOENT — the "essentials"-style Windows ffmpeg build
|
||||
// with no ffprobe.exe) and "file is genuinely unreadable" into the same NaN,
|
||||
// giving audio.mjs no way to tell "measure differently" from "give up".
|
||||
//
|
||||
// Builds an isolated PATH containing only a fake `ffmpeg` stub (no `ffprobe`
|
||||
// at all) so ffprobeDuration's spawnSync("ffprobe", ...) call ENOENTs for
|
||||
// real, then verifies it recovers the duration via the ffmpeg fallback
|
||||
// instead of returning NaN.
|
||||
test("ffprobeDuration falls back to ffmpeg when the ffprobe binary itself is missing", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "tts-ffprobe-fallback-"));
|
||||
const fakeFfmpeg = join(dir, "ffmpeg");
|
||||
writeFileSync(
|
||||
fakeFfmpeg,
|
||||
"#!/bin/sh\necho 'Duration: 00:00:02.50, start: 0.000000, bitrate: 128 kb/s' 1>&2\nexit 1\n",
|
||||
);
|
||||
chmodSync(fakeFfmpeg, 0o755);
|
||||
const originalPath = process.env.PATH;
|
||||
try {
|
||||
process.env.PATH = dir; // only the fake ffmpeg resolves; no real ffprobe on this PATH
|
||||
assert.equal(ffprobeDuration("/does/not/matter.wav"), 2.5);
|
||||
} finally {
|
||||
process.env.PATH = originalPath;
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("ffprobeDuration returns NaN when neither ffprobe nor ffmpeg resolve", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "tts-no-binaries-"));
|
||||
const originalPath = process.env.PATH;
|
||||
try {
|
||||
process.env.PATH = dir; // empty directory — nothing resolves
|
||||
assert.ok(Number.isNaN(ffprobeDuration("/does/not/matter.wav")));
|
||||
} finally {
|
||||
process.env.PATH = originalPath;
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("synthesizeOne(elevenlabs) creates the output dir before writing", async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "tts-el-mkdir-"));
|
||||
const wavAbs = join(dir, "assets", "voice", "line-0.wav"); // nested, not yet created
|
||||
const savedKey = process.env.ELEVENLABS_API_KEY;
|
||||
try {
|
||||
// Unset the key so the Python side fails fast — the mkdir must run before
|
||||
// the spawn regardless, which is what this guards.
|
||||
delete process.env.ELEVENLABS_API_KEY;
|
||||
await synthesizeOne({
|
||||
provider: "elevenlabs",
|
||||
text: "hi",
|
||||
voiceId: "v",
|
||||
wavAbs,
|
||||
hyperframesDir: dir,
|
||||
});
|
||||
assert.ok(existsSync(dirname(wavAbs)), "output directory should be created");
|
||||
} finally {
|
||||
if (savedKey === undefined) delete process.env.ELEVENLABS_API_KEY;
|
||||
else process.env.ELEVENLABS_API_KEY = savedKey;
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("synthesizeHeygen surfaces a thrown HTTP error (e.g. 402) instead of swallowing it", async () => {
|
||||
const res = await synthesizeHeygen(
|
||||
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
|
||||
{
|
||||
heygenAuthHeaders: () => ({}),
|
||||
heygenJSON: async () => {
|
||||
throw new Error("HeyGen POST /voices/speech → HTTP 402\nplan_upgrade_required");
|
||||
},
|
||||
},
|
||||
);
|
||||
assert.equal(res.ok, false);
|
||||
assert.match(res.error, /402/);
|
||||
assert.match(res.error, /plan_upgrade_required/);
|
||||
});
|
||||
|
||||
test("synthesizeHeygen surfaces a failed audio_url fetch with its status", async () => {
|
||||
const res = await synthesizeHeygen(
|
||||
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
|
||||
{
|
||||
heygenAuthHeaders: () => ({}),
|
||||
heygenJSON: async () => ({ data: { audio_url: "http://audio.example/x" } }),
|
||||
fetch: async () => ({ ok: false, status: 403 }),
|
||||
},
|
||||
);
|
||||
assert.equal(res.ok, false);
|
||||
assert.match(res.error, /HTTP 403/);
|
||||
});
|
||||
|
||||
test("synthesizeHeygen reports a missing audio_url", async () => {
|
||||
const res = await synthesizeHeygen(
|
||||
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
|
||||
{ heygenAuthHeaders: () => ({}), heygenJSON: async () => ({}) },
|
||||
);
|
||||
assert.equal(res.ok, false);
|
||||
assert.match(res.error, /no audio_url/);
|
||||
});
|
||||
|
||||
test("synthesizeHeygen reports wav transcode failures", async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-tts-test-"));
|
||||
try {
|
||||
const res = await synthesizeHeygen(
|
||||
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: join(dir, "voice.wav") },
|
||||
{
|
||||
heygenAuthHeaders: () => ({}),
|
||||
heygenJSON: async () => ({ data: { audio_url: "http://audio.example/x" } }),
|
||||
fetch: async () => ({ ok: true, status: 200, arrayBuffer: async () => new ArrayBuffer(0) }),
|
||||
transcodeToWav: () => false,
|
||||
},
|
||||
);
|
||||
assert.equal(res.ok, false);
|
||||
assert.equal(res.error, "wav transcode failed (ffmpeg)");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("synthResult names a non-zero subprocess exit", () => {
|
||||
const res = synthResult({ status: 2 }, "/tmp/none.wav", "kokoro (npx hyperframes tts)");
|
||||
assert.equal(res.ok, false);
|
||||
assert.match(res.error, /kokoro .* exited with status 2/);
|
||||
});
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate BGM using Google Lyria RealTime API.
|
||||
|
||||
Usage:
|
||||
python lyria-recipe.py --output <path> --duration <seconds> [tuning flags]
|
||||
|
||||
Requires:
|
||||
$GOOGLE_API_KEY or $GEMINI_API_KEY environment variable (treated as aliases).
|
||||
pip install google-genai python-dotenv. audio.mjs Step 4b installs these on
|
||||
demand when a key is set but google.genai is not importable; if that install
|
||||
fails it falls back to local MusicGen rather than leaving the video with no BGM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import wave
|
||||
from pathlib import Path
|
||||
|
||||
# Windows sizes stdio to the ANSI code page (cp1252). These scripts emit UTF-8 on
|
||||
# every platform; say so rather than depending on the console's code page. Carry
|
||||
# `errors` across: reconfigure() resets it to "strict", and CPython deliberately gives
|
||||
# stderr "backslashreplace" so the diagnostic path can never itself raise.
|
||||
for _stream in (sys.stdout, sys.stderr):
|
||||
if hasattr(_stream, "reconfigure"):
|
||||
_stream.reconfigure(encoding="utf-8", errors=_stream.errors)
|
||||
|
||||
DEFAULT_PROMPT = "Uplifting corporate tech, bright and modern, gentle piano with synth pads"
|
||||
SAMPLE_RATE = 48000
|
||||
CHANNELS = 2
|
||||
SAMPLE_WIDTH = 2 # 16-bit
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(description="Generate BGM via Google Lyria RealTime.")
|
||||
p.add_argument("--output", required=True, help="Output WAV path.")
|
||||
p.add_argument("--duration", type=float, required=True, help="Target duration in seconds.")
|
||||
p.add_argument("--prompt", default=DEFAULT_PROMPT, help="Mood / instrumentation prompt.")
|
||||
p.add_argument("--negative-prompt", default=None, help="Styles to exclude (optional).")
|
||||
p.add_argument("--bpm", type=int, default=110)
|
||||
p.add_argument("--brightness", type=float, default=0.8, help="0-1, higher = brighter mood.")
|
||||
p.add_argument("--density", type=float, default=0.5, help="0-1, higher = fuller mix.")
|
||||
p.add_argument(
|
||||
"--scale",
|
||||
default="MAJOR",
|
||||
help="MAJOR / MINOR / PENTATONIC / etc. — see google.genai.types.Scale. Pass empty string for none.",
|
||||
)
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
async def generate_bgm(args: argparse.Namespace) -> dict:
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
|
||||
api_key = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY") or ""
|
||||
if not api_key:
|
||||
raise RuntimeError("Neither GOOGLE_API_KEY nor GEMINI_API_KEY is set.")
|
||||
|
||||
client = genai.Client(
|
||||
api_key=api_key,
|
||||
http_options={"api_version": "v1alpha"},
|
||||
)
|
||||
|
||||
out_path = Path(args.output)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
target_bytes = int(args.duration * SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH)
|
||||
|
||||
cfg: dict = {"bpm": args.bpm, "temperature": 1.0}
|
||||
if args.density is not None:
|
||||
cfg["density"] = args.density
|
||||
if args.brightness is not None:
|
||||
cfg["brightness"] = args.brightness
|
||||
if args.scale:
|
||||
scale_enum = getattr(types.Scale, args.scale, None)
|
||||
if scale_enum:
|
||||
cfg["scale"] = scale_enum
|
||||
|
||||
prompts = [types.WeightedPrompt(text=args.prompt, weight=1.0)]
|
||||
if args.negative_prompt:
|
||||
prompts.append(types.WeightedPrompt(text=args.negative_prompt, weight=-1.0))
|
||||
|
||||
buf = bytearray()
|
||||
timeout = args.duration + 8
|
||||
|
||||
async with client.aio.live.music.connect(
|
||||
model="models/lyria-realtime-exp",
|
||||
) as session:
|
||||
await session.set_weighted_prompts(prompts=prompts)
|
||||
await session.set_music_generation_config(
|
||||
config=types.LiveMusicGenerationConfig(**cfg),
|
||||
)
|
||||
await session.play()
|
||||
|
||||
async def collect():
|
||||
while len(buf) < target_bytes:
|
||||
async for msg in session.receive():
|
||||
sc = msg.server_content
|
||||
if sc and sc.audio_chunks:
|
||||
for chunk in sc.audio_chunks:
|
||||
buf.extend(chunk.data)
|
||||
if len(buf) >= target_bytes:
|
||||
return
|
||||
await asyncio.sleep(1e-6)
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(collect(), timeout=timeout)
|
||||
except TimeoutError:
|
||||
print(f"Timeout after {timeout:.0f}s, collected {len(buf)} bytes", file=sys.stderr)
|
||||
|
||||
audio = bytes(buf[:target_bytes])
|
||||
with wave.open(str(out_path), "wb") as wf:
|
||||
wf.setnchannels(CHANNELS)
|
||||
wf.setsampwidth(SAMPLE_WIDTH)
|
||||
wf.setframerate(SAMPLE_RATE)
|
||||
wf.writeframes(audio)
|
||||
|
||||
actual_duration = len(audio) / (SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH)
|
||||
print(f"BGM: {out_path} ({actual_duration:.2f}s)")
|
||||
return {"file": str(out_path), "duration_sec": round(actual_duration, 2)}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
try:
|
||||
asyncio.run(generate_bgm(args))
|
||||
except RuntimeError as exc:
|
||||
print(f"BGM generation failed: {exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env node
|
||||
// Phase 4c pre-assemble helper — wait for detached BGM, then write status.
|
||||
//
|
||||
// audio.mjs may launch Lyria / MusicGen in a detached process so voice work can
|
||||
// keep moving. Before assemble-index.mjs decides whether to emit the BGM audio
|
||||
// track, this script gives the background renderer a bounded chance to finish
|
||||
// and converts log/process state into a small bgm_status.json file.
|
||||
//
|
||||
// Always exits 0 for normal pipeline use: missing/failed BGM should not block a
|
||||
// voice/captions/SFX render. Structural invocation errors still exit 1.
|
||||
//
|
||||
// Usage:
|
||||
// node wait-bgm.mjs --audio-meta ./audio_meta.json --hyperframes . \
|
||||
// [--timeout-ms 120000] [--interval-ms 2000] [--out ./bgm_status.json]
|
||||
|
||||
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const flag = (name, def) => {
|
||||
const i = argv.indexOf(`--${name}`);
|
||||
return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
|
||||
};
|
||||
|
||||
function die(msg) {
|
||||
console.error(`✗ wait-bgm.mjs: ${msg}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const audioMetaPath = resolve(flag("audio-meta", "./audio_meta.json"));
|
||||
const hyperframesDir = resolve(flag("hyperframes", "."));
|
||||
const outPath = resolve(flag("out", join(hyperframesDir, "bgm_status.json")));
|
||||
const timeoutMs = Math.max(0, Number(flag("timeout-ms", "120000")) || 0);
|
||||
const intervalMs = Math.max(250, Number(flag("interval-ms", "2000")) || 2000);
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise((resolveSleep) => setTimeout(resolveSleep, ms));
|
||||
}
|
||||
|
||||
function isProcessAlive(pid) {
|
||||
if (!pid || !Number.isFinite(Number(pid))) return false;
|
||||
try {
|
||||
process.kill(Number(pid), 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function readTail(path, maxChars = 6000) {
|
||||
if (!path) return "";
|
||||
try {
|
||||
const txt = readFileSync(path, "utf8");
|
||||
return txt.slice(Math.max(0, txt.length - maxChars));
|
||||
} catch (error) {
|
||||
if (error.code === "ENOENT" || error.code === "ENOTDIR") return "";
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function detectFailure(logTail) {
|
||||
if (!logTail) return "";
|
||||
const lines = logTail.split("\n");
|
||||
// Bare "out of range" over-matched benign BGM-renderer logs (e.g. a "sample rate
|
||||
// out of range, resampling" notice), mislabelling a healthy track as failed and
|
||||
// silently dropping the music. Anchor to the actual crash strings instead:
|
||||
// Python "(list) index out of range" and torch "index … out of bounds".
|
||||
const idx = lines.findIndex((line) =>
|
||||
/(Traceback|IndexError|RuntimeError|Exception|Killed|No space left|Cannot allocate|index out of range|out of bounds)/i.test(
|
||||
line,
|
||||
),
|
||||
);
|
||||
if (idx < 0) return "";
|
||||
return lines.slice(idx).join("\n").trim();
|
||||
}
|
||||
|
||||
function writeStatus(status) {
|
||||
const payload = {
|
||||
generated_at: new Date().toISOString(),
|
||||
...status,
|
||||
};
|
||||
writeFileSync(outPath, JSON.stringify(payload, null, 2) + "\n");
|
||||
return payload;
|
||||
}
|
||||
|
||||
if (!existsSync(audioMetaPath)) die(`audio_meta.json missing at ${audioMetaPath}`);
|
||||
|
||||
const audioMeta = JSON.parse(readFileSync(audioMetaPath, "utf8"));
|
||||
const bgmPath = audioMeta.bgm?.path || "";
|
||||
const bgmAbsPath = bgmPath ? join(hyperframesDir, bgmPath) : "";
|
||||
const logPath = audioMeta.bgm_log || "";
|
||||
const pid = audioMeta.bgm_pid || null;
|
||||
|
||||
const base = {
|
||||
enabled: Boolean(audioMeta.bgm_pending && bgmPath),
|
||||
provider: audioMeta.bgm_provider || null,
|
||||
mode: audioMeta.bgm_mode || null,
|
||||
path: bgmPath || null,
|
||||
log: logPath || null,
|
||||
pid,
|
||||
target_duration_s: audioMeta.bgm_target_duration_s || null,
|
||||
seed_duration_s: audioMeta.bgm_seed_duration_s || null,
|
||||
loop_count: audioMeta.bgm_loop_count || null,
|
||||
timeout_ms: timeoutMs,
|
||||
};
|
||||
|
||||
if (!base.enabled) {
|
||||
const status = writeStatus({
|
||||
...base,
|
||||
status: "disabled",
|
||||
ready: false,
|
||||
waited_ms: 0,
|
||||
message: "BGM not requested or disabled in audio_meta.json.",
|
||||
});
|
||||
console.log(`✓ bgm: ${status.status} (${status.message})`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const started = Date.now();
|
||||
let lastFailure = "";
|
||||
let lastTail = "";
|
||||
|
||||
while (Date.now() - started <= timeoutMs) {
|
||||
if (existsSync(bgmAbsPath)) {
|
||||
const size = statSync(bgmAbsPath).size;
|
||||
writeStatus({
|
||||
...base,
|
||||
status: "ready",
|
||||
ready: true,
|
||||
waited_ms: Date.now() - started,
|
||||
size_bytes: size,
|
||||
message: `BGM ready at ${bgmPath}.`,
|
||||
});
|
||||
console.log(`✓ bgm: ready (${bgmPath}, ${size}B)`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
lastTail = readTail(logPath);
|
||||
lastFailure = detectFailure(lastTail);
|
||||
const alive = isProcessAlive(pid);
|
||||
if (lastFailure || (!alive && logPath && existsSync(logPath))) {
|
||||
const message = lastFailure
|
||||
? `BGM renderer failed; see ${logPath}.`
|
||||
: `BGM renderer exited without writing ${bgmPath}; see ${logPath}.`;
|
||||
const status = writeStatus({
|
||||
...base,
|
||||
status: "failed",
|
||||
ready: false,
|
||||
waited_ms: Date.now() - started,
|
||||
process_alive: alive,
|
||||
message,
|
||||
error_tail: lastFailure || lastTail.slice(-2000),
|
||||
});
|
||||
console.log(`! bgm: failed (${status.message})`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (timeoutMs === 0) break;
|
||||
await sleep(Math.min(intervalMs, Math.max(0, timeoutMs - (Date.now() - started))));
|
||||
}
|
||||
|
||||
const status = writeStatus({
|
||||
...base,
|
||||
status: "timeout",
|
||||
ready: false,
|
||||
waited_ms: Date.now() - started,
|
||||
process_alive: isProcessAlive(pid),
|
||||
message: `Timed out waiting for ${bgmPath}; assemble-index will skip BGM if still absent.`,
|
||||
log_tail: lastTail.slice(-2000),
|
||||
});
|
||||
console.log(`! bgm: timeout after ${status.waited_ms}ms (${bgmPath})`);
|
||||
@@ -0,0 +1,65 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { test } from "node:test";
|
||||
import { runInNewContext } from "node:vm";
|
||||
|
||||
const source = readFileSync(new URL("./wait-bgm.mjs", import.meta.url), "utf8");
|
||||
const start = source.indexOf("function readTail(");
|
||||
const end = source.indexOf("\nfunction detectFailure(", start);
|
||||
function load(read) {
|
||||
return runInNewContext(`${source.slice(start, end)}; readTail`, {
|
||||
readFileSync: read,
|
||||
existsSync: () => true,
|
||||
statSync: () => {
|
||||
throw new Error("pathname stat must not precede log read");
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
test("reads a log once without a separate pathname check", () => {
|
||||
let calls = 0;
|
||||
const read = load((path, encoding) => {
|
||||
calls++;
|
||||
assert.equal(path, "renderer.log");
|
||||
assert.equal(encoding, "utf8");
|
||||
return "prefix\nRuntimeError: failed";
|
||||
});
|
||||
assert.equal(read("renderer.log", 20), "RuntimeError: failed");
|
||||
assert.equal(calls, 1);
|
||||
});
|
||||
|
||||
test("preserves UTF-16 character tail boundaries and zero-length tails", () => {
|
||||
const txt = "🎵日本語".repeat(2000);
|
||||
const read = load(() => txt);
|
||||
assert.equal(read("log"), txt.slice(-6000));
|
||||
assert.equal(read("log", 0), "");
|
||||
});
|
||||
|
||||
for (const code of ["ENOENT", "ENOTDIR"]) {
|
||||
test(`a log removed during polling is absent (${code})`, () => {
|
||||
const read = load(() => {
|
||||
throw Object.assign(new Error(code), { code });
|
||||
});
|
||||
assert.equal(read("log"), "");
|
||||
});
|
||||
}
|
||||
|
||||
test("preserves unexpected read errors", () => {
|
||||
const error = Object.assign(new Error("denied"), { code: "EACCES" });
|
||||
assert.throws(
|
||||
() =>
|
||||
load(() => {
|
||||
throw error;
|
||||
})("log"),
|
||||
(caught) => caught === error,
|
||||
);
|
||||
});
|
||||
|
||||
test("empty log path does not access the filesystem", () => {
|
||||
assert.equal(
|
||||
load(() => {
|
||||
throw new Error("unexpected read");
|
||||
})(""),
|
||||
"",
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
# LUT library (authoring)
|
||||
|
||||
`index.json` is the agent-consumed catalog of color-grade looks. Each entry resolves
|
||||
on demand — no `.cube` bodies are committed to the repo.
|
||||
|
||||
Each look has:
|
||||
|
||||
- `id`, `description`, `tags`, `intensity` — matching + application metadata.
|
||||
- `url` (optional) — a hosted `.cube` downloaded, validated, and frozen at resolve
|
||||
time, exactly like bgm/image assets.
|
||||
- `params` (optional) — a deterministic `buildCube` spec used offline (`--local-only`)
|
||||
or as a fallback if the `url` download/validation fails.
|
||||
|
||||
An entry needs at least one of `url` or `params`; prefer both (CDN url with a params
|
||||
fallback) so resolution is never blocked on the network.
|
||||
|
||||
## Hosting a new look (operators)
|
||||
|
||||
1. Generate the `.cube` (e.g. `resolve -t lut --params '{...}'` or a graded export).
|
||||
2. Upload it to the public CDN origin bucket:
|
||||
|
||||
```
|
||||
aws s3 cp <id>.cube s3://heygen-public/luts/<id>.cube
|
||||
```
|
||||
|
||||
It is then served at `https://static.heygen.ai/luts/<id>.cube` (CloudFront).
|
||||
|
||||
3. Add an entry to `index.json` with that `url` (and ideally a `params` fallback).
|
||||
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"notes": "Look entries resolve on-demand from a CDN .cube `url` (downloaded + frozen, like bgm/image); `params` is a deterministic buildCube fallback used offline (--local-only) or if the download/validation fails. No .cube bodies are committed. See README.md to author or host a new look.",
|
||||
"looks": [
|
||||
{
|
||||
"id": "teal-orange-blockbuster",
|
||||
"description": "Teal shadows and warm orange highlights for blockbuster-style cinematic footage.",
|
||||
"tags": ["teal", "orange", "blockbuster", "cinematic", "split tone", "movie"],
|
||||
"intensity": 0.85,
|
||||
"url": "https://static.heygen.ai/luts/teal-orange-blockbuster.cube",
|
||||
"params": {
|
||||
"contrast": 0.18,
|
||||
"saturation": 0.08,
|
||||
"vibrance": 0.12,
|
||||
"splitTone": {
|
||||
"intensity": 0.62,
|
||||
"balance": 0.52,
|
||||
"shadows": [-0.04, 0.05, 0.09],
|
||||
"highlights": [0.1, 0.04, -0.03]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "bleach-bypass",
|
||||
"description": "High-contrast desaturated bleach bypass look with strong blacks.",
|
||||
"tags": ["bleach", "bypass", "desaturated", "high contrast", "gritty", "film"],
|
||||
"intensity": 0.8,
|
||||
"url": "https://static.heygen.ai/luts/bleach-bypass.cube",
|
||||
"params": {
|
||||
"blacks": 0.04,
|
||||
"shadows": -0.08,
|
||||
"highlights": 0.08,
|
||||
"whites": 0.18,
|
||||
"contrast": 0.55,
|
||||
"temperature": -0.02,
|
||||
"saturation": -0.72,
|
||||
"vibrance": -0.25
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "film-fade",
|
||||
"description": "Soft faded film wash with lifted blacks and warm highlights.",
|
||||
"tags": ["film", "fade", "faded", "wash", "warm", "vintage"],
|
||||
"intensity": 0.75,
|
||||
"url": "https://static.heygen.ai/luts/film-fade.cube",
|
||||
"params": {
|
||||
"blacks": 0.35,
|
||||
"shadows": 0.18,
|
||||
"highlights": 0.02,
|
||||
"whites": -0.03,
|
||||
"contrast": -0.28,
|
||||
"temperature": 0.16,
|
||||
"saturation": -0.12,
|
||||
"vibrance": -0.08
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
# Audio engine — voiceover, music, SFX, captions, transcription
|
||||
|
||||
For a full audio pass (TTS voiceover + background music + sound effects in one
|
||||
shot), use the shared engine at `audio/scripts/audio.mjs`. It takes a neutral
|
||||
`audio_request.json` and writes `audio_meta.json` plus assets under
|
||||
`.media/audio/{voice,bgm,sfx}`:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/audio/scripts/audio.mjs --request ./audio_request.json --out ./audio_meta.json
|
||||
```
|
||||
|
||||
- **Request** `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`: `id` joins each line back to your model; `bgm.mode` = `retrieve | generate | none` (omit for auto). `--only tts,bgm,sfx` runs a subset and merges into an existing `--out`.
|
||||
- **Output** `audio_meta.json` (id-keyed): `voices[].{path,duration_s,words[]}` (word timestamps for captions), `sfx[]`, `bgm`, `total_duration_s`.
|
||||
- **HeyGen free-usage path**: HeyGen CLI auth unlocks TTS plus music/SFX retrieval. Local/provider-specific generators are explicit alternatives where installed; run `node <SKILL_DIR>/scripts/resolve.mjs --doctor` before assuming retrieval or TTS will work.
|
||||
- If BGM took the generate path (`bgm_pending: true`), run `audio/scripts/wait-bgm.mjs` before final render.
|
||||
|
||||
Single-shot helpers: `audio/scripts/heygen-tts.mjs` (one voice file). Transcription / background removal / captions use the `hyperframes` CLI (`transcribe`, `remove-background`), see the per-topic guides in `audio/references/` (`tts.md`, `bgm.md`, `sfx.md`, `transcribe.md`, `remove-background.md`, `captions/`).
|
||||
|
||||
Transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via `scripts/transcribe.mjs`, with whisper.cpp auto-fallback (see `references/operations.md`).
|
||||
@@ -0,0 +1,160 @@
|
||||
# Color grading — grade blocks and LUTs
|
||||
|
||||
Use `grade` when you need a canonical HyperFrames grading/effects payload for
|
||||
an `<img>` or `<video>`. Core presets and params-backed LUT entries resolve
|
||||
locally; future CDN-backed LUT entries require network unless already
|
||||
frozen. Persist a decided payload with the CLI rather than editing HTML by
|
||||
hand:
|
||||
|
||||
For a vague but explicit polish request, do not jump directly from intent to a
|
||||
preset name. Read `media-treatments.md`, choose a treatment whose subject and
|
||||
avoid rules match the actual media, apply its conservative base with only
|
||||
justified bounded tuning, then complete its visual verification steps. A named
|
||||
owned treatment uses the exact preset/payload in its recipe; do not run the
|
||||
generic grade/LUT resolver first.
|
||||
|
||||
Stop here and use that treatment workflow for requests such as retro, old home
|
||||
video, camcorder, film, print, ASCII, glitch, privacy, or a media reveal. Do not
|
||||
assemble those from a generic LUT plus handmade CSS vignette/grain/opacity.
|
||||
|
||||
**Never `cat`/read a `.cube` file into context.** A 3D LUT is ~size^3 lines of raw numbers (33^3 ≈ 36k lines at the default size). It bloats context and carries zero human/agent-legible signal. To understand or choose a LUT, use `hyperframes grade-compare` to see it rendered, or `cube-validate.mjs` for a one-line `{ok,size}` check. Read `.media/index.md` or `luts/index.json` for the description. Never read the LUT body itself.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json
|
||||
```
|
||||
|
||||
Preset-first output uses the core runtime vocabulary and does not freeze a file:
|
||||
|
||||
```json
|
||||
{
|
||||
"preset": "warm-daylight",
|
||||
"intensity": 1
|
||||
}
|
||||
```
|
||||
|
||||
Apply that payload to one unambiguous real media element:
|
||||
|
||||
```bash
|
||||
hyperframes media-treatment --project . --file index.html \
|
||||
--selector '#hero' \
|
||||
--grading '{"preset":"warm-daylight","intensity":1}' --apply --json
|
||||
```
|
||||
|
||||
Use `--dry-run` before writing when scope is uncertain and `--clear` to remove
|
||||
the treatment. The low-level persisted result is still normal HTML:
|
||||
|
||||
```html
|
||||
<video
|
||||
class="clip"
|
||||
src="./media/scene.mp4"
|
||||
data-color-grading='{"preset":"warm-daylight","intensity":1}'
|
||||
></video>
|
||||
```
|
||||
|
||||
Direct attribute authoring is a fallback for environments where the CLI is not
|
||||
available, not the primary agent workflow.
|
||||
|
||||
To build a treatment that is not already represented by a recipe, inspect the
|
||||
canonical toolbox first:
|
||||
|
||||
```bash
|
||||
hyperframes media-treatment --capabilities --json
|
||||
```
|
||||
|
||||
It reports a concise family map. Read `--capability grading` for the processing
|
||||
order, then request only the focused family needed to get its legal controls
|
||||
and ranges from Core. Compose one nested payload and pass it back through
|
||||
`hyperframes media-treatment`; the command rejects unknown keys before
|
||||
mutation. Do not generate or hand-edit a LUT merely to combine controls already
|
||||
owned by the realtime shader.
|
||||
|
||||
For seek-safe effect motion, animate only the runtime-supported CSS properties
|
||||
on that same real media element with its registered paused GSAP timeline:
|
||||
|
||||
| CSS property | Range |
|
||||
| ---------------------------------- | ------- |
|
||||
| `--hf-color-grading-intensity` | 0 to 1 |
|
||||
| `--hf-color-grading-lut-intensity` | 0 to 1 |
|
||||
| `--hf-color-grading-exposure` | -2 to 2 |
|
||||
| `--hf-color-grading-blur` | 0 to 1 |
|
||||
| `--hf-color-grading-bloom` | 0 to 3 |
|
||||
| `--hf-color-grading-kuwahara` | 0 to 1 |
|
||||
| `--hf-color-grading-pixelate` | 0 to 1 |
|
||||
| `--hf-color-grading-ascii` | 0 to 1 |
|
||||
| `--hf-color-grading-dither` | 0 to 1 |
|
||||
|
||||
Author the initial value directly in the media element's inline `style`, then
|
||||
use finite `tl.to()` keyframes. Do not use a frame-zero `tl.set()`, CSS
|
||||
animation clocks, timers, random values, or `onUpdate` callbacks. The static
|
||||
`data-color-grading` payload remains the fallback and source of the other
|
||||
controls.
|
||||
|
||||
For a reusable color transform beyond the preset vocabulary, freeze a validated
|
||||
`.cube` under `.media/luts/` and return a block that references it:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "teal orange blockbuster" --project . --json
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"lut": { "src": ".media/luts/grade_001.cube", "intensity": 0.85 }
|
||||
}
|
||||
```
|
||||
|
||||
Use `lut` when you only need the reusable `.cube` file:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project .
|
||||
```
|
||||
|
||||
For a describable technical look, author an explicit parametric LUT with `--params`:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type lut --params '{"contrast":0.2,"temperature":-0.3}' --project .
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type grade --params '{"exposure":0.2}' --project . --json
|
||||
```
|
||||
|
||||
For a LUT generated by your own script, ingest it with `--from`; media-use validates it before registration and rejects invalid or oversized cubes:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type lut --from custom.cube --project .
|
||||
```
|
||||
|
||||
Parametric math (`buildCube`) cannot reproduce real film stocks or emulsion
|
||||
transforms. Use a CDN-backed scanned `.cube` entry or ingest a real scanned
|
||||
`.cube` for those.
|
||||
|
||||
For visual selection, list reusable LUT candidates with
|
||||
`resolve --type grade --candidates`, write the promising entries to a
|
||||
`grades.json`, run
|
||||
`hyperframes grade-compare --for <frame> --grades grades.json`, then commit the
|
||||
winner with `resolve -t grade` as the final `data-color-grading` block.
|
||||
|
||||
For media already selected in a composition, use `media-treatment --analyze`
|
||||
when you need side-effect-free `ffmpeg`/`ffprobe` signalstats evidence. It
|
||||
returns source metadata, HDR/unknown-LOG warnings, and a bounded `adjust`
|
||||
suggestion without modifying the composition. The suggestion is a starting
|
||||
point for visual review, not an automatic neutralization of intentional color.
|
||||
|
||||
```bash
|
||||
hyperframes media-treatment --project . --file index.html \
|
||||
--selector '#hero' --analyze --json
|
||||
```
|
||||
|
||||
For an unbound source file, `resolve --type grade --for ... --analyze` remains
|
||||
available. Without `--analyze`, that resolver records a grade candidate in
|
||||
`.media`; use that form only when you intend to keep the candidate.
|
||||
|
||||
Library LUT entries live in `luts/index.json`. Each entry keeps `id`,
|
||||
`description`, `tags`, and `intensity`, then supplies either compact `params`
|
||||
for on-demand `buildCube(params)` generation or a direct CDN `url` for future
|
||||
scanned `.cube` files. Do not commit generated `.cube` bodies; resolve
|
||||
validates generated or downloaded cubes as it freezes them under
|
||||
`.media/luts/`.
|
||||
|
||||
```bash
|
||||
node skills/media-use/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project . --json
|
||||
node skills/media-use/scripts/lib/cube-validate.mjs .media/luts/lut_001.cube
|
||||
```
|
||||
@@ -0,0 +1,929 @@
|
||||
# Media treatment recipes
|
||||
|
||||
These are optional tested seeds, not the complete capability surface. Read the
|
||||
shared policy and choose one relevant section through `media-treatments.md`.
|
||||
Agents may modify or combine a seed with compatible canonical controls after
|
||||
inspecting the media, or assemble a bespoke payload from
|
||||
`hyperframes media-treatment --capabilities --json` when no seed fits.
|
||||
|
||||
## Natural Portrait
|
||||
|
||||
Use for a talking head, interview, presenter, or people-focused photo whose
|
||||
intended result is natural, polished, and restrained.
|
||||
|
||||
Do not use when the face is incidental or tiny, the source is intentionally
|
||||
neon/monochrome/strongly stylized, or the requested result is beauty retouching.
|
||||
This treatment changes the whole frame; it is not a face mask or skin-smoothing
|
||||
effect.
|
||||
|
||||
Inspect face exposure, highlight retention, shadow detail, white balance, and
|
||||
whether the existing look is intentional. Signalstats do not detect faces or
|
||||
creative intent.
|
||||
|
||||
### Base payload
|
||||
|
||||
Start here, then tune only when the sampled frames justify it:
|
||||
|
||||
```json
|
||||
{ "preset": "skin-soft", "intensity": 0.6 }
|
||||
```
|
||||
|
||||
`skin-soft` is a global tonal/color preset whose vibrance math is reduced for
|
||||
skin-like colors. It does not blur, retouch, segment, or track a face.
|
||||
|
||||
### Bounded tuning
|
||||
|
||||
Adjustment values are absolute values in the final payload, not deltas added to
|
||||
the preset. Keep changes inside these conservative ranges unless the user asks
|
||||
for a stylized result:
|
||||
|
||||
| Property | Natural Portrait range |
|
||||
| ----------- | ---------------------- |
|
||||
| intensity | 0.45 to 0.75 |
|
||||
| exposure | -0.06 to 0.14 |
|
||||
| contrast | -0.05 to 0.08 |
|
||||
| highlights | -0.18 to -0.04 |
|
||||
| shadows | 0.04 to 0.18 |
|
||||
| whites | -0.10 to 0.04 |
|
||||
| blacks | -0.06 to 0.06 |
|
||||
| temperature | -0.05 to 0.10 |
|
||||
| tint | -0.03 to 0.05 |
|
||||
| vibrance | 0 to 0.06 |
|
||||
| saturation | -0.04 to 0.06 |
|
||||
|
||||
Leave grain, blur, and pixelate at zero. A vignette is optional at `0` to
|
||||
`0.05` only when it improves subject focus without looking like an effect.
|
||||
|
||||
Manual controls must stay inside their schema section; they are never
|
||||
top-level keys. A tuned Natural Portrait payload looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"preset": "skin-soft",
|
||||
"intensity": 0.58,
|
||||
"adjust": {
|
||||
"highlights": -0.08,
|
||||
"shadows": 0.08,
|
||||
"temperature": 0.02,
|
||||
"vibrance": 0.02
|
||||
},
|
||||
"details": { "vignette": 0.03 }
|
||||
}
|
||||
```
|
||||
|
||||
Use the same nested shape in `grade-compare` candidate files. `adjust` owns
|
||||
tonal/color controls, `details` owns vignette/grain, and `effects` owns blur,
|
||||
pixelate, chroma bleed, and the advanced treatment primitives below.
|
||||
|
||||
During the common comparison, reject any result that makes skin implausible,
|
||||
loses highlight detail, flattens or desaturates dark skin, or casts clothing and
|
||||
background colors accidentally.
|
||||
|
||||
## Product Polish
|
||||
|
||||
Use for photographed or filmed physical products when the goal is clean,
|
||||
accurate, dimensional presentation. Protect product color, material texture,
|
||||
label readability, specular highlights, and intentional lighting.
|
||||
|
||||
Do not use this treatment for literal app/site screenshots or screen captures;
|
||||
follow UI Fidelity below. Do not neutralize a lifestyle scene's deliberate
|
||||
ambient color, and do not infer exact brand-color correction without a neutral
|
||||
reference or known product color.
|
||||
|
||||
Inspect the product separately from its background. Check white balance, label
|
||||
legibility, surface texture, highlight clipping, shadow detail, white point,
|
||||
and black point. Statistics cannot identify a white package, metallic
|
||||
highlight, amber glass, or intentional warm light.
|
||||
|
||||
### Base payload
|
||||
|
||||
Compare this restrained correction against the untouched source:
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 0.7,
|
||||
"adjust": {
|
||||
"exposure": 0.01,
|
||||
"contrast": 0.06,
|
||||
"highlights": -0.1,
|
||||
"shadows": 0.04,
|
||||
"whites": 0.02,
|
||||
"blacks": -0.03,
|
||||
"vibrance": 0.03,
|
||||
"saturation": 0.02
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This is a comparison starting point, not an instruction to change an already
|
||||
finished source. If the original has accurate color, clean endpoints, and good
|
||||
texture, leave the pixels unchanged and polish through framing or motion.
|
||||
|
||||
### Bounded tuning
|
||||
|
||||
| Property | Product Polish range |
|
||||
| ----------- | -------------------- |
|
||||
| intensity | 0.45 to 0.8 |
|
||||
| exposure | -0.08 to 0.1 |
|
||||
| contrast | 0 to 0.1 |
|
||||
| highlights | -0.16 to 0 |
|
||||
| shadows | 0 to 0.12 |
|
||||
| whites | -0.08 to 0.05 |
|
||||
| blacks | -0.06 to 0.04 |
|
||||
| temperature | -0.05 to 0.05 |
|
||||
| tint | -0.03 to 0.03 |
|
||||
| vibrance | 0 to 0.06 |
|
||||
| saturation | -0.04 to 0.05 |
|
||||
|
||||
Temperature and tint stay at zero unless the frames show a plausible cast.
|
||||
Leave grain, vignette, blur, and pixelate at zero for catalog/e-commerce media.
|
||||
For a lifestyle product shot, a vignette up to `0.04` is acceptable only when
|
||||
it improves focus without changing the product itself.
|
||||
|
||||
During the common comparison, reject any result that clips white packaging,
|
||||
muddies black products, shifts a known brand color, hides texture, or makes
|
||||
labels harder to read. Report when preserving the original was the deliberate
|
||||
decision.
|
||||
|
||||
## UI Fidelity
|
||||
|
||||
Use for literal app, website, dashboard, terminal, slide, or screen-recording
|
||||
pixels whose colors and readability are part of the product being shown.
|
||||
|
||||
The default payload is **none**: do not add `data-color-grading`. Global color
|
||||
changes affect brand colors, status colors, charts, screenshots, and tiny text
|
||||
together, so even a tasteful photographic look can make the demonstration less
|
||||
truthful.
|
||||
|
||||
Polish UI footage with crop, scale, pacing, cursor emphasis, surrounding DOM
|
||||
overlays, or seek-safe motion outside the captured pixels. If the user
|
||||
explicitly asks for a stylized UI look, preview it against the original and
|
||||
state that exact UI color is no longer preserved. If a camera filmed a screen,
|
||||
correct only a demonstrated capture cast or exposure issue and still verify
|
||||
text and brand colors across representative frames.
|
||||
|
||||
## Film Memory
|
||||
|
||||
Use when the story explicitly calls for a warm memory, restrained flashback,
|
||||
personal archive, or film-like recollection. This is not the default meaning of
|
||||
"cinematic", and it is not scanned-film-stock emulation.
|
||||
|
||||
Do not use for literal UI, product catalog media, technical demonstrations, or
|
||||
footage whose accurate current-day color is important. Use a separate camcorder
|
||||
treatment for VHS/REC language. Do not add dust, scratches, light leaks, film
|
||||
burns, or halation unless an owned component is available and the requested
|
||||
story actually benefits from it.
|
||||
|
||||
Check that the source has enough highlight and shadow detail to tolerate a
|
||||
faded treatment, and confirm nostalgia or temporal separation belongs in the
|
||||
story. Compare the full moving treatment, not only a still preset card.
|
||||
|
||||
### Static pixel base
|
||||
|
||||
Start with this owned shader recipe:
|
||||
|
||||
```json
|
||||
{
|
||||
"preset": "vintage-wash",
|
||||
"intensity": 0.6,
|
||||
"details": {
|
||||
"vignette": 0.12,
|
||||
"grain": 0.12,
|
||||
"grainSize": 0.2,
|
||||
"grainRoughness": 0.6
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep the static values inside these ranges:
|
||||
|
||||
| Property | Film Memory range |
|
||||
| -------------- | ----------------- |
|
||||
| intensity | 0.5 to 0.75 |
|
||||
| vignette | 0.08 to 0.16 |
|
||||
| grain | 0.08 to 0.16 |
|
||||
| grainSize | 0.16 to 0.24 |
|
||||
| grainRoughness | 0.5 to 0.7 |
|
||||
|
||||
The stronger end can flatten dark skin, black clothing, or already-faded
|
||||
footage. Compare against the source and lower strength when it does.
|
||||
|
||||
### Temporal character
|
||||
|
||||
Use the existing registered paused GSAP timeline on the same media element:
|
||||
|
||||
- author `--hf-color-grading-exposure: 0` in the media element's inline
|
||||
`style`;
|
||||
- move it through a finite irregular sequence within `-0.03` to `0.03`, using
|
||||
gentle `sine.inOut` segments around `0.45` to `0.8` seconds;
|
||||
- for gate weave, keep `x`/`y` within `0.15%` of the shorter composition edge,
|
||||
rotation within `0.03` degrees, and scale between `1.005` and `1.01` to
|
||||
protect the frame edges;
|
||||
- return close to the starting exposure and transform at the treatment end.
|
||||
|
||||
Do not use randomness, infinite CSS keyframes, timers, or `onUpdate`. Flicker
|
||||
is a gentle exposure pulse, not a flash. Weave is slight mechanical drift, not
|
||||
handheld shake.
|
||||
|
||||
Also run focused keyframe diagnostics and seek directly to the final-minus-frame
|
||||
position. Reject brightness pumping, distracting drift, clipped edges, or
|
||||
skin/detail loss. Report the motion ranges and describe this as an HF
|
||||
film-memory treatment, not camera-stock emulation. If motion reads as an effect
|
||||
before it reads as a memory, reduce or remove it.
|
||||
|
||||
## Creator Camcorder
|
||||
|
||||
Use when the story explicitly calls for a creator-camera recording, consumer
|
||||
camcorder memory, or restrained digital-video character. This treatment is a
|
||||
modern camcorder language, not VHS restoration, CRT simulation, surveillance,
|
||||
or a promise to reproduce a specific camera model.
|
||||
|
||||
Do not apply it to literal UI, product catalog media, tiny media tiles, or
|
||||
already compressed footage that has distracting color bleed. Do not add a REC
|
||||
HUD merely because the source contains a person talking; the camera-device
|
||||
language must support the story or the user's requested style.
|
||||
|
||||
Check skin, saturated edges, fine text, source compression, and whether the
|
||||
source already has a deliberate camera look. Reject softened chroma that
|
||||
damages labels, graphics, or identifying product color. Judge chroma softness
|
||||
and grain in motion, not one still.
|
||||
|
||||
### Static pixel base
|
||||
|
||||
Start with the proven shader payload below, then tune only inside the bounded
|
||||
ranges when representative frames justify it:
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 0.72,
|
||||
"adjust": {
|
||||
"contrast": 0.08,
|
||||
"highlights": -0.05,
|
||||
"shadows": 0.02,
|
||||
"whites": 0.03,
|
||||
"blacks": -0.04,
|
||||
"temperature": -0.03,
|
||||
"tint": -0.015,
|
||||
"vibrance": -0.03,
|
||||
"saturation": -0.06
|
||||
},
|
||||
"details": {
|
||||
"vignette": 0.06,
|
||||
"grain": 0.08,
|
||||
"grainSize": 0.18,
|
||||
"grainRoughness": 0.58
|
||||
},
|
||||
"effects": { "chromaBleed": 0.55 }
|
||||
}
|
||||
```
|
||||
|
||||
| Property | Creator Camcorder range |
|
||||
| -------------- | ----------------------- |
|
||||
| intensity | 0.55 to 0.8 |
|
||||
| contrast | 0.03 to 0.1 |
|
||||
| highlights | -0.1 to 0 |
|
||||
| shadows | 0 to 0.06 |
|
||||
| whites | 0 to 0.05 |
|
||||
| blacks | -0.08 to -0.01 |
|
||||
| temperature | -0.06 to 0.04 |
|
||||
| tint | -0.03 to 0.02 |
|
||||
| vibrance | -0.06 to 0.02 |
|
||||
| saturation | -0.12 to -0.02 |
|
||||
| vignette | 0.03 to 0.1 |
|
||||
| grain | 0.04 to 0.12 |
|
||||
| grainSize | 0.14 to 0.24 |
|
||||
| grainRoughness | 0.45 to 0.7 |
|
||||
| chromaBleed | 0.35 to 0.7 |
|
||||
|
||||
Leave blur and pixelate at zero. Square pixels, scanlines, RGB splitting, and
|
||||
tracking noise are different visual languages and are not defaults for this
|
||||
treatment.
|
||||
|
||||
### Optional camera HUD
|
||||
|
||||
When the narrative benefits from explicit recording-device language, install
|
||||
the Registry overlay block:
|
||||
|
||||
```bash
|
||||
npx hyperframes add camcorder-hud --no-clipboard
|
||||
```
|
||||
|
||||
Insert the printed `data-composition-src` host over the intended media range.
|
||||
Edit the displayed date/time/mode/counter in
|
||||
`compositions/camcorder-hud.html`. The block's paused GSAP timeline derives
|
||||
its counter and REC blink from composition time, so play, scrub, and render
|
||||
agree. Keep the HUD finite and scoped to the shot.
|
||||
|
||||
The HUD is an optional authored overlay. The pixel payload remains useful
|
||||
without it, and the HUD alone is not evidence that the footage was treated.
|
||||
|
||||
### Optional source-to-camera reveal
|
||||
|
||||
Global grading intensity fades only primary correction and LUT output; it does
|
||||
not fade the independent camcorder effects. For a visible source-to-camera
|
||||
mode change, use two synchronized media layers and a finite opacity crossfade
|
||||
from untreated to treated footage. Fade the HUD in on that same paused GSAP
|
||||
timeline. Do not animate shader state with callbacks or an independent clock.
|
||||
|
||||
Also verify HUD placement and framing in each aspect ratio the project supports.
|
||||
Report whether the HUD was used and describe this as an HF camcorder treatment,
|
||||
not camera/VHS emulation. If an effect artifact is more noticeable than the
|
||||
subject, reduce chroma bleed/grain or keep the source unchanged.
|
||||
|
||||
## VHS Playback
|
||||
|
||||
Use when the story explicitly calls for analog home-video tape, a dated archive,
|
||||
or a visibly degraded VHS playback. This treatment is not Creator Camcorder,
|
||||
generic pixelation, CRT display simulation, or a default retro look.
|
||||
|
||||
Do not use for literal UI, product catalog media, small text, clean modern
|
||||
creator footage, or any source whose identifying color/detail must remain exact.
|
||||
Inspect high-contrast vertical edges, faces, saturated objects, and the bottom
|
||||
of the frame in motion. Analog damage must support the story without making the
|
||||
subject hard to read.
|
||||
|
||||
### Pixel payload
|
||||
|
||||
Start with the complete proven combination, not `tapeDamage` alone:
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"adjust": { "contrast": -0.04, "saturation": -0.08 },
|
||||
"details": {
|
||||
"grain": 0.16,
|
||||
"grainSize": 0.12,
|
||||
"grainRoughness": 0.72
|
||||
},
|
||||
"effects": {
|
||||
"tapeDamage": 0.82,
|
||||
"tapeTracking": 0.85,
|
||||
"tapeNoise": 0.3,
|
||||
"tapeSpeed": 0.5,
|
||||
"chromaBleed": 0.5,
|
||||
"chromaticAberration": 0.18,
|
||||
"chromaticAngle": 0,
|
||||
"scanlines": 0.35,
|
||||
"scanlineCount": 0.17,
|
||||
"scanlineSoftness": 1,
|
||||
"digitalGlitch": 0.32,
|
||||
"digitalGlitchColorSplit": 0,
|
||||
"digitalGlitchLineTear": 0.08,
|
||||
"digitalGlitchPixelate": 0,
|
||||
"digitalGlitchBlockAmount": 0,
|
||||
"digitalGlitchBlockDisplacement": 0,
|
||||
"digitalGlitchBlockOpacity": 0,
|
||||
"digitalGlitchSpeed": 0.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Property | VHS Playback range |
|
||||
| --------------------- | ------------------ |
|
||||
| intensity | 0.75 to 1 |
|
||||
| contrast | -0.1 to 0 |
|
||||
| saturation | -0.16 to 0 |
|
||||
| grain | 0.08 to 0.18 |
|
||||
| grainSize | 0.08 to 0.18 |
|
||||
| grainRoughness | 0.55 to 0.8 |
|
||||
| tapeDamage | 0.65 to 0.9 |
|
||||
| tapeTracking | 0.5 to 0.9 |
|
||||
| tapeNoise | 0.15 to 0.45 |
|
||||
| tapeSpeed | 0.35 to 0.65 |
|
||||
| chromaBleed | 0.35 to 0.65 |
|
||||
| chromaticAberration | 0.08 to 0.22 |
|
||||
| scanlines | 0.2 to 0.4 |
|
||||
| scanlineCount | 0.14 to 0.2 |
|
||||
| digitalGlitch | 0.2 to 0.4 |
|
||||
| digitalGlitchLineTear | 0.04 to 0.1 |
|
||||
|
||||
`tapeDamage` owns deterministic horizontal line jitter, slow time-base wobble,
|
||||
bottom-edge head switching, luma bandwidth loss, restrained ghosting, noise,
|
||||
and sparse dropouts. Its subordinate tracking/noise/speed controls add bounded
|
||||
moving tape tears and control their signal character without introducing a new
|
||||
clock. `chromaBleed` separately reduces horizontal chroma detail. The restrained
|
||||
scanline and chromatic settings supply the remaining tape-playback character.
|
||||
The digital stage is used only for rare horizontal row tears: keep its color
|
||||
split, pixelation, block displacement, block opacity, and corruption values at
|
||||
zero. Leave blur, CRT curvature, generic pixelation, and a camera HUD off.
|
||||
|
||||
These values are an original HyperFrames recipe calibrated on the same public
|
||||
Orange Cat source used for the external visual reference. They are not copied
|
||||
shader code or a claim of pixel-identical output from the external reference. The scanline count is
|
||||
mapped to the reference's approximately 127-cycle primary line pattern; the HF
|
||||
tracking math stays bounded in media pixels and uses the composition clock.
|
||||
|
||||
The shader damage evolves from the existing deterministic media time, so it
|
||||
needs no CSS loop or private timeline. Global grading intensity does not fade
|
||||
tape damage or other independent effects. If the story requires a finite
|
||||
source-to-tape reveal, crossfade synchronized untreated and treated media layers
|
||||
on the host's paused GSAP timeline. During the common workflow, inspect dense
|
||||
consecutive frames and reject hard edge tearing, face
|
||||
smearing, frozen noise, square blocks, blank borders, or a bottom disturbance
|
||||
that competes with the subject.
|
||||
|
||||
## 8mm Home Movie
|
||||
|
||||
Use for personal archive, family-memory, childhood, travel-memory, or explicit
|
||||
small-gauge home-movie language. This is stronger and more materially film-like
|
||||
than Film Memory, but it is still an owned HyperFrames treatment rather than a
|
||||
claim to reproduce a named film stock, camera, or laboratory process.
|
||||
|
||||
Do not use for literal UI, technical demonstrations, catalog products, clean
|
||||
interviews, or footage where dust/scratches would imply false provenance. Check
|
||||
skin, highlights, dark clothing, and frame edges before applying it.
|
||||
|
||||
### Pixel payload
|
||||
|
||||
```json
|
||||
{
|
||||
"preset": "vintage-wash",
|
||||
"intensity": 0.72,
|
||||
"details": {
|
||||
"vignette": 0.28,
|
||||
"vignetteMidpoint": 0.54,
|
||||
"vignetteFeather": 0.72,
|
||||
"grain": 0.34,
|
||||
"grainSize": 0.18,
|
||||
"grainRoughness": 0.72
|
||||
},
|
||||
"effects": { "filmArtifacts": 0.62 }
|
||||
}
|
||||
```
|
||||
|
||||
| Property | 8mm Home Movie range |
|
||||
| -------------- | -------------------- |
|
||||
| intensity | 0.6 to 0.8 |
|
||||
| vignette | 0.18 to 0.34 |
|
||||
| grain | 0.22 to 0.42 |
|
||||
| grainSize | 0.12 to 0.24 |
|
||||
| grainRoughness | 0.6 to 0.8 |
|
||||
| filmArtifacts | 0.35 to 0.7 |
|
||||
|
||||
`filmArtifacts` owns only deterministic sparse dust and short scratches. The
|
||||
existing preset/details own color, vignette, and grain; the host's paused GSAP
|
||||
timeline owns optional gate weave. Keep weave within `0.15%` of the shorter
|
||||
composition edge, rotation within `0.03` degrees, and scale between `1.005` and
|
||||
`1.015`. Use finite `sine.inOut` segments around `0.6` to `1` second, return
|
||||
near the starting transform, and never use randomness, timers, `onUpdate`, or
|
||||
an infinite CSS animation.
|
||||
|
||||
Reject a result when dust is constantly visible, scratches persist unnaturally,
|
||||
the frame pumps, weave exposes an edge, highlights turn muddy, or the material
|
||||
artifacts are more noticeable than the memory. For a subtler nostalgic result,
|
||||
use Film Memory instead.
|
||||
|
||||
## Editorial Halftone
|
||||
|
||||
Use for print/editorial transitions, poster frames, comic/newsprint language,
|
||||
stylized product or portrait beats, and graphic sequences where visible ink
|
||||
screening is the point. This is a real four-angle CMYK raster treatment, not a
|
||||
dotted DOM overlay.
|
||||
|
||||
Do not use on literal UI, dense text, tiny labels, footage that must remain
|
||||
photorealistic, or a long talking-head segment unless the user explicitly asks
|
||||
for strong print stylization. Preserve text/captions as ungraded DOM above the
|
||||
media whenever they must stay readable.
|
||||
|
||||
### Pixel payload
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"adjust": { "contrast": 0.04, "saturation": 0.04 },
|
||||
"effects": { "halftone": 0.94, "halftoneSize": 0.36 }
|
||||
}
|
||||
```
|
||||
|
||||
| Property | Editorial Halftone range |
|
||||
| ------------ | ------------------------ |
|
||||
| intensity | 0.8 to 1 |
|
||||
| contrast | -0.02 to 0.08 |
|
||||
| saturation | -0.04 to 0.08 |
|
||||
| halftone | 0.75 to 1 |
|
||||
| halftoneSize | 0.15 to 0.55 |
|
||||
|
||||
The shader uses fixed C/M/Y/K screen angles of 15/75/0/45 degrees, separate ink
|
||||
coverage, a warm paper base, and resolution-aware dot-cell sizing. Keep those
|
||||
screen semantics fixed; tune only amount and size unless a future visual proof
|
||||
justifies a broader schema. Judge the result at final output resolution because
|
||||
browser zoom can misrepresent the screen. Reject unstable moire, unreadable
|
||||
subjects, clipped ink detail, excessive dot size, or any treatment that looks
|
||||
like a transparent dot texture laid over unchanged footage.
|
||||
|
||||
## Two-Ink Editorial Print
|
||||
|
||||
Use for poster frames, editorial portraits, music/social cutaways, zine
|
||||
graphics, and bold print-led transitions where two visible spot inks are more
|
||||
appropriate than photographic color. This is a fixed original HyperFrames
|
||||
vermilion/teal treatment, not a claim to emulate a named printer, ink set, or
|
||||
commercial print process.
|
||||
|
||||
Do not use for literal UI, brand-color-critical products, small labels, natural
|
||||
talking heads, or media that must remain photorealistic. Keep captions and
|
||||
graphics as normal DOM above the treated media.
|
||||
|
||||
### Pixel payload
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"adjust": { "contrast": 0.08, "highlights": -0.06, "shadows": 0.04 },
|
||||
"effects": { "twoInkPrint": 1, "twoInkPrintSize": 0.42 }
|
||||
}
|
||||
```
|
||||
|
||||
| Property | Two-Ink range |
|
||||
| --------------- | ------------- |
|
||||
| intensity | 0.8 to 1 |
|
||||
| contrast | 0.02 to 0.1 |
|
||||
| highlights | -0.1 to 0 |
|
||||
| shadows | 0 to 0.08 |
|
||||
| twoInkPrint | 0.8 to 1 |
|
||||
| twoInkPrintSize | 0.18 to 0.55 |
|
||||
|
||||
The shader maps warm midtones to vermilion, deep/cool shadows to teal, and
|
||||
shared dark coverage to a dark overprint on warm paper. It uses separate
|
||||
15/75-degree screens, a subtle fixed registration offset, deterministic paper
|
||||
texture, and resolution-aware dot sizing. Do not combine it with `halftone` or
|
||||
a duotone LUT: that re-separates the result and defeats the two-ink contract.
|
||||
|
||||
Judge it at output resolution and across multiple frames. Reject missing second
|
||||
ink, crushed faces, unstable moire, illegible silhouettes, or a result that
|
||||
reads as a red tint with dots rather than two screened inks.
|
||||
|
||||
## Monochrome Screen Print
|
||||
|
||||
Use for graphic portrait beats, posterized social inserts, newspaper-like
|
||||
screens, or a finite transition into visible monochrome cells. Keep captions
|
||||
and typography as normal DOM above the treated media.
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"effects": {
|
||||
"monoScreen": 1,
|
||||
"monoScreenSize": 0.35,
|
||||
"monoScreenAngle": 0.25,
|
||||
"monoScreenSpread": 0.3,
|
||||
"monoScreenShape": 0,
|
||||
"monoScreenInvert": 0
|
||||
},
|
||||
"palette": ["#111319", "#f2ecdc"]
|
||||
}
|
||||
```
|
||||
|
||||
Use `monoScreenShape` `0..4` for circle, square, diamond, triangle, or line.
|
||||
Keep cell size within `0.15..0.55` and spread within `0.15..0.55`. Reject faces
|
||||
that lose their silhouette, unstable moire, or cells too small to survive the
|
||||
final encoded resolution.
|
||||
|
||||
## Engraved Illustration
|
||||
|
||||
Use for editorial portraits, historical/technical illustration, title-card
|
||||
cutaways, or a source-to-line-art reveal. It is not routine correction and
|
||||
should not be applied to literal UI or brand-color-critical product footage.
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"effects": {
|
||||
"engraving": 1,
|
||||
"engravingSpacing": 0.4118,
|
||||
"engravingMinThickness": 0.2,
|
||||
"engravingMaxThickness": 0.4571,
|
||||
"engravingAngle": 0.25,
|
||||
"engravingContrast": 0.4667,
|
||||
"engravingSharpness": 0.59,
|
||||
"engravingWave": 0.2,
|
||||
"engravingWaveFrequency": 0.2222
|
||||
},
|
||||
"palette": ["#101216", "#f3eddf"]
|
||||
}
|
||||
```
|
||||
|
||||
Preserve the calibrated base first. Tune spacing within `0.25..0.6`, contrast
|
||||
within `0.3..0.65`, and wave within `0..0.35`. Reject squeezed framing, broken
|
||||
contours, noisy flat backgrounds, or lines that flicker across moving frames.
|
||||
|
||||
## Crosshatched Sketch
|
||||
|
||||
Use for hand-rendered editorial beats, comic/documentary cutaways, and short
|
||||
illustrative transformations where multiple line directions should preserve
|
||||
the subject contour.
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"effects": {
|
||||
"crosshatch": 1,
|
||||
"crosshatchSpacing": 0.28,
|
||||
"crosshatchThickness": 0.25,
|
||||
"crosshatchAngle": 0.25,
|
||||
"crosshatchContrast": 0.3333,
|
||||
"crosshatchEdges": 0.5,
|
||||
"crosshatchLineWeight": 0,
|
||||
"crosshatchWave": 0.33,
|
||||
"crosshatchWaveFrequency": 0.2222
|
||||
},
|
||||
"palette": ["#101216", "#f3eddf"]
|
||||
}
|
||||
```
|
||||
|
||||
Tune spacing within `0.18..0.5`, edge detail within `0.3..0.7`, and wave within
|
||||
`0.1..0.45`. Reject distorted aspect ratio, dense black fill that hides the
|
||||
subject, or temporal shimmer stronger than the intended sketch language.
|
||||
|
||||
## CRT Display
|
||||
|
||||
Use when the media is intentionally shown as an older monitor, terminal, game
|
||||
screen, or broadcast display. Curvature alone is geometry, not a complete CRT
|
||||
treatment, so pair it with restrained scanlines and only slight channel
|
||||
separation.
|
||||
|
||||
```json
|
||||
{
|
||||
"intensity": 1,
|
||||
"effects": {
|
||||
"crtCurvature": 0.2,
|
||||
"scanlines": 0.35,
|
||||
"scanlineCount": 0.17,
|
||||
"scanlineSoftness": 1,
|
||||
"chromaticAberration": 0.08,
|
||||
"chromaticAngle": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keep curvature within `0.08..0.28`, scanlines within `0.18..0.45`, and channel
|
||||
separation within `0..0.12`. Reject excessive black corners, unreadable UI,
|
||||
large color fringes, or applying the display language to ordinary footage when
|
||||
the user only asked for correction.
|
||||
|
||||
## Procedural ASCII
|
||||
|
||||
Use for a deliberate terminal, code, data, surveillance, editorial, or
|
||||
source-to-character reveal. This is a real shader-generated 5x7 glyph field,
|
||||
not monospace text placed over unchanged footage.
|
||||
|
||||
Do not use as routine talking-head polish, on literal UI or dense text, or when
|
||||
recognizing a face/product precisely matters. Keep captions and graphics as
|
||||
normal DOM above the treated media.
|
||||
|
||||
Choose one of these proven starting points:
|
||||
|
||||
```json
|
||||
{
|
||||
"effects": { "ascii": 1, "asciiSize": 0.08, "asciiInvert": 1 },
|
||||
"palette": ["#020605", "#38ff78"]
|
||||
}
|
||||
```
|
||||
|
||||
The first is **Terminal ASCII**: dark field, bright green glyphs, appropriate
|
||||
for code/data/device language. For a warmer print-like **Editorial ASCII**, use:
|
||||
|
||||
```json
|
||||
{
|
||||
"effects": { "ascii": 1, "asciiSize": 0.066, "asciiInvert": 0 },
|
||||
"palette": ["#0b0d0d", "#eee9db"]
|
||||
}
|
||||
```
|
||||
|
||||
Keep `ascii` between `0.75` and `1` for a fully readable treatment and
|
||||
`asciiSize` between `0.04` and `0.15`. A finite reveal may author
|
||||
`--hf-color-grading-ascii: 0` inline and tween it to `1` with the registered
|
||||
paused GSAP timeline. Reject unstable cells, lost silhouette/face structure,
|
||||
unreadable composition, or a palette that conflicts with the project.
|
||||
|
||||
## Ordered Palette Dither
|
||||
|
||||
Use for posterized social beats, music/editorial cutaways, pixel-art language,
|
||||
or a finite source-to-palette reveal. The shader uses a stable 4x4 Bayer
|
||||
threshold matrix and an explicit dark-to-light palette. Do not describe it as
|
||||
Floyd-Steinberg, Atkinson, or another sequential error-diffusion process.
|
||||
|
||||
Do not use on literal UI, brand-color-critical products, tiny labels, or long
|
||||
photorealistic sections. Start with one of these original palettes:
|
||||
|
||||
```json
|
||||
{
|
||||
"effects": { "dither": 1, "ditherSize": 0.25 },
|
||||
"palette": ["#17121a", "#824c50", "#e09873", "#f7ddb1"]
|
||||
}
|
||||
```
|
||||
|
||||
The four-color option is **Warm Print**. For a louder social/music beat, use
|
||||
the six-color **Electric Ink** palette:
|
||||
|
||||
```json
|
||||
{
|
||||
"effects": { "dither": 1, "ditherSize": 0.4 },
|
||||
"palette": ["#080717", "#3c185f", "#7e2278", "#d9339f", "#ff6b66", "#aafae0"]
|
||||
}
|
||||
```
|
||||
|
||||
HyperFrames also owns these named ramps. The name is an authoring shortcut;
|
||||
persist the listed colors through the existing `palette` array:
|
||||
|
||||
| Group | Palette ID | Ordered colors |
|
||||
| ----------- | ---------------- | ---------------------------------------------------------------- |
|
||||
| Classic | `noir` | `#000000`, `#ffffff` |
|
||||
| Classic | `ink-paper` | `#1a1a2e`, `#f5f5dc` |
|
||||
| Classic | `terminal` | `#001100`, `#00ff00` |
|
||||
| Classic | `amber-glow` | `#1a0f00`, `#ffcc00` |
|
||||
| Classic | `handheld-green` | `#0f380f`, `#306230`, `#8bac0f`, `#9bbc0f` |
|
||||
| Mood | `golden-hour` | `#1a1205`, `#4a3510`, `#8b6914`, `#d4a017`, `#fff8dc` |
|
||||
| Mood | `deep-sea` | `#0a1628`, `#1a3a5c`, `#2d6187`, `#5ba4c9`, `#a8dce8` |
|
||||
| Mood | `arctic-night` | `#0a0a14`, `#1a2a4a`, `#3a5a8a`, `#6a9aca`, `#cae8ff` |
|
||||
| Mood | `synthwave` | `#120458`, `#7b2cbf`, `#e040fb`, `#ff6ec7`, `#fff59d` |
|
||||
| Mood | `vaporwave` | `#1a0a2e`, `#3d1a5c`, `#ff71ce`, `#01cdfe`, `#fffb96` |
|
||||
| Mood | `forest` | `#1a2e1a`, `#2d4a2d`, `#4a7c4a`, `#7ab37a`, `#c8e6c8` |
|
||||
| Mono | `sepia` | `#1a1610`, `#3d3020`, `#6b5a40`, `#a89070`, `#e8dcc8` |
|
||||
| Mono | `blueprint` | `#001830`, `#003060`, `#0050a0`, `#0080e0`, `#e0f0ff` |
|
||||
| HyperFrames | `warm-print` | `#17121a`, `#824c50`, `#e09873`, `#f7ddb1` |
|
||||
| HyperFrames | `electric-ink` | `#080717`, `#3c185f`, `#7e2278`, `#d9339f`, `#ff6b66`, `#aafae0` |
|
||||
|
||||
Choose by inspected source and project language, not by palette name alone.
|
||||
For example, `terminal` fits device/code language, `warm-print` fits editorial
|
||||
print, and `synthwave` is an intentional stylization rather than generic polish.
|
||||
|
||||
`palette` must contain two to six exact `#RRGGBB` colors in authored order. Use
|
||||
dark-to-light order for this treatment; the runtime validates colors but does
|
||||
not reorder them, so reversing the array intentionally inverts the mapping.
|
||||
Keep `dither` between `0.7` and `1` and `ditherSize` between `0.1` and `0.5`.
|
||||
A finite reveal may author `--hf-color-grading-dither: 0` inline and tween it
|
||||
to the chosen amount with GSAP. Judge the moving result at output resolution;
|
||||
reject shimmer, lost subject structure, accidental muddy intermediate colors,
|
||||
or a palette chosen without regard to the project's design language.
|
||||
|
||||
## Cached Error Diffusion
|
||||
|
||||
Use exact error diffusion for a deliberate 1-bit Macintosh, newspaper/print,
|
||||
limited-palette game, or crunchy editorial treatment. It bakes a new image or
|
||||
MP4 because every processed block depends on error from earlier blocks; it is
|
||||
not a realtime shader setting.
|
||||
|
||||
Choose the algorithm by visible intent:
|
||||
|
||||
- `floyd-steinberg`: balanced default with organic fine texture.
|
||||
- `atkinson`: higher-contrast, more open and distinctly early-Macintosh.
|
||||
- `jarvis-judice-ninke`: smoother gradients with a wider 12-neighbor field.
|
||||
- `stucki`: smooth, slightly sharper alternative to JJN.
|
||||
- `burkes`: compact two-row texture.
|
||||
- `sierra`, `sierra-lite`, `two-row-sierra`: progressively different
|
||||
speed/texture tradeoffs; use only after comparing frames.
|
||||
|
||||
Run the exact processor and register its output through the existing media
|
||||
ledger/cache:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/dither.mjs \
|
||||
--input .media/videos/video_001.mp4 \
|
||||
--out .media/generated/video_001.atkinson.mp4 \
|
||||
--algorithm atkinson \
|
||||
--palette '#17121a,#824c50,#e09873,#f7ddb1' \
|
||||
--point-size 3
|
||||
|
||||
node <SKILL_DIR>/scripts/resolve.mjs \
|
||||
--from .media/generated/video_001.atkinson.mp4 --type video --project .
|
||||
```
|
||||
|
||||
Use the registered output path on a real `<img>` or `<video>`. Keep text,
|
||||
captions, logos, and interface graphics outside the processed media. For a
|
||||
finite reveal, overlap the original and processed media with identical framing
|
||||
and crossfade or wipe them using the registered paused GSAP timeline. Do not
|
||||
label the realtime Bayer shader as Floyd-Steinberg/Atkinson, and do not process
|
||||
PQ/HLG footage without an explicit SDR tone-map decision.
|
||||
|
||||
## Organic Light Leak
|
||||
|
||||
Use for one motivated memory beat, time shift, warm scene handoff, or tactile
|
||||
transition. It is a finite deterministic CSS/GSAP overlay, not a looping
|
||||
texture, generic flash, or film-stock emulation.
|
||||
|
||||
Install the Registry overlay block:
|
||||
|
||||
```bash
|
||||
npx hyperframes add organic-light-leak-overlay --no-clipboard
|
||||
```
|
||||
|
||||
Insert the printed `data-composition-src` host at the intended beat and keep
|
||||
its duration finite. Its paused timeline owns one rise, peak, and complete
|
||||
recovery and scales those phases to the placed duration. Inspect the source
|
||||
before, at the brightest frame, and after recovery. Reject clipped faces, an
|
||||
unmotivated warm wash, visible black from incorrect blend mode, or a leak that
|
||||
conceals the subject longer than the transition needs.
|
||||
|
||||
## Freeze-Frame Cutout
|
||||
|
||||
Use for a social introduction, speaker emphasis, chapter punctuation, sports
|
||||
or creator beat, or a scrapbook/editorial hold. This requires a real alpha
|
||||
matte; decoration may not conceal a poor subject edge.
|
||||
|
||||
Extract the exact deterministic source frame first, then remove its background:
|
||||
|
||||
```bash
|
||||
ffmpeg -ss <seconds> -i <source-video> -frames:v 1 -y .media/generated/freeze-source.png
|
||||
npx hyperframes remove-background .media/generated/freeze-source.png \
|
||||
-o .media/generated/freeze-cutout.png --json
|
||||
npx hyperframes add freeze-frame-dressing --no-clipboard
|
||||
```
|
||||
|
||||
Add the transparent result as a direct-root timed media layer and insert the
|
||||
printed overlay block above the same time range. The block owns the paper,
|
||||
tape, and flash; the host timeline only animates the real cutout:
|
||||
|
||||
```html
|
||||
<img
|
||||
id="hf-freeze-cutout"
|
||||
class="clip"
|
||||
src="./.media/generated/freeze-cutout.png"
|
||||
alt=""
|
||||
data-start="6"
|
||||
data-duration="3"
|
||||
data-track-index="20"
|
||||
/>
|
||||
```
|
||||
|
||||
```js
|
||||
tl.fromTo(
|
||||
"#hf-freeze-cutout",
|
||||
{ y: 42, scale: 0.86, rotation: -2 },
|
||||
{ y: 0, scale: 1, rotation: 0.4, duration: 0.5, ease: "back.out(1.35)" },
|
||||
freezeAt,
|
||||
);
|
||||
```
|
||||
|
||||
Inspect the matte over both light and dark temporary plates before styling it.
|
||||
Reject missing hair/fingers, background halos, a cutout that changes identity,
|
||||
overly thick outline, exposed frame edges, or a flash that obscures the reveal.
|
||||
If the matte is not acceptable, choose another frame or keep the original media.
|
||||
|
||||
## Social Flash / Editorial Reveal
|
||||
|
||||
Use this treatment for one meaningful high-energy cut, creator reveal, product
|
||||
beat, or before/after handoff. It is not a default transition for every scene.
|
||||
Avoid it for calm long-form footage, accessibility-sensitive contexts, already
|
||||
clipped highlights, literal UI that must remain readable through the cut, or
|
||||
any request for repeated strobing.
|
||||
|
||||
Inspect representative frames on both sides of the cut first. Grade each media
|
||||
layer for its own subject using the appropriate contract above; the flash is
|
||||
not a substitute for correction. For people, a restrained `skin-soft` payload
|
||||
is a safe starting point. For literal UI, preserve the pixels and use only the
|
||||
authored light/motion layers when they do not obscure required information.
|
||||
|
||||
Install the Registry overlay block:
|
||||
|
||||
```bash
|
||||
npx hyperframes add editorial-flash-overlay --no-clipboard
|
||||
```
|
||||
|
||||
Insert the printed `data-composition-src` host so the block's midpoint lands
|
||||
on the cut. Its own paused timeline drives the finite flash. The host timeline
|
||||
may coordinate outgoing and incoming media motion without reaching into the
|
||||
block:
|
||||
|
||||
```js
|
||||
tl.to(
|
||||
"#outgoing-media",
|
||||
{
|
||||
scale: 1.035,
|
||||
"--hf-color-grading-exposure": 0.82,
|
||||
duration: 0.12,
|
||||
ease: "power3.in",
|
||||
},
|
||||
cutAt - 0.16,
|
||||
);
|
||||
tl.fromTo(
|
||||
"#incoming-media",
|
||||
{ scale: 1.1 },
|
||||
{ scale: 1, duration: 0.42, ease: "power3.out" },
|
||||
cutAt,
|
||||
);
|
||||
tl.to(
|
||||
"#incoming-media",
|
||||
{
|
||||
"--hf-color-grading-exposure": 0,
|
||||
"--hf-color-grading-intensity": 0.58,
|
||||
duration: 0.24,
|
||||
ease: "power2.out",
|
||||
},
|
||||
cutAt,
|
||||
);
|
||||
```
|
||||
|
||||
When the shader steps are used, author
|
||||
`--hf-color-grading-exposure: 0.72` and
|
||||
`--hf-color-grading-intensity: 0` inline on the incoming media so a fresh seek
|
||||
has the correct start state. Set the final intensity to the source-approved
|
||||
value instead of copying `0.58` blindly. Skip the shader intensity step when
|
||||
the incoming source should remain ungraded.
|
||||
|
||||
Keep the rise between roughly `0.035` and `0.055` seconds and the recovery
|
||||
between `0.24` and `0.38` seconds. Default to one neutral/warm flash event,
|
||||
never saturated red, never a looping strobe, and never more than one authored
|
||||
flash inside a one-second treatment window. Verify frames immediately before,
|
||||
at, and after the cut, then inspect moving playback and a rendered draft. The
|
||||
peak must hide the cut; the recovery must reveal a correctly framed source with
|
||||
no retained prior canvas, clipped face, or unexpected highlight damage.
|
||||
@@ -0,0 +1,202 @@
|
||||
# Media treatments
|
||||
|
||||
A media treatment is a source-aware plan that composes existing HyperFrames
|
||||
color, effect, timeline, and Registry primitives. It is not a second runtime
|
||||
schema. Use this file to choose a primary direction. A matching recipe is an
|
||||
optional tested seed; bespoke requests may assemble a validated treatment from
|
||||
the canonical capability catalog.
|
||||
|
||||
## Permission and scope
|
||||
|
||||
- An explicit request such as "polish this", "make it look better", or "make it
|
||||
fit the topic" delegates a conservative treatment. Apply it, verify it, and
|
||||
report what changed.
|
||||
- During an unsolicited opportunity scan, show or suggest the treatment first.
|
||||
- Target meaningful photographic media. Skip text, SVG, logos, icons, UI
|
||||
chrome, and intentionally stylized footage unless the user asks.
|
||||
- Realtime grading and effects apply to the entire selected real `<img>` or
|
||||
`<video>`. They do not isolate or track a face, plate, address, or other
|
||||
region. For region-only work, first create a separate cropped/masked media
|
||||
layer or use an external segmentation/tracking tool; never imply that a
|
||||
whole-media Blur or Pixelate performed region isolation.
|
||||
- The realtime treatment path is Rec.709/SDR. Do not silently process HDR, HLG,
|
||||
PQ, or camera LOG sources through it.
|
||||
|
||||
## Classify the request
|
||||
|
||||
Choose the smallest lane that satisfies the request before choosing a recipe
|
||||
or assembling a custom treatment:
|
||||
|
||||
| User intent | Lane |
|
||||
| --------------------------------------------------- | ------------------------------------------ |
|
||||
| too dark, flat, too warm, too many shadows | correction |
|
||||
| shape shadows/highlights or selected colors | wheels, curves, or HSL secondary |
|
||||
| polished, premium, warm, cinematic, fit the topic | preset or custom treatment |
|
||||
| retro, print, ASCII, glitch, camcorder | shader effect or effect-bearing preset |
|
||||
| obscure the whole selected media | privacy Blur or Pixelate |
|
||||
| hide one face, plate, address, or screen region | separate crop/mask/asset or external tool |
|
||||
| draw attention to media without changing its pixels | framing, motion, or optional overlay |
|
||||
| reveal, focus, depixelate, fade the treatment | finite seek-safe treatment keyframes |
|
||||
| REC HUD, light leak, flash, freeze-frame cutout | Registry overlay plus any justified pixels |
|
||||
|
||||
The lane identifies the primary reason for the change; it is not a one-feature
|
||||
limit. A final treatment may combine correction, a preset, finishing, multiple
|
||||
compatible shader effects, finite keyframed values, and optional overlays when
|
||||
the inspected media and user intent justify the complete combination. Keep one
|
||||
primary intent as the creative anchor so the result remains coherent and
|
||||
deterministic.
|
||||
|
||||
Do not add a stylized Effect when correction solves the complaint. Do not
|
||||
change color when the request is only temporal, and do not install an overlay
|
||||
when the selected media alone communicates the result.
|
||||
|
||||
Match strength to intent. When the user explicitly names a bold look such as
|
||||
VHS, glitch, ASCII, halftone, camcorder, print, or engraving, apply its
|
||||
signature effects strongly enough to read unmistakably. The guard against
|
||||
unrequested additions does not mean under-delivering an effect the user asked
|
||||
for. Correction and polish stay restrained; named stylization must be obvious
|
||||
in the after-frame.
|
||||
|
||||
### Translate vague feedback conservatively
|
||||
|
||||
| User feedback | First action | Add only when the frames justify it | Never infer |
|
||||
| ------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------- |
|
||||
| too many shadows and a bit boring | lift shadows/protect highlights, then compare one restrained source-appropriate preset | mild contrast or vibrance | retro texture, HUD, or palette effect |
|
||||
| make the product footage feel premium | protect product color and labels, compare Product Polish | restrained vignette on lifestyle footage | a cinematic LUT or crushed blacks |
|
||||
| make this reveal cooler | preserve color and animate one supported effect or treatment value | a short owned overlay block | unrelated whole-clip styling |
|
||||
| make it feel like an old home video | compare 8mm and VHS language against the source | finite weave/flicker or justified HUD | that every old-video request means VHS |
|
||||
| hide this face | explain that realtime effects are whole-media; isolate the region first or use an external tool | whole-media Blur/Pixelate only when the user accepts that scope | face tracking or masking that was not performed |
|
||||
| keep the brand colors exact | leave UI/logo pixels unchanged; use framing and motion | demonstrated exposure-only correction | stylized preset, palette, or LUT |
|
||||
|
||||
When more than one lane could fit, generate at most two candidates and choose
|
||||
from inspected before/after evidence. Ordinary correction or polish starts with
|
||||
one candidate; a second candidate is an escalation, not the default. Do not
|
||||
stack effects merely to make the answer look more sophisticated.
|
||||
|
||||
## Seed or assemble
|
||||
|
||||
Use the table when a tested recipe directly fits. Read only that recipe
|
||||
section. Recipes are optional macros, not a closed list of allowed results.
|
||||
|
||||
| Intent or source | Recipe heading to read |
|
||||
| -------------------------------------------------------- | ------------------------------------------------- |
|
||||
| Talking head, interview, presenter, people-focused photo | `Natural Portrait` |
|
||||
| Product footage, lifestyle footage, clean social polish | `Product Polish` |
|
||||
| Screen capture, dashboard, website, app UI | `UI Fidelity` |
|
||||
| Warm memory, restrained nostalgia | `Film Memory` |
|
||||
| Creator/UGC handheld camera character | `Creator Camcorder` |
|
||||
| Analog tape playback | `VHS Playback` |
|
||||
| Small-gauge home-movie character | `8mm Home Movie` |
|
||||
| Editorial dots or ink print | `Editorial Halftone` or `Two-Ink Editorial Print` |
|
||||
| Monochrome dot/line screen print | `Monochrome Screen Print` |
|
||||
| Engraved or hand-hatched illustration | `Engraved Illustration` or `Crosshatched Sketch` |
|
||||
| Curved scanlined display | `CRT Display` |
|
||||
| Glyph-based art | `Procedural ASCII` |
|
||||
| Realtime palette quantization | `Ordered Palette Dither` |
|
||||
| Exact historical error diffusion | `Cached Error Diffusion` |
|
||||
| Finite warm flare layer | `Organic Light Leak` |
|
||||
| Held-frame graphic interruption | `Freeze-Frame Cutout` |
|
||||
| Short exposure-flash transition | `Social Flash / Editorial Reveal` |
|
||||
|
||||
Use `rg -n '^## <heading>$' <SKILL_DIR>/references/media-treatment-recipes.md`,
|
||||
then read only from that heading to the next `##`. Do not load the entire
|
||||
cookbook for one request.
|
||||
|
||||
When source intent is unclear, inspect the concise capability overview:
|
||||
|
||||
```bash
|
||||
hyperframes media-treatment --capabilities --json
|
||||
```
|
||||
|
||||
It lists the complete surface by family with one-line descriptions. Then load
|
||||
only the family, effect, preset, or palette relevant to the inspected source:
|
||||
|
||||
```bash
|
||||
hyperframes media-treatment --capability <id> --json
|
||||
```
|
||||
|
||||
The focused result provides legal controls, recommended apply values, render
|
||||
cost, palette support, and the exact animation contract when supported. Use
|
||||
`--all` only for tooling/tests or a genuinely exhaustive audit. Compose one
|
||||
nested payload from these existing parts. Recipes and catalog-built payloads
|
||||
use the same renderer and persistence contract.
|
||||
|
||||
Treat `renderLane: "multipass"` as a cost signal. Blur, Bloom, and Kuwahara are
|
||||
bounded but more expensive than single-pass effects; avoid stacking several of
|
||||
them across many simultaneous media elements unless the composition needs it,
|
||||
then verify playback and a draft render.
|
||||
Cost follows treated pixel area as well as element count. More than two
|
||||
simultaneously visible full-frame multipass media layers requires a continuous
|
||||
playback check on the target machine; simplify or pre-render the stack if it
|
||||
drops frames. Do not impose or claim a universal hard cap from one machine.
|
||||
|
||||
## Common workflow
|
||||
|
||||
1. Confirm the target is a real `<img>` or `<video>` and inspect source color
|
||||
metadata.
|
||||
2. For an image, read it once. For video, capture early/middle/late output as
|
||||
one labeled sheet and read that one image:
|
||||
|
||||
```bash
|
||||
hyperframes snapshot <project> --frames 3 --no-end --describe false \
|
||||
--output snapshots/treatment-before
|
||||
```
|
||||
|
||||
Read `snapshots/treatment-before/contact-sheet.jpg`; do not spend separate
|
||||
model turns reading each frame unless the sheet exposes a specific problem.
|
||||
Do not infer semantics from signal statistics alone.
|
||||
|
||||
3. Choose one primary lane. Use one matching recipe as a tested seed, or read
|
||||
the overview and one focused capability detail when the request is bespoke.
|
||||
A seed may be changed or combined with compatible catalog controls when the
|
||||
contact sheet justifies it. Do not invent keys, exceed reported ranges, or
|
||||
stack effects without a visual reason. Do not run the generic grade/LUT
|
||||
resolver first; it adds irrelevant candidates and may download an unused
|
||||
LUT. Use `media-treatment --selector "#hero" --analyze --json` only when
|
||||
correction needs measured signal evidence.
|
||||
4. Persist pixel settings with `hyperframes media-treatment`; it validates and
|
||||
merges a patch into the existing nested `data-color-grading` contract. Use registered
|
||||
GSAP only for supported animated values and Registry overlay blocks only
|
||||
for authored dressing.
|
||||
```bash
|
||||
hyperframes media-treatment --selector "<unique selector>" \
|
||||
--grading '<nested JSON patch>' --apply --json
|
||||
```
|
||||
For a temporal reveal, use the focused capability result's `animation`
|
||||
contract. If it is `null`, the capability is static. Author the starting CSS
|
||||
property inline on the real media element and return temporary treatment
|
||||
values to neutral so the finished shot preserves its existing pixels.
|
||||
Prefer that bounded media animation first; if an overlay is justified,
|
||||
install the owned Registry block instead of recreating it with bespoke
|
||||
overlay markup.
|
||||
For correction and ordinary polish, keep those values as editable
|
||||
preset/adjustment JSON; do not generate a LUT for controls the realtime shader
|
||||
already owns.
|
||||
Use the canonical `details`/`effects` fields for vignette, grain, blur,
|
||||
pixelate, and related primitives. Do not duplicate them with CSS filters,
|
||||
SVG turbulence, opacity, or decorative DOM overlays.
|
||||
5. When the treatment calls for an overlay, install that named block with
|
||||
`hyperframes add <name> --dir <project> --no-clipboard --json`, inspect its
|
||||
returned `data-composition-src` host, and place it once using the block's
|
||||
timing contract. Check for the installed file and host element
|
||||
ID before insertion; never duplicate an existing overlay block. This is one
|
||||
treatment workflow: do not make the user discover Catalog or separately ask
|
||||
for the recipe's justified overlay.
|
||||
6. For ordinary correction/polish, capture one after-sheet with the same three
|
||||
timestamps under `snapshots/treatment-after`, compare it to the before-sheet,
|
||||
and stop when the result is clearly better. Run the normal project check;
|
||||
do not encode a draft solely to prove a static correction.
|
||||
7. Escalate only when evidence requires it. Read individual frames to diagnose a
|
||||
specific visual problem. Preview and render moving evidence when judging
|
||||
treatment keyframes, glitch/tape motion, overlays, playback smoothness, LUT
|
||||
timing, or any other temporal behavior. HDR/LOG, privacy, and brand-sensitive
|
||||
work also require the existing explicit caveats and stronger verification.
|
||||
If the treatment is not clearly better, keep the source unchanged.
|
||||
8. Report the selected media, primary intent, recipe seed if used, final
|
||||
composed controls, optional overlays, and the frames/render that were
|
||||
actually checked. Do not report visual quality from command success alone.
|
||||
|
||||
`media-treatment --analyze` provides deterministic clipping and signal
|
||||
evidence for local composition media, not subject recognition or automatic
|
||||
taste. It reports HDR/metadata caveats and a bounded primary-correction patch;
|
||||
it does not invent wheels, curves, or HSL selections from statistics.
|
||||
@@ -0,0 +1,36 @@
|
||||
# User memory — preferences and recipes
|
||||
|
||||
## Preferences — remembered defaults
|
||||
|
||||
The lightweight tier of user memory: confirmed brief answers (destination, aspect, language, flow, storyboard, voice, style preset) persisted on the same two-tier split as assets — project `.media/preferences.json` (committed, the team inherits it) and personal `~/.media/preferences.json`. A value earns the personal tier by being confirmed in **two different projects**, so a one-off choice never pollutes the global defaults.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/prefs.mjs get --hyperframes . --json # merged view (project overrides user)
|
||||
node <SKILL_DIR>/scripts/prefs.mjs record --hyperframes . --key destination --value x-feed
|
||||
node <SKILL_DIR>/scripts/prefs.mjs record --hyperframes . --key style_preset --value pin-and-paper --workflow faceless-explainer
|
||||
```
|
||||
|
||||
Only what the user actually confirmed gets recorded — never an inferred or defaulted value. How workflows consume these (a remembered value becomes the recommended default with a receipt, and never skips a question) is the brief contract's rule: `hyperframes-core/references/brief-contract.md` § 2, Remembered defaults.
|
||||
|
||||
## Recipes — frozen video bundles
|
||||
|
||||
The heavyweight tier of user memory: one approved run frozen as a named, versioned bundle — `frame.md`, the storyboard skeleton (structure kept, content blanked to per-frame fill-ins), the brief skeleton (from `BRIEF.md` when the project has one — reusable frontmatter kept, run-shape and prose blanked), and the confirmed brief values. Same two tiers: project `.media/recipes/<name>/` (committed) and `~/.media/recipes/<name>/` (a freeze is already a confirmed bundle, so it promotes immediately — no two-project rule). Re-freezing a name bumps `version` and archives the old folder as `<name>@v<N>`.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/recipe.mjs freeze --hyperframes . --name weekly-promo # workflow read from BRIEF.md (--workflow only for briefless projects)
|
||||
node <SKILL_DIR>/scripts/recipe.mjs list --hyperframes . --workflow product-launch-video
|
||||
node <SKILL_DIR>/scripts/recipe.mjs use --hyperframes . --name weekly-promo # also: resolve.mjs --type recipe --entity weekly-promo
|
||||
```
|
||||
|
||||
The freeze is offered once after the final approval (`hyperframes-core/references/review-loop.md` § 4), and the intent layer (`/hyperframes` → `references/intent-interview.md`, step 1) checks for a match before its first question. Adopting a recipe fills the brief, the design spec, and the storyboard skeleton — and unlike preferences it may skip the questions it answers: the bundle was approved as a whole, and adoption itself is the question.
|
||||
|
||||
## Files
|
||||
|
||||
- `.media/manifest.jsonl`: machine SSOT, one JSON record per line
|
||||
- `.media/index.md`: agent-readable table (id, type, dur, dims, path, description)
|
||||
- `.media/preferences.json`: the project's remembered defaults (committed)
|
||||
- `~/.media/`: global cross-project reuse cache (content-addressed, SHA-256)
|
||||
- `~/.media/preferences.json`: personal remembered defaults (promoted after two projects)
|
||||
- `.media/recipes/<name>/`: frozen video bundles — recipe.json + frame.md + storyboard skeleton (committed)
|
||||
- `~/.media/recipes/<name>/`: personal recipe tier (promoted on freeze)
|
||||
- `~/.media/misses.jsonl`: local-only resolve misses, including intent text for `--stats`
|
||||
@@ -0,0 +1,61 @@
|
||||
# Ownership matrix, usage stats, telemetry, privacy
|
||||
|
||||
Maintainer-facing reference. Nothing here changes how you resolve or operate on media.
|
||||
|
||||
## What it owns (the gaps HyperFrames leaves)
|
||||
|
||||
HyperFrames owns media _playback_; media-use owns everything else. Each row is enforced by `scripts/lib/coverage.test.mjs` so the claim can't rot.
|
||||
|
||||
| HyperFrames gap | media-use owns it via |
|
||||
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Audio-only, no image/icon | `resolve --type image\|icon` (heygen asset search) |
|
||||
| No third-party brand logos | `resolve --type logo` (svgl → simple-icons → GitHub org avatar → domain favicon) |
|
||||
| No voice / audio generation | `resolve --type voice` (HeyGen TTS free-usage path; optional local Kokoro) + the audio engine (`audio/scripts/audio.mjs`) |
|
||||
| Scattered/duplicated audio engine | one consolidated engine under `audio/` (hyperframes-media retired) |
|
||||
| No agent media-ops (cut/reframe/transform) | `references/operations.md` + `resolve --from` to register outputs |
|
||||
| No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists |
|
||||
| No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes |
|
||||
| No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) |
|
||||
| Grade recipes and LUT freezing | `resolve --type grade` emits a paste-ready recipe and `resolve --type lut` freezes validated `.cube` files; direct element analysis/authoring lives in `hyperframes media-treatment` |
|
||||
| No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) |
|
||||
| No video generation | `resolve --type video` — HeyGen avatar video first (free-usage path, sign-in nudge on auth failure), local LTX fallback (`videogen` in `scripts/lib/local-models.mjs`); image-to-video, photo-avatar, dub/translate remain manual `heygen` CLI recipes (`references/operations.md`) |
|
||||
| Weak local-model defaults | HeyGen free-usage path via the `heygen` CLI; local open-source tools only as opt-in alternatives (`scripts/lib/local-run.mjs`) |
|
||||
|
||||
## Usage stats
|
||||
|
||||
Use `resolve --stats` for a local, shareable report over the current project's `.media/` manifest, the global `~/.media/` cache, and local resolve misses. Human output is compact; add `--json` for a single machine-readable object, and `--days N` to window timestamped records.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --stats --project . --days 7
|
||||
# media-use stats
|
||||
# total resolves: 12
|
||||
# misses: 2
|
||||
# hit rate: 86%
|
||||
```
|
||||
|
||||
## Telemetry
|
||||
|
||||
`resolve` and the edit tools (transcribe / transcript-cut / audio-duck) send an
|
||||
anonymous usage event to PostHog (`scripts/lib/telemetry.mjs`), so we can see
|
||||
which capabilities are actually used. It records only the media TYPE, the
|
||||
resolution SOURCE, and the winning PROVIDER: never the intent text, file names,
|
||||
or paths, and `$ip:null` so no IP is stored. Best-effort and non-blocking (a
|
||||
resolve never waits on or fails from telemetry).
|
||||
|
||||
Opt out with `DO_NOT_TRACK=1` or `HYPERFRAMES_NO_TELEMETRY=1` (also off in CI and
|
||||
dev). Same public PostHog project key and opt-outs as the `hyperframes` CLI.
|
||||
|
||||
HeyGen request tagging: every generating `heygen` call (TTS, avatar video, catalog
|
||||
search) carries the allowlisted `X-HeyGen-Client-Source: media-use` header, sourced
|
||||
from one shared constant (`HEYGEN_CLIENT_SOURCE_ARGV` in `scripts/lib/heygen-cli.mjs`)
|
||||
so a future call site can't silently ship untagged. Read-only discovery calls
|
||||
(`voice list`, `avatar list`) are intentionally left untagged.
|
||||
|
||||
## Privacy
|
||||
|
||||
media-use uses the same shared install id as the `hyperframes` CLI/studio
|
||||
(`~/.hyperframes/config.json`). When you are signed in to HeyGen, usage is
|
||||
linked to your account email, or username when email is unavailable, matching
|
||||
the CLI behavior. The events stay coarse: media type, source, provider, and
|
||||
small counts only; intent text and paths stay local. Disable telemetry with
|
||||
`HYPERFRAMES_NO_TELEMETRY=1` or `DO_NOT_TRACK=1`.
|
||||
@@ -0,0 +1,325 @@
|
||||
# Media operations: agent guidance
|
||||
|
||||
media-use resolves and remembers assets. For **operating** on them: cutting,
|
||||
reframing, stitching, transforming, it does not wrap every action as a bespoke
|
||||
command. Instead it points you at the right local tool (decision OP1). Run the
|
||||
tool, then register the output with `resolve --from <output> --type <type>` so the
|
||||
result lands in the ledger and the global cache like any other asset.
|
||||
|
||||
All tools below are local and free. ffmpeg is assumed present (it backs the
|
||||
engine already).
|
||||
|
||||
## Cut / trim: keep a slice
|
||||
|
||||
```bash
|
||||
ffmpeg -i in.mp4 -ss 00:00:12 -to 00:00:20 -c copy out.mp4 # 0:12–0:20, no re-encode
|
||||
```
|
||||
|
||||
In-composition trimming usually needs **no new file**: a clip plays a sub-window
|
||||
via `data-media-start` + `data-duration` (see hyperframes-core). Only cut a
|
||||
physical file when exporting/assembling outside the composition.
|
||||
|
||||
## Reframe / crop: change aspect ratio
|
||||
|
||||
```bash
|
||||
# 16:9 -> 9:16, crop centered
|
||||
ffmpeg -i in.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" out.mp4
|
||||
```
|
||||
|
||||
For a non-destructive crop, set a `clip-path` on the element in the composition
|
||||
itself (render-time, source file untouched) instead of re-encoding with ffmpeg.
|
||||
|
||||
## Montage / stitch: join clips
|
||||
|
||||
```bash
|
||||
printf "file '%s'\n" a.mp4 b.mp4 c.mp4 > list.txt
|
||||
ffmpeg -f concat -safe 0 -i list.txt -c copy out.mp4
|
||||
```
|
||||
|
||||
## Silence-cut / highlight: trim dead air, grab the best moment
|
||||
|
||||
```bash
|
||||
auto-editor in.mp4 --edit audio:threshold=4% -o tight.mp4 # pip install auto-editor
|
||||
scenedetect -i in.mp4 detect-adaptive list-scenes # pip install scenedetect
|
||||
```
|
||||
|
||||
## Transforms with a quality choice (process)
|
||||
|
||||
These have a local option AND a higher-quality HeyGen-CLI option. Run the local
|
||||
one for free/offline; use the HeyGen CLI when quality matters. Showing the user
|
||||
a **side-by-side** (local vs HeyGen) is the honest way to let them choose.
|
||||
|
||||
| Op | Local (free) | HeyGen CLI (quality) |
|
||||
| ------------------ | -------------------------------------------------- | --------------------------- |
|
||||
| Background removal | `hyperframes remove-background in.png` (u2net) | `heygen background-removal` |
|
||||
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
|
||||
| Lipsync (dub) | n/a | `heygen lipsync` |
|
||||
| Translate | n/a | `heygen video-translate` |
|
||||
|
||||
After any op: `resolve --from out.ext --type <type>` to register the derived
|
||||
asset (it records provenance and auto-promotes to the global cache).
|
||||
|
||||
> ponytail: media-use doesn't re-wrap ffmpeg/heygen here, that's deliberate
|
||||
> (OP1). The value it adds is the ledger + global reuse on the _output_, via
|
||||
> `--from`. Add a thin `process` verb only if agents repeatedly fumble these
|
||||
> recipes.
|
||||
|
||||
## Exact error-diffusion dither
|
||||
|
||||
Use the local processor when the requested look specifically calls for
|
||||
Floyd-Steinberg, Atkinson/Macintosh, Jarvis-Judice-Ninke, Stucki, Burkes, or a
|
||||
Sierra variant. These are sequential error-diffusion algorithms, not the
|
||||
realtime Bayer `effects.dither` shader.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/dither.mjs \
|
||||
--input source.mp4 \
|
||||
--out source.atkinson.mp4 \
|
||||
--algorithm atkinson \
|
||||
--palette '#0f380f,#306230,#8bac0f,#9bbc0f' \
|
||||
--point-size 3
|
||||
|
||||
node <SKILL_DIR>/scripts/resolve.mjs \
|
||||
--from source.atkinson.mp4 --type video --project .
|
||||
```
|
||||
|
||||
Available algorithms: `floyd-steinberg`, `atkinson`,
|
||||
`jarvis-judice-ninke`, `stucki`, `burkes`, `sierra`, `sierra-lite`, and
|
||||
`two-row-sierra`. The default is balanced Floyd-Steinberg with a black/white
|
||||
palette. Palettes contain 2-6 `#rrggbb` colors in authored dark-to-light order;
|
||||
reversing the order intentionally inverts the mapping. `--point-size` controls
|
||||
1-20px blocks; `--brightness` and `--contrast` accept 0.5-2; `--detail` accepts
|
||||
0.1-1.
|
||||
|
||||
The processor supports ordinary SDR images and MP4 video, preserves video
|
||||
audio, and emits BT.709 MP4. It rejects tagged PQ/HLG input rather than silently
|
||||
tone-mapping it. To animate the transformation, keep the original and processed
|
||||
files as two real media layers and use the seek-safe GSAP timeline to reveal or
|
||||
crossfade between them. Use the realtime Bayer shader instead when the dither
|
||||
amount itself must animate continuously.
|
||||
|
||||
## Transcription (default: Parakeet, better than whisper.cpp)
|
||||
|
||||
`transcribe.mjs` is the default local transcription path. It runs **NVIDIA
|
||||
Parakeet-TDT via parakeet-mlx**, which beats whisper.cpp on the Open ASR
|
||||
Leaderboard (avg WER ~6.05% vs 7.44%; on NOISY audio 4.73% vs 5.96%, where
|
||||
whisper-large-v3 hallucinated to 308% WER on meetings) and is 5-10x faster.
|
||||
It emits `{ text, words:[{text,start,end}] }` with word timestamps (merged from
|
||||
Parakeet's sub-word tokens), feeding transcript-cut, captions, and the audio
|
||||
engine directly.
|
||||
|
||||
```bash
|
||||
# install once: uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx
|
||||
node <SKILL_DIR>/scripts/transcribe.mjs --input talk.mp4 --out talk.transcribe.json
|
||||
|
||||
# equivalently, the hyperframes CLI has Parakeet built in (auto-detects it, whisper fallback):
|
||||
npx hyperframes transcribe talk.mp4 --engine parakeet # or --engine auto (default)
|
||||
```
|
||||
|
||||
VERIFIED on 24GB: accurate, ~3s (cached) for 8s audio. Parakeet covers English +
|
||||
25 European languages. For other languages, or when parakeet-mlx is not
|
||||
installed, transcribe.mjs auto-falls-back to whisper.cpp (99 languages) via
|
||||
`hyperframes transcribe`. `--engine parakeet|whisper` forces one. (Cohere
|
||||
Transcribe tops the leaderboard on paper but its mlx-audio quants produced
|
||||
garbage and ran 40-70x slower on a Mac in testing, so it is not wired in.)
|
||||
|
||||
## Text-based editing (transcript cut)
|
||||
|
||||
`transcript-cut.mjs` is a compiler, not a wrapper: it turns word timestamps and
|
||||
agent cut decisions into exact kept segments. It is provided even though the rest
|
||||
of this file is guidance-only.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/transcript-cut.mjs \
|
||||
--input talk.mp4 \
|
||||
--transcript talk.transcribe.json \
|
||||
--remove "12.41-15.02,88.3-91.7" \
|
||||
--remove-fillers "um,uh,like" \
|
||||
--cut-silence 0.8 \
|
||||
--out talk.cut.mp4
|
||||
|
||||
resolve --from talk.cut.mp4 --type video
|
||||
```
|
||||
|
||||
Use `--plan` first when you want to inspect the kept segment JSON before encoding.
|
||||
|
||||
## Ducking (declare in-composition / bake for export)
|
||||
|
||||
B1, declare ducking in the composition. `audio-duck.mjs` emits GSAP volume
|
||||
keyframes. Paste them into the composition timeline, the source file stays
|
||||
untouched.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/audio-duck.mjs \
|
||||
--meta audio_meta.json \
|
||||
--target "#bgm" \
|
||||
--composition index.html
|
||||
```
|
||||
|
||||
```js
|
||||
// auto-duck: #bgm under narration (generated; base volume 0.6)
|
||||
tl.to("#bgm", { volume: 0.15, duration: 0.15 }, 3.42);
|
||||
tl.to("#bgm", { volume: 0.6, duration: 0.4 }, 9.87);
|
||||
```
|
||||
|
||||
B2, bake ducking only for exported or standalone files.
|
||||
|
||||
```bash
|
||||
ffmpeg -i bgm.mp3 -i voice.wav \
|
||||
-filter_complex "[0][1]sidechaincompress=threshold=0.03:ratio=8:attack=200:release=400[ducked]" \
|
||||
-map "[ducked]" bgm.ducked.wav
|
||||
```
|
||||
|
||||
Declare inside compositions. Bake only for assets leaving the hyperframes
|
||||
pipeline.
|
||||
|
||||
## Publish loudness
|
||||
|
||||
Two-pass `loudnorm` measures first, then applies the measured values with the
|
||||
target LUFS baked in.
|
||||
|
||||
Socials target, -14 LUFS:
|
||||
|
||||
```bash
|
||||
ffmpeg -i mix.wav \
|
||||
-af loudnorm=I=-14:TP=-1.5:LRA=11:print_format=json \
|
||||
-f null -
|
||||
|
||||
ffmpeg -i mix.wav \
|
||||
-af loudnorm=I=-14:TP=-1.5:LRA=11:measured_I=<input_i>:measured_TP=<input_tp>:measured_LRA=<input_lra>:measured_thresh=<input_thresh>:offset=<target_offset>:linear=true:print_format=summary \
|
||||
mix.social.wav
|
||||
```
|
||||
|
||||
Podcast target, -16 LUFS:
|
||||
|
||||
```bash
|
||||
ffmpeg -i mix.wav \
|
||||
-af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
|
||||
-f null -
|
||||
|
||||
ffmpeg -i mix.wav \
|
||||
-af loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=<input_i>:measured_TP=<input_tp>:measured_LRA=<input_lra>:measured_thresh=<input_thresh>:offset=<target_offset>:linear=true:print_format=summary \
|
||||
mix.podcast.wav
|
||||
```
|
||||
|
||||
## Generate: images (local first, cloud upsell)
|
||||
|
||||
`resolve --type image` retrieves from the HeyGen catalog first; on a miss it
|
||||
GENERATES. Two paths, best-for-the-machine picked automatically:
|
||||
|
||||
1. **Local (default, free, private): mflux** (FLUX-on-MLX). `resolve` spec-checks
|
||||
AVAILABLE RAM and runs the best FLUX-class model that fits, via
|
||||
`scripts/lib/local-models.mjs` (`imagegen` ladder) + `mflux-provider.mjs`.
|
||||
The RAM ladder (agent sees it via `describeModelLadder("imagegen", specs)`):
|
||||
|
||||
| Tier | Model | Needs (available RAM) | Notes |
|
||||
| ------ | -------------------- | --------------------- | ----------------------------------- |
|
||||
| medium | FLUX.1 schnell int4 | ~8GB (`--low-ram`) | ~20s/512px on 24GB. VERIFIED. Fast. |
|
||||
| large | FLUX.2 Klein 4B int4 | ~32GB | higher quality, full-resident |
|
||||
| xlarge | Qwen-Image | ~64GB | top quality, 64GB+ Macs only |
|
||||
|
||||
Gotchas baked into the table: the official FLUX repos are HF-gated, so it
|
||||
points at non-gated community 4-bit re-uploads; and `--low-ram` is MANDATORY
|
||||
at the medium tier (without it a 768x512 run swap-thrashed to 90 minutes on
|
||||
24GB; with it, 20 seconds).
|
||||
|
||||
2. **Cloud upsell (better quality): the `codex` CLI** `image_gen` tool, on the
|
||||
user's ChatGPT subscription (codex owns auth, no key here, no per-call
|
||||
charge). It is the automatic fallback when no local model fits AND the
|
||||
explicit "make it better" choice on any machine. Users who just want codex
|
||||
can ask for it directly. Verified: prompt -> raster -> frozen + ledgered.
|
||||
|
||||
`--local-only` keeps mflux (once cached) and skips codex (network).
|
||||
|
||||
## Generate: video (`resolve --type video`, HeyGen avatar first)
|
||||
|
||||
`resolve --type video "<intent>"` is the default path. It generates a
|
||||
script-driven HeyGen avatar video first (the free-usage allowance — OAuth
|
||||
sessions ride the web-plan free avatar-video quota where eligible, API keys
|
||||
follow normal API billing), falling back to local generative LTX only when
|
||||
HeyGen is unavailable, uncredentialed, or `--local-only` is passed. The two
|
||||
are non-substitutable outputs (a real presenter vs. a generic generative
|
||||
clip), so treat the fallback as "HeyGen wasn't reachable," not "upgrade the
|
||||
quality":
|
||||
|
||||
- **HeyGen avatar video (default, free for new API users):**
|
||||
`heygenVideoGenerate` (`scripts/lib/heygen-video-provider.mjs`) shells the
|
||||
`heygen` CLI — never the raw API — auto-picking a public avatar and a
|
||||
starfish voice (override with `--avatar-id`/`--voice-id`, threaded through
|
||||
as `ctx.avatarId`/`ctx.voiceId`). If the CLI reports `not_authenticated`,
|
||||
the provider prints an onboarding recommendation (avatar video is free for
|
||||
new API users — sign in) to stderr and falls through to LTX instead of
|
||||
hard-failing.
|
||||
- **Local fallback: LTX 2.3 on MLX** via `dgrauet/ltx-2-mlx`, the `videogen`
|
||||
ladder in `local-models.mjs` (`ltx-video-provider.mjs`). Generative clips
|
||||
(t2v), spec-gated to RAM. Verified on 24GB: 512x320 x 33f with audio.
|
||||
|
||||
Every generating `heygen` call from media-use — TTS, avatar video, and
|
||||
catalog search — sends the allowlisted `X-HeyGen-Client-Source: media-use`
|
||||
header (persistent flag, works on every subcommand) via the shared
|
||||
`HEYGEN_CLIENT_SOURCE_ARGV` constant (`scripts/lib/heygen-cli.mjs`), so usage
|
||||
tags correctly in billing/resource meta and shows up in the API dashboards.
|
||||
Read-only discovery (`avatar list`, `voice list`) doesn't need it.
|
||||
|
||||
For structured bodies `resolve --type video` doesn't expose yet (a specific
|
||||
`avatar_id`/`voice_id` combination beyond the ctx overrides, or a
|
||||
pre-recorded `audio_url` instead of a script), the raw `heygen video create`
|
||||
recipe below remains the escape hatch:
|
||||
|
||||
```bash
|
||||
# discover an avatar + a starfish voice, then create + wait
|
||||
heygen avatar list --ownership public --limit 5
|
||||
heygen voice list --engine starfish --limit 5
|
||||
heygen video create --headers "X-HeyGen-Client-Source: media-use" --wait -d '{
|
||||
"type": "avatar",
|
||||
"avatar_id": "<avatar-id>",
|
||||
"script": "Your narration here.",
|
||||
"voice_id": "<voice-id>"
|
||||
}'
|
||||
```
|
||||
|
||||
Avatar videos are deterministic + script-driven (lip-sync from a script or a
|
||||
pre-recorded `audio_url`), distinct from the generative LTX clips. After a
|
||||
manual recipe renders, `resolve --from <downloaded.mp4> --type video` to
|
||||
ledger it (not needed when generating via `resolve --type video` directly —
|
||||
that already ledgers the result).
|
||||
|
||||
### Image-to-video (animate any still into a talking clip)
|
||||
|
||||
Not wired into `resolve --type video` (deferred — the `avatar` type covers
|
||||
the default script-driven case). `heygen video create` takes the raw
|
||||
`POST /v3/videos` body, so switching `type`
|
||||
from `avatar` to `image` animates **any image of a person** into a lip-synced
|
||||
talking video, with no avatar/photo-avatar creation step first. Point `image` at a
|
||||
public URL or an uploaded `asset_id`, and drive speech with a `script`+`voice_id`
|
||||
or a pre-recorded `audio_url`:
|
||||
|
||||
```bash
|
||||
heygen video create --headers "X-HeyGen-Client-Source: media-use" --wait -d '{
|
||||
"type": "image",
|
||||
"image": { "type": "url", "url": "https://example.com/person.jpg" },
|
||||
"script": "Your narration here.",
|
||||
"voice_id": "<voice-id>"
|
||||
}'
|
||||
```
|
||||
|
||||
Common optional fields: `title`, `resolution` (`4k`/`1080p`/`720p`),
|
||||
`aspect_ratio`, `remove_background`, `background`, `voice_settings`,
|
||||
`motion_prompt` + `expressiveness` (photo-avatar animation), and
|
||||
`callback_url`/`callback_id` for webhooks. Don't hardcode these from memory: the
|
||||
CLI self-documents the full, current body with
|
||||
`heygen video create --request-schema` (a discriminated union keyed on `type`),
|
||||
so read the schema rather than trusting a stale field list. For a still you'll
|
||||
reuse across many scripts, create a reusable **Photo Avatar** once instead
|
||||
(`heygen avatar create`). Ledger the result with
|
||||
`resolve --from <downloaded.mp4> --type video`. Docs:
|
||||
<https://developers.heygen.com/image-to-video>.
|
||||
|
||||
## HEVC / H.265 sources
|
||||
|
||||
HEVC/H.265 sources need no conversion for **render** (FFmpeg pre-decodes all
|
||||
input video) or for **preview** (auto-proxy transcodes and caches an H.264
|
||||
copy on first use, disable with `--no-proxy` or `media.autoProxy: false` in
|
||||
hyperframes.json). A manual H.264 proxy via `ffmpeg -i in.mp4 -c:v libx264
|
||||
-crf 18 proxy.mp4`, registered with `resolve --from`, remains available for
|
||||
edge cases (e.g. auto-proxy disabled, or ffmpeg unavailable at preview time).
|
||||
@@ -0,0 +1,139 @@
|
||||
# Resolve — command, flags, reuse, adopt, inventory
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type <type> --intent "<description>" --project <dir>
|
||||
```
|
||||
|
||||
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`
|
||||
|
||||
## Types
|
||||
|
||||
| Type | What it finds | Provider / cascade |
|
||||
| ------- | -------------------------------- | ------------------------------------------------------------ |
|
||||
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
|
||||
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
|
||||
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
|
||||
| `icon` | Icons, symbols | HeyGen asset search (type=icon) |
|
||||
| `logo` | Official brand marks | svgl → simple-icons → GitHub org avatar → domain favicon |
|
||||
| `voice` | TTS voiceover | HeyGen TTS free-usage path; optional local Kokoro |
|
||||
| `grade` | HyperFrames color-grading blocks | Core preset → look index params/CDN LUT → deterministic cube |
|
||||
| `lut` | Reusable `.cube` LUT files | Look index params/CDN LUT → deterministic cube |
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Background music
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --project .
|
||||
# → resolved bgm_001 → .media/audio/bgm/bgm_001.mp3 (bgm, 25s)
|
||||
|
||||
# Sound effect
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type sfx --intent "whoosh" --project .
|
||||
# → resolved sfx_001 → .media/audio/sfx/sfx_001.mp3 (sfx, 0.57s)
|
||||
|
||||
# Image
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type image --intent "gradient tech background" --project .
|
||||
# → resolved image_001 → .media/images/image_001.jpg (image)
|
||||
|
||||
# Icon
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type icon --intent "rocket" --project .
|
||||
# → resolved icon_001 → .media/images/icon_001.png (icon, transparent)
|
||||
|
||||
# Brand logo (official mark — never redrawn by hand)
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type logo --entity linkedin --intent "LinkedIn logo" --project .
|
||||
# → resolved logo_001 → .media/images/logo_001.svg (logo, official mark)
|
||||
|
||||
# Color grade block
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json
|
||||
# → {"ok":true,"preset":"warm-daylight","grading":{"preset":"warm-daylight","intensity":1},...}
|
||||
|
||||
# LUT file
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project .
|
||||
# → resolved lut_001 → .media/luts/lut_001.cube (lut)
|
||||
```
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------ |
|
||||
| `--type, -t` | Media type: bgm, sfx, image, icon, logo, voice, grade, lut |
|
||||
| `--intent, -i` | What you need (natural language) |
|
||||
| `--entity, -e` | Entity name for cache matching (optional) |
|
||||
| `--project, -p` | Project directory (default: .) |
|
||||
| `--candidates` | List reusable assets (project + global cache) for `--type`; no download, no mutation |
|
||||
| `--reuse <sha>` | Import a specific global-cache asset (by content sha/prefix, from `--candidates`) |
|
||||
| `--from` | Freeze a local file or direct public URL (ingest) |
|
||||
| `--for` | Analyze a local image/video and add measured adjust suggestions (`grade` only) |
|
||||
| `--local-only` | Offline: skip every network provider (cache + local only) |
|
||||
| `--provider` | Force one generator (e.g. `codex`, `mflux`, `kokoro`, `heygen`) |
|
||||
| `--adopt` | Bulk-import existing assets/ into manifest |
|
||||
| `--doctor` | Check local CLI dependencies; no manifest changes |
|
||||
| `--stats` | Print local usage stats from `.media/` and `~/.media`; no manifest changes |
|
||||
| `--days N` | Limit `--stats` to timestamped records/misses from the last N days |
|
||||
| `--json` | Output JSON instead of one-line result |
|
||||
|
||||
## Reuse before you resolve
|
||||
|
||||
Before resolving bgm/sfx/image/icon/logo/grade/lut, **check what already exists and reuse it when it fits.** media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide.
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --candidates --project .
|
||||
# [project] upbeat tech launch (25s, heygen.audio.sounds)
|
||||
# .media/audio/bgm/bgm_001.wav
|
||||
# [global] energetic tech intro (22s, heygen.audio.sounds)
|
||||
# --reuse 06e052c075fd2b80
|
||||
```
|
||||
|
||||
Read the list and judge semantic fit yourself — "upbeat tech launch" ≈ "energetic tech intro" is a call only you can make from the descriptions. Then:
|
||||
|
||||
- **A project candidate fits** → just reference its path in your composition. Nothing else to run.
|
||||
- **A global candidate fits** → `resolve --type bgm --reuse <sha>` copies it into this project (self-contained render) and records it.
|
||||
- **Nothing fits** → resolve fresh (`--type ... --intent ...`).
|
||||
|
||||
**Trust guardrail — when unsure, resolve fresh.** A redundant download is cheap; shipping the wrong asset is not. Judge fit from description + prompt + type + duration/dims. For **brand/entity** assets, reuse a _global_ candidate only when the entity matches exactly — the global cache aggregates every project you have worked on, so a `--candidates` list can surface another client's brand mark and its prompt text. Never reuse a cross-project brand asset on a loose match.
|
||||
|
||||
The deterministic floor still runs automatically: an identical (case/whitespace-insensitive) repeat auto-reuses with no `--candidates` step. `--candidates` is only for the semantic layer above that floor — and a fuzzy match is **never** auto-applied; reuse is always your explicit call. On a resolve that misses the floor and is about to fetch, media-use prints a one-line stderr hint when similar cached assets exist, pointing you back here.
|
||||
|
||||
## How it works
|
||||
|
||||
`resolve` runs an automatic floor, then falls through to fetching:
|
||||
|
||||
1. Check project `.media/manifest.jsonl` for a prompt match (case- and whitespace-insensitive) — auto-reuse
|
||||
2. Scan existing `assets/` directory for unregistered files that share a word with the need
|
||||
3. Check global cache `~/.media/` for a reusable asset matched on the same normalized prompt — auto-reuse
|
||||
4. Search via provider (HeyGen audio catalog, HeyGen asset search), or resolve color locally
|
||||
5. Freeze file to `.media/<type>/`, register in manifest, regenerate `index.md`, auto-promote to `~/.media/`
|
||||
|
||||
Steps 1 and 3 are the **deterministic floor**: they only auto-reuse an exact-normalized match, never a fuzzy one. Semantic reuse ("close enough") is the agent's explicit call via [Reuse before you resolve](#reuse-before-you-resolve) — it never happens automatically. The agent gets back **one line**; candidates, scores, provenance stay on disk.
|
||||
|
||||
## Adopt existing projects
|
||||
|
||||
Most HyperFrames projects already have assets in `assets/`. media-use adopts them:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --adopt --project .
|
||||
# → adopted 9 assets from assets/
|
||||
# bgm_001 → assets/bgm/mango-fizz.mp3 (bgm, 146.6s)
|
||||
# image_001 → assets/images/avatar.jpg (image, 400×400)
|
||||
```
|
||||
|
||||
`ffprobe` extracts real duration and dimensions. During resolve, unregistered files in `assets/` matching the intent are adopted on the fly.
|
||||
|
||||
## Reading the inventory
|
||||
|
||||
After resolve or adopt, read `.media/index.md` for the full inventory:
|
||||
|
||||
```
|
||||
# .media · 4 assets
|
||||
|
||||
id type dur dims path description
|
||||
bgm_001 bgm 25s - .media/audio/bgm/bgm_001.mp3 upbeat tech launch
|
||||
sfx_001 sfx 0.6s - .media/audio/sfx/sfx_001.mp3 whoosh
|
||||
image_001 image - 1920×1080 .media/images/image_001.jpg gradient tech background
|
||||
icon_001 icon - 200×200 .media/images/icon_001.png rocket
|
||||
```
|
||||
|
||||
## Cross-project reuse
|
||||
|
||||
Assets are cached automatically on resolve. Every resolved/ingested asset is auto-promoted to the global cache at `~/.media/`, so subsequent resolves for the same (or near-identical) prompt, in any project, hit the cache with no re-download and no provider call.
|
||||
|
||||
For a _semantically_ similar (not identical) need in another project, the exact-match floor won't fire — use [Reuse before you resolve](#reuse-before-you-resolve): `--candidates` lists the global assets, and `--reuse <sha>` imports the one you pick. This is how a track resolved in one project gets reused in the next when the wording differs.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Setup and providers — install, auth, RAM ladders, forcing a provider
|
||||
|
||||
## Setup — install heygen first (free-usage path)
|
||||
|
||||
Install the HeyGen CLI through its [verified release instructions](https://developers.heygen.com/cli), then run:
|
||||
|
||||
```bash
|
||||
heygen update # free usage needs the OAuth-capable CLI (v0.3.0+)
|
||||
heygen auth login --oauth # OAuth = free subscription credits; --api-key bills API credits
|
||||
```
|
||||
|
||||
This unlocks the FREE path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar videos. Sign in with `--oauth` — the free allowance rides on the OAuth session (an API key bills API credits instead). **media-use requires heygen >= v0.3.0 uniformly** (the OAuth free-usage path needs it), so `--doctor` nudges older CLIs to update even for API-key-only use. Before resolving anything, verify setup with:
|
||||
|
||||
```bash
|
||||
node <SKILL_DIR>/scripts/resolve.mjs --doctor
|
||||
```
|
||||
|
||||
## Providers
|
||||
|
||||
media-use holds no keys; every external tool owns its auth. Generation is
|
||||
centered on the HeyGen CLI free-usage path. Install and authenticate `heygen`
|
||||
before resolving bgm/sfx/image/icon/voice/avatar-video. Local tools are opt-in
|
||||
alternatives where they exist: mflux for image, Kokoro for voice, Parakeet for
|
||||
transcription, and LTX for local video generation. `resolve` spec-checks
|
||||
AVAILABLE RAM for those local ladders (`describeModelLadder`); the agent can
|
||||
see the ladder and override.
|
||||
|
||||
| Type | Provider / path |
|
||||
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| bgm/sfx | heygen catalog free-usage path |
|
||||
| image | heygen search free-usage path; optional local mflux; codex `image_gen` upsell |
|
||||
| voice | heygen tts free-usage path; optional local **Kokoro** (free, on-device) |
|
||||
| icon | heygen asset search free-usage path |
|
||||
| logo | svgl, then simple-icons, then GitHub org avatar, then domain favicon (all free) |
|
||||
| grade/lut | local core-preset map, params/CDN look index, deterministic `buildCube` fallback |
|
||||
| video | heygen avatar video free-usage path (sign-in nudge on auth failure); optional local LTX (`videogen` ladder). Image-to-video / photo-avatar / dub stay manual `heygen` recipes |
|
||||
|
||||
Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
|
||||
private, offline once cached). The `codex` CLI remains the ChatGPT-sub image
|
||||
upsell. Cost rule (X4): the agent confirms before an agent-initiated paid call;
|
||||
a user-requested one just runs — `heygen.video` is flagged paid (metered free
|
||||
allowance) so an agent-initiated `resolve --type video` confirms first.
|
||||
|
||||
To force a specific generator (e.g. a user says "make this image with codex"),
|
||||
pass `--provider codex`: it pins resolution to that provider and skips the
|
||||
free-usage default. See `references/operations.md` for the RAM ladders and
|
||||
provider recipes.
|
||||
|
||||
`--local-only` skips every network provider, including the free HeyGen ones,
|
||||
leaving the project + global cache and any installed local provider. For
|
||||
HeyGen-only types, that means no fresh resolve.
|
||||
|
||||
## CLI tools used (what to run, and how to enable each)
|
||||
|
||||
`resolve` auto-cascades; each provider shells one CLI. HeyGen is the
|
||||
free-usage path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar
|
||||
video, so those capabilities need `heygen` installed and authenticated. Local
|
||||
tools are OPT-IN alternatives where they exist; install one to unlock its free,
|
||||
private, on-device path instead of or ahead of HeyGen for that type. Only
|
||||
`ffmpeg`/`ffprobe` are strictly required for the tool to run at all.
|
||||
|
||||
| Tool | Serves | Install |
|
||||
| ------------------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
|
||||
| `heygen` | catalog (bgm/sfx/image/icon) + TTS (voice) + avatar video — the free-usage path | install through [verified HeyGen release instructions](https://developers.heygen.com/cli), then `heygen auth login --oauth` (needs >= v0.3.0) |
|
||||
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
|
||||
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
|
||||
| `parakeet-mlx` | local transcription (default ASR, best) | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
|
||||
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
|
||||
| `npx hyperframes` | Kokoro TTS (voice), whisper.cpp (transcribe fallback), remove-background | via the hyperframes CLI; whisper.cpp is built on first use (Homebrew on macOS, else git+cmake), models download from HuggingFace |
|
||||
|
||||
The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
|
||||
`scripts/lib/local-models.mjs` (the agent can read `describeModelLadder(cap, specs)`
|
||||
to see which model fits this machine). Without a tool on PATH, its provider
|
||||
prints a one-line diagnostic to stderr and resolve falls through where another
|
||||
provider exists (e.g. no `mflux` -> codex image upsell; no `parakeet-mlx` -> whisper.cpp).
|
||||
|
||||
`heygen asset search` is a pre-launch command hidden from `heygen --help`, but it
|
||||
runs; providers tag requests with the allowlisted `X-HeyGen-Client-Source` header
|
||||
(v0.3.0+).
|
||||
@@ -0,0 +1,49 @@
|
||||
# media-use usage dashboard
|
||||
|
||||
Reproducible definition of the media-use usage dashboard. The dashboard answers
|
||||
"how much is media-use used, for what, is reuse working, and what can't it
|
||||
satisfy" from the telemetry `scripts/lib/telemetry.mjs` already emits. Build it
|
||||
in an authorized HyperFrames analytics project; this doc is the source of truth
|
||||
so it can be recreated. Local complement: `resolve --stats` (same questions,
|
||||
from `.media/` + `~/.media`, no dashboard access needed).
|
||||
|
||||
## Identity (see `scripts/lib/telemetry.mjs`)
|
||||
|
||||
Events attribute to the **same person as the hyperframes CLI and studio**
|
||||
— the shared install id in `~/.hyperframes/config.json` (`anonymousId`), stitched
|
||||
to the HeyGen account (`$identify`, `distinct_id` = email/username) on sign-in.
|
||||
Not fully anonymous by design; pseudonymous before sign-in, account-linked after.
|
||||
`$ip:null`. Opt-out: `HYPERFRAMES_NO_TELEMETRY=1` / `DO_NOT_TRACK=1` (also CI, dev).
|
||||
|
||||
## Event catalog (verified present in-project)
|
||||
|
||||
Every event carries `surface: "media-use"`. Event **properties are coarse** —
|
||||
never intent text, file names, or paths.
|
||||
|
||||
| Event | Fires on | Key properties |
|
||||
| ---------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
|
||||
| `media_use_resolve` | a resolve that produced/returned an asset | `type`, `source`, `provider`, `via`, `local_only`, `provider_override` |
|
||||
| `media_use_resolve_miss` | a resolve that found nothing | `type`, `local_only`, `provider_override` (no intent) |
|
||||
| `media_use_candidates` | `--candidates` / `--dry-run` listing | `type`, counts |
|
||||
| `media_use_doctor_run` | `--doctor` | `ok`, `checks_failed`, `failed[]` |
|
||||
| `media_use_compare` | `grade-compare` / `compare` | `command`, `cells`, `truncated`, `total`, `render_ready_timed_out` |
|
||||
| `media_use_transcribe` · `media_use_duck` · `media_use_transcript_cut` | audio-engine ops | op-specific |
|
||||
|
||||
## Dashboard tiles
|
||||
|
||||
1. **Invocation volume** — `query-trends`, count of `media_use_resolve` over time (daily). "How much."
|
||||
2. **By media type** — `media_use_resolve` broken down by `type` (bgm/sfx/image/icon/logo/voice/grade/lut). "For what."
|
||||
3. **Resolve hit-rate** — trends formula: `A / (A + B)` where A = `media_use_resolve`, B = `media_use_resolve_miss`. "Is the catalog covering needs."
|
||||
4. **Provider mix** — `media_use_resolve` broken down by `provider`; a second tile by `via` (`url` / `params-fallback` / `params`) to catch CDN→params LUT downgrades.
|
||||
5. **Top misses** — `media_use_resolve_miss` broken down by `type` (the tuning signal — pair with local `resolve --stats`, which also shows the missed _intents_ that telemetry deliberately omits).
|
||||
6. **Doctor health** — `media_use_doctor_run` broken down by `failed[]` (which dependency check fails most) + `checks_failed` distribution.
|
||||
7. **Compare cost** — `media_use_compare` by `command`, plus `truncated` / `render_ready_timed_out` rates (observe before lifting the 16-cell cap).
|
||||
8. **Adoption (optional)** — if the `first_run` property ships (plan U5), segment `media_use_resolve` first-run vs repeat.
|
||||
|
||||
## Recreate in an analytics dashboard
|
||||
|
||||
For each tile, confirm the event/property schema, build its trend or breakdown,
|
||||
then add it to a dashboard. Keep names prefixed `media-use:` so the dashboard is
|
||||
greppable. Cross-surface note: because identity is shared with CLI/studio, you
|
||||
can also break these down by the same person across `cli_command*` and `studio:*`
|
||||
events.
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import { parseArgs } from "node:util";
|
||||
import { duckKeyframes, speechSpans } from "./lib/duck.mjs";
|
||||
import { track } from "./lib/telemetry.mjs";
|
||||
|
||||
const { values: args } = parseArgs({
|
||||
options: {
|
||||
meta: { type: "string" },
|
||||
target: { type: "string" },
|
||||
duck: { type: "string", default: "0.25" },
|
||||
attack: { type: "string", default: "0.15" },
|
||||
release: { type: "string", default: "0.4" },
|
||||
"merge-gap": { type: "string", default: "0.6" },
|
||||
sequential: { type: "boolean", default: false },
|
||||
gap: { type: "string", default: "0" },
|
||||
offsets: { type: "string" },
|
||||
composition: { type: "string" },
|
||||
json: { type: "boolean", default: false },
|
||||
help: { type: "boolean", short: "h", default: false },
|
||||
},
|
||||
strict: true,
|
||||
});
|
||||
|
||||
if (args.help) {
|
||||
console.log(`media-use audio-duck — generate GSAP volume ducking keyframes
|
||||
|
||||
Usage:
|
||||
node audio-duck.mjs --meta audio_meta.json --target "#bgm"
|
||||
|
||||
Options:
|
||||
--meta audio_meta.json or JSON word transcript
|
||||
--target GSAP selector for the background audio element
|
||||
--duck Duck multiplier (default: 0.25)
|
||||
--attack Duck-in duration seconds (default: 0.15)
|
||||
--release Restore duration seconds (default: 0.4)
|
||||
--merge-gap Bridge speech gaps smaller than this many seconds (default: 0.6)
|
||||
--sequential Place multi-line meta back to back at composition time
|
||||
--gap Extra seconds between sequential lines (default: 0)
|
||||
--offsets Explicit placement, "l1=0,l2=3.4" (voice id = start seconds)
|
||||
--composition Read target data-volume from this HTML file
|
||||
--json Output { spans, keyframes }
|
||||
--help, -h Show this help`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
try {
|
||||
run();
|
||||
await track("media_use_duck", { sequential: !!args.sequential });
|
||||
} catch (err) {
|
||||
if (args.json) console.log(JSON.stringify({ ok: false, error: err.message }));
|
||||
else console.error(`error: ${err.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function run() {
|
||||
if (!args.meta || !args.target) throw new Error("--meta and --target are required");
|
||||
const meta = JSON.parse(readFileSync(resolve(args.meta), "utf8"));
|
||||
const target = args.target;
|
||||
const baseVolume = readBaseVolume(args.composition, target);
|
||||
const offsets = args.offsets
|
||||
? Object.fromEntries(
|
||||
args.offsets.split(",").map((pair) => {
|
||||
const [id, t] = pair.split("=");
|
||||
return [id.trim(), Number(t)];
|
||||
}),
|
||||
)
|
||||
: undefined;
|
||||
const spans = speechSpans(meta, {
|
||||
mergeGap: Number(args["merge-gap"]),
|
||||
sequential: args.sequential,
|
||||
gap: Number(args.gap),
|
||||
offsets,
|
||||
});
|
||||
const keyframes = duckKeyframes(spans, {
|
||||
duck: Number(args.duck),
|
||||
attack: Number(args.attack),
|
||||
release: Number(args.release),
|
||||
baseVolume,
|
||||
});
|
||||
|
||||
if (args.json) {
|
||||
console.log(JSON.stringify({ spans, keyframes }));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(
|
||||
`// auto-duck: ${target} under narration (generated; base volume ${fmt(baseVolume)})`,
|
||||
);
|
||||
for (const keyframe of keyframes) {
|
||||
console.log(
|
||||
`tl.to(${JSON.stringify(target)}, { volume: ${fmt(keyframe.volume)}, duration: ${fmt(
|
||||
keyframe.duration,
|
||||
)} }, ${fmt(keyframe.time)});`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function readBaseVolume(composition, target) {
|
||||
if (!composition || !target.startsWith("#")) return 1;
|
||||
const id = target.slice(1);
|
||||
const html = readFileSync(resolve(composition), "utf8");
|
||||
// ponytail: regex is enough here because this only reads one attribute from
|
||||
// one user-authored composition element, not arbitrary HTML.
|
||||
const tag = html.match(new RegExp(`<[^>]*\\bid=["']${escapeRegExp(id)}["'][^>]*>`, "i"))?.[0];
|
||||
const raw = tag?.match(/\bdata-volume=["']([^"']+)["']/i)?.[1];
|
||||
const volume = Number(raw);
|
||||
return Number.isFinite(volume) ? volume : 1;
|
||||
}
|
||||
|
||||
function escapeRegExp(value) {
|
||||
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
}
|
||||
|
||||
function fmt(n) {
|
||||
return Number(n)
|
||||
.toFixed(3)
|
||||
.replace(/\.?0+$/, "");
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { execFileSync, spawn } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
import { existsSync, mkdirSync, renameSync, rmSync } from "node:fs";
|
||||
import { dirname, extname, resolve } from "node:path";
|
||||
import { text } from "node:stream/consumers";
|
||||
import { parseArgs } from "node:util";
|
||||
import {
|
||||
ERROR_DIFFUSION_ALGORITHMS,
|
||||
applyErrorDiffusionRgba,
|
||||
errorDiffusionBufferLength,
|
||||
} from "./lib/error-diffusion.mjs";
|
||||
|
||||
const IMAGE_EXTENSIONS = new Set([".png", ".jpg", ".jpeg", ".webp", ".bmp", ".tif", ".tiff"]);
|
||||
const OUTPUT_IMAGE_EXTENSIONS = new Set([".png", ".jpg", ".jpeg", ".webp"]);
|
||||
|
||||
const { values: args } = parseArgs({
|
||||
options: {
|
||||
input: { type: "string", short: "i" },
|
||||
out: { type: "string", short: "o" },
|
||||
algorithm: { type: "string", short: "a", default: "floyd-steinberg" },
|
||||
palette: { type: "string", default: "#000000,#ffffff" },
|
||||
"point-size": { type: "string", default: "3" },
|
||||
brightness: { type: "string", default: "1" },
|
||||
contrast: { type: "string", default: "1.2" },
|
||||
detail: { type: "string", default: "1" },
|
||||
json: { type: "boolean", default: false },
|
||||
help: { type: "boolean", short: "h", default: false },
|
||||
},
|
||||
strict: true,
|
||||
});
|
||||
|
||||
if (args.help) {
|
||||
console.log(`media-use dither — exact cached error-diffusion for image or MP4 media
|
||||
|
||||
Usage:
|
||||
node dither.mjs --input in.mp4 --out out.mp4 [options]
|
||||
|
||||
Options:
|
||||
--algorithm, -a ${Object.keys(ERROR_DIFFUSION_ALGORITHMS).join(" | ")}
|
||||
--palette 2-6 authored-order #rrggbb colors, comma-separated
|
||||
--point-size Block size in pixels, 1-20 (default: 3)
|
||||
--brightness 0.5-2 (default: 1)
|
||||
--contrast 0.5-2 (default: 1.2)
|
||||
--detail Diffused-error strength, 0.1-1 (default: 1)
|
||||
--json Output JSON status
|
||||
--help, -h Show this help
|
||||
|
||||
Video output uses the source average frame rate as CFR; VFR cadence is normalized.
|
||||
|
||||
After processing, register the output with:
|
||||
node resolve.mjs --from <output> --type image|video`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await run();
|
||||
if (args.json) console.log(JSON.stringify({ ok: true, ...result }));
|
||||
else {
|
||||
console.log(`dithered ${result.input} -> ${result.out} (${result.algorithm})`);
|
||||
console.log(`next: resolve --from ${result.out} --type ${result.type}`);
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (args.json) console.log(JSON.stringify({ ok: false, error: message }));
|
||||
else console.error(`error: ${message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function run() {
|
||||
if (!args.input || !args.out) throw new Error("--input and --out are required");
|
||||
const inputPath = resolve(args.input);
|
||||
const outPath = resolve(args.out);
|
||||
if (!existsSync(inputPath)) throw new Error(`input file not found: ${inputPath}`);
|
||||
if (inputPath === outPath) throw new Error("--out must differ from --input");
|
||||
|
||||
const metadata = probe(inputPath);
|
||||
if (metadata.colorTransfer === "smpte2084" || metadata.colorTransfer === "arib-std-b67") {
|
||||
throw new Error(
|
||||
`HDR ${metadata.colorTransfer} input is not supported by the 8-bit SDR dither processor; tone-map to Rec.709 first`,
|
||||
);
|
||||
}
|
||||
const options = {
|
||||
algorithm: args.algorithm,
|
||||
palette: args.palette.split(",").map((color) => color.trim()),
|
||||
pointSize: Number(args["point-size"]),
|
||||
brightness: Number(args.brightness),
|
||||
contrast: Number(args.contrast),
|
||||
detail: Number(args.detail),
|
||||
};
|
||||
// Validate before starting FFmpeg or creating an output file.
|
||||
applyErrorDiffusionRgba(new Uint8ClampedArray(4), 1, 1, options, new Float32Array(3));
|
||||
|
||||
mkdirSync(dirname(outPath), { recursive: true });
|
||||
const inputIsImage = IMAGE_EXTENSIONS.has(extname(inputPath).toLowerCase());
|
||||
if (inputIsImage) {
|
||||
if (!OUTPUT_IMAGE_EXTENSIONS.has(extname(outPath).toLowerCase())) {
|
||||
throw new Error("image output must use .png, .jpg, .jpeg, or .webp");
|
||||
}
|
||||
processImage(inputPath, outPath, metadata, options);
|
||||
} else {
|
||||
if (extname(outPath).toLowerCase() !== ".mp4") throw new Error("video output must use .mp4");
|
||||
await processVideo(inputPath, outPath, metadata, options);
|
||||
}
|
||||
|
||||
return {
|
||||
input: inputPath,
|
||||
out: outPath,
|
||||
type: inputIsImage ? "image" : "video",
|
||||
algorithm: options.algorithm,
|
||||
palette: options.palette,
|
||||
point_size: options.pointSize,
|
||||
brightness: options.brightness,
|
||||
contrast: options.contrast,
|
||||
detail: options.detail,
|
||||
};
|
||||
}
|
||||
|
||||
function probe(filePath) {
|
||||
const raw = execFileSync(
|
||||
"ffprobe",
|
||||
["-v", "error", "-print_format", "json", "-show_streams", "-show_format", "--", filePath],
|
||||
{ encoding: "utf8", timeout: 10_000 },
|
||||
);
|
||||
const parsed = JSON.parse(raw);
|
||||
const video = parsed.streams?.find((stream) => stream.codec_type === "video");
|
||||
if (!video?.width || !video?.height)
|
||||
throw new Error(`no readable video/image stream: ${filePath}`);
|
||||
const fps = usableFrameRate(video.avg_frame_rate) ?? usableFrameRate(video.r_frame_rate) ?? "30";
|
||||
return {
|
||||
width: video.width,
|
||||
height: video.height,
|
||||
fps,
|
||||
colorTransfer: video.color_transfer || "",
|
||||
};
|
||||
}
|
||||
|
||||
function processImage(inputPath, outPath, metadata, options) {
|
||||
const frameBytes = metadata.width * metadata.height * 4;
|
||||
const rgba = execFileSync(
|
||||
"ffmpeg",
|
||||
[
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-nostdin",
|
||||
"-i",
|
||||
inputPath,
|
||||
"-frames:v",
|
||||
"1",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgba",
|
||||
"-",
|
||||
],
|
||||
{ maxBuffer: frameBytes + 1024 },
|
||||
);
|
||||
if (rgba.length !== frameBytes)
|
||||
throw new Error(`decoded ${rgba.length} bytes; expected ${frameBytes}`);
|
||||
applyErrorDiffusionRgba(rgba, metadata.width, metadata.height, options);
|
||||
|
||||
const temporary = temporaryOutput(outPath);
|
||||
try {
|
||||
execFileSync(
|
||||
"ffmpeg",
|
||||
[
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgba",
|
||||
"-s:v",
|
||||
`${metadata.width}x${metadata.height}`,
|
||||
"-i",
|
||||
"-",
|
||||
"-frames:v",
|
||||
"1",
|
||||
temporary,
|
||||
],
|
||||
{ input: rgba, maxBuffer: frameBytes + 1024 },
|
||||
);
|
||||
renameSync(temporary, outPath);
|
||||
} finally {
|
||||
rmSync(temporary, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function processVideo(inputPath, outPath, metadata, options) {
|
||||
const temporary = temporaryOutput(outPath);
|
||||
const frameBytes = metadata.width * metadata.height * 4;
|
||||
const keyframeInterval = String(Math.max(1, Math.round(frameRateNumber(metadata.fps))));
|
||||
const errors = new Float32Array(
|
||||
errorDiffusionBufferLength(metadata.width, metadata.height, options.pointSize),
|
||||
);
|
||||
const decoder = spawn("ffmpeg", [
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-nostdin",
|
||||
"-i",
|
||||
inputPath,
|
||||
"-map",
|
||||
"0:v:0",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgba",
|
||||
"-",
|
||||
]);
|
||||
const encoder = spawn("ffmpeg", [
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgba",
|
||||
"-s:v",
|
||||
`${metadata.width}x${metadata.height}`,
|
||||
"-r",
|
||||
metadata.fps,
|
||||
"-i",
|
||||
"-",
|
||||
"-i",
|
||||
inputPath,
|
||||
"-map",
|
||||
"0:v:0",
|
||||
"-map",
|
||||
"1:a?",
|
||||
"-map_metadata",
|
||||
"1",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-preset",
|
||||
"veryfast",
|
||||
"-crf",
|
||||
"18",
|
||||
"-g",
|
||||
keyframeInterval,
|
||||
"-keyint_min",
|
||||
keyframeInterval,
|
||||
"-sc_threshold",
|
||||
"0",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-x264-params",
|
||||
"colorprim=bt709:transfer=bt709:colormatrix=bt709",
|
||||
"-color_primaries:v",
|
||||
"bt709",
|
||||
"-color_trc:v",
|
||||
"bt709",
|
||||
"-colorspace:v",
|
||||
"bt709",
|
||||
"-color_range",
|
||||
"tv",
|
||||
"-c:a",
|
||||
"aac",
|
||||
"-b:a",
|
||||
"192k",
|
||||
"-shortest",
|
||||
"-movflags",
|
||||
"+faststart",
|
||||
temporary,
|
||||
]);
|
||||
const decoderError = text(decoder.stderr);
|
||||
const encoderError = text(encoder.stderr);
|
||||
const decoderDone = once(decoder, "close").then(([code]) => code ?? 1);
|
||||
const encoderDone = once(encoder, "close").then(([code]) => code ?? 1);
|
||||
|
||||
try {
|
||||
const frame = Buffer.allocUnsafe(frameBytes);
|
||||
let frameOffset = 0;
|
||||
for await (const chunk of decoder.stdout) {
|
||||
let chunkOffset = 0;
|
||||
while (chunkOffset < chunk.length) {
|
||||
const length = Math.min(frameBytes - frameOffset, chunk.length - chunkOffset);
|
||||
chunk.copy(frame, frameOffset, chunkOffset, chunkOffset + length);
|
||||
chunkOffset += length;
|
||||
frameOffset += length;
|
||||
if (frameOffset !== frameBytes) continue;
|
||||
applyErrorDiffusionRgba(frame, metadata.width, metadata.height, options, errors);
|
||||
await writeFrame(encoder.stdin, frame);
|
||||
frameOffset = 0;
|
||||
}
|
||||
}
|
||||
if (frameOffset)
|
||||
throw new Error(`decoder returned a partial RGBA frame (${frameOffset} bytes)`);
|
||||
encoder.stdin.end();
|
||||
const [decoderCode, encoderCode] = await Promise.all([decoderDone, encoderDone]);
|
||||
if (decoderCode !== 0) throw new Error(`FFmpeg decode failed: ${(await decoderError).trim()}`);
|
||||
if (encoderCode !== 0) throw new Error(`FFmpeg encode failed: ${(await encoderError).trim()}`);
|
||||
renameSync(temporary, outPath);
|
||||
} catch (error) {
|
||||
decoder.kill("SIGKILL");
|
||||
encoder.kill("SIGKILL");
|
||||
throw error;
|
||||
} finally {
|
||||
rmSync(temporary, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function writeFrame(stream, frame) {
|
||||
return new Promise((resolveWrite, reject) => {
|
||||
stream.write(frame, (error) => (error ? reject(error) : resolveWrite()));
|
||||
});
|
||||
}
|
||||
|
||||
function usableFrameRate(value) {
|
||||
if (!value || value === "0/0") return null;
|
||||
const number = frameRateNumber(value);
|
||||
return Number.isFinite(number) && number > 0 ? value : null;
|
||||
}
|
||||
|
||||
function frameRateNumber(value) {
|
||||
const [numerator, denominator = "1"] = value.split("/");
|
||||
return Number(numerator) / Number(denominator);
|
||||
}
|
||||
|
||||
function temporaryOutput(outPath) {
|
||||
const extension = extname(outPath);
|
||||
return `${outPath.slice(0, -extension.length)}.part-${process.pid}${extension}`;
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const SCRIPT = fileURLToPath(new URL("./dither.mjs", import.meta.url));
|
||||
const HAS_FFMPEG =
|
||||
spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0 &&
|
||||
spawnSync("ffprobe", ["-version"], { stdio: "ignore" }).status === 0;
|
||||
|
||||
function fixture() {
|
||||
const dir = mkdtempSync(join(tmpdir(), "media-use-dither-"));
|
||||
return { dir, cleanup: () => rmSync(dir, { recursive: true, force: true }) };
|
||||
}
|
||||
|
||||
function ffmpeg(args) {
|
||||
execFileSync("ffmpeg", ["-y", "-hide_banner", "-loglevel", "error", ...args]);
|
||||
}
|
||||
|
||||
function run(args) {
|
||||
return spawnSync(process.execPath, [SCRIPT, ...args, "--json"], { encoding: "utf8" });
|
||||
}
|
||||
|
||||
test("processes an image into the requested ordered palette", { skip: !HAS_FFMPEG }, (t) => {
|
||||
const { dir, cleanup } = fixture();
|
||||
t.after(cleanup);
|
||||
const input = join(dir, "source.png");
|
||||
const output = join(dir, "dithered.png");
|
||||
ffmpeg(["-f", "lavfi", "-i", "testsrc2=size=16x12:rate=1", "-frames:v", "1", input]);
|
||||
|
||||
const result = run([
|
||||
"--input",
|
||||
input,
|
||||
"--out",
|
||||
output,
|
||||
"--algorithm",
|
||||
"floyd-steinberg",
|
||||
"--palette",
|
||||
"#000000,#ffffff",
|
||||
"--point-size",
|
||||
"3",
|
||||
]);
|
||||
assert.equal(result.status, 0, result.stderr || result.stdout);
|
||||
assert.equal(JSON.parse(result.stdout).type, "image");
|
||||
|
||||
const rgb = execFileSync("ffmpeg", [
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-i",
|
||||
output,
|
||||
"-frames:v",
|
||||
"1",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgb24",
|
||||
"-",
|
||||
]);
|
||||
for (let index = 0; index < rgb.length; index += 3) {
|
||||
const value = rgb[index];
|
||||
assert.ok(value === 0 || value === 255);
|
||||
assert.equal(rgb[index + 1], value);
|
||||
assert.equal(rgb[index + 2], value);
|
||||
}
|
||||
});
|
||||
|
||||
test("processes moving MP4 frames, audio, and BT.709 metadata", { skip: !HAS_FFMPEG }, (t) => {
|
||||
const { dir, cleanup } = fixture();
|
||||
t.after(cleanup);
|
||||
const input = join(dir, "source.mp4");
|
||||
const output = join(dir, "dithered.mp4");
|
||||
ffmpeg([
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=size=16x12:rate=3:duration=2",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"sine=frequency=440:duration=3",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-c:a",
|
||||
"aac",
|
||||
input,
|
||||
]);
|
||||
|
||||
const result = run([
|
||||
"--input",
|
||||
input,
|
||||
"--out",
|
||||
output,
|
||||
"--algorithm",
|
||||
"atkinson",
|
||||
"--palette",
|
||||
"#0f380f,#306230,#8bac0f,#9bbc0f",
|
||||
]);
|
||||
assert.equal(result.status, 0, result.stderr || result.stdout);
|
||||
const probe = JSON.parse(
|
||||
execFileSync(
|
||||
"ffprobe",
|
||||
["-v", "error", "-print_format", "json", "-show_streams", "-show_format", "--", output],
|
||||
{
|
||||
encoding: "utf8",
|
||||
},
|
||||
),
|
||||
);
|
||||
const video = probe.streams.find((stream) => stream.codec_type === "video");
|
||||
const audio = probe.streams.find((stream) => stream.codec_type === "audio");
|
||||
assert.equal(video.width, 16);
|
||||
assert.equal(video.height, 12);
|
||||
assert.equal(video.nb_frames, "6");
|
||||
assert.equal(video.color_space, "bt709");
|
||||
assert.equal(video.color_transfer, "bt709");
|
||||
assert.equal(video.color_primaries, "bt709");
|
||||
assert.equal(video.color_range, "tv");
|
||||
assert.equal(audio.codec_name, "aac");
|
||||
assert.ok(Number(probe.format.duration) < 2.5, "audio must not outlive processed video");
|
||||
const keyframes = execFileSync(
|
||||
"ffprobe",
|
||||
[
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v:0",
|
||||
"-skip_frame",
|
||||
"nokey",
|
||||
"-show_entries",
|
||||
"frame=best_effort_timestamp_time",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
"--",
|
||||
output,
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
)
|
||||
.trim()
|
||||
.split(/\s+/)
|
||||
.map((value) => Number.parseFloat(value));
|
||||
assert.deepEqual(keyframes, [0, 1]);
|
||||
});
|
||||
|
||||
test("rejects tagged PQ or HLG instead of silently producing SDR", { skip: !HAS_FFMPEG }, (t) => {
|
||||
const { dir, cleanup } = fixture();
|
||||
t.after(cleanup);
|
||||
const input = join(dir, "hlg.mp4");
|
||||
ffmpeg([
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=white:size=16x12:rate=1:duration=1",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p10le",
|
||||
"-x264-params",
|
||||
"colorprim=bt2020:transfer=arib-std-b67:colormatrix=bt2020nc",
|
||||
"-color_primaries:v",
|
||||
"bt2020",
|
||||
"-color_trc:v",
|
||||
"arib-std-b67",
|
||||
"-colorspace:v",
|
||||
"bt2020nc",
|
||||
input,
|
||||
]);
|
||||
|
||||
const result = run(["--input", input, "--out", join(dir, "wrong.mp4")]);
|
||||
assert.equal(result.status, 1);
|
||||
assert.match(JSON.parse(result.stdout).error, /HDR arib-std-b67 input is not supported/);
|
||||
});
|
||||
@@ -0,0 +1,369 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* media-use eval — compare baseline (no media-use) vs. with media-use
|
||||
* on real registry blocks. Produces an HTML report.
|
||||
*/
|
||||
|
||||
import {
|
||||
mkdtempSync,
|
||||
cpSync,
|
||||
rmSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
existsSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join, basename, resolve, dirname } from "node:path";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(SCRIPT_DIR, "..", "..", "..");
|
||||
const RESOLVE_SCRIPT = join(SCRIPT_DIR, "resolve.mjs");
|
||||
|
||||
const TEST_BLOCKS = [
|
||||
"registry/blocks/nyc-paris-flight",
|
||||
"registry/blocks/macos-tahoe-liquid-glass",
|
||||
"registry/blocks/blue-sweater-intro-video",
|
||||
"registry/blocks/vpn-youtube-spot",
|
||||
"registry/blocks/apple-money-count",
|
||||
"registry/blocks/liquid-glass-notification",
|
||||
"registry/blocks/instagram-follow",
|
||||
];
|
||||
|
||||
// Run resolve.mjs with args as a literal argv array (no shell), so values
|
||||
// interpolated from manifest metadata (--intent prompt, --type) can't inject
|
||||
// shell. Mirrors the execFileSync fix in probe.mjs / heygen-search.mjs.
|
||||
function run(args, opts = {}) {
|
||||
try {
|
||||
return {
|
||||
ok: true,
|
||||
output: execFileSync(process.execPath, [RESOLVE_SCRIPT, ...args], {
|
||||
encoding: "utf8",
|
||||
timeout: 15000,
|
||||
stdio: "pipe",
|
||||
...opts,
|
||||
}).trim(),
|
||||
};
|
||||
} catch (err) {
|
||||
return { ok: false, output: (err.stdout || "") + (err.stderr || ""), code: err.status };
|
||||
}
|
||||
}
|
||||
|
||||
function countAssetFiles(dir) {
|
||||
const assetsDir = join(dir, "assets");
|
||||
if (!existsSync(assetsDir)) return { count: 0, files: [] };
|
||||
const files = [];
|
||||
function walk(d, base = "") {
|
||||
for (const e of readdirSync(d, { withFileTypes: true })) {
|
||||
const rel = base ? `${base}/${e.name}` : e.name;
|
||||
if (e.isDirectory()) walk(join(d, e.name), rel);
|
||||
else files.push(rel);
|
||||
}
|
||||
}
|
||||
walk(assetsDir);
|
||||
return { count: files.length, files };
|
||||
}
|
||||
|
||||
function evalBlock(blockPath) {
|
||||
const fullPath = join(REPO_ROOT, blockPath);
|
||||
if (!existsSync(fullPath)) return null;
|
||||
|
||||
const name = basename(blockPath);
|
||||
const tmp = mkdtempSync(join(tmpdir(), `mu-eval-${name}-`));
|
||||
|
||||
try {
|
||||
cpSync(fullPath, tmp, { recursive: true });
|
||||
|
||||
// baseline: what the agent sees WITHOUT media-use
|
||||
const baseline = countAssetFiles(tmp);
|
||||
const htmlFiles = readdirSync(tmp).filter((f) => f.endsWith(".html"));
|
||||
|
||||
// parse compositions for asset references
|
||||
const assetRefs = [];
|
||||
for (const hf of htmlFiles) {
|
||||
const html = readFileSync(join(tmp, hf), "utf8");
|
||||
const srcMatches = html.matchAll(/src=["']([^"']+?)["']/g);
|
||||
for (const m of srcMatches) {
|
||||
const ref = m[1];
|
||||
if (ref.startsWith("data:") || ref.startsWith("http")) continue;
|
||||
assetRefs.push({ composition: hf, ref });
|
||||
}
|
||||
const urlMatches = html.matchAll(/url\(["']?([^"')]+?)["']?\)/g);
|
||||
for (const m of urlMatches) {
|
||||
const ref = m[1];
|
||||
if (ref.startsWith("data:") || ref.startsWith("http") || ref.startsWith("#")) continue;
|
||||
assetRefs.push({ composition: hf, ref });
|
||||
}
|
||||
}
|
||||
|
||||
// with media-use: run --adopt
|
||||
const adoptResult = run(["--adopt", "--project", tmp, "--json"]);
|
||||
let adopted = { ok: false, adopted: 0, assets: [] };
|
||||
if (adoptResult.ok) {
|
||||
try {
|
||||
adopted = JSON.parse(adoptResult.output);
|
||||
} catch {
|
||||
/* */
|
||||
}
|
||||
}
|
||||
|
||||
// read the generated index
|
||||
const indexPath = join(tmp, ".media", "index.md");
|
||||
const indexContent = existsSync(indexPath)
|
||||
? readFileSync(indexPath, "utf8")
|
||||
: "(no index generated)";
|
||||
|
||||
// read manifest for detail
|
||||
const manifestPath = join(tmp, ".media", "manifest.jsonl");
|
||||
const manifest = existsSync(manifestPath)
|
||||
? readFileSync(manifestPath, "utf8")
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((l) => {
|
||||
try {
|
||||
return JSON.parse(l);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})
|
||||
.filter(Boolean)
|
||||
: [];
|
||||
|
||||
// test resolve cache hit: try resolving something that was adopted
|
||||
let resolveTest = null;
|
||||
if (manifest.length > 0) {
|
||||
const first = manifest[0];
|
||||
const prompt = first.provenance?.prompt || first.description;
|
||||
const r = run(["--type", first.type, "--intent", prompt, "--project", tmp, "--json"]);
|
||||
if (r.ok) {
|
||||
try {
|
||||
resolveTest = JSON.parse(r.output);
|
||||
} catch {
|
||||
/* */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// test resolve miss: try resolving something that doesn't exist
|
||||
const missResult = run([
|
||||
"--type",
|
||||
"bgm",
|
||||
"--intent",
|
||||
"nonexistent query xyz",
|
||||
"--project",
|
||||
tmp,
|
||||
"--json",
|
||||
]);
|
||||
let resolveMiss = null;
|
||||
if (!missResult.ok) {
|
||||
try {
|
||||
resolveMiss = JSON.parse(missResult.output);
|
||||
} catch {
|
||||
/* */
|
||||
}
|
||||
}
|
||||
|
||||
// coverage: which composition refs are covered by the manifest
|
||||
const manifestPaths = new Set(manifest.map((m) => m.path));
|
||||
const coverage = assetRefs.map((r) => ({
|
||||
...r,
|
||||
covered: manifestPaths.has(r.ref),
|
||||
}));
|
||||
|
||||
return {
|
||||
name,
|
||||
baseline: { fileCount: baseline.count, files: baseline.files, htmlCount: htmlFiles.length },
|
||||
compositions: htmlFiles,
|
||||
assetRefs: coverage,
|
||||
adopted: { count: adopted.adopted, assets: adopted.assets || [] },
|
||||
index: indexContent,
|
||||
manifest,
|
||||
resolveTest,
|
||||
resolveMiss,
|
||||
};
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function generateReport(results) {
|
||||
const all = results.filter(Boolean);
|
||||
const passed = all.filter((r) => r.adopted.count > 0);
|
||||
|
||||
const rows = results
|
||||
.filter(Boolean)
|
||||
.map((r) => {
|
||||
const hasMetadata = r.manifest.some((m) => m.duration || m.width);
|
||||
const cacheHit = r.resolveTest?._source === "cached";
|
||||
const missHandled = r.resolveMiss?.ok === false;
|
||||
|
||||
return `<tr>
|
||||
<td><strong>${r.name}</strong></td>
|
||||
<td>${r.baseline.fileCount} files, ${r.baseline.htmlCount} comp${r.baseline.htmlCount === 1 ? "" : "s"}</td>
|
||||
<td>${r.adopted.count} adopted</td>
|
||||
<td>${hasMetadata ? "<span class='pass'>with metadata</span>" : "<span class='warn'>no metadata</span>"}</td>
|
||||
<td>${cacheHit ? "<span class='pass'>cache hit</span>" : "<span class='warn'>no hit</span>"}</td>
|
||||
<td>${missHandled ? "<span class='pass'>handled</span>" : "<span class='fail'>unexpected</span>"}</td>
|
||||
</tr>`;
|
||||
})
|
||||
.join("\n");
|
||||
|
||||
const details = results
|
||||
.filter(Boolean)
|
||||
.filter((r) => r.adopted.count > 0)
|
||||
.map((r) => {
|
||||
const assetRows = r.manifest
|
||||
.map((m) => {
|
||||
const dur = m.duration != null ? `${m.duration}s` : "—";
|
||||
const dims = m.width && m.height ? `${m.width}×${m.height}` : "—";
|
||||
return `<tr><td>${m.id}</td><td>${m.type}</td><td>${dur}</td><td>${dims}</td><td class="path">${m.path}</td><td>${m.description || ""}</td></tr>`;
|
||||
})
|
||||
.join("\n");
|
||||
|
||||
const coveredCount = r.assetRefs.filter((c) => c.covered).length;
|
||||
const totalRefs = r.assetRefs.length;
|
||||
const coveragePct = totalRefs > 0 ? Math.round((coveredCount / totalRefs) * 100) : 100;
|
||||
|
||||
const refRows = r.assetRefs
|
||||
.map(
|
||||
(c) =>
|
||||
`<tr><td class="path">${c.composition}</td><td class="path">${c.ref}</td><td>${c.covered ? "<span class='pass'>covered</span>" : "<span class='warn'>not in manifest</span>"}</td></tr>`,
|
||||
)
|
||||
.join("\n");
|
||||
|
||||
return `<div class="block-detail">
|
||||
<h3>${r.name}</h3>
|
||||
<p style="font-size:13px;color:var(--muted)">${r.compositions.length} composition${r.compositions.length === 1 ? "" : "s"}: ${r.compositions.join(", ")}</p>
|
||||
|
||||
<div class="comparison">
|
||||
<div class="col">
|
||||
<h4>Baseline (no media-use)</h4>
|
||||
<p>Agent sees: ${r.baseline.fileCount} raw files in assets/<br>No metadata, no type info, no relationship to compositions.</p>
|
||||
<pre class="file-list">${r.baseline.files.join("\n") || "(no assets)"}</pre>
|
||||
</div>
|
||||
<div class="col">
|
||||
<h4>With media-use (after --adopt)</h4>
|
||||
<p>Agent reads index.md — structured, typed, with metadata:</p>
|
||||
<pre class="index">${escapeHtml(r.index)}</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
${
|
||||
totalRefs > 0
|
||||
? `<h4>Composition → asset coverage <span class="${coveragePct === 100 ? "pass" : "warn"}">${coveragePct}%</span> (${coveredCount}/${totalRefs} refs)</h4>
|
||||
<table class="manifest">
|
||||
<thead><tr><th>composition</th><th>asset reference</th><th>in manifest?</th></tr></thead>
|
||||
<tbody>${refRows}</tbody>
|
||||
</table>`
|
||||
: ""
|
||||
}
|
||||
|
||||
<h4>Manifest records</h4>
|
||||
<table class="manifest">
|
||||
<thead><tr><th>id</th><th>type</th><th>dur</th><th>dims</th><th>path</th><th>description</th></tr></thead>
|
||||
<tbody>${assetRows}</tbody>
|
||||
</table>
|
||||
</div>`;
|
||||
})
|
||||
.join("\n");
|
||||
|
||||
return `<title>media-use eval report</title>
|
||||
<style>
|
||||
:root { --bg: #fafaf7; --text: #1b1b18; --muted: #7a756a; --accent: #0d7377; --good: #1a7a3a; --warn: #b45309; --fail: #dc2626; --border: #e8e5df; --surface: #fff; --mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; --sans: system-ui, -apple-system, sans-serif; --serif: Georgia, serif }
|
||||
* { box-sizing: border-box; margin: 0 } body { background: var(--bg); color: var(--text); font-family: var(--serif); line-height: 1.6; font-size: 15px; padding: 40px 24px }
|
||||
.wrap { max-width: 1100px; margin: 0 auto }
|
||||
h1 { font-family: var(--sans); font-size: 28px; font-weight: 700; margin-bottom: 8px; letter-spacing: -.02em }
|
||||
h2 { font-family: var(--sans); font-size: 20px; font-weight: 650; margin: 32px 0 12px; letter-spacing: -.01em }
|
||||
h3 { font-family: var(--sans); font-size: 17px; font-weight: 650; margin: 24px 0 8px }
|
||||
h4 { font-family: var(--sans); font-size: 14px; font-weight: 600; margin: 16px 0 6px; color: var(--muted) }
|
||||
p { margin-bottom: 10px }
|
||||
.meta { font-family: var(--mono); font-size: 12px; color: var(--muted); margin-bottom: 24px }
|
||||
.summary { display: flex; gap: 16px; margin: 16px 0; flex-wrap: wrap }
|
||||
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 18px; flex: 1; min-width: 140px }
|
||||
.stat .num { font-family: var(--sans); font-size: 28px; font-weight: 700; color: var(--accent) }
|
||||
.stat .label { font-family: var(--mono); font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: .1em }
|
||||
table { width: 100%; border-collapse: collapse; font-size: 13px; font-family: var(--sans); margin: 8px 0 }
|
||||
th { text-align: left; font-family: var(--mono); font-size: 10px; letter-spacing: .08em; text-transform: uppercase; color: var(--muted); border-bottom: 2px solid var(--border); padding: 6px 8px; font-weight: 700 }
|
||||
td { border-bottom: 1px solid var(--border); padding: 7px 8px; vertical-align: top }
|
||||
td.path { font-family: var(--mono); font-size: 12px; color: var(--muted); max-width: 300px; overflow: hidden; text-overflow: ellipsis }
|
||||
.pass { color: var(--good); font-weight: 600 } .warn { color: var(--warn); font-weight: 600 } .fail { color: var(--fail); font-weight: 600 }
|
||||
.comparison { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin: 12px 0 }
|
||||
@media(max-width:700px) { .comparison { grid-template-columns: 1fr } }
|
||||
.col { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 16px }
|
||||
.col h4 { margin-top: 0 }
|
||||
pre { font-family: var(--mono); font-size: 12px; background: #1b1b18; color: #d4d0c8; border-radius: 6px; padding: 12px 14px; overflow-x: auto; margin: 6px 0; line-height: 1.5 }
|
||||
pre.file-list { background: var(--bg); color: var(--muted); border: 1px solid var(--border) }
|
||||
pre.index { white-space: pre; }
|
||||
.block-detail { border-top: 1px solid var(--border); padding-top: 20px; margin-top: 20px }
|
||||
.verdict { margin-top: 24px; padding: 16px 20px; border-radius: 8px; font-family: var(--sans); font-size: 15px }
|
||||
.verdict.ship { background: #edfbf0; border: 1px solid #1a7a3a; color: #1a7a3a }
|
||||
.verdict.wait { background: #fff3ec; border: 1px solid #d94f04; color: #d94f04 }
|
||||
</style>
|
||||
<div class="wrap">
|
||||
<h1>media-use eval report</h1>
|
||||
<p class="meta">${new Date().toISOString().slice(0, 10)} · ${all.length} blocks evaluated · baseline vs. media-use --adopt</p>
|
||||
|
||||
<div class="summary">
|
||||
<div class="stat"><div class="num">${all.length}</div><div class="label">blocks tested</div></div>
|
||||
<div class="stat"><div class="num">${passed.length}</div><div class="label">with assets</div></div>
|
||||
<div class="stat"><div class="num">${all.reduce((s, r) => s + r.adopted.count, 0)}</div><div class="label">assets adopted</div></div>
|
||||
<div class="stat"><div class="num">${all.filter((r) => r.manifest.some((m) => m.duration || m.width)).length}</div><div class="label">with ffprobe metadata</div></div>
|
||||
<div class="stat"><div class="num">${(() => {
|
||||
const refs = all.flatMap((r) => r.assetRefs);
|
||||
const covered = refs.filter((c) => c.covered).length;
|
||||
return refs.length > 0 ? Math.round((covered / refs.length) * 100) + "%" : "—";
|
||||
})()}</div><div class="label">composition coverage</div></div>
|
||||
</div>
|
||||
|
||||
<h2>Results matrix</h2>
|
||||
<table>
|
||||
<thead><tr><th>Block</th><th>Baseline</th><th>Adopted</th><th>Metadata</th><th>Cache hit</th><th>Miss handling</th></tr></thead>
|
||||
<tbody>${rows}</tbody>
|
||||
</table>
|
||||
|
||||
<h2>Before / after comparisons</h2>
|
||||
${details}
|
||||
|
||||
<div class="verdict ${passed.length >= 3 ? "ship" : "wait"}">
|
||||
${
|
||||
passed.length >= 3
|
||||
? `<strong>Ship it.</strong> ${passed.length}/${all.length} blocks adopted successfully with metadata. Resolve cache hits work. Miss handling is clean.`
|
||||
: `<strong>Needs work.</strong> Only ${passed.length} blocks adopted. Check the failures above.`
|
||||
}
|
||||
</div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function escapeHtml(str) {
|
||||
return str.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
|
||||
}
|
||||
|
||||
console.log("media-use eval · running against registry blocks...\n");
|
||||
|
||||
const results = [];
|
||||
for (const block of TEST_BLOCKS) {
|
||||
const fullPath = join(REPO_ROOT, block);
|
||||
if (!existsSync(fullPath)) {
|
||||
console.log(` skip ${basename(block)} (not found)`);
|
||||
results.push(null);
|
||||
continue;
|
||||
}
|
||||
process.stdout.write(` ${basename(block)}...`);
|
||||
const result = evalBlock(block);
|
||||
if (result) {
|
||||
console.log(
|
||||
` ${result.adopted.count} adopted, ${result.manifest.filter((m) => m.duration || m.width).length} with metadata`,
|
||||
);
|
||||
} else {
|
||||
console.log(" failed");
|
||||
}
|
||||
results.push(result);
|
||||
}
|
||||
|
||||
const report = generateReport(results);
|
||||
const outPath = join(SCRIPT_DIR, "..", "eval-report.html");
|
||||
writeFileSync(outPath, report);
|
||||
console.log(`\nReport: ${outPath}`);
|
||||
@@ -0,0 +1,128 @@
|
||||
import { readdirSync, statSync, existsSync } from "node:fs";
|
||||
import { join, extname, basename } from "node:path";
|
||||
import { readManifest, appendRecord, nextId } from "./manifest.mjs";
|
||||
import { regenerateIndex } from "./index-gen.mjs";
|
||||
import { probe } from "./probe.mjs";
|
||||
import { matchTokens } from "./match.mjs";
|
||||
|
||||
const AUDIO_EXT = new Set([".mp3", ".wav", ".ogg", ".m4a", ".aac"]);
|
||||
const IMAGE_EXT = new Set([".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg", ".ico"]);
|
||||
const VIDEO_EXT = new Set([".mp4", ".webm", ".mov"]);
|
||||
|
||||
function inferType(filePath) {
|
||||
const ext = extname(filePath).toLowerCase();
|
||||
if (AUDIO_EXT.has(ext)) {
|
||||
const lower = filePath.toLowerCase();
|
||||
if (lower.includes("/bgm/") || lower.includes("/music/") || lower.startsWith("bgm/"))
|
||||
return "bgm";
|
||||
if (lower.includes("/sfx/") || lower.includes("/sound") || lower.startsWith("sfx/"))
|
||||
return "sfx";
|
||||
if (lower.includes("/voice/") || lower.includes("/narrat") || lower.startsWith("voice/"))
|
||||
return "voice";
|
||||
return "bgm";
|
||||
}
|
||||
if (IMAGE_EXT.has(ext)) {
|
||||
if (ext === ".svg" || ext === ".ico") return "icon";
|
||||
return "image";
|
||||
}
|
||||
if (VIDEO_EXT.has(ext)) return "video";
|
||||
return null;
|
||||
}
|
||||
|
||||
function walkDir(dir, base = "") {
|
||||
const files = [];
|
||||
if (!existsSync(dir)) return files;
|
||||
for (const entry of readdirSync(dir, { withFileTypes: true })) {
|
||||
const rel = base ? `${base}/${entry.name}` : entry.name;
|
||||
if (entry.isDirectory()) {
|
||||
files.push(...walkDir(join(dir, entry.name), rel));
|
||||
} else {
|
||||
files.push(rel);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
export function scanExistingAssets(projectDir) {
|
||||
const assetsDir = join(projectDir, "assets");
|
||||
if (!existsSync(assetsDir)) return [];
|
||||
|
||||
const files = walkDir(assetsDir);
|
||||
const found = [];
|
||||
for (const rel of files) {
|
||||
const type = inferType(rel);
|
||||
if (!type) continue;
|
||||
const fullPath = join(assetsDir, rel);
|
||||
const stat = statSync(fullPath);
|
||||
if (stat.size === 0) {
|
||||
// A 0-byte asset would register clean but fail at render — skip it loudly
|
||||
// rather than adopt a broken file.
|
||||
console.error(`media-use: skipping 0-byte asset assets/${rel}`);
|
||||
continue;
|
||||
}
|
||||
const meta = probe(fullPath);
|
||||
found.push({
|
||||
relativePath: `assets/${rel}`,
|
||||
type,
|
||||
size: stat.size,
|
||||
name: basename(rel, extname(rel)),
|
||||
...meta,
|
||||
});
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
export function adoptExistingAssets(projectDir) {
|
||||
const existing = scanExistingAssets(projectDir);
|
||||
if (existing.length === 0) return [];
|
||||
|
||||
const manifest = readManifest(projectDir);
|
||||
const knownPaths = new Set(manifest.map((r) => r.path));
|
||||
|
||||
const adopted = [];
|
||||
for (const asset of existing) {
|
||||
if (knownPaths.has(asset.relativePath)) continue;
|
||||
|
||||
const id = nextId(projectDir, asset.type);
|
||||
const record = {
|
||||
id,
|
||||
type: asset.type,
|
||||
path: asset.relativePath,
|
||||
source: "existing",
|
||||
description: asset.name.replace(/[-_]/g, " "),
|
||||
...(asset.duration != null && { duration: asset.duration }),
|
||||
...(asset.width != null && { width: asset.width }),
|
||||
...(asset.height != null && { height: asset.height }),
|
||||
provenance: { provider: "local", adopted: true },
|
||||
};
|
||||
appendRecord(projectDir, record);
|
||||
adopted.push(record);
|
||||
}
|
||||
|
||||
if (adopted.length > 0) regenerateIndex(projectDir);
|
||||
return adopted;
|
||||
}
|
||||
|
||||
// Adopt a pre-existing assets/ file only when it shares a meaningful word with
|
||||
// the intent. The old test — `name.includes(intent) || intent.includes(name)` —
|
||||
// silently returned the WRONG file: "whoosh" grabbed a stray who.mp3, and a
|
||||
// one-letter filename matched every intent. A false negative just falls through
|
||||
// to a catalog search (safe); a false positive ships the wrong asset. So bias to
|
||||
// precision: require a shared token, don't guess from substrings.
|
||||
export function findExistingAsset(projectDir, intent, type) {
|
||||
const assetsDir = join(projectDir, "assets");
|
||||
if (!existsSync(assetsDir)) return null;
|
||||
const intentTokens = matchTokens(intent);
|
||||
if (intentTokens.size === 0) return null;
|
||||
for (const rel of walkDir(assetsDir)) {
|
||||
const t = inferType(rel);
|
||||
if (!t || (type && t !== type)) continue;
|
||||
const stem = basename(rel, extname(rel));
|
||||
for (const tok of matchTokens(stem)) {
|
||||
if (intentTokens.has(tok)) {
|
||||
return { relativePath: `assets/${rel}`, type: t, name: stem };
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { findExistingAsset, adoptExistingAssets } from "./adopt.mjs";
|
||||
|
||||
let tmp;
|
||||
function setup() {
|
||||
tmp = mkdtempSync(join(tmpdir(), "mu-adopt-test-"));
|
||||
}
|
||||
function cleanup() {
|
||||
if (tmp) rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
function drop(rel) {
|
||||
const full = join(tmp, "assets", rel);
|
||||
mkdirSync(join(full, ".."), { recursive: true });
|
||||
writeFileSync(full, "x");
|
||||
}
|
||||
|
||||
function runTests() {
|
||||
const tests = [];
|
||||
const test = (name, fn) => tests.push({ name, fn });
|
||||
|
||||
test("does NOT false-match a short filename stem (who.mp3 vs 'whoosh')", () => {
|
||||
setup();
|
||||
drop("sfx/who.mp3");
|
||||
assert.equal(findExistingAsset(tmp, "whoosh", "sfx"), null);
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("does NOT match a one-letter filename against any intent", () => {
|
||||
setup();
|
||||
drop("images/a.jpg");
|
||||
assert.equal(findExistingAsset(tmp, "gradient tech background", "image"), null);
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("matches on a shared meaningful word", () => {
|
||||
setup();
|
||||
drop("images/hero-shot.jpg");
|
||||
const hit = findExistingAsset(tmp, "hero image", "image");
|
||||
assert.ok(hit);
|
||||
assert.equal(hit.relativePath, "assets/images/hero-shot.jpg");
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("matches multi-word overlap", () => {
|
||||
setup();
|
||||
drop("images/gradient-tech-bg.jpg");
|
||||
assert.ok(findExistingAsset(tmp, "gradient tech background", "image"));
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("a shared stopword alone does not match", () => {
|
||||
setup();
|
||||
drop("video/the-video.mp4");
|
||||
assert.equal(findExistingAsset(tmp, "the rocket", null), null);
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("respects the type filter", () => {
|
||||
setup();
|
||||
drop("sfx/rocket.mp3");
|
||||
assert.equal(findExistingAsset(tmp, "rocket", "image"), null, "wrong type is skipped");
|
||||
assert.ok(findExistingAsset(tmp, "rocket", "sfx"), "right type matches");
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("adoptExistingAssets still imports every typed file (unaffected by match rule)", () => {
|
||||
setup();
|
||||
drop("sfx/who.mp3");
|
||||
drop("images/a.jpg");
|
||||
assert.equal(adoptExistingAssets(tmp).length, 2);
|
||||
cleanup();
|
||||
});
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
for (const { name, fn } of tests) {
|
||||
try {
|
||||
fn();
|
||||
passed++;
|
||||
console.log(` \x1b[32m✓\x1b[0m ${name}`);
|
||||
} catch (err) {
|
||||
failed++;
|
||||
console.log(` \x1b[31m✗\x1b[0m ${name}`);
|
||||
console.log(` ${err.message}`);
|
||||
}
|
||||
}
|
||||
console.log(`\n${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) process.exit(1);
|
||||
}
|
||||
|
||||
console.log("media-use · adopt / findExistingAsset tests\n");
|
||||
runTests();
|
||||
@@ -0,0 +1,20 @@
|
||||
import { heygenSearch } from "./heygen-search.mjs";
|
||||
|
||||
export const bgmProvider = {
|
||||
async search(intent) {
|
||||
const results = heygenSearch("audio sounds list", intent, { type: "music" });
|
||||
if (!results) return null;
|
||||
const best = results[0];
|
||||
return {
|
||||
url: best.audio_url,
|
||||
source: "search",
|
||||
// ext derived from audio_url by resolve.mjs — catalog tracks are .mp3 or .wav
|
||||
metadata: {
|
||||
description: best.description || intent,
|
||||
duration: best.duration || null,
|
||||
provider: "heygen.audio.sounds",
|
||||
provenance: { track_id: best.id, score: best.score, query: intent },
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,59 @@
|
||||
import { readFileSync, existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
function findDesignSpec(projectDir) {
|
||||
for (const name of ["frame.md", "design.md", "DESIGN.md"]) {
|
||||
const p = join(projectDir, name);
|
||||
if (existsSync(p)) return { path: p, name };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseFrontmatter(content) {
|
||||
const match = content.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (!match) return null;
|
||||
const yaml = match[1];
|
||||
const tokens = {};
|
||||
for (const line of yaml.split("\n")) {
|
||||
const m = line.match(/^\s*(\w[\w-]*):\s*(.+)/);
|
||||
if (m) tokens[m[1]] = m[2].trim().replace(/^["']|["']$/g, "");
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
|
||||
function extractColors(tokens) {
|
||||
const colors = [];
|
||||
for (const [k, v] of Object.entries(tokens)) {
|
||||
if (typeof v === "string" && /^#[0-9a-fA-F]{3,8}$/.test(v)) {
|
||||
colors.push({ name: k, hex: v });
|
||||
}
|
||||
}
|
||||
return colors;
|
||||
}
|
||||
|
||||
export const brandProvider = {
|
||||
async search(intent, { projectDir } = {}) {
|
||||
if (!projectDir) return null;
|
||||
const spec = findDesignSpec(projectDir);
|
||||
if (!spec) return null;
|
||||
const content = readFileSync(spec.path, "utf8");
|
||||
const tokens = parseFrontmatter(content);
|
||||
if (!tokens) return null;
|
||||
const colors = extractColors(tokens);
|
||||
return {
|
||||
localPath: spec.path,
|
||||
source: "local",
|
||||
ext: ".md",
|
||||
metadata: {
|
||||
description: "Brand tokens from " + spec.name,
|
||||
provider: "design_spec",
|
||||
provenance: {
|
||||
file: spec.name,
|
||||
colors,
|
||||
font: tokens.font || tokens.typography || null,
|
||||
logo: tokens.logo || null,
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,108 @@
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { extname, join } from "node:path";
|
||||
|
||||
const LIB_DIR =
|
||||
process.env.HYPERFRAMES_MEDIA_USE_SFX_DIR ||
|
||||
join(import.meta.dirname, "..", "..", "audio", "assets", "sfx");
|
||||
|
||||
export const BUNDLED_SFX_RECOVERY_COMMAND = "npx hyperframes skills update media-use";
|
||||
|
||||
export class BundledSfxAssetsError extends Error {
|
||||
constructor(health) {
|
||||
super(
|
||||
`bundled SFX assets are missing or incomplete (${health.detail}). Repair the installed media-use skill: ${health.fix}`,
|
||||
);
|
||||
this.name = "BundledSfxAssetsError";
|
||||
this.code = health.code;
|
||||
this.fix = health.fix;
|
||||
}
|
||||
}
|
||||
|
||||
function unhealthy(detail) {
|
||||
return {
|
||||
ok: false,
|
||||
code: "bundled_sfx_assets_missing",
|
||||
detail,
|
||||
fix: BUNDLED_SFX_RECOVERY_COMMAND,
|
||||
};
|
||||
}
|
||||
|
||||
export function inspectBundledSfxAssets(libraryDir = LIB_DIR) {
|
||||
const manifestPath = join(libraryDir, "manifest.json");
|
||||
if (!existsSync(manifestPath)) return unhealthy(`manifest not found: ${manifestPath}`);
|
||||
|
||||
let manifest;
|
||||
try {
|
||||
manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
|
||||
} catch {
|
||||
return unhealthy(`manifest is not valid JSON: ${manifestPath}`);
|
||||
}
|
||||
if (!manifest || typeof manifest !== "object" || Array.isArray(manifest)) {
|
||||
return unhealthy(`manifest must contain an object: ${manifestPath}`);
|
||||
}
|
||||
|
||||
const entries = Object.entries(manifest);
|
||||
if (entries.length === 0) return unhealthy(`manifest contains no SFX entries: ${manifestPath}`);
|
||||
for (const [key, entry] of entries) {
|
||||
if (!entry?.file || typeof entry.file !== "string") {
|
||||
return unhealthy(`manifest entry "${key}" has no file`);
|
||||
}
|
||||
const assetPath = join(libraryDir, entry.file);
|
||||
if (!existsSync(assetPath)) return unhealthy(`asset not found: ${assetPath}`);
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
count: entries.length,
|
||||
detail: `${entries.length} bundled SFX asset${entries.length === 1 ? "" : "s"} available`,
|
||||
fix: "",
|
||||
};
|
||||
}
|
||||
|
||||
const normalize = (value) =>
|
||||
String(value)
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, " ")
|
||||
.trim();
|
||||
|
||||
export function extensionForBundledSfxFile(filename) {
|
||||
return extname(filename) || ".mp3";
|
||||
}
|
||||
|
||||
function score(intent, key, entry) {
|
||||
const query = normalize(intent);
|
||||
const name = normalize(key);
|
||||
if (query === name) return 100;
|
||||
if (query.includes(name) || name.includes(query)) return 50;
|
||||
const haystack = new Set(normalize(`${key} ${entry.description || ""}`).split(/\s+/));
|
||||
return query.split(/\s+/).filter((token) => token && haystack.has(token)).length;
|
||||
}
|
||||
|
||||
export const bundledSfxProvider = {
|
||||
async search(intent, ctx = {}) {
|
||||
const libraryDir = ctx.libraryDir || LIB_DIR;
|
||||
const health = inspectBundledSfxAssets(libraryDir);
|
||||
if (!health.ok) throw new BundledSfxAssetsError(health);
|
||||
const manifest = JSON.parse(readFileSync(join(libraryDir, "manifest.json"), "utf8"));
|
||||
|
||||
const ranked = Object.entries(manifest)
|
||||
.map(([key, entry]) => ({ key, entry, score: score(intent, key, entry) }))
|
||||
.filter(({ entry, score }) => entry?.file && score > 0)
|
||||
.sort((a, b) => b.score - a.score || a.key.localeCompare(b.key));
|
||||
const best = ranked[0];
|
||||
if (!best) return null;
|
||||
|
||||
const localPath = join(libraryDir, best.entry.file);
|
||||
return {
|
||||
localPath,
|
||||
ext: extensionForBundledSfxFile(best.entry.file),
|
||||
source: "bundled",
|
||||
metadata: {
|
||||
description: best.entry.description || best.key,
|
||||
duration: best.entry.duration ?? null,
|
||||
provider: "bundled.sfx",
|
||||
provenance: { library_key: best.key },
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,85 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
BUNDLED_SFX_RECOVERY_COMMAND,
|
||||
BundledSfxAssetsError,
|
||||
bundledSfxProvider,
|
||||
extensionForBundledSfxFile,
|
||||
inspectBundledSfxAssets,
|
||||
} from "./bundled-sfx-provider.mjs";
|
||||
|
||||
test("derives bundled SFX extension from the manifest filename", () => {
|
||||
assert.equal(extensionForBundledSfxFile("impact.wav"), ".wav");
|
||||
assert.equal(extensionForBundledSfxFile("whoosh.ogg"), ".ogg");
|
||||
assert.equal(extensionForBundledSfxFile("extensionless"), ".mp3");
|
||||
});
|
||||
|
||||
test("reports an agent-friendly recovery when the bundled SFX manifest is absent", () => {
|
||||
const libraryDir = mkdtempSync(join(tmpdir(), "media-use-sfx-missing-"));
|
||||
try {
|
||||
const health = inspectBundledSfxAssets(libraryDir);
|
||||
assert.equal(health.ok, false);
|
||||
assert.equal(health.code, "bundled_sfx_assets_missing");
|
||||
assert.match(health.detail, /manifest\.json/);
|
||||
assert.match(health.fix, /hyperframes skills update media-use/);
|
||||
assert.equal(health.fix, BUNDLED_SFX_RECOVERY_COMMAND);
|
||||
} finally {
|
||||
rmSync(libraryDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("reports the exact missing file from an incomplete bundled SFX install", () => {
|
||||
const libraryDir = mkdtempSync(join(tmpdir(), "media-use-sfx-incomplete-"));
|
||||
try {
|
||||
writeFileSync(
|
||||
join(libraryDir, "manifest.json"),
|
||||
JSON.stringify({ whoosh: { file: "whoosh.mp3", description: "transition" } }),
|
||||
);
|
||||
const health = inspectBundledSfxAssets(libraryDir);
|
||||
assert.equal(health.ok, false);
|
||||
assert.equal(health.code, "bundled_sfx_assets_missing");
|
||||
assert.match(health.detail, /whoosh\.mp3/);
|
||||
} finally {
|
||||
rmSync(libraryDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("bundled provider raises a typed install error instead of a generic catalog miss", async () => {
|
||||
const libraryDir = mkdtempSync(join(tmpdir(), "media-use-sfx-provider-"));
|
||||
try {
|
||||
await assert.rejects(
|
||||
() => bundledSfxProvider.search("whoosh", { libraryDir }),
|
||||
(error) => {
|
||||
assert.ok(error instanceof BundledSfxAssetsError);
|
||||
assert.equal(error.code, "bundled_sfx_assets_missing");
|
||||
assert.match(error.message, /hyperframes skills update media-use/);
|
||||
return true;
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
rmSync(libraryDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("accepts a complete bundled SFX library", () => {
|
||||
const libraryDir = mkdtempSync(join(tmpdir(), "media-use-sfx-complete-"));
|
||||
try {
|
||||
mkdirSync(libraryDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(libraryDir, "manifest.json"),
|
||||
JSON.stringify({ whoosh: { file: "whoosh.mp3", description: "transition" } }),
|
||||
);
|
||||
writeFileSync(join(libraryDir, "whoosh.mp3"), "audio");
|
||||
assert.deepEqual(inspectBundledSfxAssets(libraryDir), {
|
||||
ok: true,
|
||||
count: 1,
|
||||
detail: "1 bundled SFX asset available",
|
||||
fix: "",
|
||||
});
|
||||
} finally {
|
||||
rmSync(libraryDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,150 @@
|
||||
import { readFileSync, writeFileSync, mkdirSync, existsSync, copyFileSync } from "node:fs";
|
||||
import { join, basename } from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import { homedir } from "node:os";
|
||||
import { readManifest, appendRecord, normalizePrompt } from "./manifest.mjs";
|
||||
|
||||
const SCHEMA_PREFIX = "mu-v1-";
|
||||
const KEY_HEX_CHARS = 16;
|
||||
const COMPLETE_SENTINEL = ".hf-complete";
|
||||
|
||||
export function globalMediaDir() {
|
||||
return join(homedir(), ".media");
|
||||
}
|
||||
|
||||
export function contentHash(filePath) {
|
||||
const bytes = readFileSync(filePath);
|
||||
return createHash("sha256").update(bytes).digest("hex");
|
||||
}
|
||||
|
||||
function cacheEntryDir(rootDir, sha) {
|
||||
return join(rootDir, SCHEMA_PREFIX + sha.slice(0, KEY_HEX_CHARS));
|
||||
}
|
||||
|
||||
function isComplete(entryDir) {
|
||||
return existsSync(join(entryDir, COMPLETE_SENTINEL));
|
||||
}
|
||||
|
||||
function markComplete(entryDir) {
|
||||
writeFileSync(join(entryDir, COMPLETE_SENTINEL), "", "utf8");
|
||||
}
|
||||
|
||||
// The manifest helpers append their own ".media" to the dir they get, so the
|
||||
// global manifest must be addressed by HOME, not by globalMediaDir() — passing
|
||||
// the latter nested it at ~/.media/.media/manifest.jsonl, invisible to the
|
||||
// Studio /api/assets/global route (which reads the documented flat path).
|
||||
export function readGlobalManifest() {
|
||||
return readManifest(homedir());
|
||||
}
|
||||
|
||||
// Resolve a content-sha (full or unambiguous prefix) to a reusable global-cache
|
||||
// record, for `resolve --reuse <sha>`. Returns null on no match, or
|
||||
// { ambiguous: true, count } when a prefix matches multiple distinct entries.
|
||||
// Completeness (the .hf-complete sentinel) is left to importFromCache so the
|
||||
// caller can surface an "incomplete cache entry" error distinctly from a miss.
|
||||
export function findGlobalBySha(shaPrefix) {
|
||||
const p = String(shaPrefix || "")
|
||||
.toLowerCase()
|
||||
.trim();
|
||||
if (!p) return null;
|
||||
const matches = readGlobalManifest().filter(
|
||||
(r) => r.reusable && typeof r.sha === "string" && r.sha.startsWith(p),
|
||||
);
|
||||
if (matches.length === 0) return null;
|
||||
if (matches.length > 1) {
|
||||
const exact = matches.find((r) => r.sha === p);
|
||||
if (exact) return exact;
|
||||
return { ambiguous: true, count: matches.length };
|
||||
}
|
||||
return matches[0];
|
||||
}
|
||||
|
||||
function validateCacheHit(match) {
|
||||
if (!match?.sha) return null;
|
||||
return isComplete(cacheEntryDir(globalMediaDir(), match.sha)) ? match : null;
|
||||
}
|
||||
|
||||
export function cacheGet(prompt, type) {
|
||||
const key = normalizePrompt(prompt);
|
||||
if (!key) return null;
|
||||
return validateCacheHit(
|
||||
readGlobalManifest().find(
|
||||
(r) =>
|
||||
r.reusable &&
|
||||
normalizePrompt(r.provenance?.prompt) === key &&
|
||||
(type == null || r.type === type),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export function cacheGetByEntity(entity) {
|
||||
const lower = entity.toLowerCase();
|
||||
return validateCacheHit(
|
||||
readGlobalManifest().find((r) => r.reusable && r.entity && r.entity.toLowerCase() === lower),
|
||||
);
|
||||
}
|
||||
|
||||
export function cachePut(filePath, record) {
|
||||
const sha = contentHash(filePath);
|
||||
// Idempotent: same content already promoted -> don't duplicate the global
|
||||
// record. ponytail: skips usage_count bump; add it when the metric is needed.
|
||||
const existing = readGlobalManifest().find((r) => r.sha === sha);
|
||||
if (existing) return { sha, cached_path: existing.cached_path, deduped: true };
|
||||
|
||||
const dir = globalMediaDir();
|
||||
const entryDir = cacheEntryDir(dir, sha);
|
||||
mkdirSync(entryDir, { recursive: true });
|
||||
|
||||
const dest = join(entryDir, basename(filePath));
|
||||
copyFileSync(filePath, dest);
|
||||
markComplete(entryDir);
|
||||
|
||||
const globalRecord = {
|
||||
...record,
|
||||
sha,
|
||||
reusable: true,
|
||||
cached_path: dest,
|
||||
};
|
||||
appendRecord(homedir(), globalRecord);
|
||||
return { sha, cached_path: dest };
|
||||
}
|
||||
|
||||
export function importFromCache(cacheRecord, projectDir, localId, localPath) {
|
||||
const sha = cacheRecord.sha;
|
||||
const entryDir = cacheEntryDir(globalMediaDir(), sha);
|
||||
if (!isComplete(entryDir)) return null;
|
||||
|
||||
const cachedFile = cacheRecord.cached_path;
|
||||
if (!cachedFile || !existsSync(cachedFile)) return null;
|
||||
|
||||
mkdirSync(join(projectDir, ".media"), { recursive: true });
|
||||
const fullDest = join(projectDir, localPath);
|
||||
mkdirSync(join(fullDest, ".."), { recursive: true });
|
||||
copyFileSync(cachedFile, fullDest);
|
||||
|
||||
const projectRecord = {
|
||||
...cacheRecord,
|
||||
id: localId,
|
||||
path: localPath,
|
||||
provenance: {
|
||||
...cacheRecord.provenance,
|
||||
imported_from: sha,
|
||||
},
|
||||
};
|
||||
delete projectRecord.sha;
|
||||
delete projectRecord.reusable;
|
||||
delete projectRecord.cached_path;
|
||||
|
||||
return projectRecord;
|
||||
}
|
||||
|
||||
export function promote(projectDir, id) {
|
||||
const records = readManifest(projectDir);
|
||||
const record = records.find((r) => r.id === id);
|
||||
if (!record) throw new Error(`asset not found in project manifest: ${id}`);
|
||||
|
||||
const filePath = join(projectDir, record.path);
|
||||
if (!existsSync(filePath)) throw new Error(`asset file not found: ${filePath}`);
|
||||
|
||||
return cachePut(filePath, record);
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
// Reuse candidates: a side-effect-free view of assets already available to this
|
||||
// project (its own manifest) and across every project (the global ~/.media
|
||||
// cache), so the calling agent can judge semantic fit itself. No download, no
|
||||
// provider, no mutation. The ranker only *surfaces* — it orders by lexical
|
||||
// overlap but never filters a candidate out on zero overlap (that would
|
||||
// pre-empt the agent's judgment); the agent does the semantic call.
|
||||
|
||||
import { readManifest } from "./manifest.mjs";
|
||||
import { readGlobalManifest } from "./cache.mjs";
|
||||
import { tokenOverlap, typesMatch } from "./match.mjs";
|
||||
|
||||
export const CANDIDATE_CAP = 8;
|
||||
|
||||
function shape(record, scope, intent) {
|
||||
const description = record.description || record.provenance?.prompt || "";
|
||||
const prompt = record.provenance?.prompt || null;
|
||||
return {
|
||||
id: record.id,
|
||||
type: record.type,
|
||||
scope,
|
||||
description,
|
||||
prompt,
|
||||
provider: record.provenance?.provider || null,
|
||||
duration: record.duration ?? null,
|
||||
width: record.width ?? null,
|
||||
height: record.height ?? null,
|
||||
// Only global records carry a content sha — it is the stable reuse handle
|
||||
// for `resolve --reuse <sha>`. Project assets are reused by referencing
|
||||
// their path directly, so they need no handle.
|
||||
sha: scope === "global" ? record.sha || null : null,
|
||||
path: scope === "project" ? record.path : null,
|
||||
score: intent ? tokenOverlap(intent, `${description} ${prompt || ""}`) : 0,
|
||||
};
|
||||
}
|
||||
|
||||
// Rank one scope: type-matched (icon<->image aware), newest-first within equal
|
||||
// overlap, ordered by overlap desc. Returns the full ranked list (uncapped).
|
||||
function rankScope(records, scope, type, intent) {
|
||||
return records
|
||||
.filter((r) => typesMatch(r.type, type))
|
||||
.reverse() // manifest is append-order (oldest first); newest-first at equal score
|
||||
.map((r) => shape(r, scope, intent))
|
||||
.sort((a, b) => b.score - a.score); // Array.sort is stable → recency preserved
|
||||
}
|
||||
|
||||
// List reuse candidates for `type`, capped per scope. Returns:
|
||||
// candidates: capped project candidates followed by capped global candidates
|
||||
// truncated: true if either scope had more than `cap`
|
||||
// total: { project, global } counts before the cap (machine-readable)
|
||||
// similar: count of candidates with lexical overlap > 0 (drives the nudge)
|
||||
export function listCandidates({ projectDir, type, intent = "", cap = CANDIDATE_CAP }) {
|
||||
const project = rankScope(readManifest(projectDir), "project", type, intent);
|
||||
const global = rankScope(readGlobalManifest(), "global", type, intent);
|
||||
const candidates = [...project.slice(0, cap), ...global.slice(0, cap)];
|
||||
return {
|
||||
candidates,
|
||||
truncated: project.length > cap || global.length > cap,
|
||||
total: { project: project.length, global: global.length },
|
||||
similar: [...project, ...global].filter((c) => c.score > 0).length,
|
||||
};
|
||||
}
|
||||
|
||||
function meta(c) {
|
||||
const parts = [];
|
||||
if (c.duration != null) parts.push(`${c.duration}s`);
|
||||
if (c.width && c.height) parts.push(`${c.width}x${c.height}`);
|
||||
if (c.provider) parts.push(c.provider);
|
||||
return parts.join(", ");
|
||||
}
|
||||
|
||||
// Human-readable listing. The agent can read this directly; --json is for
|
||||
// programmatic use. Reuse handle differs by scope: path for project, sha for
|
||||
// global.
|
||||
export function formatCandidates(candidates, { truncated, total } = {}) {
|
||||
if (candidates.length === 0) return "no reuse candidates found (project or global cache)";
|
||||
const lines = [`${candidates.length} reuse candidate${candidates.length === 1 ? "" : "s"}:`, ""];
|
||||
for (const c of candidates) {
|
||||
const handle =
|
||||
c.scope === "global" ? `--reuse ${String(c.sha).slice(0, 16)}` : c.path || `manifest:${c.id}`;
|
||||
const m = meta(c);
|
||||
lines.push(` [${c.scope}] ${c.description}${m ? ` (${m})` : ""}`);
|
||||
lines.push(` ${handle}`);
|
||||
}
|
||||
if (truncated && total) {
|
||||
lines.push("");
|
||||
lines.push(
|
||||
` (showing top ${CANDIDATE_CAP} per scope; ${total.project} project / ${total.global} global total — refine --intent to narrow)`,
|
||||
);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
import { test } from "node:test";
|
||||
import { strict as assert } from "node:assert";
|
||||
import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { listCandidates, formatCandidates, CANDIDATE_CAP } from "./candidates.mjs";
|
||||
import { findGlobalBySha } from "./cache.mjs";
|
||||
|
||||
// candidates + findGlobalBySha are offline (no heygen), so we can override HOME
|
||||
// to a temp dir and seed a fake global ~/.media manifest deterministically.
|
||||
function sandbox() {
|
||||
const root = mkdtempSync(join(tmpdir(), "mu-cand-"));
|
||||
const project = join(root, "proj");
|
||||
const home = join(root, "home");
|
||||
process.env.HOME = home;
|
||||
return { root, project, home };
|
||||
}
|
||||
function seedManifest(dir, records) {
|
||||
const md = join(dir, ".media");
|
||||
mkdirSync(md, { recursive: true });
|
||||
writeFileSync(md + "/manifest.jsonl", records.map((r) => JSON.stringify(r)).join("\n") + "\n");
|
||||
}
|
||||
function proj(id, type, description, prompt) {
|
||||
return { id, type, path: `.media/audio/bgm/${id}.wav`, description, provenance: { prompt } };
|
||||
}
|
||||
function glob(id, type, description, prompt, sha) {
|
||||
return {
|
||||
id,
|
||||
type,
|
||||
sha,
|
||||
reusable: true,
|
||||
cached_path: `/x/${sha}/${id}.wav`,
|
||||
description,
|
||||
provenance: { prompt, provider: "heygen.audio.sounds" },
|
||||
};
|
||||
}
|
||||
|
||||
test("ranks project + global by overlap, tags scope", () => {
|
||||
const { root, project, home } = sandbox();
|
||||
try {
|
||||
seedManifest(project, [proj("bgm_001", "bgm", "calm ambient piano", "calm ambient piano")]);
|
||||
seedManifest(home, [
|
||||
glob("bgm_009", "bgm", "energetic tech launch", "energetic tech launch", "a".repeat(64)),
|
||||
glob("bgm_010", "bgm", "sad corporate piano", "sad corporate piano", "b".repeat(64)),
|
||||
]);
|
||||
const { candidates } = listCandidates({
|
||||
projectDir: project,
|
||||
type: "bgm",
|
||||
intent: "tech launch",
|
||||
});
|
||||
assert.equal(candidates[0].scope, "project"); // project listed first
|
||||
const g = candidates.filter((c) => c.scope === "global");
|
||||
assert.equal(g[0].description, "energetic tech launch"); // higher overlap ranks first
|
||||
assert.ok(g[0].score >= g[1].score);
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("zero-overlap intent still lists candidates (no hard filter)", () => {
|
||||
const { root, project, home } = sandbox();
|
||||
try {
|
||||
seedManifest(project, []);
|
||||
seedManifest(home, [glob("bgm_009", "bgm", "driving synth", "driving synth", "c".repeat(64))]);
|
||||
const { candidates, similar } = listCandidates({
|
||||
projectDir: project,
|
||||
type: "bgm",
|
||||
intent: "totally unrelated words xyz",
|
||||
});
|
||||
assert.equal(candidates.length, 1, "listed despite zero overlap");
|
||||
assert.equal(candidates[0].score, 0);
|
||||
assert.equal(similar, 0, "similar counts only overlap>0");
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("caps per scope and reports truncation + totals", () => {
|
||||
const { root, project, home } = sandbox();
|
||||
try {
|
||||
const many = Array.from({ length: CANDIDATE_CAP + 3 }, (_, i) =>
|
||||
glob(`bgm_${i}`, "bgm", `track ${i}`, `track ${i}`, String(i).padStart(64, "0")),
|
||||
);
|
||||
seedManifest(project, []);
|
||||
seedManifest(home, many);
|
||||
const { candidates, truncated, total } = listCandidates({ projectDir: project, type: "bgm" });
|
||||
assert.equal(candidates.length, CANDIDATE_CAP);
|
||||
assert.equal(truncated, true);
|
||||
assert.equal(total.global, CANDIDATE_CAP + 3);
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("honors icon<->image adjacency", () => {
|
||||
const { root, project, home } = sandbox();
|
||||
try {
|
||||
seedManifest(project, []);
|
||||
seedManifest(home, [glob("image_1", "image", "rocket logo", "rocket logo", "d".repeat(64))]);
|
||||
const { candidates } = listCandidates({ projectDir: project, type: "icon", intent: "rocket" });
|
||||
assert.equal(candidates.length, 1, "image asset surfaces for icon request");
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("sha only on global, path only on project", () => {
|
||||
const { root, project, home } = sandbox();
|
||||
try {
|
||||
seedManifest(project, [proj("bgm_001", "bgm", "x", "x")]);
|
||||
seedManifest(home, [glob("bgm_009", "bgm", "y", "y", "e".repeat(64))]);
|
||||
const { candidates } = listCandidates({ projectDir: project, type: "bgm" });
|
||||
const p = candidates.find((c) => c.scope === "project");
|
||||
const g = candidates.find((c) => c.scope === "global");
|
||||
assert.ok(p.path && !p.sha);
|
||||
assert.ok(g.sha && !g.path);
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("findGlobalBySha resolves unique prefix, flags ambiguity, misses cleanly", () => {
|
||||
const { root, project, home } = sandbox();
|
||||
try {
|
||||
seedManifest(project, []);
|
||||
seedManifest(home, [
|
||||
glob("bgm_1", "bgm", "a", "a", "abc" + "0".repeat(61)),
|
||||
glob("bgm_2", "bgm", "b", "b", "abd" + "0".repeat(61)),
|
||||
glob("bgm_3", "bgm", "c", "c", "fff" + "0".repeat(61)),
|
||||
]);
|
||||
assert.equal(findGlobalBySha("fff").id, "bgm_3", "unique prefix resolves");
|
||||
assert.deepEqual(
|
||||
{ ambiguous: findGlobalBySha("ab").ambiguous, count: findGlobalBySha("ab").count },
|
||||
{ ambiguous: true, count: 2 },
|
||||
"ambiguous prefix flagged",
|
||||
);
|
||||
assert.equal(findGlobalBySha("zzz"), null, "miss returns null");
|
||||
assert.equal(findGlobalBySha(""), null, "empty returns null");
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test("formatCandidates shows reuse handles by scope; empty message", () => {
|
||||
const { candidates } = {
|
||||
candidates: [
|
||||
{ scope: "project", description: "p", path: ".media/audio/bgm/bgm_001.wav" },
|
||||
{ scope: "global", description: "g", sha: "f".repeat(64) },
|
||||
],
|
||||
};
|
||||
const out = formatCandidates(candidates, {});
|
||||
assert.match(out, /\.media\/audio\/bgm\/bgm_001\.wav/);
|
||||
assert.match(out, /--reuse ffffffffffffffff/);
|
||||
assert.match(formatCandidates([], {}), /no reuse candidates/);
|
||||
});
|
||||
@@ -0,0 +1,161 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { copyFileSync, existsSync, readdirSync, statSync, unlinkSync } from "node:fs";
|
||||
import { homedir, tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
// Image generation via the OpenAI Codex CLI's built-in image tool (gpt-image-2)
|
||||
// on the user's ChatGPT subscription: the codex CLI owns auth, media-use holds
|
||||
// no key (CLI-only). The image UPSELL behind local mflux; skipped by --local-only.
|
||||
//
|
||||
// Retrieval mirrors illo-skill rather than trusting the model to save a file:
|
||||
// `--enable image_generation` makes the built-in tool drop the rendered artifact
|
||||
// into $CODEX_HOME/generated_images/, and we fetch the freshest file that
|
||||
// postdates this run. The save-to-path instruction is only a best-effort
|
||||
// verify-first.
|
||||
//
|
||||
// The flag was called `imagegenext` until Codex CLI ~0.145, which renamed it to
|
||||
// `image_generation` and dropped the old row from `codex features list`. Both
|
||||
// the gate below and the --enable arg track the new name, with a fallback to
|
||||
// the old one so older CLIs keep working.
|
||||
|
||||
const TIMEOUT_MS = 600000; // codex exec round-trips the sub; first-run tool spin-up is slow
|
||||
const MTIME_SKEW_MS = 2000; // tolerate mtime granularity / clock skew (illo uses 2s)
|
||||
|
||||
function codexGeneratedDir() {
|
||||
// Codex relocates CODEX_HOME on some hosts, so resolve it at run time.
|
||||
return join(process.env.CODEX_HOME || join(homedir(), ".codex"), "generated_images");
|
||||
}
|
||||
|
||||
// Newest artifact that postdates `sinceMs` (minus skew), so a stale prior render
|
||||
// or a concurrent session's file can't be mistaken for this run's output.
|
||||
function freshestGeneratedImage(sinceMs) {
|
||||
const dir = codexGeneratedDir();
|
||||
if (!existsSync(dir)) return null;
|
||||
const floor = sinceMs - MTIME_SKEW_MS;
|
||||
let best = null;
|
||||
for (const name of readdirSync(dir)) {
|
||||
let st;
|
||||
try {
|
||||
st = statSync(join(dir, name));
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (!st.isFile() || st.mtimeMs < floor) continue;
|
||||
if (!best || st.mtimeMs > best.mtimeMs) best = { path: join(dir, name), mtimeMs: st.mtimeMs };
|
||||
}
|
||||
return best?.path ?? null;
|
||||
}
|
||||
|
||||
// Short `codex` subcommand → combined stdout+stderr, or null if it can't run.
|
||||
function codexRun(args) {
|
||||
try {
|
||||
return execFileSync("codex", args, {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
timeout: 10000,
|
||||
});
|
||||
} catch (err) {
|
||||
return `${err.stdout?.toString() ?? ""}${err.stderr?.toString() ?? ""}` || null;
|
||||
}
|
||||
}
|
||||
|
||||
// Name of the feature flag this CLI uses for the extended image tool, or null if
|
||||
// it exposes neither. Newer CLIs list only `image_generation`; older ones list
|
||||
// `imagegenext` alongside it and reject the new name on --enable.
|
||||
export function imageGenFlag(feats) {
|
||||
if (/\bimagegenext\b/.test(feats)) return "imagegenext";
|
||||
if (/\bimage_generation\b/.test(feats)) return "image_generation";
|
||||
return null;
|
||||
}
|
||||
|
||||
// Fail-fast host check (mirrors illo): don't burn a minutes-long exec when Codex
|
||||
// isn't usable. Returns { flag } when ready, else { reason }. The flag ships
|
||||
// default-disabled, so we check the ROW is present (the capability signal) — the
|
||||
// exec enables it per-render with --enable.
|
||||
function codexUnavailableReason() {
|
||||
try {
|
||||
const which = process.platform === "win32" ? "where" : "which";
|
||||
execFileSync(which, ["codex"], { stdio: ["ignore", "ignore", "ignore"], timeout: 5000 });
|
||||
} catch {
|
||||
// A shell alias (e.g. `codex → /Applications/Codex.app/...`) is NOT enough:
|
||||
// aliases live only in the interactive shell, so a spawned subprocess's PATH
|
||||
// lookup can't see them. Symlink the real binary onto PATH.
|
||||
return {
|
||||
reason:
|
||||
'codex CLI not reachable on PATH (a shell alias won\'t work — spawned processes can\'t see aliases; symlink the real binary onto PATH, e.g. ln -s "$(readlink -f "$(command -v codex)")" ~/.local/bin/codex)',
|
||||
};
|
||||
}
|
||||
// Auth marker: presence of the credentials file, NOT `codex login status`.
|
||||
// That command prints "Logged in using ChatGPT" only to a human stream
|
||||
// (stderr / TTY) and exits 0, so its piped stdout — how media-use spawns it —
|
||||
// is empty, and the gate falsely reported "not logged in", blocking codex
|
||||
// image gen in every headless / CI / agent run even when fully authed.
|
||||
// auth.json is the durable, TTY-independent signal; token validity is proven
|
||||
// by the exec itself, which fails cleanly if the login is stale.
|
||||
const authPath = join(process.env.CODEX_HOME || join(homedir(), ".codex"), "auth.json");
|
||||
if (!existsSync(authPath)) return { reason: "codex not logged in (run: codex login)" };
|
||||
const feats = codexRun(["features", "list"]);
|
||||
if (feats == null) return { reason: "could not read `codex features list`" };
|
||||
const flag = imageGenFlag(feats);
|
||||
if (!flag) return { reason: "codex image generation feature unavailable (upgrade Codex CLI)" };
|
||||
return { flag };
|
||||
}
|
||||
|
||||
export async function codexImageGenerate(intent) {
|
||||
const { reason, flag } = codexUnavailableReason();
|
||||
if (reason) {
|
||||
console.error(`media-use: codex image upsell unavailable: ${reason}`);
|
||||
return null;
|
||||
}
|
||||
const outPath = join(tmpdir(), `media-use-codex-${process.pid}-${Date.now()}.png`);
|
||||
const prompt =
|
||||
`${intent}\n\n` +
|
||||
`Use your built-in image generation tool to render this, then save the image ` +
|
||||
`to ${outPath} (overwrite if it exists). Do not ask for confirmation. ` +
|
||||
`If you have no built-in image tool, do nothing (no PIL/matplotlib/SVG substitute).`;
|
||||
try {
|
||||
unlinkSync(outPath); // clear any prior file so verify-first can't accept a stale render
|
||||
} catch {
|
||||
/* no prior file */
|
||||
}
|
||||
const started = Date.now();
|
||||
try {
|
||||
execFileSync(
|
||||
"codex",
|
||||
[
|
||||
"exec",
|
||||
"--cd",
|
||||
tmpdir(),
|
||||
"-s",
|
||||
"workspace-write",
|
||||
"--skip-git-repo-check",
|
||||
"--enable",
|
||||
flag,
|
||||
"-",
|
||||
],
|
||||
{ input: prompt, encoding: "utf8", timeout: TIMEOUT_MS, stdio: ["pipe", "pipe", "pipe"] },
|
||||
);
|
||||
} catch (err) {
|
||||
console.error(
|
||||
`media-use: \`codex exec\` image generation failed: ${err.stderr?.toString().trim().slice(-200) || err.message}`,
|
||||
);
|
||||
return null;
|
||||
}
|
||||
// Verify-first (save-to-path may have worked), else fetch the image-tool artifact.
|
||||
const produced =
|
||||
existsSync(outPath) && statSync(outPath).size > 0 ? outPath : freshestGeneratedImage(started);
|
||||
if (!produced) return null;
|
||||
if (produced !== outPath) {
|
||||
try {
|
||||
copyFileSync(produced, outPath);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return {
|
||||
localPath: outPath,
|
||||
ext: ".png",
|
||||
source: "generated",
|
||||
metadata: { description: intent, provider: "codex.image_gen", provenance: { prompt: intent } },
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert";
|
||||
import { imageGenFlag } from "./codex-provider.mjs";
|
||||
|
||||
// Real `codex features list` output is three space-padded columns:
|
||||
// <name> <stage> <enabled>
|
||||
const OLD_CLI = [
|
||||
"image_detail_original removed false",
|
||||
"image_generation stable true",
|
||||
"imagegenext under development false",
|
||||
].join("\n");
|
||||
|
||||
const NEW_CLI = [
|
||||
"image_detail_original removed false",
|
||||
"image_generation stable true",
|
||||
].join("\n");
|
||||
|
||||
test("older CLIs keep the old flag — they reject --enable image_generation", () => {
|
||||
assert.equal(imageGenFlag(OLD_CLI), "imagegenext");
|
||||
});
|
||||
|
||||
test("Codex CLI >=0.145 dropped the imagegenext row; fall back to its new name", () => {
|
||||
// The regression this guards: gating on `imagegenext` alone made the provider
|
||||
// report "unavailable" on every up-to-date CLI, without attempting a render.
|
||||
assert.equal(imageGenFlag(NEW_CLI), "image_generation");
|
||||
});
|
||||
|
||||
test("no image feature at all is unavailable, not a silent default", () => {
|
||||
assert.equal(imageGenFlag("apps stable true\nhooks stable true"), null);
|
||||
});
|
||||
|
||||
test("substring lookalikes do not count as the feature row", () => {
|
||||
assert.equal(imageGenFlag("image_generation_v2 stable true"), null);
|
||||
});
|
||||
@@ -0,0 +1,117 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { test } from "node:test";
|
||||
import { existsSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { listTypes, getProviders } from "./registry.mjs";
|
||||
import { CAPABILITIES, listModels } from "./local-models.mjs";
|
||||
|
||||
// Capstone: media-use must actually OWN each hyperframes media weakness. This
|
||||
// test enforces the weakness→owner matrix in references/meta.md so a claim can't rot — if
|
||||
// a capability's entrypoint disappears, this fails.
|
||||
|
||||
const SKILL = join(dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
|
||||
test("weakness: audio-only → media-use resolves image + icon", () => {
|
||||
for (const t of ["image", "icon"]) {
|
||||
assert.ok(getProviders(t).length > 0, `no provider for ${t}`);
|
||||
}
|
||||
});
|
||||
|
||||
test("weakness: no third-party brand logos → media-use resolves logo", () => {
|
||||
assert.ok(listTypes().includes("logo"), "logo type missing");
|
||||
assert.ok(getProviders("logo").length >= 4, "logo cascade incomplete");
|
||||
});
|
||||
|
||||
test("weakness: no voice/audio gen → media-use exposes voice + the audio engine", () => {
|
||||
assert.ok(listTypes().includes("voice"), "voice type missing");
|
||||
assert.ok(getProviders("voice").length > 0, "no enabled voice provider (Bin approved)");
|
||||
assert.ok(existsSync(join(SKILL, "audio", "scripts", "audio.mjs")), "audio engine missing");
|
||||
});
|
||||
|
||||
test("weakness: scattered audio engine → consolidated under media-use (hyperframes-media gone)", () => {
|
||||
assert.ok(existsSync(join(SKILL, "audio", "scripts", "lib", "tts.mjs")), "tts engine missing");
|
||||
assert.ok(
|
||||
existsSync(join(SKILL, "audio", "assets", "sfx", "manifest.json")),
|
||||
"bundled SFX missing",
|
||||
);
|
||||
});
|
||||
|
||||
test("weakness: no media-ops → ops guidance reference exists", () => {
|
||||
assert.ok(existsSync(join(SKILL, "references", "operations.md")), "operations.md missing");
|
||||
});
|
||||
|
||||
test("weakness: no transcript-driven cutting → cut compiler entrypoints exist", async () => {
|
||||
assert.ok(existsSync(join(SKILL, "scripts", "transcript-cut.mjs")), "transcript-cut missing");
|
||||
assert.ok(existsSync(join(SKILL, "scripts", "lib", "cutlist.mjs")), "cutlist lib missing");
|
||||
const cutlist = await import("./cutlist.mjs");
|
||||
assert.equal(typeof cutlist.compileCutList, "function");
|
||||
});
|
||||
|
||||
test("weakness: whisper.cpp is weak → better local ASR (Parakeet) entrypoint exists", async () => {
|
||||
assert.ok(existsSync(join(SKILL, "scripts", "transcribe.mjs")), "transcribe.mjs missing");
|
||||
const pw = await import("./parakeet-words.mjs");
|
||||
assert.equal(typeof pw.mergeTokensToWords, "function", "token->word merge missing");
|
||||
const lm = await import("./local-models.mjs");
|
||||
const asr = lm.listModels("asr");
|
||||
const parakeet = asr.find((m) => m.id === "parakeet-mlx");
|
||||
assert.ok(parakeet && parakeet.rank === 0, "Parakeet must be the rank-0 preferred ASR");
|
||||
});
|
||||
|
||||
test("weakness: no auto-duck/loudness → duck compiler and recipes exist", async () => {
|
||||
assert.ok(existsSync(join(SKILL, "scripts", "audio-duck.mjs")), "audio-duck missing");
|
||||
assert.ok(existsSync(join(SKILL, "scripts", "lib", "duck.mjs")), "duck lib missing");
|
||||
assert.ok(existsSync(join(SKILL, "references", "operations.md")), "operations.md missing");
|
||||
const duck = await import("./duck.mjs");
|
||||
assert.equal(typeof duck.speechSpans, "function");
|
||||
assert.equal(typeof duck.duckKeyframes, "function");
|
||||
});
|
||||
|
||||
test("weakness: no cross-project memory → global cache + ingest entrypoints exist", async () => {
|
||||
const cache = await import("./cache.mjs");
|
||||
assert.equal(typeof cache.cachePut, "function");
|
||||
assert.equal(typeof cache.promote, "function");
|
||||
assert.equal(typeof cache.globalMediaDir, "function");
|
||||
const freeze = await import("./freeze.mjs");
|
||||
assert.equal(typeof freeze.isDirectMediaUrl, "function", "ingest URL guard missing");
|
||||
});
|
||||
|
||||
// Wenbo (06-29): heygen free-usage is the default; local models are the opt-out
|
||||
// fallback ("if user no, then local"). We still assert the fallback table is
|
||||
// populated so the opt-out path stays real.
|
||||
test("weakness: weak local defaults → local models exist as the opt-out fallback (tts/asr/upscale)", () => {
|
||||
for (const cap of ["tts", "asr", "upscale"]) {
|
||||
assert.ok(CAPABILITIES.includes(cap), `capability ${cap} missing`);
|
||||
assert.ok(listModels(cap).length > 0, `no local models for ${cap}`);
|
||||
}
|
||||
});
|
||||
|
||||
test("weakness: no image generation → local mflux (RAM-graded) + codex upsell", async () => {
|
||||
const ps = getProviders("image");
|
||||
assert.ok(
|
||||
ps.some((p) => p.name === "mflux.local" && typeof p.generate === "function"),
|
||||
"local image gen missing",
|
||||
);
|
||||
assert.ok(
|
||||
ps.some((p) => p.name === "codex.image_gen" && typeof p.generate === "function"),
|
||||
"codex image upsell missing",
|
||||
);
|
||||
const lm = await import("./local-models.mjs");
|
||||
assert.ok(lm.CAPABILITIES.includes("imagegen"), "imagegen capability missing");
|
||||
assert.ok(lm.listModels("imagegen").length >= 3, "imagegen RAM ladder too small");
|
||||
assert.equal(typeof lm.describeModelLadder, "function", "agent-facing ladder missing");
|
||||
});
|
||||
|
||||
test("weakness: no video generation → local videogen ladder + heygen avatar upsell", async () => {
|
||||
const lm = await import("./local-models.mjs");
|
||||
assert.ok(lm.CAPABILITIES.includes("videogen"), "videogen capability missing");
|
||||
assert.ok(lm.listModels("videogen").length >= 2, "videogen ladder too small");
|
||||
const ops = existsSync(join(SKILL, "references", "operations.md"));
|
||||
assert.ok(ops, "operations.md (avatar-upsell recipe) missing");
|
||||
});
|
||||
|
||||
test("every resolve type has at least one enabled provider", () => {
|
||||
for (const t of listTypes()) {
|
||||
assert.ok(getProviders(t).length > 0, `type ${t} has no enabled provider`);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,181 @@
|
||||
const DEFAULT_SIZE = 33;
|
||||
const MAX_SIZE = 64;
|
||||
|
||||
function clamp(value, min, max) {
|
||||
if (!Number.isFinite(value)) return 0;
|
||||
return Math.min(max, Math.max(min, value));
|
||||
}
|
||||
|
||||
function clampUnit(value) {
|
||||
return clamp(value, 0, 1);
|
||||
}
|
||||
|
||||
function readParam(params, key, min, max) {
|
||||
return clamp(Number(params?.[key] ?? 0), min, max);
|
||||
}
|
||||
|
||||
function luma([r, g, b]) {
|
||||
// Rec.709 luma weightings (matches the color space the grading runtime uses).
|
||||
return r * 0.2126 + g * 0.7152 + b * 0.0722;
|
||||
}
|
||||
|
||||
function smoothstep(edge0, edge1, value) {
|
||||
const t = clampUnit((value - edge0) / (edge1 - edge0));
|
||||
return t * t * (3 - 2 * t);
|
||||
}
|
||||
|
||||
function applyLiftGain(color, params) {
|
||||
const y = luma(color);
|
||||
const blacks = readParam(params, "blacks", -1, 1);
|
||||
const shadows = readParam(params, "shadows", -1, 1);
|
||||
const highlights = readParam(params, "highlights", -1, 1);
|
||||
const whites = readParam(params, "whites", -1, 1);
|
||||
const shadowMask = 1 - smoothstep(0.18, 0.62, y);
|
||||
const highlightMask = smoothstep(0.38, 0.82, y);
|
||||
const offset =
|
||||
blacks * 0.08 + shadows * 0.12 * shadowMask + highlights * 0.12 * highlightMask + whites * 0.08;
|
||||
return color.map((channel) => clampUnit(channel + offset));
|
||||
}
|
||||
|
||||
function applyExposure(color, params) {
|
||||
const exposure = readParam(params, "exposure", -2, 2);
|
||||
const gain = 2 ** exposure;
|
||||
const lift = Math.max(0, exposure) * 0.015;
|
||||
return color.map((channel) => clampUnit(channel * gain + lift));
|
||||
}
|
||||
|
||||
function applyContrast(color, params) {
|
||||
const contrast = readParam(params, "contrast", -1, 1);
|
||||
if (contrast === 0) return color;
|
||||
const factor = 1 + contrast * 1.2;
|
||||
return color.map((channel) => clampUnit(0.5 + (channel - 0.5) * factor));
|
||||
}
|
||||
|
||||
function applyWhiteBalance(color, params) {
|
||||
const temperature = readParam(params, "temperature", -1, 1);
|
||||
const tint = readParam(params, "tint", -1, 1);
|
||||
const redScale = 1 + temperature * 0.28 + tint * 0.08;
|
||||
const greenScale = 1 - Math.abs(tint) * 0.1 - tint * 0.08;
|
||||
const blueScale = 1 - temperature * 0.28 + tint * 0.08;
|
||||
return [
|
||||
clampUnit(color[0] * redScale),
|
||||
clampUnit(color[1] * greenScale),
|
||||
clampUnit(color[2] * blueScale),
|
||||
];
|
||||
}
|
||||
|
||||
function applySplitTone(color, params) {
|
||||
const split = params?.splitTone;
|
||||
if (!split) return color;
|
||||
const intensity = clampUnit(Number(split.intensity ?? 0));
|
||||
if (intensity === 0) return color;
|
||||
const balance = clampUnit(Number(split.balance ?? 0.5));
|
||||
const y = luma(color);
|
||||
const shadowMask = 1 - smoothstep(balance - 0.25, balance + 0.2, y);
|
||||
const highlightMask = smoothstep(balance - 0.2, balance + 0.25, y);
|
||||
const shadows = Array.isArray(split.shadows) ? split.shadows : [0, 0, 0];
|
||||
const highlights = Array.isArray(split.highlights) ? split.highlights : [0, 0, 0];
|
||||
return color.map((channel, i) =>
|
||||
clampUnit(
|
||||
channel +
|
||||
Number(shadows[i] ?? 0) * shadowMask * intensity +
|
||||
Number(highlights[i] ?? 0) * highlightMask * intensity,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
function applySaturation(color, params) {
|
||||
const saturation = readParam(params, "saturation", -1, 1);
|
||||
const vibrance = readParam(params, "vibrance", -1, 1);
|
||||
if (saturation === 0 && vibrance === 0) return color;
|
||||
const y = luma(color);
|
||||
const currentSat = Math.max(
|
||||
Math.abs(color[0] - y),
|
||||
Math.abs(color[1] - y),
|
||||
Math.abs(color[2] - y),
|
||||
);
|
||||
const vibranceWeight = 1 - clampUnit(currentSat * 2);
|
||||
const factor = clamp(1 + saturation + vibrance * vibranceWeight, 0, 2.5);
|
||||
return color.map((channel) => clampUnit(y + (channel - y) * factor));
|
||||
}
|
||||
|
||||
function applyParams(color, params) {
|
||||
let out = applyLiftGain(color, params);
|
||||
out = applyExposure(out, params);
|
||||
out = applyContrast(out, params);
|
||||
out = applyWhiteBalance(out, params);
|
||||
out = applySplitTone(out, params);
|
||||
out = applySaturation(out, params);
|
||||
return out;
|
||||
}
|
||||
|
||||
function formatNumber(value) {
|
||||
return clampUnit(value).toFixed(6);
|
||||
}
|
||||
|
||||
export function buildCube(params = {}, size = DEFAULT_SIZE) {
|
||||
if (!Number.isInteger(size) || size < 2 || size > MAX_SIZE) {
|
||||
throw new Error(`LUT size must be an integer from 2 to ${MAX_SIZE}`);
|
||||
}
|
||||
const lines = [
|
||||
`TITLE "media-use parametric grade"`,
|
||||
"DOMAIN_MIN 0 0 0",
|
||||
"DOMAIN_MAX 1 1 1",
|
||||
`LUT_3D_SIZE ${size}`,
|
||||
];
|
||||
const denom = size - 1;
|
||||
for (let b = 0; b < size; b++) {
|
||||
for (let g = 0; g < size; g++) {
|
||||
for (let r = 0; r < size; r++) {
|
||||
const out = applyParams([r / denom, g / denom, b / denom], params);
|
||||
lines.push(`${formatNumber(out[0])} ${formatNumber(out[1])} ${formatNumber(out[2])}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
|
||||
export function paramsFromIntent(intent) {
|
||||
const text = String(intent ?? "").toLowerCase();
|
||||
const params = {};
|
||||
let matched = false;
|
||||
|
||||
if (/\b(warm|golden|sunlit|sunny)\b/.test(text)) {
|
||||
params.temperature = 0.18;
|
||||
matched = true;
|
||||
} else if (/\b(cool|blue|icy|crisp)\b/.test(text)) {
|
||||
params.temperature = -0.16;
|
||||
matched = true;
|
||||
}
|
||||
|
||||
if (/\b(cinematic|film|movie)\b/.test(text)) {
|
||||
params.contrast = 0.08;
|
||||
params.saturation = 0.04;
|
||||
matched = true;
|
||||
}
|
||||
if (/\b(punchy|contrast|dramatic|bold)\b/.test(text)) {
|
||||
params.contrast = Math.max(params.contrast ?? 0, 0.22);
|
||||
matched = true;
|
||||
}
|
||||
if (/\b(bright|airy|lift)\b/.test(text)) {
|
||||
params.exposure = 0.16;
|
||||
params.shadows = 0.08;
|
||||
matched = true;
|
||||
}
|
||||
if (/\b(dark|moody|low-key)\b/.test(text)) {
|
||||
params.exposure = -0.12;
|
||||
params.contrast = Math.max(params.contrast ?? 0, 0.12);
|
||||
matched = true;
|
||||
}
|
||||
if (/\b(vibrant|saturated|colorful)\b/.test(text)) {
|
||||
params.saturation = Math.max(params.saturation ?? 0, 0.16);
|
||||
params.vibrance = 0.12;
|
||||
matched = true;
|
||||
}
|
||||
if (/\b(muted|desaturated|washed)\b/.test(text)) {
|
||||
params.saturation = Math.min(params.saturation ?? 0, -0.16);
|
||||
matched = true;
|
||||
}
|
||||
|
||||
return matched ? params : null;
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { test } from "node:test";
|
||||
import { buildCube, paramsFromIntent } from "./cube-build.mjs";
|
||||
import { validateCube } from "./cube-validate.mjs";
|
||||
|
||||
function rows(cube) {
|
||||
return cube
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => /^[+-]?(?:\d|\.\d)/.test(line))
|
||||
.map((line) => line.split(/\s+/).map(Number));
|
||||
}
|
||||
|
||||
function rowAt(cubeRows, size, r, g, b) {
|
||||
return cubeRows[(b * size + g) * size + r];
|
||||
}
|
||||
|
||||
function luma(row) {
|
||||
return row[0] * 0.2126 + row[1] * 0.7152 + row[2] * 0.0722;
|
||||
}
|
||||
|
||||
test("all-zero params produce a near-identity LUT", () => {
|
||||
const cube = buildCube({}, 3);
|
||||
assert.equal(validateCube(cube).ok, true);
|
||||
const parsed = rows(cube);
|
||||
for (let b = 0; b < 3; b++) {
|
||||
for (let g = 0; g < 3; g++) {
|
||||
for (let r = 0; r < 3; r++) {
|
||||
const row = rowAt(parsed, 3, r, g, b);
|
||||
assert.ok(Math.abs(row[0] - r / 2) < 0.000001);
|
||||
assert.ok(Math.abs(row[1] - g / 2) < 0.000001);
|
||||
assert.ok(Math.abs(row[2] - b / 2) < 0.000001);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test("positive exposure increases unclipped output luma", () => {
|
||||
const identity = rows(buildCube({}, 5));
|
||||
const exposed = rows(buildCube({ exposure: 0.3 }, 5));
|
||||
for (let i = 0; i < identity.length; i++) {
|
||||
const before = luma(identity[i]);
|
||||
if (before > 0.02 && before < 0.95) {
|
||||
assert.ok(luma(exposed[i]) > before, `row ${i} should brighten`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test("positive temperature warms mid-gray", () => {
|
||||
const parsed = rows(buildCube({ temperature: 0.2 }, 3));
|
||||
const mid = rowAt(parsed, 3, 1, 1, 1);
|
||||
assert.ok(mid[0] > 0.5, "red channel should rise");
|
||||
assert.ok(mid[2] < 0.5, "blue channel should fall");
|
||||
});
|
||||
|
||||
test("positive contrast darkens shadows and brightens highlights", () => {
|
||||
const parsed = rows(buildCube({ contrast: 0.3 }, 5));
|
||||
const shadow = rowAt(parsed, 5, 1, 1, 1);
|
||||
const highlight = rowAt(parsed, 5, 3, 3, 3);
|
||||
assert.ok(luma(shadow) < 0.25, "below-mid gray should darken");
|
||||
assert.ok(luma(highlight) > 0.75, "above-mid gray should brighten");
|
||||
});
|
||||
|
||||
test("outputs validate at the default size and are deterministic", () => {
|
||||
const params = { exposure: 0.15, contrast: 0.2, temperature: -0.1, saturation: 0.12 };
|
||||
const a = buildCube(params);
|
||||
const b = buildCube(params);
|
||||
assert.equal(a, b);
|
||||
assert.equal(validateCube(a).ok, true);
|
||||
assert.equal(validateCube(a).size, 33);
|
||||
});
|
||||
|
||||
test("paramsFromIntent declines zero-overlap prompts and maps technical words", () => {
|
||||
assert.equal(paramsFromIntent("zqxv imaginary neutron look"), null);
|
||||
assert.deepEqual(paramsFromIntent("warm cinematic"), {
|
||||
temperature: 0.18,
|
||||
contrast: 0.08,
|
||||
saturation: 0.04,
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Standalone mirror of packages/core/src/colorLuts.ts. media-use cannot import
|
||||
// the TypeScript source at runtime, so cube-validate.test.mjs mirrors core
|
||||
// parser cases to catch drift in accepted .cube files before freezing them.
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { resolve as resolvePath } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export const DEFAULT_MAX_CUBE_LUT_SIZE = 64;
|
||||
|
||||
const DEFAULT_DOMAIN_MIN = [0, 0, 0];
|
||||
const DEFAULT_DOMAIN_MAX = [1, 1, 1];
|
||||
|
||||
class CubeValidateError extends Error {
|
||||
constructor(message, lineNumber = null) {
|
||||
super(lineNumber == null ? message : `${message} at line ${lineNumber}`);
|
||||
this.name = "CubeValidateError";
|
||||
this.lineNumber = lineNumber;
|
||||
}
|
||||
}
|
||||
|
||||
function stripComment(line) {
|
||||
let inQuote = false;
|
||||
for (let i = 0; i < line.length; i++) {
|
||||
const char = line[i];
|
||||
if (char === '"') inQuote = !inQuote;
|
||||
if (char === "#" && !inQuote) return line.slice(0, i);
|
||||
}
|
||||
return line;
|
||||
}
|
||||
|
||||
function parseFiniteNumber(value, lineNumber) {
|
||||
const parsed = Number(value);
|
||||
if (!Number.isFinite(parsed)) {
|
||||
throw new CubeValidateError(`Invalid number "${value}"`, lineNumber);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function parseVec3(parts, keyword, lineNumber) {
|
||||
if (parts.length !== 3) {
|
||||
throw new CubeValidateError(`${keyword} expects three numbers`, lineNumber);
|
||||
}
|
||||
return [
|
||||
parseFiniteNumber(parts[0], lineNumber),
|
||||
parseFiniteNumber(parts[1], lineNumber),
|
||||
parseFiniteNumber(parts[2], lineNumber),
|
||||
];
|
||||
}
|
||||
|
||||
function parseSize(value, keyword, lineNumber) {
|
||||
if (!value) throw new CubeValidateError(`${keyword} expects a size`, lineNumber);
|
||||
const parsed = Number(value);
|
||||
if (!Number.isInteger(parsed) || parsed < 2) {
|
||||
throw new CubeValidateError(`${keyword} must be an integer greater than 1`, lineNumber);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function validateDomain(domainMin, domainMax) {
|
||||
if (
|
||||
domainMax[0] <= domainMin[0] ||
|
||||
domainMax[1] <= domainMin[1] ||
|
||||
domainMax[2] <= domainMin[2]
|
||||
) {
|
||||
throw new CubeValidateError("DOMAIN_MAX values must be greater than DOMAIN_MIN values");
|
||||
}
|
||||
}
|
||||
|
||||
function isNumericDataLine(token) {
|
||||
return /^[+-]?(?:\d|\.\d)/.test(token);
|
||||
}
|
||||
|
||||
function parseCube(input, options = {}) {
|
||||
const maxSize = options.maxSize ?? DEFAULT_MAX_CUBE_LUT_SIZE;
|
||||
let domainMin = DEFAULT_DOMAIN_MIN;
|
||||
let domainMax = DEFAULT_DOMAIN_MAX;
|
||||
let lut1dSize = null;
|
||||
let lut3dSize = null;
|
||||
let rows = 0;
|
||||
|
||||
const lines = String(input)
|
||||
.replace(/^\uFEFF/, "")
|
||||
.split(/\r?\n/);
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const lineNumber = i + 1;
|
||||
const line = stripComment(lines[i] ?? "").trim();
|
||||
if (!line) continue;
|
||||
const parts = line.split(/\s+/);
|
||||
const keyword = (parts[0] ?? "").toUpperCase();
|
||||
const rest = parts.slice(1);
|
||||
|
||||
if (keyword === "TITLE") continue;
|
||||
if (keyword === "DOMAIN_MIN") {
|
||||
domainMin = parseVec3(rest, keyword, lineNumber);
|
||||
continue;
|
||||
}
|
||||
if (keyword === "DOMAIN_MAX") {
|
||||
domainMax = parseVec3(rest, keyword, lineNumber);
|
||||
continue;
|
||||
}
|
||||
if (keyword === "LUT_3D_INPUT_RANGE") {
|
||||
if (rest.length !== 2) {
|
||||
throw new CubeValidateError(`${keyword} expects two numbers`, lineNumber);
|
||||
}
|
||||
const min = parseFiniteNumber(rest[0], lineNumber);
|
||||
const max = parseFiniteNumber(rest[1], lineNumber);
|
||||
if (max <= min) {
|
||||
throw new CubeValidateError("LUT_3D_INPUT_RANGE max must exceed min", lineNumber);
|
||||
}
|
||||
domainMin = [min, min, min];
|
||||
domainMax = [max, max, max];
|
||||
continue;
|
||||
}
|
||||
if (keyword === "LUT_1D_SIZE") {
|
||||
lut1dSize = parseSize(rest[0], keyword, lineNumber);
|
||||
continue;
|
||||
}
|
||||
if (keyword === "LUT_3D_SIZE") {
|
||||
lut3dSize = parseSize(rest[0], keyword, lineNumber);
|
||||
if (lut3dSize > maxSize) {
|
||||
throw new CubeValidateError(`LUT_3D_SIZE ${lut3dSize} exceeds max ${maxSize}`, lineNumber);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!isNumericDataLine(keyword)) {
|
||||
if (keyword.startsWith("LUT_")) {
|
||||
throw new CubeValidateError(`Unsupported cube keyword ${keyword}`, lineNumber);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!lut3dSize) {
|
||||
if (lut1dSize) {
|
||||
throw new CubeValidateError("1D cube LUTs are not supported yet", lineNumber);
|
||||
}
|
||||
throw new CubeValidateError("LUT data appears before LUT_3D_SIZE", lineNumber);
|
||||
}
|
||||
if (parts.length !== 3) {
|
||||
throw new CubeValidateError("LUT data rows must contain three numbers", lineNumber);
|
||||
}
|
||||
parseFiniteNumber(parts[0], lineNumber);
|
||||
parseFiniteNumber(parts[1], lineNumber);
|
||||
parseFiniteNumber(parts[2], lineNumber);
|
||||
rows++;
|
||||
}
|
||||
|
||||
if (lut1dSize && lut3dSize) {
|
||||
throw new CubeValidateError("Mixed 1D and 3D cube LUTs are not supported yet");
|
||||
}
|
||||
if (!lut3dSize) {
|
||||
if (lut1dSize) throw new CubeValidateError("1D cube LUTs are not supported yet");
|
||||
throw new CubeValidateError("Missing LUT_3D_SIZE");
|
||||
}
|
||||
validateDomain(domainMin, domainMax);
|
||||
|
||||
const expectedRows = lut3dSize * lut3dSize * lut3dSize;
|
||||
if (rows !== expectedRows) {
|
||||
throw new CubeValidateError(
|
||||
`Expected ${expectedRows} LUT rows for size ${lut3dSize}, found ${rows}`,
|
||||
);
|
||||
}
|
||||
|
||||
return { size: lut3dSize };
|
||||
}
|
||||
|
||||
export function validateCube(input, options = {}) {
|
||||
try {
|
||||
const parsed = parseCube(input, options);
|
||||
return { ok: true, size: parsed.size };
|
||||
} catch (err) {
|
||||
return { ok: false, error: err.message };
|
||||
}
|
||||
}
|
||||
|
||||
export function validateCubeFile(filePath, options = {}) {
|
||||
return validateCube(readFileSync(filePath, "utf8"), options);
|
||||
}
|
||||
|
||||
function main(argv) {
|
||||
const file = argv[2];
|
||||
if (!file) {
|
||||
console.error("usage: cube-validate.mjs <file.cube>");
|
||||
process.exit(2);
|
||||
}
|
||||
const result = validateCubeFile(file);
|
||||
if (!result.ok) {
|
||||
console.error(`error: ${result.error}`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(`ok: LUT_3D_SIZE ${result.size}`);
|
||||
}
|
||||
|
||||
if (process.argv[1] && resolvePath(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main(process.argv);
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { test } from "node:test";
|
||||
import { validateCube } from "./cube-validate.mjs";
|
||||
|
||||
const IDENTITY_2 = `
|
||||
# comment
|
||||
TITLE "Identity 2"
|
||||
DOMAIN_MIN 0 0 0
|
||||
DOMAIN_MAX 1 1 1
|
||||
LUT_3D_SIZE 2
|
||||
0 0 0
|
||||
1 0 0
|
||||
0 1 0
|
||||
1 1 0
|
||||
0 0 1
|
||||
1 0 1
|
||||
0 1 1
|
||||
1 1 1
|
||||
`;
|
||||
|
||||
test("accepts a valid minimal 3D cube LUT", () => {
|
||||
const result = validateCube(IDENTITY_2);
|
||||
assert.deepEqual(result, { ok: true, size: 2 });
|
||||
});
|
||||
|
||||
test("rejects oversize LUTs with the core parser message", () => {
|
||||
const result = validateCube("LUT_3D_SIZE 65", { maxSize: 64 });
|
||||
assert.equal(result.ok, false);
|
||||
assert.match(result.error, /LUT_3D_SIZE 65 exceeds max 64/);
|
||||
});
|
||||
|
||||
test("rejects data rows before LUT_3D_SIZE", () => {
|
||||
const result = validateCube("0 0 0\nLUT_3D_SIZE 2");
|
||||
assert.equal(result.ok, false);
|
||||
assert.match(result.error, /LUT data appears before LUT_3D_SIZE/);
|
||||
});
|
||||
|
||||
test("rejects missing LUT_3D_SIZE", () => {
|
||||
const result = validateCube('TITLE "No Size"');
|
||||
assert.equal(result.ok, false);
|
||||
assert.match(result.error, /Missing LUT_3D_SIZE/);
|
||||
});
|
||||
|
||||
test("rejects row count mismatches with the core parser message", () => {
|
||||
const result = validateCube("LUT_3D_SIZE 2\n0 0 0");
|
||||
assert.equal(result.ok, false);
|
||||
assert.match(result.error, /Expected 8 LUT rows/);
|
||||
});
|
||||
|
||||
test("rejects inverted domains", () => {
|
||||
const result = validateCube(`
|
||||
DOMAIN_MIN 0 0 0
|
||||
DOMAIN_MAX 1 0 1
|
||||
LUT_3D_SIZE 2
|
||||
0 0 0
|
||||
1 0 0
|
||||
0 1 0
|
||||
1 1 0
|
||||
0 0 1
|
||||
1 0 1
|
||||
0 1 1
|
||||
1 1 1
|
||||
`);
|
||||
assert.equal(result.ok, false);
|
||||
assert.match(result.error, /DOMAIN_MAX values must be greater than DOMAIN_MIN values/);
|
||||
});
|
||||
|
||||
test("rejects unsupported 1D and mixed cube LUTs", () => {
|
||||
const oneD = validateCube("LUT_1D_SIZE 2\n0 0 0\n1 1 1");
|
||||
assert.equal(oneD.ok, false);
|
||||
assert.match(oneD.error, /1D cube LUTs are not supported yet/);
|
||||
|
||||
const mixed = validateCube(`
|
||||
LUT_1D_SIZE 2
|
||||
LUT_3D_SIZE 2
|
||||
0 0 0
|
||||
1 0 0
|
||||
0 1 0
|
||||
1 1 0
|
||||
0 0 1
|
||||
1 0 1
|
||||
0 1 1
|
||||
1 1 1
|
||||
`);
|
||||
assert.equal(mixed.ok, false);
|
||||
assert.match(mixed.error, /Mixed 1D and 3D cube LUTs are not supported yet/);
|
||||
});
|
||||
|
||||
test("CLI exits zero for valid files and non-zero for invalid files", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-cube-validate-"));
|
||||
try {
|
||||
const valid = join(dir, "valid.cube");
|
||||
const invalid = join(dir, "invalid.cube");
|
||||
writeFileSync(valid, IDENTITY_2);
|
||||
writeFileSync(invalid, "LUT_3D_SIZE 65");
|
||||
|
||||
const out = execFileSync(
|
||||
process.execPath,
|
||||
[new URL("./cube-validate.mjs", import.meta.url).pathname, valid],
|
||||
{
|
||||
encoding: "utf8",
|
||||
},
|
||||
);
|
||||
assert.match(out, /ok: LUT_3D_SIZE 2/);
|
||||
|
||||
assert.throws(
|
||||
() =>
|
||||
execFileSync(
|
||||
process.execPath,
|
||||
[new URL("./cube-validate.mjs", import.meta.url).pathname, invalid],
|
||||
{
|
||||
encoding: "utf8",
|
||||
stdio: "pipe",
|
||||
},
|
||||
),
|
||||
(err) => err.status === 1 && String(err.stderr).includes("exceeds max 64"),
|
||||
);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,184 @@
|
||||
import { normalizeWords } from "./words.mjs";
|
||||
|
||||
const MIN_SEGMENT_SECONDS = 0.2;
|
||||
const SILENCE_PAD_SECONDS = 0.15;
|
||||
|
||||
export function compileCutList(transcript, opts = {}) {
|
||||
const words = normalizeWords(transcript);
|
||||
if (opts.keep != null && hasRemovalSource(opts)) {
|
||||
throw new Error("--keep is mutually exclusive with removal options");
|
||||
}
|
||||
|
||||
if (opts.keep != null) {
|
||||
const duration = durationFrom(words, opts);
|
||||
const ranges = parseTimeRanges(opts.keep);
|
||||
return finalizeKept(duration != null ? clampRanges(ranges, duration) : ranges);
|
||||
}
|
||||
|
||||
const duration = durationFrom(words, opts);
|
||||
if (!duration) return [];
|
||||
|
||||
const removals = [
|
||||
...parseTimeRanges(opts.remove),
|
||||
...wordIndexRanges(words, opts.removeWords),
|
||||
...fillerRanges(words, opts.removeFillers),
|
||||
...silenceRanges(words, opts.cutSilence),
|
||||
];
|
||||
const mergedRemovals = mergeRanges(clampRanges(removals, duration));
|
||||
return finalizeKept(invertRanges(mergedRemovals, duration));
|
||||
}
|
||||
|
||||
function hasRemovalSource(opts) {
|
||||
return (
|
||||
opts.remove != null ||
|
||||
opts.removeWords != null ||
|
||||
opts.removeFillers != null ||
|
||||
opts.cutSilence != null
|
||||
);
|
||||
}
|
||||
|
||||
function durationFrom(words, opts) {
|
||||
const explicit = Number(opts.duration ?? opts.totalDuration);
|
||||
if (Number.isFinite(explicit) && explicit > 0) return explicit;
|
||||
const last = words.at(-1);
|
||||
return last && Number.isFinite(last.end) && last.end > 0 ? last.end : null;
|
||||
}
|
||||
|
||||
function parseTimeRanges(value) {
|
||||
if (value == null || value === false || value === "") return [];
|
||||
if (typeof value === "string") {
|
||||
return value
|
||||
.split(",")
|
||||
.map((part) => part.trim())
|
||||
.filter(Boolean)
|
||||
.map(parseRangeString);
|
||||
}
|
||||
if (!Array.isArray(value)) throw new Error("range list must be a string or array");
|
||||
return value.map((range) => {
|
||||
if (Array.isArray(range)) return cleanRange(Number(range[0]), Number(range[1]));
|
||||
return cleanRange(Number(range?.start), Number(range?.end));
|
||||
});
|
||||
}
|
||||
|
||||
function parseRangeString(value) {
|
||||
const match = value.match(/^([0-9]*\.?[0-9]+)\s*-\s*([0-9]*\.?[0-9]+)$/);
|
||||
if (!match) throw new Error(`invalid range: ${value}`);
|
||||
return cleanRange(Number(match[1]), Number(match[2]));
|
||||
}
|
||||
|
||||
function cleanRange(start, end) {
|
||||
if (!Number.isFinite(start) || !Number.isFinite(end)) {
|
||||
throw new Error("range start/end must be finite numbers");
|
||||
}
|
||||
if (end < start) throw new Error(`range end ${end} is before start ${start}`);
|
||||
return { start, end };
|
||||
}
|
||||
|
||||
function wordIndexRanges(words, value) {
|
||||
if (value == null || value === false || value === "") return [];
|
||||
const ranges = typeof value === "string" ? value.split(",") : value;
|
||||
if (!Array.isArray(ranges)) throw new Error("--remove-words must be a string or array");
|
||||
return ranges
|
||||
.map((range) => (typeof range === "string" ? range.trim() : range))
|
||||
.filter(Boolean)
|
||||
.map((range) => {
|
||||
const [first, last = first] =
|
||||
typeof range === "string" ? range.split("-").map((n) => n.trim()) : range;
|
||||
const startIndex = Number(first);
|
||||
const endIndex = Number(last);
|
||||
if (!Number.isInteger(startIndex) || !Number.isInteger(endIndex)) {
|
||||
throw new Error(`invalid word range: ${range}`);
|
||||
}
|
||||
if (startIndex < 0 || endIndex < startIndex || endIndex >= words.length) {
|
||||
throw new Error(`word range out of bounds: ${range}`);
|
||||
}
|
||||
return { start: words[startIndex].start, end: words[endIndex].end };
|
||||
});
|
||||
}
|
||||
|
||||
function fillerRanges(words, value) {
|
||||
if (value == null || value === false || value === "") return [];
|
||||
const fillers = Array.isArray(value)
|
||||
? value
|
||||
: String(value)
|
||||
.split(",")
|
||||
.map((s) => s.trim());
|
||||
const set = new Set(fillers.filter(Boolean).map(bareToken));
|
||||
if (set.size === 0) return [];
|
||||
// Whisper emits words with attached punctuation and arbitrary case
|
||||
// ("UM," / "Um."), so compare bare tokens.
|
||||
return words
|
||||
.filter((word) => set.has(bareToken(word.text)))
|
||||
.map((word) => ({ start: word.start, end: word.end }));
|
||||
}
|
||||
|
||||
function bareToken(text) {
|
||||
return String(text)
|
||||
.toLowerCase()
|
||||
.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, "");
|
||||
}
|
||||
|
||||
function silenceRanges(words, value) {
|
||||
if (value == null || value === false || value === "") return [];
|
||||
const threshold = Number(value);
|
||||
if (!Number.isFinite(threshold) || threshold <= 0) {
|
||||
throw new Error("--cut-silence must be a positive number");
|
||||
}
|
||||
const ranges = [];
|
||||
for (let i = 0; i < words.length - 1; i++) {
|
||||
const current = words[i];
|
||||
const next = words[i + 1];
|
||||
const gap = next.start - current.end;
|
||||
if (gap <= threshold) continue;
|
||||
const start = current.end + SILENCE_PAD_SECONDS;
|
||||
const end = next.start - SILENCE_PAD_SECONDS;
|
||||
if (end > start) ranges.push({ start, end });
|
||||
}
|
||||
return ranges;
|
||||
}
|
||||
|
||||
function clampRanges(ranges, duration) {
|
||||
return ranges
|
||||
.map((range) => ({
|
||||
start: Math.max(0, Math.min(duration, range.start)),
|
||||
end: Math.max(0, Math.min(duration, range.end)),
|
||||
}))
|
||||
.filter((range) => range.end > range.start);
|
||||
}
|
||||
|
||||
function mergeRanges(ranges) {
|
||||
const sorted = ranges
|
||||
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
|
||||
.sort((a, b) => a.start - b.start || a.end - b.end);
|
||||
const merged = [];
|
||||
for (const range of sorted) {
|
||||
const prev = merged.at(-1);
|
||||
if (prev && range.start <= prev.end) {
|
||||
prev.end = Math.max(prev.end, range.end);
|
||||
} else {
|
||||
merged.push({ ...range });
|
||||
}
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
function invertRanges(removals, duration) {
|
||||
const kept = [];
|
||||
let cursor = 0;
|
||||
for (const range of removals) {
|
||||
if (range.start > cursor) kept.push({ start: cursor, end: range.start });
|
||||
cursor = Math.max(cursor, range.end);
|
||||
}
|
||||
if (cursor < duration) kept.push({ start: cursor, end: duration });
|
||||
return kept;
|
||||
}
|
||||
|
||||
function finalizeKept(ranges) {
|
||||
return mergeRanges(ranges)
|
||||
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
|
||||
.filter((range) => round3(range.end - range.start) >= MIN_SEGMENT_SECONDS);
|
||||
}
|
||||
|
||||
function round3(n) {
|
||||
return Math.round(Number(n) * 1000) / 1000;
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { test } from "node:test";
|
||||
import { compileCutList } from "./cutlist.mjs";
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const SCRIPT = join(HERE, "..", "transcript-cut.mjs");
|
||||
|
||||
test("explicit --remove ranges invert to kept segments", () => {
|
||||
const transcript = [
|
||||
word("w0", "alpha", 0, 1),
|
||||
word("w1", "beta", 1.2, 2),
|
||||
word("w2", "gamma", 2.2, 5),
|
||||
];
|
||||
|
||||
assert.deepEqual(compileCutList(transcript, { remove: "1-2.5" }), [
|
||||
{ start: 0, end: 1 },
|
||||
{ start: 2.5, end: 5 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("--remove-words resolves inclusive word-index ranges to time ranges", () => {
|
||||
const transcript = [
|
||||
word("w0", "zero", 0, 0.5),
|
||||
word("w1", "one", 0.6, 1),
|
||||
word("w2", "two", 1.1, 1.5),
|
||||
word("w3", "three", 2, 3),
|
||||
];
|
||||
|
||||
assert.deepEqual(compileCutList(transcript, { removeWords: "1-2" }), [
|
||||
{ start: 0, end: 0.6 },
|
||||
{ start: 1.5, end: 3 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("--remove-fillers drops case-insensitive matching words", () => {
|
||||
const transcript = [
|
||||
word("w0", "Hello", 0, 0.5),
|
||||
word("w1", "Um", 0.5, 0.7),
|
||||
word("w2", "world", 0.8, 1.2),
|
||||
word("w3", "LIKE", 1.3, 1.5),
|
||||
word("w4", "done", 1.6, 2),
|
||||
];
|
||||
|
||||
assert.deepEqual(compileCutList(transcript, { removeFillers: "um,like" }), [
|
||||
{ start: 0, end: 0.5 },
|
||||
{ start: 0.7, end: 1.3 },
|
||||
{ start: 1.5, end: 2 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("--cut-silence removes only the center of long inter-word gaps", () => {
|
||||
const transcript = [word("w0", "a", 0, 0.5), word("w1", "b", 2, 2.5), word("w2", "c", 2.7, 3)];
|
||||
|
||||
assert.deepEqual(compileCutList(transcript, { cutSilence: 0.8 }), [
|
||||
{ start: 0, end: 0.65 },
|
||||
{ start: 1.85, end: 3 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("overlapping removal sources merge before inversion", () => {
|
||||
const transcript = [
|
||||
word("w0", "start", 0, 0.5),
|
||||
word("w1", "um", 0.9, 1.1),
|
||||
word("w2", "middle", 2.5, 2.8),
|
||||
word("w3", "more", 3.1, 3.4),
|
||||
word("w4", "end", 5.5, 6),
|
||||
];
|
||||
|
||||
assert.deepEqual(
|
||||
compileCutList(transcript, {
|
||||
remove: "1-2.7",
|
||||
removeWords: "2-3",
|
||||
removeFillers: "um",
|
||||
}),
|
||||
[
|
||||
{ start: 0, end: 0.9 },
|
||||
{ start: 3.4, end: 6 },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test("kept slivers shorter than 0.2s are dropped", () => {
|
||||
const transcript = [word("w0", "start", 0, 0.5), word("w1", "end", 2.5, 3)];
|
||||
|
||||
assert.deepEqual(compileCutList(transcript, { remove: "0.1-2.95" }), []);
|
||||
});
|
||||
|
||||
test("--keep is inverse mode and coalesces direct kept ranges", () => {
|
||||
const transcript = [word("w0", "start", 0, 0.5), word("w1", "end", 4.5, 5)];
|
||||
|
||||
assert.deepEqual(compileCutList(transcript, { keep: "3-4,1-2,1.5-2.5,4.1-4.2" }), [
|
||||
{ start: 1, end: 2.5 },
|
||||
{ start: 3, end: 4 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("--plan on a fixture transcript prints the exact segment JSON", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "media-use-cutlist-"));
|
||||
try {
|
||||
const transcriptPath = join(dir, "fixture.json");
|
||||
writeFileSync(
|
||||
transcriptPath,
|
||||
JSON.stringify([
|
||||
word("w0", "hello", 0, 0.4),
|
||||
word("w1", "um", 0.5, 0.65),
|
||||
word("w2", "there", 0.7, 1),
|
||||
word("w3", "pause", 2.2, 2.5),
|
||||
word("w4", "end", 2.7, 3.2),
|
||||
]),
|
||||
);
|
||||
|
||||
const out = execFileSync(
|
||||
process.execPath,
|
||||
[
|
||||
SCRIPT,
|
||||
"--input",
|
||||
"ignored.mp4",
|
||||
"--transcript",
|
||||
transcriptPath,
|
||||
"--remove",
|
||||
"0.9-1.2",
|
||||
"--remove-fillers",
|
||||
"um",
|
||||
"--cut-silence",
|
||||
"0.8",
|
||||
"--plan",
|
||||
],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
|
||||
assert.deepEqual(JSON.parse(out), [
|
||||
{ start: 0, end: 0.5 },
|
||||
{ start: 0.65, end: 0.9 },
|
||||
{ start: 2.05, end: 3.2 },
|
||||
]);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
function word(id, text, start, end) {
|
||||
return { id, text, start, end };
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
import { wordListsFromMediaMeta } from "./words.mjs";
|
||||
|
||||
/**
|
||||
* Speech spans from word timestamps.
|
||||
*
|
||||
* audio_meta.json word times are relative to EACH LINE'S OWN FILE, not to the
|
||||
* composition. Without placement info, multiple lines would overlap at t=0 and
|
||||
* merge into one bogus span. Placement options:
|
||||
* offsets: { [voiceId]: startSeconds } explicit composition placement
|
||||
* sequential: stack lines back to back (plus `gap` seconds between lines)
|
||||
* A single word list (bare transcript) needs neither.
|
||||
*/
|
||||
export function speechSpans(meta, { mergeGap = 0.6, offsets, sequential = false, gap = 0 } = {}) {
|
||||
const merge = Number(mergeGap);
|
||||
const lists = wordListsFromMediaMeta(meta);
|
||||
const voices = Array.isArray(meta?.voices) ? meta.voices : [];
|
||||
if (lists.length > 1 && !offsets && !sequential) {
|
||||
throw new Error(
|
||||
"audio_meta has multiple voice lines with file-relative times; pass --sequential or --offsets so spans land at composition time",
|
||||
);
|
||||
}
|
||||
const intervals = [];
|
||||
let cursor = 0;
|
||||
for (let i = 0; i < lists.length; i++) {
|
||||
const voice = voices[i];
|
||||
let offset = 0;
|
||||
if (offsets) {
|
||||
const id = voice?.id ?? String(i);
|
||||
if (!(id in offsets)) throw new Error(`--offsets is missing voice "${id}"`);
|
||||
offset = Number(offsets[id]) || 0;
|
||||
} else if (sequential) {
|
||||
offset = cursor;
|
||||
const lineDuration = Number(voice?.duration_s) || Math.max(...lists[i].map((w) => w.end), 0);
|
||||
cursor += lineDuration + (Number(gap) || 0);
|
||||
}
|
||||
for (const word of lists[i]) {
|
||||
if (word.end > word.start)
|
||||
intervals.push({ start: word.start + offset, end: word.end + offset });
|
||||
}
|
||||
}
|
||||
return mergeIntervals(intervals, Number.isFinite(merge) && merge >= 0 ? merge : 0.6);
|
||||
}
|
||||
|
||||
export function duckKeyframes(
|
||||
spans,
|
||||
{ duck = 0.25, attack = 0.15, release = 0.4, baseVolume = 1 } = {},
|
||||
) {
|
||||
const base = finiteOr(baseVolume, 1);
|
||||
const ducked = round3(base * finiteOr(duck, 0.25));
|
||||
const keyframes = [];
|
||||
for (const span of spans) {
|
||||
keyframes.push({
|
||||
time: round3(Math.max(0, finiteOr(span.start, 0))),
|
||||
volume: ducked,
|
||||
duration: round3(finiteOr(attack, 0.15)),
|
||||
});
|
||||
keyframes.push({
|
||||
time: round3(Math.max(0, finiteOr(span.end, 0))),
|
||||
volume: round3(base),
|
||||
duration: round3(finiteOr(release, 0.4)),
|
||||
});
|
||||
}
|
||||
return keyframes.sort((a, b) => a.time - b.time);
|
||||
}
|
||||
|
||||
function mergeIntervals(intervals, mergeGap) {
|
||||
const sorted = intervals
|
||||
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
|
||||
.sort((a, b) => a.start - b.start || a.end - b.end);
|
||||
const merged = [];
|
||||
for (const range of sorted) {
|
||||
const prev = merged.at(-1);
|
||||
if (prev && (range.start <= prev.end || range.start - prev.end < mergeGap)) {
|
||||
prev.end = Math.max(prev.end, range.end);
|
||||
} else {
|
||||
merged.push({ ...range });
|
||||
}
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
function finiteOr(value, fallback) {
|
||||
const n = Number(value);
|
||||
return Number.isFinite(n) ? n : fallback;
|
||||
}
|
||||
|
||||
function round3(n) {
|
||||
return Math.round(Number(n) * 1000) / 1000;
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
|
||||
import { join, dirname } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { test } from "node:test";
|
||||
import { duckKeyframes, speechSpans } from "./duck.mjs";
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const SCRIPT = join(HERE, "..", "audio-duck.mjs");
|
||||
|
||||
test("speechSpans bridges gaps smaller than mergeGap", () => {
|
||||
const meta = {
|
||||
words: [word("w0", "one", 0, 0.5), word("w1", "two", 0.8, 1), word("w2", "three", 2, 2.2)],
|
||||
};
|
||||
|
||||
assert.deepEqual(speechSpans(meta, { mergeGap: 0.4 }), [
|
||||
{ start: 0, end: 1 },
|
||||
{ start: 2, end: 2.2 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("speechSpans refuses multi-line meta without placement (file-relative times)", () => {
|
||||
const meta = {
|
||||
voices: [
|
||||
{ id: "a", words: [word("w0", "one", 0, 1)] },
|
||||
{ id: "b", words: [word("w1", "two", 0, 1)] },
|
||||
],
|
||||
};
|
||||
assert.throws(() => speechSpans(meta, { mergeGap: 0.2 }), /--sequential or --offsets/);
|
||||
});
|
||||
|
||||
test("speechSpans sequential stacks lines by duration plus gap", () => {
|
||||
const meta = {
|
||||
voices: [
|
||||
{ id: "a", duration_s: 2, words: [word("w0", "one", 0.1, 1.9)] },
|
||||
{ id: "b", duration_s: 1, words: [word("w1", "two", 0.1, 0.9)] },
|
||||
],
|
||||
};
|
||||
assert.deepEqual(speechSpans(meta, { mergeGap: 0.2, sequential: true, gap: 0.5 }), [
|
||||
{ start: 0.1, end: 1.9 },
|
||||
{ start: 2.6, end: 3.4 },
|
||||
]);
|
||||
});
|
||||
|
||||
test("speechSpans explicit offsets place each line at composition time", () => {
|
||||
const meta = {
|
||||
voices: [
|
||||
{ id: "a", words: [word("w0", "one", 0, 1)] },
|
||||
{ id: "b", words: [word("w1", "two", 0, 1)] },
|
||||
],
|
||||
};
|
||||
assert.deepEqual(speechSpans(meta, { mergeGap: 0.2, offsets: { a: 0, b: 4 } }), [
|
||||
{ start: 0, end: 1 },
|
||||
{ start: 4, end: 5 },
|
||||
]);
|
||||
assert.throws(() => speechSpans(meta, { offsets: { a: 0 } }), /missing voice "b"/);
|
||||
});
|
||||
|
||||
test("speechSpans returns empty spans for empty input", () => {
|
||||
assert.deepEqual(speechSpans({ voices: [] }, { mergeGap: 0.6 }), []);
|
||||
});
|
||||
|
||||
test("duckKeyframes shapes attack and release from base volume", () => {
|
||||
assert.deepEqual(
|
||||
duckKeyframes([{ start: 3, end: 5 }], {
|
||||
duck: 0.25,
|
||||
attack: 0.15,
|
||||
release: 0.4,
|
||||
baseVolume: 0.6,
|
||||
}),
|
||||
[
|
||||
{ time: 3, volume: 0.15, duration: 0.15 },
|
||||
{ time: 5, volume: 0.6, duration: 0.4 },
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test("--json spans match --merge-gap semantics exactly", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "media-use-duck-"));
|
||||
try {
|
||||
const metaPath = join(dir, "audio_meta.json");
|
||||
writeFileSync(
|
||||
metaPath,
|
||||
JSON.stringify({
|
||||
voices: [
|
||||
{
|
||||
id: "narration",
|
||||
words: [
|
||||
word("w0", "one", 0, 0.4),
|
||||
word("w1", "two", 0.9, 1.2),
|
||||
word("w2", "three", 1.8, 2.1),
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
|
||||
const out = execFileSync(
|
||||
process.execPath,
|
||||
[SCRIPT, "--meta", metaPath, "--target", "#bgm", "--merge-gap", "0.6", "--json"],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
|
||||
const parsed = JSON.parse(out);
|
||||
assert.deepEqual(parsed.spans, [
|
||||
{ start: 0, end: 1.2 },
|
||||
{ start: 1.8, end: 2.1 },
|
||||
]);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
function word(id, text, start, end) {
|
||||
return { id, text, start, end };
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
export const ERROR_DIFFUSION_ALGORITHMS = {
|
||||
"floyd-steinberg": {
|
||||
kernel: [
|
||||
[1, 0, 7],
|
||||
[-1, 1, 3],
|
||||
[0, 1, 5],
|
||||
[1, 1, 1],
|
||||
],
|
||||
divisor: 16,
|
||||
},
|
||||
atkinson: {
|
||||
kernel: [
|
||||
[1, 0, 1],
|
||||
[2, 0, 1],
|
||||
[-1, 1, 1],
|
||||
[0, 1, 1],
|
||||
[1, 1, 1],
|
||||
[0, 2, 1],
|
||||
],
|
||||
divisor: 8,
|
||||
},
|
||||
"jarvis-judice-ninke": {
|
||||
kernel: [
|
||||
[1, 0, 7],
|
||||
[2, 0, 5],
|
||||
[-2, 1, 3],
|
||||
[-1, 1, 5],
|
||||
[0, 1, 7],
|
||||
[1, 1, 5],
|
||||
[2, 1, 3],
|
||||
[-2, 2, 1],
|
||||
[-1, 2, 3],
|
||||
[0, 2, 5],
|
||||
[1, 2, 3],
|
||||
[2, 2, 1],
|
||||
],
|
||||
divisor: 48,
|
||||
},
|
||||
stucki: {
|
||||
kernel: [
|
||||
[1, 0, 8],
|
||||
[2, 0, 4],
|
||||
[-2, 1, 2],
|
||||
[-1, 1, 4],
|
||||
[0, 1, 8],
|
||||
[1, 1, 4],
|
||||
[2, 1, 2],
|
||||
[-2, 2, 1],
|
||||
[-1, 2, 2],
|
||||
[0, 2, 4],
|
||||
[1, 2, 2],
|
||||
[2, 2, 1],
|
||||
],
|
||||
divisor: 42,
|
||||
},
|
||||
burkes: {
|
||||
kernel: [
|
||||
[1, 0, 8],
|
||||
[2, 0, 4],
|
||||
[-2, 1, 2],
|
||||
[-1, 1, 4],
|
||||
[0, 1, 8],
|
||||
[1, 1, 4],
|
||||
[2, 1, 2],
|
||||
],
|
||||
divisor: 32,
|
||||
},
|
||||
sierra: {
|
||||
kernel: [
|
||||
[1, 0, 5],
|
||||
[2, 0, 3],
|
||||
[-2, 1, 2],
|
||||
[-1, 1, 4],
|
||||
[0, 1, 5],
|
||||
[1, 1, 4],
|
||||
[2, 1, 2],
|
||||
[-1, 2, 2],
|
||||
[0, 2, 3],
|
||||
[1, 2, 2],
|
||||
],
|
||||
divisor: 32,
|
||||
},
|
||||
"sierra-lite": {
|
||||
kernel: [
|
||||
[1, 0, 2],
|
||||
[-1, 1, 1],
|
||||
[0, 1, 1],
|
||||
],
|
||||
divisor: 4,
|
||||
},
|
||||
"two-row-sierra": {
|
||||
kernel: [
|
||||
[1, 0, 4],
|
||||
[2, 0, 3],
|
||||
[-2, 1, 1],
|
||||
[-1, 1, 2],
|
||||
[0, 1, 3],
|
||||
[1, 1, 2],
|
||||
[2, 1, 1],
|
||||
],
|
||||
divisor: 16,
|
||||
},
|
||||
};
|
||||
|
||||
const DEFAULTS = {
|
||||
algorithm: "floyd-steinberg",
|
||||
brightness: 1,
|
||||
contrast: 1.2,
|
||||
detail: 1,
|
||||
palette: ["#000000", "#ffffff"],
|
||||
pointSize: 3,
|
||||
};
|
||||
|
||||
export function errorDiffusionBufferLength(width, height, pointSize) {
|
||||
return Math.ceil(width / pointSize) * Math.ceil(height / pointSize) * 3;
|
||||
}
|
||||
|
||||
export function applyErrorDiffusionRgba(data, width, height, options = {}, errorBuffer) {
|
||||
if (!Number.isInteger(width) || width < 1 || !Number.isInteger(height) || height < 1) {
|
||||
throw new Error("width and height must be positive integers");
|
||||
}
|
||||
if (!data || data.length !== width * height * 4) {
|
||||
throw new Error(`RGBA data must contain ${width * height * 4} bytes`);
|
||||
}
|
||||
|
||||
const algorithm = options.algorithm ?? DEFAULTS.algorithm;
|
||||
const diffusion = ERROR_DIFFUSION_ALGORITHMS[algorithm];
|
||||
if (!diffusion) throw new Error(`unknown error-diffusion algorithm: ${algorithm}`);
|
||||
|
||||
const pointSize = integerInRange(options.pointSize ?? DEFAULTS.pointSize, 1, 20, "pointSize");
|
||||
const brightness = numberInRange(options.brightness ?? DEFAULTS.brightness, 0.5, 2, "brightness");
|
||||
const contrast = numberInRange(options.contrast ?? DEFAULTS.contrast, 0.5, 2, "contrast");
|
||||
const detail = numberInRange(options.detail ?? DEFAULTS.detail, 0.1, 1, "detail");
|
||||
const palette = parsePalette(options.palette ?? DEFAULTS.palette);
|
||||
const blockColumns = Math.ceil(width / pointSize);
|
||||
const blockRows = Math.ceil(height / pointSize);
|
||||
const errorLength = errorDiffusionBufferLength(width, height, pointSize);
|
||||
const errors = errorBuffer ?? new Float32Array(errorLength);
|
||||
if (!(errors instanceof Float32Array) || errors.length !== errorLength) {
|
||||
throw new Error(`errorBuffer must be a Float32Array of length ${errorLength}`);
|
||||
}
|
||||
errors.fill(0);
|
||||
|
||||
const centerOffset = Math.floor(pointSize / 2);
|
||||
for (let blockRow = 0; blockRow < blockRows; blockRow++) {
|
||||
const blockY = blockRow * pointSize;
|
||||
for (let blockColumn = 0; blockColumn < blockColumns; blockColumn++) {
|
||||
const blockX = blockColumn * pointSize;
|
||||
const centerX = Math.min(blockX + centerOffset, width - 1);
|
||||
const centerY = Math.min(blockY + centerOffset, height - 1);
|
||||
const rgbaIndex = (centerY * width + centerX) * 4;
|
||||
const errorIndex = (blockRow * blockColumns + blockColumn) * 3;
|
||||
const red = correctedChannel(data[rgbaIndex], errors[errorIndex], brightness, contrast);
|
||||
const green = correctedChannel(
|
||||
data[rgbaIndex + 1],
|
||||
errors[errorIndex + 1],
|
||||
brightness,
|
||||
contrast,
|
||||
);
|
||||
const blue = correctedChannel(
|
||||
data[rgbaIndex + 2],
|
||||
errors[errorIndex + 2],
|
||||
brightness,
|
||||
contrast,
|
||||
);
|
||||
const luminance = 0.299 * red + 0.587 * green + 0.114 * blue;
|
||||
const output = palette[Math.min(palette.length - 1, Math.floor(luminance * palette.length))];
|
||||
|
||||
for (let y = blockY; y < Math.min(blockY + pointSize, height); y++) {
|
||||
for (let x = blockX; x < Math.min(blockX + pointSize, width); x++) {
|
||||
const outputIndex = (y * width + x) * 4;
|
||||
data[outputIndex] = Math.round(output[0] * 255);
|
||||
data[outputIndex + 1] = Math.round(output[1] * 255);
|
||||
data[outputIndex + 2] = Math.round(output[2] * 255);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [dx, dy, weight] of diffusion.kernel) {
|
||||
const targetColumn = blockColumn + dx;
|
||||
const targetRow = blockRow + dy;
|
||||
if (
|
||||
targetColumn < 0 ||
|
||||
targetColumn >= blockColumns ||
|
||||
targetRow < 0 ||
|
||||
targetRow >= blockRows
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
const target = (targetRow * blockColumns + targetColumn) * 3;
|
||||
const scale = (weight / diffusion.divisor) * detail;
|
||||
errors[target] += (red - output[0]) * scale;
|
||||
errors[target + 1] += (green - output[1]) * scale;
|
||||
errors[target + 2] += (blue - output[2]) * scale;
|
||||
}
|
||||
}
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
function correctedChannel(byte, error, brightness, contrast) {
|
||||
return Math.min(1, Math.max(0, ((byte / 255 - 0.5) * contrast + 0.5) * brightness + error));
|
||||
}
|
||||
|
||||
function parsePalette(colors) {
|
||||
if (!Array.isArray(colors) || colors.length < 2 || colors.length > 6) {
|
||||
throw new Error("palette must contain 2 to 6 colors");
|
||||
}
|
||||
return colors.map((color) => {
|
||||
const match = /^#([0-9a-f]{6})$/i.exec(color);
|
||||
if (!match) throw new Error(`palette color must use #rrggbb: ${color}`);
|
||||
const value = Number.parseInt(match[1], 16);
|
||||
return [(value >> 16) / 255, ((value >> 8) & 255) / 255, (value & 255) / 255];
|
||||
});
|
||||
}
|
||||
|
||||
function numberInRange(value, min, max, name) {
|
||||
const number = Number(value);
|
||||
if (!Number.isFinite(number) || number < min || number > max) {
|
||||
throw new Error(`${name} must be between ${min} and ${max}`);
|
||||
}
|
||||
return number;
|
||||
}
|
||||
|
||||
function integerInRange(value, min, max, name) {
|
||||
const number = Number(value);
|
||||
if (!Number.isInteger(number) || number < min || number > max) {
|
||||
throw new Error(`${name} must be an integer between ${min} and ${max}`);
|
||||
}
|
||||
return number;
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { ERROR_DIFFUSION_ALGORITHMS, applyErrorDiffusionRgba } from "./error-diffusion.mjs";
|
||||
|
||||
const EXPECTED_GRADIENTS = {
|
||||
"floyd-steinberg": "00000101/00010101/00100101/00010111/01010101/01011011",
|
||||
atkinson: "00000011/00001100/00010011/00010111/01001101/00111011",
|
||||
"jarvis-judice-ninke": "00000011/00001011/00011001/00101111/00100111/01011011",
|
||||
stucki: "00000011/00010101/00010110/00101011/00101101/01010111",
|
||||
burkes: "00000101/00010011/00010110/00101011/01010111/00101011",
|
||||
sierra: "00000011/00010101/00010110/00100111/00110111/00101101",
|
||||
"sierra-lite": "00000101/00010101/00100101/00010110/01010111/01010101",
|
||||
"two-row-sierra": "00000101/00010011/00010110/00101011/00101101/01011011",
|
||||
};
|
||||
|
||||
test("exposes the eight article error-diffusion algorithms", () => {
|
||||
assert.deepEqual(Object.keys(ERROR_DIFFUSION_ALGORITHMS), Object.keys(EXPECTED_GRADIENTS));
|
||||
});
|
||||
|
||||
test("matches deterministic golden patterns for every diffusion kernel", () => {
|
||||
const width = 8;
|
||||
const height = 6;
|
||||
const source = new Uint8ClampedArray(width * height * 4);
|
||||
for (let y = 0; y < height; y++) {
|
||||
for (let x = 0; x < width; x++) {
|
||||
const value = Math.round((255 * (x + y * 0.7)) / (width - 1 + (height - 1) * 0.7));
|
||||
const offset = (y * width + x) * 4;
|
||||
source[offset] = value;
|
||||
source[offset + 1] = Math.round(value * 0.8);
|
||||
source[offset + 2] = Math.round(value * 0.55);
|
||||
source[offset + 3] = 17 + x + y;
|
||||
}
|
||||
}
|
||||
|
||||
for (const [algorithm, expected] of Object.entries(EXPECTED_GRADIENTS)) {
|
||||
const output = source.slice();
|
||||
applyErrorDiffusionRgba(output, width, height, {
|
||||
algorithm,
|
||||
brightness: 1,
|
||||
contrast: 1,
|
||||
detail: 1,
|
||||
palette: ["#000000", "#ffffff"],
|
||||
pointSize: 1,
|
||||
});
|
||||
const rows = [];
|
||||
for (let y = 0; y < height; y++) {
|
||||
let row = "";
|
||||
for (let x = 0; x < width; x++) row += output[(y * width + x) * 4] ? "1" : "0";
|
||||
rows.push(row);
|
||||
}
|
||||
assert.equal(rows.join("/"), expected, algorithm);
|
||||
for (let i = 0; i < width * height; i++) assert.equal(output[i * 4 + 3], source[i * 4 + 3]);
|
||||
}
|
||||
});
|
||||
|
||||
test("fills point-size blocks from their center sample", () => {
|
||||
const data = new Uint8ClampedArray([
|
||||
0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 255, 255, 255, 6, 0, 0, 0, 7, 255,
|
||||
255, 255, 8,
|
||||
]);
|
||||
applyErrorDiffusionRgba(
|
||||
data,
|
||||
4,
|
||||
2,
|
||||
{
|
||||
algorithm: "floyd-steinberg",
|
||||
brightness: 1,
|
||||
contrast: 1,
|
||||
detail: 1,
|
||||
palette: ["#000000", "#ffffff"],
|
||||
pointSize: 2,
|
||||
},
|
||||
new Float32Array(6),
|
||||
);
|
||||
assert.deepEqual(
|
||||
[...data],
|
||||
[
|
||||
255, 255, 255, 1, 255, 255, 255, 2, 255, 255, 255, 3, 255, 255, 255, 4, 255, 255, 255, 5, 255,
|
||||
255, 255, 6, 255, 255, 255, 7, 255, 255, 255, 8,
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test("preserves authored palette order and validates the public contract", () => {
|
||||
const reversed = new Uint8ClampedArray([0, 0, 0, 255]);
|
||||
applyErrorDiffusionRgba(reversed, 1, 1, {
|
||||
palette: ["#ffffff", "#000000"],
|
||||
});
|
||||
assert.deepEqual([...reversed], [255, 255, 255, 255]);
|
||||
|
||||
assert.throws(
|
||||
() => applyErrorDiffusionRgba(new Uint8ClampedArray(4), 1, 1, { palette: ["#000000"] }),
|
||||
/2 to 6 colors/,
|
||||
);
|
||||
assert.throws(
|
||||
() =>
|
||||
applyErrorDiffusionRgba(new Uint8ClampedArray(4), 1, 1, {
|
||||
algorithm: "ordered-bayer",
|
||||
}),
|
||||
/unknown error-diffusion algorithm/,
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
import { writeFileSync, copyFileSync, mkdirSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
// ponytail: bound the download so a hostile/runaway URL can't fill the disk.
|
||||
// 256MB covers any real media asset; raise if 4K video sources ever exceed it.
|
||||
const MAX_FREEZE_BYTES = 256 * 1024 * 1024;
|
||||
|
||||
export async function freezeUrl(url, destPath) {
|
||||
const where = String(url).slice(0, 80);
|
||||
const res = await fetch(url);
|
||||
if (!res.ok) throw new Error(`freeze failed: HTTP ${res.status} for ${where}`);
|
||||
|
||||
// Fail fast on an advertised oversize body before reading a single byte.
|
||||
const declared = Number(res.headers.get("content-length"));
|
||||
if (declared > MAX_FREEZE_BYTES)
|
||||
throw new Error(
|
||||
`freeze failed: ${declared} bytes exceeds ${MAX_FREEZE_BYTES} cap for ${where}`,
|
||||
);
|
||||
|
||||
// Stream and abort once the cap is crossed, so a lying/chunked hostile URL
|
||||
// can't buffer the whole payload into memory before the check (M1).
|
||||
const chunks = [];
|
||||
let total = 0;
|
||||
for await (const chunk of res.body) {
|
||||
total += chunk.length;
|
||||
if (total > MAX_FREEZE_BYTES)
|
||||
throw new Error(`freeze failed: stream exceeds ${MAX_FREEZE_BYTES} cap for ${where}`);
|
||||
chunks.push(chunk);
|
||||
}
|
||||
if (total === 0) throw new Error(`freeze failed: empty response for ${where}`);
|
||||
|
||||
mkdirSync(dirname(destPath), { recursive: true });
|
||||
writeFileSync(destPath, Buffer.concat(chunks, total));
|
||||
return total;
|
||||
}
|
||||
|
||||
export function freezeLocalFile(srcPath, destPath) {
|
||||
mkdirSync(dirname(destPath), { recursive: true });
|
||||
copyFileSync(srcPath, destPath);
|
||||
}
|
||||
|
||||
// Ingest accepts a DIRECT public media URL only — not a platform page. yt-dlp is
|
||||
// deliberately out (cloud IPs get blocked, and it's brittle); the supported case
|
||||
// is "user points at their own file or a direct asset link". A direct URL is a
|
||||
// non-platform host whose path ends in a known media extension.
|
||||
const PLATFORM_HOSTS =
|
||||
/(^|\.)(youtube\.com|youtu\.be|vimeo\.com|tiktok\.com|instagram\.com|twitter\.com|x\.com|facebook\.com|dailymotion\.com)$/i;
|
||||
const MEDIA_EXT = /\.(mp3|wav|m4a|aac|ogg|flac|mp4|mov|webm|mkv|png|jpe?g|webp|gif|svg|avif)$/i;
|
||||
|
||||
// SSRF guard (m11): a user-supplied --from URL must not point at the local host
|
||||
// or a private network. Blocks loopback/localhost, RFC1918, link-local, and the
|
||||
// IPv6 equivalents on the literal hostname.
|
||||
// ponytail: literal-host check only; a DNS name that *resolves* to a private IP
|
||||
// (rebinding) still passes — add resolve-then-check if --from ever fetches from
|
||||
// untrusted hostnames at scale.
|
||||
const PRIVATE_HOST =
|
||||
/^(localhost|.*\.local|.*\.internal|127\.|10\.|0\.|169\.254\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.|\[?(::1|::ffff:127\.|f[cd][0-9a-f]{2}:|fe80:))/i;
|
||||
|
||||
export function isDirectMediaUrl(u) {
|
||||
let url;
|
||||
try {
|
||||
url = new URL(u);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (url.protocol !== "http:" && url.protocol !== "https:") return false;
|
||||
if (PLATFORM_HOSTS.test(url.hostname)) return false;
|
||||
if (PRIVATE_HOST.test(url.hostname)) return false;
|
||||
return MEDIA_EXT.test(url.pathname);
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { test } from "node:test";
|
||||
import { isDirectMediaUrl } from "./freeze.mjs";
|
||||
|
||||
test("accepts direct public media URLs", () => {
|
||||
assert.equal(isDirectMediaUrl("https://cdn.example.com/clip.mp4"), true);
|
||||
assert.equal(isDirectMediaUrl("https://example.com/a/b/track.mp3"), true);
|
||||
assert.equal(isDirectMediaUrl("http://example.com/logo.svg"), true);
|
||||
});
|
||||
|
||||
test("rejects platform pages (no yt-dlp)", () => {
|
||||
assert.equal(isDirectMediaUrl("https://www.youtube.com/watch?v=abc"), false);
|
||||
assert.equal(isDirectMediaUrl("https://youtu.be/abc"), false);
|
||||
assert.equal(isDirectMediaUrl("https://vimeo.com/12345"), false);
|
||||
assert.equal(isDirectMediaUrl("https://x.com/u/status/1"), false);
|
||||
});
|
||||
|
||||
test("rejects non-direct / non-media URLs", () => {
|
||||
assert.equal(isDirectMediaUrl("https://example.com/page"), false, "no media extension");
|
||||
assert.equal(isDirectMediaUrl("ftp://example.com/a.mp4"), false, "non-http(s)");
|
||||
assert.equal(isDirectMediaUrl("not a url"), false);
|
||||
});
|
||||
|
||||
test("rejects local / private hosts (SSRF guard, m11)", () => {
|
||||
for (const u of [
|
||||
"http://localhost/a.mp4",
|
||||
"http://127.0.0.1/a.mp4",
|
||||
"http://127.1.2.3/a.mp4",
|
||||
"http://0.0.0.0/a.mp4",
|
||||
"http://10.0.0.5/a.mp4",
|
||||
"http://192.168.1.1/a.mp4",
|
||||
"http://172.16.0.1/a.mp4",
|
||||
"http://172.31.255.255/a.mp4",
|
||||
"http://169.254.169.254/a.mp4", // cloud metadata endpoint
|
||||
"http://printer.local/a.mp4",
|
||||
"http://svc.internal/a.mp4",
|
||||
"http://[::1]/a.mp4",
|
||||
"http://[fe80::1]/a.mp4",
|
||||
"http://[fd00::1]/a.mp4",
|
||||
]) {
|
||||
assert.equal(isDirectMediaUrl(u), false, `should block ${u}`);
|
||||
}
|
||||
// A public host that merely starts with similar digits is still allowed.
|
||||
assert.equal(isDirectMediaUrl("https://172.40.0.1/a.mp4"), true, "172.40 is public");
|
||||
assert.equal(isDirectMediaUrl("https://11.example.com/a.mp4"), true);
|
||||
});
|
||||
@@ -0,0 +1,265 @@
|
||||
// Vendored plain-JS copy of packages/core/src/mediaGradeAnalyzer.ts.
|
||||
// packages/core/src/mediaGradeAnalyzer.vendoredParity.test.ts guards behavior drift.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { basename, extname } from "node:path";
|
||||
|
||||
const IMAGE_EXT = new Set([".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tif", ".tiff"]);
|
||||
const SAMPLE_FRAMES = 5;
|
||||
const DEFAULT_TIMEOUT_MS = 15_000;
|
||||
|
||||
const ADJUST_LIMITS = {
|
||||
exposure: { min: -2, max: 2 },
|
||||
contrast: { min: -1, max: 1 },
|
||||
whites: { min: -1, max: 1 },
|
||||
blacks: { min: -1, max: 1 },
|
||||
temperature: { min: -1, max: 1 },
|
||||
tint: { min: -1, max: 1 },
|
||||
};
|
||||
|
||||
function clamp(value, key) {
|
||||
const limit = ADJUST_LIMITS[key];
|
||||
if (!Number.isFinite(value)) return 0;
|
||||
return Math.min(limit.max, Math.max(limit.min, value));
|
||||
}
|
||||
|
||||
function round(value) {
|
||||
const rounded = Math.round(value * 1000) / 1000;
|
||||
return Object.is(rounded, -0) ? 0 : rounded;
|
||||
}
|
||||
|
||||
function average(values) {
|
||||
return values.reduce((sum, value) => sum + value, 0) / Math.max(1, values.length);
|
||||
}
|
||||
|
||||
function probeMedia(mediaPath, ffprobePath) {
|
||||
try {
|
||||
const raw = execFileSync(
|
||||
ffprobePath,
|
||||
[
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v:0",
|
||||
"-show_entries",
|
||||
"stream=color_space,color_transfer,color_primaries,pix_fmt,duration:format=duration",
|
||||
"-of",
|
||||
"json",
|
||||
"--",
|
||||
mediaPath,
|
||||
],
|
||||
{ encoding: "utf8", timeout: 5_000, stdio: ["ignore", "pipe", "pipe"] },
|
||||
);
|
||||
const parsed = JSON.parse(raw);
|
||||
const stream = Array.isArray(parsed.streams) ? (parsed.streams[0] ?? {}) : {};
|
||||
const duration = Number(stream.duration ?? parsed.format?.duration);
|
||||
const text = (key) =>
|
||||
typeof stream[key] === "string" && stream[key] ? stream[key] : "unknown";
|
||||
return {
|
||||
duration: Number.isFinite(duration) && duration > 0 ? duration : null,
|
||||
colorSpace: text("color_space"),
|
||||
transfer: text("color_transfer"),
|
||||
primaries: text("color_primaries"),
|
||||
pixelFormat: text("pix_fmt"),
|
||||
};
|
||||
} catch {
|
||||
return {
|
||||
duration: null,
|
||||
colorSpace: "unknown",
|
||||
transfer: "unknown",
|
||||
primaries: "unknown",
|
||||
pixelFormat: "unknown",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export function parseMediaTreatmentSignalStats(raw) {
|
||||
const frames = [];
|
||||
let current = null;
|
||||
for (const line of String(raw).split(/\r?\n/)) {
|
||||
// One digit is enough: .* consumes the rest without overlapping repetitions.
|
||||
const frame = line.match(/^frame:\d.*pts_time:([+-]?(?:\d+(?:\.\d+)?|\.\d+))/);
|
||||
if (frame) {
|
||||
if (current) frames.push(current);
|
||||
current = { ptsTime: Number(frame[1]) };
|
||||
continue;
|
||||
}
|
||||
const stat = line.match(/lavfi\.signalstats\.([A-Z]+)=([+-]?(?:\d+(?:\.\d+)?|\.\d+))/);
|
||||
if (!stat) continue;
|
||||
current ??= {};
|
||||
current[stat[1]] = Number(stat[2]);
|
||||
}
|
||||
if (current) frames.push(current);
|
||||
return frames.filter(
|
||||
(frame) =>
|
||||
Number.isFinite(frame.YMIN) &&
|
||||
Number.isFinite(frame.YLOW) &&
|
||||
Number.isFinite(frame.YAVG) &&
|
||||
Number.isFinite(frame.YHIGH) &&
|
||||
Number.isFinite(frame.YMAX) &&
|
||||
Number.isFinite(frame.UAVG) &&
|
||||
Number.isFinite(frame.VAVG),
|
||||
);
|
||||
}
|
||||
|
||||
function summarizeFrames(frames) {
|
||||
if (frames.length === 0) throw new Error("FFmpeg returned no analyzable video frames");
|
||||
return {
|
||||
frames: frames.length,
|
||||
yMin: Math.min(...frames.map((frame) => frame.YMIN)),
|
||||
yLow: average(frames.map((frame) => frame.YLOW)),
|
||||
yAvg: average(frames.map((frame) => frame.YAVG)),
|
||||
yHigh: average(frames.map((frame) => frame.YHIGH)),
|
||||
yMax: Math.max(...frames.map((frame) => frame.YMAX)),
|
||||
uAvg: average(frames.map((frame) => frame.UAVG)),
|
||||
vAvg: average(frames.map((frame) => frame.VAVG)),
|
||||
satAvg: average(frames.map((frame) => frame.SATAVG ?? 0)),
|
||||
shadowClipRisk: average(frames.map((frame) => (frame.YLOW <= 16 ? 1 : 0))),
|
||||
highlightClipRisk: average(frames.map((frame) => (frame.YHIGH >= 235 ? 1 : 0))),
|
||||
};
|
||||
}
|
||||
|
||||
function suggestedExposure(normalizedAverage, yLow, yHigh) {
|
||||
if (normalizedAverage < 0.28 && yHigh / 255 < 0.65) {
|
||||
return clamp((0.32 - normalizedAverage) * 1.2, "exposure");
|
||||
}
|
||||
if (normalizedAverage > 0.72 && yLow / 255 > 0.3) {
|
||||
return clamp((0.68 - normalizedAverage) * 1.2, "exposure");
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function suggestedTemperature(uAverage, vAverage) {
|
||||
const chromaWarmth = (vAverage - 128 + (128 - uAverage)) / 128;
|
||||
return Math.abs(chromaWarmth) >= 0.08 ? clamp(-chromaWarmth * 0.25, "temperature") : 0;
|
||||
}
|
||||
|
||||
function suggestedTint(uAverage, vAverage) {
|
||||
const cast = uAverage + vAverage - 256;
|
||||
return Math.abs(cast) >= 10 ? clamp(-cast / 512, "tint") : 0;
|
||||
}
|
||||
|
||||
export function statsToAdjust(stats) {
|
||||
const yMin = Number(stats.yMin);
|
||||
const yLow = Number(stats.yLow ?? stats.yMin);
|
||||
const yMax = Number(stats.yMax);
|
||||
const yHigh = Number(stats.yHigh ?? stats.yMax);
|
||||
const yAvg = Number(stats.yAvg);
|
||||
const uAvg = Number(stats.uAvg);
|
||||
const vAvg = Number(stats.vAvg);
|
||||
const shadowClipRisk = Number(stats.shadowClipRisk ?? (yLow <= 16 ? 1 : 0));
|
||||
const highlightClipRisk = Number(stats.highlightClipRisk ?? (yHigh >= 235 ? 1 : 0));
|
||||
const percentileSpread = (yHigh - yLow) / 255;
|
||||
const normalizedAverage = yAvg / 255;
|
||||
const contrast = percentileSpread < 0.35 ? clamp((0.35 - percentileSpread) * 0.4, "contrast") : 0;
|
||||
|
||||
return {
|
||||
adjust: {
|
||||
exposure: round(suggestedExposure(normalizedAverage, yLow, yHigh)),
|
||||
contrast: round(contrast),
|
||||
blacks: round(clamp(shadowClipRisk * 0.08, "blacks")),
|
||||
whites: round(clamp(-highlightClipRisk * 0.08, "whites")),
|
||||
temperature: round(suggestedTemperature(uAvg, vAvg)),
|
||||
tint: round(suggestedTint(uAvg, vAvg)),
|
||||
},
|
||||
measured: {
|
||||
frames: Number(stats.frames ?? 1),
|
||||
yMin: round(yMin),
|
||||
yLow: round(yLow),
|
||||
yAvg: round(yAvg),
|
||||
yHigh: round(yHigh),
|
||||
yMax: round(yMax),
|
||||
uAvg: round(uAvg),
|
||||
vAvg: round(vAvg),
|
||||
satAvg: round(Number(stats.satAvg ?? 0)),
|
||||
shadowClipRisk: round(shadowClipRisk),
|
||||
highlightClipRisk: round(highlightClipRisk),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function summarizeMediaTreatmentAnalysis(probe, frames) {
|
||||
const result = statsToAdjust(summarizeFrames(frames));
|
||||
const hdr = ["smpte2084", "arib-std-b67"].includes(probe.transfer);
|
||||
const diagnosis = [];
|
||||
if (result.measured.shadowClipRisk > 0) {
|
||||
diagnosis.push("sampled frames contain deep or clipped shadows");
|
||||
}
|
||||
if (result.measured.highlightClipRisk > 0) {
|
||||
diagnosis.push("sampled frames contain bright or clipped highlights");
|
||||
}
|
||||
if (diagnosis.length === 0) diagnosis.push("no obvious technical imbalance in sampled frames");
|
||||
|
||||
const warnings = [];
|
||||
if (hdr) warnings.push("HDR transfer detected; the realtime treatment path is SDR/Rec.709.");
|
||||
if (probe.colorSpace === "unknown" || probe.transfer === "unknown") {
|
||||
warnings.push(
|
||||
"Source color metadata is incomplete; camera LOG cannot be identified reliably from container metadata alone.",
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
...result,
|
||||
source: {
|
||||
colorSpace: probe.colorSpace,
|
||||
transfer: probe.transfer,
|
||||
primaries: probe.primaries,
|
||||
pixelFormat: probe.pixelFormat,
|
||||
hdr,
|
||||
log: "unknown",
|
||||
},
|
||||
diagnosis,
|
||||
warnings,
|
||||
};
|
||||
}
|
||||
|
||||
export function analyzeMediaGrade(
|
||||
mediaPath,
|
||||
{ ffmpegPath = "ffmpeg", ffprobePath = "ffprobe" } = {},
|
||||
) {
|
||||
try {
|
||||
const probe = probeMedia(mediaPath, ffprobePath);
|
||||
const isImage = IMAGE_EXT.has(extname(mediaPath).toLowerCase());
|
||||
const fps =
|
||||
!isImage && probe.duration
|
||||
? Math.max(0.1, Math.min(2, SAMPLE_FRAMES / probe.duration))
|
||||
: null;
|
||||
const filters = [
|
||||
fps ? `fps=${fps.toFixed(4)}` : null,
|
||||
"format=yuv444p",
|
||||
"signalstats",
|
||||
"metadata=print:file=-",
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(",");
|
||||
const raw = execFileSync(
|
||||
ffmpegPath,
|
||||
[
|
||||
"-hide_banner",
|
||||
"-nostdin",
|
||||
"-v",
|
||||
"error",
|
||||
"-i",
|
||||
mediaPath,
|
||||
"-vf",
|
||||
filters,
|
||||
"-frames:v",
|
||||
String(SAMPLE_FRAMES),
|
||||
"-f",
|
||||
"null",
|
||||
"-",
|
||||
],
|
||||
{
|
||||
encoding: "utf8",
|
||||
timeout: Number(process.env.HYPERFRAMES_ANALYZE_TIMEOUT_MS) || DEFAULT_TIMEOUT_MS,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
},
|
||||
);
|
||||
return summarizeMediaTreatmentAnalysis(probe, parseMediaTreatmentSignalStats(raw));
|
||||
} catch (error) {
|
||||
throw new Error(`grade analysis failed for ${mediaPath}: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function formatMeasuredNote(mediaPath, measured) {
|
||||
return `media-use: measured ${basename(mediaPath)}: frames=${measured.frames}, YMIN=${measured.yMin}, YLOW=${measured.yLow}, YAVG=${measured.yAvg}, YHIGH=${measured.yHigh}, YMAX=${measured.yMax}, UAVG=${measured.uAvg}, VAVG=${measured.vAvg}; adjust is a starting suggestion`;
|
||||
}
|
||||
@@ -0,0 +1,285 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { existsSync, mkdtempSync, rmSync, unlinkSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
analyzeMediaGrade,
|
||||
formatMeasuredNote,
|
||||
parseMediaTreatmentSignalStats,
|
||||
statsToAdjust,
|
||||
summarizeMediaTreatmentAnalysis,
|
||||
} from "./grade-analyzer.mjs";
|
||||
|
||||
const SIGNALSTATS = `frame:0 pts:0 pts_time:0
|
||||
lavfi.signalstats.YMIN=0
|
||||
lavfi.signalstats.YLOW=8
|
||||
lavfi.signalstats.YAVG=100
|
||||
lavfi.signalstats.YHIGH=240
|
||||
lavfi.signalstats.YMAX=255
|
||||
lavfi.signalstats.UAVG=120
|
||||
lavfi.signalstats.VAVG=140
|
||||
lavfi.signalstats.SATAVG=40
|
||||
frame:1 pts:2000 pts_time:2
|
||||
lavfi.signalstats.YMIN=10
|
||||
lavfi.signalstats.YLOW=20
|
||||
lavfi.signalstats.YAVG=130
|
||||
lavfi.signalstats.YHIGH=220
|
||||
lavfi.signalstats.YMAX=245
|
||||
lavfi.signalstats.UAVG=130
|
||||
lavfi.signalstats.VAVG=125
|
||||
lavfi.signalstats.SATAVG=60`;
|
||||
|
||||
// The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by
|
||||
// design — skills tests are meant to be node-builtin-only). Tests that shell to
|
||||
// ffmpeg skip there and run wherever ffmpeg is present (locally, dev).
|
||||
const FFMPEG_SKIP =
|
||||
spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0
|
||||
? false
|
||||
: "ffmpeg not on PATH";
|
||||
|
||||
const ADJUST_LIMITS = {
|
||||
exposure: { min: -2, max: 2 },
|
||||
contrast: { min: -1, max: 1 },
|
||||
highlights: { min: -1, max: 1 },
|
||||
shadows: { min: -1, max: 1 },
|
||||
whites: { min: -1, max: 1 },
|
||||
blacks: { min: -1, max: 1 },
|
||||
temperature: { min: -1, max: 1 },
|
||||
tint: { min: -1, max: 1 },
|
||||
vibrance: { min: -1, max: 1 },
|
||||
saturation: { min: -1, max: 1 },
|
||||
};
|
||||
|
||||
function makeFrame(dir, name, color) {
|
||||
const out = join(dir, name);
|
||||
execFileSync(
|
||||
"ffmpeg",
|
||||
[
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
`color=c=${color}:s=64x64`,
|
||||
"-frames:v",
|
||||
"1",
|
||||
"-y",
|
||||
out,
|
||||
],
|
||||
{ stdio: "pipe" },
|
||||
);
|
||||
return out;
|
||||
}
|
||||
|
||||
function assertWithinLimits(adjust) {
|
||||
for (const [key, value] of Object.entries(adjust)) {
|
||||
const limit = ADJUST_LIMITS[key];
|
||||
assert.ok(limit, `unexpected adjust key ${key}`);
|
||||
assert.ok(value >= limit.min && value <= limit.max, `${key} out of range: ${value}`);
|
||||
}
|
||||
}
|
||||
|
||||
test("under-exposed synthetic frame suggests positive exposure", { skip: FFMPEG_SKIP }, () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-grade-under-"));
|
||||
try {
|
||||
const file = makeFrame(dir, "under.png", "0x202020");
|
||||
const { adjust, measured } = analyzeMediaGrade(file);
|
||||
assert.ok(measured.frames >= 1);
|
||||
assert.ok(adjust.exposure > 0, `expected positive exposure, got ${adjust.exposure}`);
|
||||
assertWithinLimits(adjust);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test(
|
||||
"over-exposed synthetic frame pulls exposure down without inventing clipping",
|
||||
{ skip: FFMPEG_SKIP },
|
||||
() => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-grade-over-"));
|
||||
try {
|
||||
const file = makeFrame(dir, "over.png", "white");
|
||||
const { adjust } = analyzeMediaGrade(file);
|
||||
assert.ok(adjust.exposure < 0, `expected negative exposure, got ${adjust.exposure}`);
|
||||
assert.ok(adjust.whites <= 0, `expected non-positive whites, got ${adjust.whites}`);
|
||||
assertWithinLimits(adjust);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test(
|
||||
"warm-cast synthetic frame suggests negative temperature correction",
|
||||
{ skip: FFMPEG_SKIP },
|
||||
() => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-grade-warm-"));
|
||||
try {
|
||||
const file = makeFrame(dir, "warm.png", "orange");
|
||||
const { adjust } = analyzeMediaGrade(file);
|
||||
assert.ok(adjust.temperature < 0, `expected cooling correction, got ${adjust.temperature}`);
|
||||
assertWithinLimits(adjust);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test("low-spread stats suggest positive contrast", () => {
|
||||
const { adjust } = statsToAdjust({
|
||||
frames: 1,
|
||||
yMin: 104,
|
||||
yMax: 116,
|
||||
yAvg: 110,
|
||||
uAvg: 128,
|
||||
vAvg: 128,
|
||||
});
|
||||
assert.ok(adjust.contrast > 0, `expected positive contrast, got ${adjust.contrast}`);
|
||||
assertWithinLimits(adjust);
|
||||
});
|
||||
|
||||
test("malformed media fails cleanly", () => {
|
||||
assert.throws(
|
||||
() => analyzeMediaGrade(join(tmpdir(), "does-not-exist.png")),
|
||||
/grade analysis failed/,
|
||||
);
|
||||
});
|
||||
|
||||
test(
|
||||
"media path with shell metacharacters is passed as argv, not a shell string",
|
||||
{
|
||||
skip: FFMPEG_SKIP,
|
||||
},
|
||||
() => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "mu-grade-shell-"));
|
||||
const sentinel = join(process.cwd(), "mu-grade-shell-sentinel.png");
|
||||
try {
|
||||
if (existsSync(sentinel)) unlinkSync(sentinel);
|
||||
const file = makeFrame(dir, "frame; touch mu-grade-shell-sentinel.png", "orange");
|
||||
const result = analyzeMediaGrade(file);
|
||||
assert.ok(result.measured.frames >= 1);
|
||||
assert.equal(existsSync(sentinel), false);
|
||||
} finally {
|
||||
if (existsSync(sentinel)) unlinkSync(sentinel);
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
test("measured note is a stderr-safe single-line summary", () => {
|
||||
const note = formatMeasuredNote("/tmp/frame.png", {
|
||||
frames: 1,
|
||||
yMin: 10,
|
||||
yMax: 240,
|
||||
yAvg: 80,
|
||||
uAvg: 120,
|
||||
vAvg: 140,
|
||||
});
|
||||
assert.match(note, /^media-use: measured /);
|
||||
assert.match(note, /YMIN=10/);
|
||||
assert.match(note, /YMAX=240/);
|
||||
assert.match(note, /YAVG=80/);
|
||||
assert.equal(note.includes("\n"), false);
|
||||
});
|
||||
|
||||
test("parses deterministic FFmpeg signalstats frames", () => {
|
||||
const frames = parseMediaTreatmentSignalStats(SIGNALSTATS);
|
||||
assert.equal(frames.length, 2);
|
||||
assert.deepEqual(
|
||||
{
|
||||
ptsTime: frames[0]?.ptsTime,
|
||||
YLOW: frames[0]?.YLOW,
|
||||
YAVG: frames[0]?.YAVG,
|
||||
YHIGH: frames[0]?.YHIGH,
|
||||
SATAVG: frames[0]?.SATAVG,
|
||||
},
|
||||
{ ptsTime: 0, YLOW: 8, YAVG: 100, YHIGH: 240, SATAVG: 40 },
|
||||
);
|
||||
});
|
||||
|
||||
test("keeps adjust output while adding HDR metadata and warnings", () => {
|
||||
const result = summarizeMediaTreatmentAnalysis(
|
||||
{
|
||||
duration: 4,
|
||||
colorSpace: "bt2020nc",
|
||||
transfer: "smpte2084",
|
||||
primaries: "bt2020",
|
||||
pixelFormat: "yuv420p10le",
|
||||
},
|
||||
parseMediaTreatmentSignalStats(SIGNALSTATS),
|
||||
);
|
||||
|
||||
assert.deepEqual(
|
||||
{
|
||||
blacks: result.adjust.blacks,
|
||||
whites: result.adjust.whites,
|
||||
frames: result.measured.frames,
|
||||
yLow: result.measured.yLow,
|
||||
yHigh: result.measured.yHigh,
|
||||
satAvg: result.measured.satAvg,
|
||||
shadowClipRisk: result.measured.shadowClipRisk,
|
||||
highlightClipRisk: result.measured.highlightClipRisk,
|
||||
},
|
||||
{
|
||||
blacks: 0.04,
|
||||
whites: -0.04,
|
||||
frames: 2,
|
||||
yLow: 14,
|
||||
yHigh: 230,
|
||||
satAvg: 50,
|
||||
shadowClipRisk: 0.5,
|
||||
highlightClipRisk: 0.5,
|
||||
},
|
||||
);
|
||||
assert.equal(result.source.hdr, true);
|
||||
assert.equal(result.source.log, "unknown");
|
||||
assert.match(result.warnings[0] ?? "", /HDR/);
|
||||
});
|
||||
|
||||
test("does not force middle-gray exposure onto healthy intentional contrast", () => {
|
||||
const result = summarizeMediaTreatmentAnalysis(
|
||||
{
|
||||
duration: 4,
|
||||
colorSpace: "bt709",
|
||||
transfer: "bt709",
|
||||
primaries: "bt709",
|
||||
pixelFormat: "yuv420p",
|
||||
},
|
||||
[
|
||||
{
|
||||
YMIN: 20,
|
||||
YLOW: 42,
|
||||
YAVG: 122,
|
||||
YHIGH: 190,
|
||||
YMAX: 220,
|
||||
UAVG: 128,
|
||||
VAVG: 128,
|
||||
SATAVG: 40,
|
||||
},
|
||||
{
|
||||
YMIN: 18,
|
||||
YLOW: 38,
|
||||
YAVG: 118,
|
||||
YHIGH: 196,
|
||||
YMAX: 225,
|
||||
UAVG: 128,
|
||||
VAVG: 128,
|
||||
SATAVG: 42,
|
||||
},
|
||||
],
|
||||
);
|
||||
|
||||
assert.deepEqual(
|
||||
{
|
||||
exposure: result.adjust.exposure,
|
||||
blacks: result.adjust.blacks,
|
||||
whites: result.adjust.whites,
|
||||
temperature: result.adjust.temperature,
|
||||
tint: result.adjust.tint,
|
||||
},
|
||||
{ exposure: 0, blacks: 0, whites: 0, temperature: 0, tint: 0 },
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,191 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { track } from "./telemetry.mjs";
|
||||
|
||||
// v0.3.0 is the first CLI that can use an OAuth session; v0.1.x/0.2.x reject it
|
||||
// ("heygen-cli can't use OAuth yet"), and OAuth is what the free-usage path
|
||||
// needs — so anything below this can't authenticate for free usage at all.
|
||||
export const HEYGEN_MIN_VERSION = "0.3.0";
|
||||
// Free-usage path is OAuth (`--oauth` → subscription/free credits); `--api-key`
|
||||
// bills API credits, so the onboarding steers to OAuth. Keep pipe-to-shell
|
||||
// installer text out of the runtime module; the docs are the safer source of
|
||||
// truth for platform-specific setup.
|
||||
export const HEYGEN_INSTALL_COMMAND =
|
||||
"Install the CLI from https://developers.heygen.com/cli, then run: heygen auth login --oauth";
|
||||
export const HEYGEN_AUTH_COMMAND = "heygen auth login --oauth";
|
||||
export const HEYGEN_UPDATE_COMMAND = "heygen update";
|
||||
export const HEYGEN_CLIENT_SOURCE_ARGV = ["--headers", "X-HeyGen-Client-Source: media-use"];
|
||||
|
||||
export const HEYGEN_NOT_FOUND_MESSAGE = `media-use: heygen CLI not found — it's the free path for bgm/image/voice/avatar-video. ${HEYGEN_INSTALL_COMMAND}`;
|
||||
export const HEYGEN_NOT_AUTHENTICATED_MESSAGE = `media-use: heygen CLI not authenticated (free usage) — run: ${HEYGEN_AUTH_COMMAND}`;
|
||||
export const HEYGEN_OUTDATED_MESSAGE = `media-use: heygen CLI is outdated — run: ${HEYGEN_UPDATE_COMMAND} (need >= v${HEYGEN_MIN_VERSION})`;
|
||||
|
||||
const ACTIONABLE_MESSAGES = new Set([
|
||||
HEYGEN_NOT_FOUND_MESSAGE,
|
||||
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
|
||||
HEYGEN_OUTDATED_MESSAGE,
|
||||
]);
|
||||
|
||||
export function classifyHeygenError(err) {
|
||||
return classifyHeygenErrorResult(err).message;
|
||||
}
|
||||
|
||||
export function classifyHeygenErrorCode(err) {
|
||||
return classifyHeygenErrorResult(err).code;
|
||||
}
|
||||
|
||||
function classifyHeygenErrorResult(err) {
|
||||
const detail = heygenErrorDetail(err);
|
||||
const text = [err?.stderr, err?.stdout, err?.message, detail]
|
||||
.map((value) => textOf(value))
|
||||
.filter(Boolean)
|
||||
.join("\n");
|
||||
const lower = text.toLowerCase();
|
||||
|
||||
// Only ENOENT (spawn of a missing binary) or a shell's "command not found"
|
||||
// mean the CLI itself is absent. A bare "not found" would misfire on the CLI's
|
||||
// own resource errors (e.g. a stale voiceId → "voice not found"), whose message
|
||||
// embeds the `heygen ...` command line — sending users to reinstall a CLI they
|
||||
// just ran successfully. Keep this narrow.
|
||||
if (err?.code === "ENOENT" || lower.includes("command not found")) {
|
||||
return { code: "not_found", message: HEYGEN_NOT_FOUND_MESSAGE };
|
||||
}
|
||||
|
||||
if (
|
||||
lower.includes("unauthorized") ||
|
||||
lower.includes("unauthenticated") ||
|
||||
// \b401\b, not a bare "401" substring — otherwise request IDs (req-401abc),
|
||||
// URLs, and retry-after headers would misclassify as an auth failure.
|
||||
/\b401\b/.test(lower) ||
|
||||
lower.includes("not logged in") ||
|
||||
lower.includes("no api key") ||
|
||||
lower.includes("missing api key") ||
|
||||
lower.includes("invalid api key") ||
|
||||
lower.includes("login required") ||
|
||||
lower.includes("auth required") ||
|
||||
lower.includes("authentication required")
|
||||
) {
|
||||
return { code: "not_authenticated", message: HEYGEN_NOT_AUTHENTICATED_MESSAGE };
|
||||
}
|
||||
|
||||
const version = firstSemver(text);
|
||||
if (version && versionLessThan(version, HEYGEN_MIN_VERSION)) {
|
||||
return { code: "outdated", message: HEYGEN_OUTDATED_MESSAGE };
|
||||
}
|
||||
|
||||
if (
|
||||
lower.includes("rate limit") ||
|
||||
lower.includes("quota") ||
|
||||
lower.includes("insufficient credit") ||
|
||||
lower.includes("too many requests") ||
|
||||
lower.includes("throttled") ||
|
||||
/\b429\b/.test(lower)
|
||||
) {
|
||||
return { code: "rate_limited", message: detail };
|
||||
}
|
||||
|
||||
return { code: "other", message: detail };
|
||||
}
|
||||
|
||||
// reportHeygenFailure's callers (voice-provider.mjs, heygen-search.mjs) are
|
||||
// synchronous and several layers below the CLI's process.exit() calls, so
|
||||
// they can't await this tracking call themselves. Stash each attempt's
|
||||
// promise here so a caller closer to exit (resolve.mjs) can join it first —
|
||||
// same "awaited so a short-lived run flushes it" discipline telemetry.mjs's
|
||||
// track() already documents, just reachable from a sync call site.
|
||||
const pendingFailureTracking = new Set();
|
||||
// resolve.mjs is a single-shot CLI (one resolve per process), so one shared
|
||||
// consume-once slot is sufficient. If resolve becomes an in-process/concurrent
|
||||
// API, move this state into a per-resolve context before reusing that path.
|
||||
let pendingRemediation = null;
|
||||
|
||||
export function consumeHeygenRemediation() {
|
||||
const remediation = pendingRemediation;
|
||||
pendingRemediation = null;
|
||||
return remediation;
|
||||
}
|
||||
|
||||
export function reportHeygenFailure(err, context, trackEvent = track) {
|
||||
const { code, message } = classifyHeygenErrorResult(err);
|
||||
if (code === "not_found" || code === "outdated") {
|
||||
pendingRemediation = { code, message };
|
||||
}
|
||||
if (ACTIONABLE_MESSAGES.has(message)) {
|
||||
console.error(message);
|
||||
} else {
|
||||
console.error(`media-use: \`${context}\` failed: ${message}`);
|
||||
}
|
||||
try {
|
||||
const tracked = Promise.resolve(
|
||||
trackEvent("media_use_provider_error", { provider: "heygen", reason: code }),
|
||||
).catch(() => {});
|
||||
pendingFailureTracking.add(tracked);
|
||||
void tracked.finally(() => pendingFailureTracking.delete(tracked));
|
||||
return tracked;
|
||||
} catch {
|
||||
// Telemetry must never affect the provider failure path.
|
||||
return Promise.resolve();
|
||||
}
|
||||
}
|
||||
|
||||
// Awaits every provider-error track fired since the last flush, so a caller
|
||||
// about to process.exit() doesn't orphan one mid-request (both are separate,
|
||||
// non-keepalive HTTP connections with no ordering guarantee otherwise).
|
||||
// Never rejects: each tracked promise already swallows its own failure.
|
||||
export async function flushHeygenFailureTracking() {
|
||||
if (pendingFailureTracking.size === 0) return;
|
||||
await Promise.all(pendingFailureTracking);
|
||||
}
|
||||
|
||||
// Shared discovery/generation helper for the CLI-shelling providers (voice,
|
||||
// avatar-video): run a heygen JSON subcommand, report+classify on failure,
|
||||
// and hand the caller the classified reason via onError (used by callers that
|
||||
// need to distinguish e.g. not_authenticated from other failures).
|
||||
export function runHeygenJson(bin, argv, label, onError) {
|
||||
let out;
|
||||
try {
|
||||
out = execFileSync(bin, argv, {
|
||||
encoding: "utf8",
|
||||
timeout: 120000,
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
} catch (err) {
|
||||
reportHeygenFailure(err, `${bin} ${label}`);
|
||||
onError?.(classifyHeygenErrorCode(err));
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return JSON.parse(out);
|
||||
} catch {
|
||||
console.error(`media-use: \`${bin} ${label}\` returned non-JSON output`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function firstSemver(text) {
|
||||
const match = String(text || "").match(/\bv?(\d+)\.(\d+)\.(\d+)\b/);
|
||||
return match ? `${match[1]}.${match[2]}.${match[3]}` : null;
|
||||
}
|
||||
|
||||
export function versionLessThan(version, minimum) {
|
||||
const left = versionParts(version);
|
||||
const right = versionParts(minimum);
|
||||
if (!left || !right) return false;
|
||||
for (let i = 0; i < 3; i++) {
|
||||
if (left[i] < right[i]) return true;
|
||||
if (left[i] > right[i]) return false;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function heygenErrorDetail(err) {
|
||||
return textOf(err?.stderr) || textOf(err?.stdout) || err?.message || String(err);
|
||||
}
|
||||
|
||||
function textOf(value) {
|
||||
return value == null ? "" : String(value).trim();
|
||||
}
|
||||
|
||||
function versionParts(version) {
|
||||
const match = String(version || "").match(/^v?(\d+)\.(\d+)\.(\d+)$/);
|
||||
return match ? match.slice(1).map((part) => Number.parseInt(part, 10)) : null;
|
||||
}
|
||||
@@ -0,0 +1,301 @@
|
||||
import { strict as assert } from "node:assert";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
classifyHeygenError,
|
||||
classifyHeygenErrorCode,
|
||||
consumeHeygenRemediation,
|
||||
flushHeygenFailureTracking,
|
||||
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
|
||||
HEYGEN_NOT_FOUND_MESSAGE,
|
||||
HEYGEN_OUTDATED_MESSAGE,
|
||||
reportHeygenFailure,
|
||||
} from "./heygen-cli.mjs";
|
||||
|
||||
function captureFailureReport(err, context, trackEvent) {
|
||||
const originalError = console.error;
|
||||
const stderrCalls = [];
|
||||
console.error = (...args) => stderrCalls.push(args);
|
||||
try {
|
||||
if (trackEvent) {
|
||||
reportHeygenFailure(err, context, trackEvent);
|
||||
} else {
|
||||
reportHeygenFailure(err, context);
|
||||
}
|
||||
} finally {
|
||||
console.error = originalError;
|
||||
}
|
||||
return stderrCalls;
|
||||
}
|
||||
|
||||
test("classifies ENOENT-style missing heygen errors with install instructions", () => {
|
||||
const message = classifyHeygenError({ code: "ENOENT", message: "spawn heygen ENOENT" });
|
||||
|
||||
assert.equal(message, HEYGEN_NOT_FOUND_MESSAGE);
|
||||
});
|
||||
|
||||
test("classifies auth failures with login instructions", () => {
|
||||
const message = classifyHeygenError({ stderr: Buffer.from("Error: not logged in") });
|
||||
|
||||
assert.equal(message, HEYGEN_NOT_AUTHENTICATED_MESSAGE);
|
||||
});
|
||||
|
||||
test("classifies a real 401 as auth, but not a bare 401 substring in prose", () => {
|
||||
assert.equal(
|
||||
classifyHeygenError({ stderr: Buffer.from("HTTP 401 Unauthorized") }),
|
||||
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
|
||||
);
|
||||
// A request id that merely contains "401" must NOT read as an auth failure.
|
||||
const noise = classifyHeygenError({ stderr: Buffer.from("upload failed (request req-401abc)") });
|
||||
assert.notEqual(noise, HEYGEN_NOT_AUTHENTICATED_MESSAGE);
|
||||
});
|
||||
|
||||
test("classifies old heygen versions with update instructions", () => {
|
||||
const message = classifyHeygenError({
|
||||
stderr: Buffer.from("heygen v0.1.5 does not support --headers"),
|
||||
});
|
||||
|
||||
assert.equal(message, HEYGEN_OUTDATED_MESSAGE);
|
||||
});
|
||||
|
||||
test("does not misclassify a resource 'not found' error as a missing CLI", () => {
|
||||
// A stale voiceId makes `heygen voice speech create` fail with "voice not
|
||||
// found"; the error message embeds the `heygen ...` command line. This must
|
||||
// pass through as detail, not send the user to reinstall a working CLI.
|
||||
const message = classifyHeygenError({
|
||||
stderr: Buffer.from("Error: voice not found (id: stale-123)"),
|
||||
message: "Command failed: heygen voice speech create --voice stale-123",
|
||||
});
|
||||
|
||||
assert.notEqual(message, HEYGEN_NOT_FOUND_MESSAGE);
|
||||
assert.equal(message, "Error: voice not found (id: stale-123)");
|
||||
});
|
||||
|
||||
test("classifies a shell 'command not found' as a missing CLI", () => {
|
||||
const message = classifyHeygenError({ stderr: Buffer.from("bash: heygen: command not found") });
|
||||
|
||||
assert.equal(message, HEYGEN_NOT_FOUND_MESSAGE);
|
||||
});
|
||||
|
||||
test("passes through unrelated errors", () => {
|
||||
const message = classifyHeygenError({
|
||||
stderr: Buffer.from("rate limit exceeded"),
|
||||
message: "Command failed",
|
||||
});
|
||||
|
||||
assert.equal(message, "rate limit exceeded");
|
||||
});
|
||||
|
||||
test("classifies existing HeyGen failures with stable reason codes", () => {
|
||||
assert.equal(classifyHeygenErrorCode({ code: "ENOENT" }), "not_found");
|
||||
assert.equal(
|
||||
classifyHeygenErrorCode({ stderr: Buffer.from("HTTP 401 Unauthorized") }),
|
||||
"not_authenticated",
|
||||
);
|
||||
assert.equal(
|
||||
classifyHeygenErrorCode({ stderr: Buffer.from("heygen v0.1.5 is unsupported") }),
|
||||
"outdated",
|
||||
);
|
||||
assert.equal(classifyHeygenErrorCode({ stderr: Buffer.from("provider unavailable") }), "other");
|
||||
});
|
||||
|
||||
test("classifies rate-limit text case-insensitively", () => {
|
||||
assert.equal(
|
||||
classifyHeygenErrorCode({ stderr: Buffer.from("RATE LIMIT exceeded") }),
|
||||
"rate_limited",
|
||||
);
|
||||
});
|
||||
|
||||
test("classifies quota and insufficient-credit errors as rate limited", () => {
|
||||
for (const detail of ["Quota exhausted", "INSUFFICIENT CREDIT remaining"]) {
|
||||
assert.equal(classifyHeygenErrorCode({ stderr: Buffer.from(detail) }), "rate_limited");
|
||||
}
|
||||
});
|
||||
|
||||
test("classifies the literal 429 reason phrase and throttling language as rate limited", () => {
|
||||
for (const detail of ["Too Many Requests", "Error: throttled by upstream, retry later"]) {
|
||||
assert.equal(classifyHeygenErrorCode({ stderr: Buffer.from(detail) }), "rate_limited");
|
||||
}
|
||||
});
|
||||
|
||||
test("does not misclassify unrelated errors that share a word with the new phrasing", () => {
|
||||
// Shares "too many" with "too many requests" but is a distinct failure (fd
|
||||
// exhaustion, not a rate limit) — the match must require the full phrase.
|
||||
assert.equal(
|
||||
classifyHeygenErrorCode({ stderr: Buffer.from("Too many open file descriptors") }),
|
||||
"other",
|
||||
);
|
||||
});
|
||||
|
||||
test("classifies a bare 429 as rate limited without matching request IDs", () => {
|
||||
assert.equal(
|
||||
classifyHeygenErrorCode({ stderr: Buffer.from("HTTP 429 Too Many Requests") }),
|
||||
"rate_limited",
|
||||
);
|
||||
assert.equal(
|
||||
classifyHeygenErrorCode({ stderr: Buffer.from("request req-429abc failed") }),
|
||||
"other",
|
||||
);
|
||||
});
|
||||
|
||||
test("tracks not-found failures without changing actionable output", () => {
|
||||
const trackingCalls = [];
|
||||
const stderrCalls = captureFailureReport({ code: "ENOENT" }, "heygen asset search", (...args) =>
|
||||
trackingCalls.push(args),
|
||||
);
|
||||
|
||||
assert.deepEqual(stderrCalls, [[HEYGEN_NOT_FOUND_MESSAGE]]);
|
||||
assert.deepEqual(trackingCalls, [
|
||||
["media_use_provider_error", { provider: "heygen", reason: "not_found" }],
|
||||
]);
|
||||
});
|
||||
|
||||
test("records missing and outdated CLI remediation once", () => {
|
||||
consumeHeygenRemediation();
|
||||
captureFailureReport({ code: "ENOENT" }, "heygen audio sounds list", () => {});
|
||||
assert.deepEqual(consumeHeygenRemediation(), {
|
||||
code: "not_found",
|
||||
message: HEYGEN_NOT_FOUND_MESSAGE,
|
||||
});
|
||||
assert.equal(consumeHeygenRemediation(), null);
|
||||
|
||||
captureFailureReport(
|
||||
{ stderr: Buffer.from("heygen v0.1.5 does not support --headers") },
|
||||
"heygen audio sounds list",
|
||||
() => {},
|
||||
);
|
||||
assert.deepEqual(consumeHeygenRemediation(), {
|
||||
code: "outdated",
|
||||
message: HEYGEN_OUTDATED_MESSAGE,
|
||||
});
|
||||
assert.equal(consumeHeygenRemediation(), null);
|
||||
});
|
||||
|
||||
test("does not record non-install remediation", () => {
|
||||
consumeHeygenRemediation();
|
||||
for (const error of [
|
||||
{ stderr: Buffer.from("HTTP 401 Unauthorized") },
|
||||
{ stderr: Buffer.from("quota exhausted") },
|
||||
{ stderr: Buffer.from("provider unavailable") },
|
||||
]) {
|
||||
captureFailureReport(error, "heygen audio sounds list", () => {});
|
||||
assert.equal(consumeHeygenRemediation(), null);
|
||||
}
|
||||
});
|
||||
|
||||
test("tracks generic failures without including raw detail", () => {
|
||||
const trackingCalls = [];
|
||||
const stderrCalls = captureFailureReport(
|
||||
{ stderr: Buffer.from("private provider detail") },
|
||||
"heygen asset search",
|
||||
(...args) => trackingCalls.push(args),
|
||||
);
|
||||
|
||||
assert.deepEqual(stderrCalls, [
|
||||
["media-use: `heygen asset search` failed: private provider detail"],
|
||||
]);
|
||||
assert.deepEqual(trackingCalls, [
|
||||
["media_use_provider_error", { provider: "heygen", reason: "other" }],
|
||||
]);
|
||||
});
|
||||
|
||||
test("keeps failure output observable when telemetry is opted out", () => {
|
||||
const previousOptOut = process.env.HYPERFRAMES_NO_TELEMETRY;
|
||||
process.env.HYPERFRAMES_NO_TELEMETRY = "1";
|
||||
try {
|
||||
const stderrCalls = captureFailureReport({ code: "ENOENT" }, "heygen asset search");
|
||||
|
||||
assert.deepEqual(stderrCalls, [[HEYGEN_NOT_FOUND_MESSAGE]]);
|
||||
} finally {
|
||||
if (previousOptOut === undefined) {
|
||||
delete process.env.HYPERFRAMES_NO_TELEMETRY;
|
||||
} else {
|
||||
process.env.HYPERFRAMES_NO_TELEMETRY = previousOptOut;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test("keeps failure output observable when tracking throws synchronously", () => {
|
||||
const originalError = console.error;
|
||||
const stderrCalls = [];
|
||||
let thrown;
|
||||
console.error = (...args) => stderrCalls.push(args);
|
||||
try {
|
||||
try {
|
||||
reportHeygenFailure({ code: "ENOENT" }, "heygen asset search", () => {
|
||||
throw new Error("tracking failed");
|
||||
});
|
||||
} catch (err) {
|
||||
thrown = err;
|
||||
}
|
||||
} finally {
|
||||
console.error = originalError;
|
||||
}
|
||||
|
||||
assert.deepEqual(stderrCalls, [[HEYGEN_NOT_FOUND_MESSAGE]]);
|
||||
assert.equal(thrown, undefined);
|
||||
});
|
||||
|
||||
test("does not leave rejected tracking promises unhandled", () => {
|
||||
const moduleUrl = new URL("./heygen-cli.mjs", import.meta.url).href;
|
||||
const script = `
|
||||
import { reportHeygenFailure } from ${JSON.stringify(moduleUrl)};
|
||||
reportHeygenFailure(
|
||||
{ stderr: "provider unavailable" },
|
||||
"heygen asset search",
|
||||
() => Promise.reject(new Error("tracking failed")),
|
||||
);
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
`;
|
||||
const child = spawnSync(
|
||||
process.execPath,
|
||||
["--unhandled-rejections=strict", "--input-type=module", "--eval", script],
|
||||
{ encoding: "utf8", timeout: 5000 },
|
||||
);
|
||||
|
||||
assert.equal(child.error, undefined);
|
||||
assert.equal(child.signal, null);
|
||||
assert.equal(child.status, 0, child.stderr);
|
||||
assert.equal(child.stderr, "media-use: `heygen asset search` failed: provider unavailable\n");
|
||||
});
|
||||
|
||||
test("flushHeygenFailureTracking waits for a pending report before resolving", async () => {
|
||||
const events = [];
|
||||
let releaseTrack;
|
||||
const gate = new Promise((resolve) => {
|
||||
releaseTrack = resolve;
|
||||
});
|
||||
|
||||
// Mirrors the real call sites (voice-provider.mjs, heygen-search.mjs):
|
||||
// fire-and-forget, the return value is never awaited by the caller.
|
||||
reportHeygenFailure({ code: "ENOENT" }, "heygen voice speech", () =>
|
||||
gate.then(() => {
|
||||
events.push("track-settled");
|
||||
}),
|
||||
);
|
||||
|
||||
const flushed = flushHeygenFailureTracking().then(() => {
|
||||
events.push("flush-resolved");
|
||||
});
|
||||
|
||||
// Let several pending microtasks drain before releasing the gate, so this
|
||||
// proves flush is genuinely still waiting on the tracked promise -- not
|
||||
// merely that it hasn't had a tick yet.
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
assert.deepEqual(events, [], "flush must not resolve while the tracked promise is still pending");
|
||||
|
||||
releaseTrack();
|
||||
await flushed;
|
||||
|
||||
assert.deepEqual(
|
||||
events,
|
||||
["track-settled", "flush-resolved"],
|
||||
"flush must resolve only after the pending track settles, in that order",
|
||||
);
|
||||
});
|
||||
|
||||
test("flushHeygenFailureTracking resolves immediately when nothing is pending", async () => {
|
||||
await flushHeygenFailureTracking();
|
||||
});
|
||||
@@ -0,0 +1,45 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { HEYGEN_CLIENT_SOURCE_ARGV, reportHeygenFailure } from "./heygen-cli.mjs";
|
||||
|
||||
export function heygenSearch(subcommand, query, { type, limit = 5, minScore } = {}) {
|
||||
// execFileSync with an argv array (no shell), so query/type/etc. are passed as
|
||||
// literal arguments — no quoting tricks, no command injection. subcommand is a
|
||||
// hardcoded multi-word string (e.g. "audio sounds list"), split into tokens.
|
||||
// Tag the caller via the CLI's allowlisted attribution header (heygen >= v0.3.0).
|
||||
const args = [...HEYGEN_CLIENT_SOURCE_ARGV, ...subcommand.split(" "), "--query", query];
|
||||
if (type) args.push("--type", type);
|
||||
args.push("--limit", String(limit));
|
||||
// Server-side score floor. Honored by `audio sounds list`; the `asset search`
|
||||
// backend rejects it, so only audio providers pass minScore (see image-provider).
|
||||
if (minScore != null) args.push("--min-score", String(minScore));
|
||||
|
||||
let out;
|
||||
try {
|
||||
out = execFileSync("heygen", args, {
|
||||
encoding: "utf8",
|
||||
timeout: 15000,
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
} catch (err) {
|
||||
// Don't swallow a broken command / auth failure as "no results" — that turns
|
||||
// a typo or expired key into a silent dead end. Surface it, then give up.
|
||||
reportHeygenFailure(err, `heygen ${subcommand}`);
|
||||
return null;
|
||||
}
|
||||
|
||||
let parsed;
|
||||
try {
|
||||
parsed = JSON.parse(out);
|
||||
} catch {
|
||||
console.error(`media-use: \`heygen ${subcommand}\` returned non-JSON output`);
|
||||
return null;
|
||||
}
|
||||
if (parsed?.error) {
|
||||
const e = parsed.error;
|
||||
console.error(`media-use: \`heygen ${subcommand}\` error: ${e.message ?? JSON.stringify(e)}`);
|
||||
return null;
|
||||
}
|
||||
|
||||
const data = parsed?.data;
|
||||
return Array.isArray(data) && data.length > 0 ? data : null;
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user