# Resonant Garden — Handoff

A still painterly pond at dawn that **awakens through music** — live piano (mic) or a
MIDI file. Built in **ChucK + ChuGL 0.3.0**, single file (`main.ck`), audio + graphics
in one program.

---

## 0. Where it lives (read this first)
```
~/projects/ccrma-workshop/projects/resonant-garden/
```
**NOT** `~/projects/synopsis2025` — that's an unrelated PCB-detection project. Terminal
sessions sometimes launch from synopsis2025 by accident; all garden work is in the
ccrma-workshop path above.

- ChucK binary: `~/projects/ccrma-workshop/chuck/src/chuck`
- ChuGL chugin: `~/.chuck/lib/ChuGL.chug`
- ChuGL examples (API reference): `~/projects/ccrma-workshop/chugl/examples`
- ChuGL source (definitive API): `~/projects/ccrma-workshop/chugl/src`

## 1. Run
```sh
cd ~/projects/ccrma-workshop/projects/resonant-garden
./run.sh              # live mic — play piano to wake the garden
./run.sh keys         # live MIDI keyboard — play a digital piano in real time (keys=N picks input port; default 0)
./run.sh midi         # play bundled data/demo.mid (Mompou, "Secreto") — no mic, with sound
./run.sh midi=foo.mid # play your own .mid (project-relative or absolute)
./run.sh preview      # forced lit/warm scene + auto-ripples, no audio needed
./run.sh full         # fullscreen (combine: ./run.sh midi full)
```
Flags are forwarded by `run.sh` as ChucK colon-args (`main.ck:midi:full`). Dev-only flags:
`capture` (render one PNG to /tmp/garden_frame.png then quit), `seq` (with capture: save a
numbered burst to /tmp/seq for a motion clip), `capwait=N` (frames before capture fires),
`dawn=X` (pin the dawn level to preview the palette).

## 2. The concept / mappings (the "awakening")
Audio features drive a shared **feature bus** (file-scope globals) that the render loop reads:
| input | → bus → | visual |
|---|---|---|
| note onset | `g_onPitch/g_onAmp/g_onWrite` ring → `addRipple` | ripple; **pitch → x position** (low=left, high=right) |
| loudness | `g_brightness` | brightness / sparkle |
| silence | `g_brightness→0` | desaturates toward cool twilight |
| cumulative energy | `g_dawn` (render integrates) | palette warms, sun rises, stars fade |
| harmony | `g_harmony` (circle-of-fifths) | sky hue |

Three input engines write the SAME bus, so the render loop is identical for all:
- **`audioEngine()`** — mic → FFT → RMS/Centroid/Flux + chroma + adaptive onset detection.
- **`playMidi()`** — parses a .mid (per-track, k-way merge), feeds note-ons to the bus, AND
  synthesizes each note (a `Rhodey` FM-Rhodes voice → `JCRev` → dac) so you HEAR it.
  `midiEnvelope()` smooths brightness/harmony decay.
- **`playMidiKeyboard()`** — live `MidiIn`: opens a MIDI input port (`keys`/`keys=N`), drains
  note-ons in the canonical `min => now; while(min.recv(msg))` loop, and calls the SAME
  `midiNote()` the file path uses (ripple + brightness + harmony + audible Rhodes). Sporks
  `midiEnvelope()` only after the port opens; falls back to the mic if no device is found.

## 3. Scene (all in `main.ck`)
- **Sky** = big plane + custom WGSL shader (`SKY_WGSL`): vertical gradient, sun/moon
  (rises+warms with dawn), twinkling stars (fade at dawn), **drifting clouds** (fbm),
  **flocking birds** (`birdMark`: one wandering centroid + fixed V offsets + per-bird wander,
  in 2 depth layers — 8 near + 4 far/smaller/paler for parallax), **swaying forest** treeline
  (layered fbm canopy that sways). All in screen-space so features never land off-camera.
- **Pond** = plane + ripple shader (`RIPPLE_WGSL`): storage-buffer ripple field, reflects
  the live sky, sun glitter column (screen-space, under the sun), a **reflected boat
  silhouette** (mirrored below the boat's projected screen pos, sheared by the ripple field,
  bindings 5/6/7), and **foreground reeds/cattails** (`reedField`, screen-space, swaying,
  clustered at the L/R foreground edges, composited LAST as the nearest element). Extended to
  depth 16 + shifted +3 z so water fills the foreground (no sky-bleed band).
- **Boat** = custom low-poly canoe `Geometry` (11 verts / 18 tris across 5 stations: a curved
  sheer that rises to bow+stern and a curved rocker bottom — no longer a slab) + `FlatMaterial`
  silhouette + a triangular **sail** (child GMesh). Drifts, bobs, rolls; bobs more on note
  activity. Its world pos is projected to screen each frame (`cam.worldPosToScreenCoord`) to
  anchor the water reflection.
- **Motes** = `GPoints` with a soft round-dot texture (`data/dot.png`), drift upward.
- Output: `GG.bloom(true)` (sun/glints glow), `GG.outputPass().tonemap(0)` + **`.gamma(0)`**
  (see gotcha #1).

## 4. Files
- `main.ck` — everything (~666 lines): config+flags, feature bus, helpers, audioEngine,
  MIDI engine+synth, shaders, scene, render loop, capture.
- `audio_test.ck` — headless DSP check: `chuck --silent audio_test.ck` (synthetic SinOsc →
  onsets fire, pitch rises with frequency, harmony settles).
- `run.sh` — launcher (forwards flags).
- `data/demo.mid` (Mompou "Secreto"), `data/bwv772.mid` (Bach), `data/dot.png` (mote sprite).
- `README.md` — user-facing run/tuning docs.

## 5. ⚠️ Gotchas (hard-won — will save you hours)
1. **ChuGL gamma is a 0/1 TOGGLE, and we run with it OFF.** The scene renders linear; the
   OutputPass normally applies gamma 2.2 (sRGB) → washed-out pastel. We call
   `GG.outputPass().gamma(0)` to display the deep painterly palette directly. The palette
   values are therefore tuned for LINEAR display — if you ever turn gamma back on, the whole
   palette must be re-graded.
2. **Offscreen capture must match the window.** `GG.scenePass().colorOutput(capTex)` captures
   PRE-bloom AND linear → does NOT look like the window. We instead add a 2nd `OutputPass`
   tapping `GG.bloomPass().colorOutput()` with `tonemap(0)+gamma(0)` into capTex. Keep the
   capture pass's settings identical to the window's output pass or captures lie.
3. **WGSL lives inside a ChucK `"..."` string — never put a `"` in the shader, even in a
   comment.** A `"` (e.g. a comment saying `"v"`) silently terminates the ChucK string and
   produces a confusing `syntax error` at the string's start line. Use single quotes / none.
4. **ChucK `Centroid.fval(0)` is normalized [0,1] (fraction of Nyquist), never Hz.**
   `pitchHeight` converts it back to Hz before the musical log-frequency map.
5. **`FLUX_FLOOR` matters a lot.** Flux is bounded ~[0,1.4]; the old default 0.55 suppressed
   ~80% of real onsets. Now 0.08. Onsets are also gated by `rmsSmooth > NOISE_FLOOR` to avoid
   the numerical instability of Flux on near-silence.
6. **zsh ate `./run.sh midi # comment`** → `unknown file attribute: ~` (interactive zsh
   doesn't treat `#` as a comment; `(~95s)` parsed as a glob qualifier). Run commands without
   trailing comments, or `setopt interactive_comments`.
7. **Every declared WGSL `@group(1)` uniform must be USED** or the GPU prunes it and the bind
   group mismatches → validation error. Workaround: `let _ = uniformName;`.
8. **The live GLFW window can open on another macOS Space** (esp. fullscreen) — swipe via
   Mission Control. That's why the offscreen capture + `screencapture` exist.

## 6. Verifying changes without a mic/piano (the dev loop)
- **Audio/DSP:** `chuck --silent audio_test.ck` (headless, instant).
- **Visuals:** render an offscreen PNG and Read it:
  `chuck --chugin:~/.chuck/lib/ChuGL.chug main.ck:preview:capture:dawn=0.6` → `/tmp/garden_frame.png`.
  Use `dawn=X` to preview any point of the sunrise; `preview` forces a lit scene + auto-ripples.
- **End-to-end (real path, no mic):** `main.ck:midi:capture:capwait=3600` runs the actual
  pipeline driven by the MIDI file; the console monitor (`rms/pitch/harm/dawn/onsets`) proves
  the bus is being driven; `capwait=N` grabs a developed frame (dawn rises ~full by ~95s).
- **Motion clip:** `main.ck:midi:capture:seq:capwait=3600` saves a burst to `/tmp/seq`, then
  assemble with ffmpeg/PIL (ffmpeg is at /opt/homebrew/bin/ffmpeg; PIL is available). See §8.

## 7. Tuning knobs (top of `main.ck`, or as noted)
- `NOISE_FLOOR` (#1) — gates brightness AND onsets; set just above silent-room rms.
- `RMS_GAIN`, `FLUX_FLOOR` — loudness sensitivity / onset floor.
- `DAWN_RATE`, `DAWN_DECAY` — how fast playing warms the scene / settles in silence.
- `SYNTH_GAIN` (in the MIDI section) — MIDI synth volume.
- Palette `COLD_*/WARM_*` vec3s, `desat()` cool-tint target, pond reflection mix, glint/sun-
  column strength — all graded for gamma-off linear display.

## 8. Status / next steps
**Working & verified:** compiles & runs clean (mic + MIDI + keys + preview + capture paths);
audio test green (24 onsets); MIDI drives the full night→dawn arc with sound; scene has the
canoe (+ water reflection), flocking birds, swaying forest, drifting clouds, ripples, reeds, motes.

**Added & verified this session** (compile + preview capture + MIDI end-to-end + adversarial review):
- **Live MIDI keyboard** (`keys`/`keys=N`, `playMidiKeyboard()`): real-time digital-piano input
  into the shared bus. Verified the open/fallback path (gracefully falls to mic when no device);
  **live note-triggering still needs a physical/virtual MIDI device** (none was connected — see
  `chuck --probe`; this is the one untested-on-hardware piece).
- **Boat reflection** in the pond (`RIPPLE_WGSL` bindings 5/6/7 + per-frame screen projection).
- **Flocking birds** (coherent V around a wandering centroid, 2 depth layers) in `SKY_WGSL`.
- **Reeds/cattails** foreground (`reedField`, composited last) for depth.
- **Retuned hull** (11-vert curved-sheer canoe; the old slab is gone).

**Latest demo:** `~/Desktop/resonant_garden_midi.{mp4,gif}` — a ~4s MIDI-driven clip of the
current livelier scene (boat, birds, swaying forest, clouds, ripples) at mid-dawn. To
regenerate after changes: render `main.ck:midi:capture:seq:capwait=3600` (burst → `/tmp/seq`),
then with PIL sort `g_*.png` numerically → resize → padded `f%03d.png` in `/tmp/seqp`, then
`ffmpeg -y -framerate 12 -i /tmp/seqp/f%03d.png -c:v libx264 -pix_fmt yuv420p out.mp4`.

**Candidate next steps:**
- **Test the live keyboard on real hardware** — plug in a MIDI keyboard (or a virtual port like
  the macOS IAC Driver), `chuck --probe` for its port, then `./run.sh keys` (or `keys=N`).
- The real-world milestone still untested: live acoustic **piano into a mic** with NOISE_FLOOR/
  RMS_GAIN/FLUX_FLOOR tuned to the room.
- Reflection polish (optional): it currently ignores the boat's small roll/yaw/pitch (axis-aligned
  mast/beam projection) — a deliberate impressionistic simplification; transform the offsets by the
  boat's rotation if you want the reflection to track tilt exactly. Also possible: reflect the hull
  *color* rather than a flat silhouette.
- Richer bird behavior still possible (true boids via a storage buffer of bird state, vs the
  current stateless shader-side flock).

## 9. Notes
- No git commits made this session (the ccrma-workshop repo is rooted oddly at $HOME; commit
  deliberately if desired).
- Memory file `project-resonant-garden.md` records the project location + run flags.
