Extract the store engines' shared core

A store engine reads a WarpEngine catalog and writes some host's own game
library format. Only that last step is about the host: the catalog, the
release selection, the machine matching, the state and the HTTP were the
same code in every engine, and until now there was one engine, so they
lived inside it.

A second engine — RetroArch — made the seam visible, so the shared part
moves here as a single file both engines fetch at install time. An adapter
now supplies three things: a DEFAULT_CONFIG describing its host, an
accept() callback deciding which catalog entries that host can run, and
the code that writes the host's library format.

Two generalisations came out of having two hosts rather than one:

- `resolve_for_arch` becomes `resolve_for_host`. Batocera only ever
  answered "linux", so architecture was the only variable; a RetroArch
  host can be Windows, macOS or Android, and that decides such things as
  what a libretro core file is called. Keys are now os-arch, arch, os, *.
- state records are keyed by `scope`, not by ES `system`. For Batocera the
  two are the same string, so an installed box keeps working without a
  migration — `record_scope()` falls back to `system` for records written
  before this existed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-17 17:59:53 +02:00
co-authored by Claude Opus 5
commit 5b02633fb8
3 changed files with 669 additions and 0 deletions
+112
View File
@@ -0,0 +1,112 @@
# warpstore — the shared core of the WarpEngine store engines
A **store engine** reads a [WarpEngine](https://git.teletypegames.org/engines/warp_engine)
catalog and writes some host platform's own game library format. Only that last
step differs between hosts. Everything before it — talking to the API, picking
which release to install, matching the machine, remembering what was put where —
is this module.
```
┌──────────────────────────────┐
│ warpstore.py │
│ catalog · releases · host │
│ state · config · HTTP │
└──────────────┬───────────────┘
┌──────────────┴───────────────┐
▼ ▼
warp-engine-batocera-store warp-engine-retroarch-store
ROM folders + gamelist.xml .lpl playlists + thumbnails
│ │
▼ ▼
EmulationStation RetroArch
```
Single file, Python 3 standard library only. The hosts range from a Batocera box
(`python3`, no `pip`) to a desktop, and neither may be asked to install anything.
An engine's installer drops `warpstore.py` next to the adapter script and the
adapter does `import warpstore`.
## What an adapter has to supply
Three things, and nothing else:
1. **A `DEFAULT_CONFIG`** describing its host — where things live, which catalog
platform maps to what.
2. **An `accept(spec, sw)` callback** deciding which catalog entries that host can
run, and how they are grouped.
3. **The code that writes the host's library format.**
```python
import warpstore as ws
def accept(spec, sw):
"""Return (scope, extras), or raise ws.Skip(reason) to reject the title."""
if spec["system"] not in available_systems:
raise ws.Skip(f"no '{spec['system']}' ROM folder on this box")
return spec["system"], {"system": spec["system"]}
cfg = ws.load_config(path, DEFAULT_CONFIG, required=("paths.subfolder",))
ws.init(path)
games, skipped = ws.select_games(cfg, ws.fetch_catalog(cfg, use_cache=True), accept)
```
`scope` is how the host groups its library — a Batocera system, a RetroArch
playlist. It is what `state.json` is keyed by (`<scope>:<name>`), so two hosts
can carry a game of the same name without colliding.
## What is in here
| Area | Functions |
|---|---|
| Logging | `set_tag`, `set_verbose`, `log`, `debug`, `die` |
| Files | `load_json`, `write_json`, `write_atomic`, `within` |
| Config | `deep_merge`, `load_config` |
| Store home | `init`, then `HOME`, `CONFIG_PATH`, `STATE_PATH`, `CATALOG_CACHE` |
| HTTP | `http_get`, `http_download`, `api_url`, `download_url`, `user_agent` |
| Host | `machine`, `host_os`, `host`, `resolve_for_host` |
| Catalog | `fetch_catalog`, `select_games`, `pick_release`, `asset_basename`, `download_image` |
| State | `load_state`, `save_state`, `game_key`, `record_scope`, `records_by_scope`, `match_keys`, `limit_to_names` |
Three of them carry decisions worth knowing about.
**`resolve_for_host(value, host)`** — a config value that may depend on the
machine. A plain string is the same everywhere (that is what a `cartridge` is:
data for an emulator). Where it differs, the value is a map and the most specific
key wins: `{"linux-aarch64": …, "aarch64": …, "linux": …, "*": …}`. With no
matching key and no `*` the answer is `None`, and the caller reports the title as
skipped rather than installing something that cannot run.
**`within(path, root)`** — every delete a store performs is guarded by this. A
store may only remove files from the subtree it owns, never from the user's own
library and never from another store's.
**`write_atomic`** — nothing is ever written in place. A store interrupted
mid-sync would otherwise leave a half-written playlist or gamelist, which is
worse than an old one.
## State
`state.json`, version 2, keyed `<scope>:<name>`:
```json
{
"version": 2,
"installed": {
"c64:blessingofra": { "name": "blessingofra", "scope": "c64", "asset": "blessingofra-2.0.0.prg", "…": "…" }
}
}
```
`record_scope()` falls back to a record's `system` field, which is what the
Batocera store wrote before this module existed — and there the two were the same
string, so an installed box keeps working with no migration. A `version: 1` file,
keyed by bare software name, is re-keyed on first run.
## Users
- [`warp-engine-batocera-store`](https://git.teletypegames.org/tools/warp-engine-batocera-store) — EmulationStation ROM folders, `gamelist.xml`, Ports launchers
- [`warp-engine-retroarch-store`](https://git.teletypegames.org/tools/warp-engine-retroarch-store) — RetroArch `.lpl` playlists and thumbnail folders
Both pin nothing: they fetch `warpstore.py` from this repository's `master` at
install time. A change here therefore reaches every store on its next install —
which is the point, and also the reason to keep the surface above small.