stores/warp-engine-batocera-store
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give the store its own EmulationStation menu entry

Browse Source 
The games were scattered across the box's own systems — a folder inside
c64, another inside tic80, the launchers inside Ports, and the sync a
fourth entry somewhere else. There was no one place where the store
appeared, and living in the box's folders is what forced the gamelist
merging, the backups and every "never step outside your subfolder" check.

Now the store gets a folder of its own, /userdata/roms/<subfolder>, with a
directory per platform, and an es_systems_<id>.cfg overlay declares one ES
system per platform under a shared <group> — which EmulationStation draws
as a single carousel entry holding a folder per system. The gamelists are
ours, the box's systems are untouched, and the sync is startable from
inside the entry: Store -> "Update <store>".

Nothing about emulators is hardcoded. The launch command, the extensions,
the platform, the theme and the emulator list are copied off the box's own
es_systems.cfg — a file Batocera generated for this image, so it lists
exactly what got built into it. %SYSTEM% is resolved to the source system
name on the way, since ES would otherwise substitute ttg-c64, which
configgen has never heard of.

The old behaviour is emulationstation.menu.mode: "merge", and switching
either way migrates: the layout the files are in is recorded in
state.json, and a mismatch removes the old one and downloads again rather
than shuffling files half-way. purge follows that record too, so an
uninstall after an upgrade whose first sync never ran still finds the
games where they actually are.

Every path now goes through a Layout, so the two modes cannot drift apart.
Also: a gamelist or es_systems file whose content has not changed is not
rewritten at all, and the tags ES owns — favourite, playcount, lastplayed
— are carried over onto the nodes we rewrite, which merge mode used to
lose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
mr.zero
2026-08-23 12:57:25 +02:00
co-authored by Claude Opus 5
parent e2551d5fd3
commit 3b72cf81cc
5 changed files with 987 additions and 172 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+174 -60
View File
@@ -1,9 +1,9 @@
# warp-engine-batocera-store — a Batocera store engine for WarpEngine sites
Pulls games from a [WarpEngine](https://git.teletypegames.org/tools/warp_engine)
catalog straight into a Batocera box's ROM folders, with EmulationStation
metadata and box art. Triggered from the **Ports** menu on the device, or from
the CLI over SSH.
catalog straight onto a Batocera box, with EmulationStation metadata and box art,
and gives the store **its own entry in the EmulationStation menu**. Started from
that entry on the device, or from the CLI over SSH.
This repository is the engine only. It knows the WarpEngine API but nothing
about any particular site: the host, the store's name and where its games land
@@ -31,18 +31,65 @@ The reference store is
[`ttg-batocera-store`](https://git.teletypegames.org/stores/ttg-batocera-store).
```
Ports ▸ "<store name>"
"<store name>" ▸ Store ▸ "Update <store name>"
│
├─ GET /api/software the whole catalog
├─ keep platforms this box can run c64 → c64, tic80 → tic80
├─ pick the newest non-dev release that carries the right asset
├─ GET /api/download?path=<asset> .prg / .tic into the ROM folder
├─ GET /api/download?path=<asset> .prg / .tic into the store's folder
├─ GET <software.imageUrl> box art
└─ merge gamelist.xml title, desc, author, image
├─ write gamelist.xml title, desc, author, image
└─ write es_systems_<id>.cfg the store's menu entry
```
Python 3 standard library only — Batocera ships python3 and no pip.
## The store's own menu entry
Everything the store installs lives in one folder of its own,
`/userdata/roms/<subfolder>`, with a directory per platform inside it. An
`es_systems_<id>.cfg` in EmulationStation's user config declares one ES system
per platform, all carrying the same `<group>`, and ES turns a group into a single
carousel entry holding a folder per system:
```
main carousel: … ▸ Teletype Games ▸ ┌ Commodore 64
├ TIC-80
├ Ports
└ Store → "Update Teletype Games"
```
Nothing about emulators is hardcoded here. The engine reads the box's own
`/usr/share/emulationstation/es_systems.cfg` and copies the launch command, the
extensions, the platform, the theme and the emulator list off the system it is
borrowing — a file Batocera generated for this particular image, so it lists
exactly the emulators the box has. The one thing that is rewritten is `%SYSTEM%`:
ES replaces it with the name of the system being launched, which for us would be
`ttg-c64`, a name configgen has never heard of. The box's own system name goes in
instead, and a game of ours is launched exactly as the box would launch it, with
the emulator the user configured for that system.
Three details worth knowing, all of them EmulationStation's:
- **The menu entry is declared, not implied.** ES would invent the group parent
by itself, but then its theme folder would be the group's name, which no theme
has — and `HideUniqueGroups` (on by default) dissolves a group that has a
single system in it unless a system by that name exists. The declared parent
fixes both: it carries a `<theme>` (`ports` by default, since every theme has
that one) and keeps the entry in place on a box that only has c64 games.
- **An empty system does not show.** Only the platforms that actually have
something installed become folders, so the entry is never full of dead ends.
- **A restart is needed for a new system.** The engine restarts ES itself after a
sync that added one.
The store id names the entry. If the box already has a system by that name — a
store called `nes`, say — the entry is called `<id>-store` instead, and the log
says so: an overlay whose `<name>` matches an existing system would *modify that
system* rather than add one.
For the old behaviour — games inside the box's own systems, under a subfolder,
merged into the box's gamelists — set `emulationstation.menu.mode` to `merge`.
## Installing a store
The installer takes the store as a parameter, so it works with any config:
@@ -60,9 +107,9 @@ ssh root@batocera 'STORE_CONFIG=/tmp/my-config.json /tmp/warp-engine-batocera-st
```
It reads `store.id` from the config, installs the engine and the config into
`/userdata/system/batocera-store/<id>/`, writes a `<id>-store` launcher and a
Ports entry, and runs the first sync. Restart EmulationStation
(`batocera-es-swissknife --restart`) to see the games.
`/userdata/system/batocera-store/<id>/`, writes a `<id>-store` launcher, and runs
the first sync — which is what creates the menu entry. Restart EmulationStation
(`batocera-es-swissknife --restart`) to see it.
It installs two files: `store.py` from this repository and `warpstore.py` from
the [shared core](https://git.teletypegames.org/engines/warpstore). They have to
@@ -73,6 +120,23 @@ Installer knobs, all environment variables: `STORE_CONFIG` (required),
`ENGINE_RAW_BASE`, `WARPSTORE_RAW_BASE`, and `WARPSTORE_SRC` to install a local
`warpstore.py` instead of downloading it.
## Upgrading a store that was installed before the menu entry
Engine 4.0 is the layout change, so the first sync after the upgrade moves the
store off the box's own systems. It does that by **removing the old install and
downloading again**, not by carrying files across: the ROMs, box art and port
payloads go, our nodes come out of the box's gamelists, the empty folders are
swept, the Ports entry is taken away — and then the sync fills the store's own
folder. The catalog is small enough that this is minutes, and there is no
half-moved state to end up in.
What is kept: your gamelists (only our nodes go — play counts and favourites of
your own games are untouched), and the `gamelist.xml.<id>-backup` copies, which
are listed at the end of the run for you to delete or keep.
`-n` shows the whole thing without doing any of it. To stay on the old layout,
set `emulationstation.menu.mode` to `merge` before the first sync.
## Writing a store repository
Four small files:
@@ -92,7 +156,6 @@ my-batocera-store/
set -euo pipefail
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/stores/warp-engine-batocera-store/raw/branch/master}"
export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-batocera-store/raw/branch/master/config.json}"
export BATOCERA_PORT_NAME="My Store"
curl -fsSL "$ENGINE_RAW_BASE/install.sh" | sh
```
@@ -105,13 +168,15 @@ curl -fsSL "$ENGINE_RAW_BASE/uninstall.sh" | sh
```
Pick a `store.id` and a `paths.subfolder` nobody else uses — they are what keeps
two stores on the same box from deleting each other's games.
two stores on the same box out of each other's way, and the id is what names the
menu entry and its `es_systems` file.
## Use
**On the device.** *Ports ▸ "\<store name\>"*. The script downloads anything
new, then restarts EmulationStation so the games show up. Ports scripts get no
console in Batocera, so the output goes to `store.log` in the store home.
**On the device.** *"\<store name\>" ▸ Store ▸ "Update \<store name\>"*. It
downloads anything new, then restarts EmulationStation so the games show up. ES
gives a launched script no console, so the output goes to `store.log` in the
store home.
**Over SSH**, through the launcher the installer wrote:
@@ -119,12 +184,12 @@ console in Batocera, so the output goes to `store.log` in the store home.
S=/userdata/system/batocera-store/example-store
$S list # compatible catalog entries; * installed, ^ update available
$S sync # download everything new, refresh gamelists
$S sync # download everything new, refresh gamelists and the menu
$S sync blessingofra # just one title (never prunes)
$S -n sync # dry run
$S remove c64:demo # uninstall (bare name works too)
$S purge # uninstall everything this store installed
$S config # effective configuration
$S config # effective configuration, and where the menu entry lives
```
Global flags go **before** the subcommand: `$S --roms-root /tmp/roms sync`.
@@ -136,12 +201,12 @@ ssh root@batocera 'curl -fsSL https://git.teletypegames.org/stores/warp-engine-b
```
or `./uninstall.sh` from a checkout on the box. It removes the games first,
through the engine — `purge` — and only then the Ports entry, the launcher and the
store home: the shell script has no way of knowing which ROMs, box art, Ports
payloads and gamelist entries were the store's, and the engine's `state.json`
does. If the engine cannot finish — an unmounted ROMs root, say — nothing else is
touched, so you are never left with the files but not the engine that knows about
them.
through the engine — `purge` — and only then the launcher, any Ports entry and
the store home: the shell script has no way of knowing which ROMs, box art, port
payloads, gamelist entries and ES systems were the store's, and the engine's
`state.json` does. If the engine cannot finish — an unmounted ROMs root, say —
nothing else is touched, so you are never left with the files but not the engine
that knows about them.
| Environment variable | Meaning |
|---|---|
@@ -151,11 +216,19 @@ them.
| `KEEP_HOME` | `1` to keep the store home (state, catalog cache, log) for a reinstall |
| `BATOCERA_STORE_ROOT`, `BATOCERA_PORTS_DIR`, `BATOCERA_PORT_NAME` | as for the installer |
What it deliberately leaves behind:
In the store's own layout an uninstall is total: the store's ROM folder, its
gamelists and its `es_systems_<id>.cfg` all go, and the box is left as it was.
In `merge` mode there is more to be careful about, and these stay behind:
- **ROMs you put in the store's subfolder yourself.** Directories are removed only when empty, so one stray file keeps its directory. `ports/.data` itself also stays: another store may still keep a payload there.
- **your gamelists.** They belong to the box: our `<game>` and `<folder>` nodes go, the rest — your own games, their play counts and favourites — is written back untouched. A gamelist we *created* and that is left empty is deleted, since it was never yours.
- **`gamelist.xml.<store id>-backup`**, listed at the end of the run: your gamelists as we first found them. Deleting them is your call.
- **ROMs you put in the store's subfolder yourself.** Directories are removed
only when empty, so one stray file keeps its directory. `ports/.data` itself
also stays: another store may still keep a payload there.
- **your gamelists.** They belong to the box: our `<game>` and `<folder>` nodes
go, the rest — your own games, their play counts and favourites — is written
back untouched. A gamelist we *created* and that is left empty is deleted,
since it was never yours.
- **`gamelist.xml.<store id>-backup`**, listed at the end of the run: your
gamelists as we first found them. Deleting them is your call.
A dry run cannot predict which directories will end up empty, so it lists the
files but not the directory removals.
@@ -178,18 +251,38 @@ curl -fsSL https://git.teletypegames.org/engines/warpstore/raw/branch/master/uni
├── config.json the store that selected it
└── state.json, catalog.json, store.log
/userdata/roms/c64/example/ blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
/userdata/roms/c64/example/images/ blessingofra.png, …
/userdata/roms/c64/gamelist.xml our entries merged in
/userdata/roms/ports/Example Store.sh
/userdata/system/configs/emulationstation/es_systems_example.cfg the menu entry
/userdata/roms/example/ the store's own ROM folder
├── c64/
│ ├── blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
│ ├── images/blessingofra.png, …
│ └── gamelist.xml
├── ports/
│ ├── <name>.sh the launchers ES lists
│ ├── images/<name>.png
│ ├── .data/<name>/ the unpacked programs
│ └── gamelist.xml
└── store/
├── update.sh the sync, as the menu starts it
└── gamelist.xml
```
Everything a store installs lives under its own `paths.subfolder`, so a prune
can never reach ROMs you put there yourself — or another store's games. The
first time a `gamelist.xml` is touched it is copied to
`gamelist.xml.<store id>-backup`; on every merge only `<game>`/`<folder>` nodes
under that store's subfolder are rewritten, so your own entries keep their play
counts, favourites and scraped media, and two stores can share one gamelist.
The box's own system folders are not touched at all, which is what makes a prune
safe: nothing outside the store's folder can be reached, and inside it only what
`state.json` says we installed is ever deleted — a ROM you drop in there
yourself keeps its file *and* its gamelist entry.
In `merge` mode the same content goes into `<roms_root>/<system>/<subfolder>/`
and our nodes are merged into the box's `gamelist.xml`. The first time one is
touched it is copied to `gamelist.xml.<store id>-backup`; on every merge only
`<game>`/`<folder>` nodes under that store's subfolder are rewritten, so your own
entries keep their play counts, favourites and scraped media, and two stores can
share one gamelist.
Either way, the tags EmulationStation owns — `favorite`, `playcount`,
`lastplayed`, `rating` — are carried over onto the node we rewrite. A favourite
stays a favourite across a sync.
## Configuration
@@ -197,15 +290,22 @@ counts, favourites and scraped media, and two stores can share one gamelist.
| Key | Default | Meaning |
| --- | --- | --- |
| `store.id` | `warp` | slug: names the store home, log prefix and gamelist backup |
| `store.name` | `WarpEngine Store` | display name for the Ports entry |
| `store.id` | `warp` | slug: names the store home, the log prefix, the menu entry and its `es_systems` file |
| `store.name` | `WarpEngine Store` | display name of the menu entry |
| `store.base_url` | — | WarpEngine host |
| `store.api.catalog` | `/api/software` | catalog endpoint, if the engine is mounted elsewhere |
| `store.api.download` | `/api/download` | download endpoint |
| `paths.roms_root` | `/userdata/roms` | where systems live |
| `paths.subfolder` | `warp` | this store's subfolder inside each system |
| `emulationstation.folder_name` | `WarpEngine Store` | display name of that folder in ES |
| `paths.subfolder` | `warp` | the store's own folder inside `roms_root` (in `merge` mode: its subfolder inside each system) |
| `emulationstation.menu.mode` | `system` | `system`: the store gets its own menu entry. `merge`: install into the box's own systems |
| `emulationstation.menu.name` | `null` → `store.name` | label of the menu entry |
| `emulationstation.menu.theme` | `ports` | theme folder the entry borrows its logo from |
| `emulationstation.menu.updater` | `true` | an *Update …* entry inside the menu that runs the sync |
| `emulationstation.menu.labels` | `{}` | per-system folder labels; a system not listed keeps the box's own `<fullname>` |
| `emulationstation.ports_entry` | `null` → `false` in `system` mode, `true` in `merge` mode | also write a Ports entry that runs the sync |
| `emulationstation.folder_name` | `WarpEngine Store` | `merge` mode only: display name of our folder inside the box's gamelist; `null` hides it |
| `emulationstation.restart` | `true` | restart EmulationStation after a change |
| `emulationstation.config_dir` | `null` → `/userdata/system/configs/emulationstation` | where ES keeps its user config |
| `catalog.statuses` | `["released", "archived"]` | catalog `status` values to install |
| `catalog.owner_id` | `null` | restrict to one publisher (`/api/software?owner_id=`) |
| `catalog.only` / `catalog.exclude` | `[]` | software-name allow / deny lists |
@@ -214,16 +314,24 @@ counts, favourites and scraped media, and two stores can share one gamelist.
| `behavior.timeout` | `30` | HTTP timeout, seconds |
| `behavior.insecure` | `false` | skip TLS verification (self-hosted test instances) |
`BATOCERA_ES_CONFIG_DIR` and `BATOCERA_ES_SYSTEMS_CFG` override the two
EmulationStation paths from the environment, which is what makes a staging root
testable off a real box.
### Platform mapping
A catalog platform is installable when its release asset is a file the Batocera
system can boot directly:
A catalog platform is installable when its release asset is a file a system this
box has can boot directly:
| Catalog platform | Asset kind | Batocera system | Extension |
| --- | --- | --- | --- |
| `c64` | `cartridge` | `c64` (VICE) | `.prg` |
| `tic80` | `cartridge` | `tic80` | `.tic` |
"Has" means declared in the box's own `es_systems.cfg` — an emulator that got
built into this image. (A ROM folder is a poorer test: it can be there with
nothing behind it. The folders are the fallback when the file cannot be read.)
The other WarpEngine platforms (`ebitengine`, `love`, `godot`, `bevy`,
`phaser`) ship `html` and per-OS zip archives, not a ROM a Batocera system
launches, so they are not mapped by default. Adding one is a config edit:
@@ -234,9 +342,10 @@ launches, so they are not mapped by default. Adding one is a config edit:
### Native games as Ports
A cartridge is booted by an emulator, so it goes into that system's ROM folder.
A native build is a program: Batocera runs Linux, and its **Ports** system takes
`.sh` launchers. Set `install` to `port` and point `system` at `ports`:
A cartridge is booted by an emulator, so it goes to the folder of the system that
boots it. A native build is a program: Batocera runs Linux, and its **Ports**
system takes `.sh` launchers. Set `install` to `port` and point `system` at
`ports`:
```json
"platforms": {
@@ -253,15 +362,15 @@ A native build is a program: Batocera runs Linux, and its **Ports** system takes
The zip is unpacked and a launcher written:
```
/userdata/roms/ports/<subfolder>/<name>.sh the launcher ES lists
/userdata/roms/ports/<subfolder>/images/<name>.png box art
/userdata/roms/ports/.data/<subfolder>/<name>/ the unpacked program
/userdata/roms/<subfolder>/ports/<name>.sh the launcher ES lists
/userdata/roms/<subfolder>/ports/images/<name>.png box art
/userdata/roms/<subfolder>/ports/.data/<name>/ the unpacked program
```
The payload sits under `.data`, whose leading dot is what hides it from
EmulationStation — the launchers themselves are in a normal subfolder, scanned
like any other system's. Both paths carry the store's subfolder, so an uninstall
or a prune can only ever reach this store's own files.
EmulationStation — only the `.sh` next to it is meant to be listed. `system:
"ports"` still names where the launcher *definition* comes from: our own ports
system copies the box's, which is `sh`, so the launcher is simply run.
The launcher `cd`s into the directory holding the executable before running it:
a game loads its `assets/` relative to the working directory. Python's zip
@@ -297,26 +406,31 @@ The engine reports the architecture it detected — `x86_64`, `aarch64`, `armhf`
entry for the running machine and no `*`, the title is skipped and says so:
`skipped bevydemo: bevy has no asset kind for aarch64`.
A platform is also skipped when the box has no ROM folder for its system —
`list` reports that as `skipped <name>: no '<system>' ROM folder`.
A platform is also skipped when the box does not have its system —
`list` reports that as `skipped <name>: this box has no '<system>' system`.
## How a sync stays safe next to EmulationStation
EmulationStation keeps gamelists in memory and writes them back when it exits,
so a merge done while it runs can be clobbered. Launched from the Ports menu
so a merge done while it runs can be clobbered. Launched from the menu
(`sync-from-es`), the engine therefore downloads first, then hands the gamelist
merge to a detached `apply-gamelists --wait-pid <es-pid>` child that waits for
the old ES process to die before writing. From SSH, with no ES running, the
merge happens inline.
Re-running a sync is idempotent: files already present are left alone, and a
release bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new
one. `GET /api/download` is used rather than `/file/`, so downloads count
The `es_systems_<id>.cfg` needs none of that: ES reads it and never writes it,
so it is written straight away — it just does not take effect until ES restarts.
Re-running a sync is idempotent: files already present are left alone, a release
bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new one, and a
gamelist or `es_systems` file whose content has not changed is not rewritten at
all. `GET /api/download` is used rather than `/file/`, so downloads count
towards the catalog's statistics.
## State
`state.json` records what this store installed, keyed `<system>:<name>` so the
same software name on two systems stays two entries. A `version: 1` state file
written by the pre-split `ttg-store` is migrated to that key format on first
run.
same software name on two systems stays two entries. `layout` says which
directory layout those records belong to, which is how the upgrade to the store's
own folder knows it has to run once. A `version: 1` state file written by the
pre-split `ttg-store` is migrated to the current key format on first run.