warp-engine-batocera-store/README.md

warp-engine-batocera-store — a Batocera store engine for WarpEngine sites

Pulls games from a WarpEngine catalog straight onto a Batocera box, with EmulationStation metadata and box art, and gives the store its own entry in the EmulationStation menu. Started from that entry on the device, or from the CLI over SSH.

This repository is the engine only. It knows the WarpEngine API but nothing about any particular site: the host, the store's name and where its games land all come from a config.json that lives in a separate store repository. The part that is not about Batocera — catalog, releases, host matching, state, HTTP — is warpstore, shared with the RetroArch engine.

warpstore                           the shared core — warpstore.py
        ▲
warp-engine-batocera-store          this engine — store.py, install.sh
        ▲
        │ config.json
        │
  ┌─────┴─────┬───────────────┐
  │           │               │
ttg-…       my-…            other-…       store repositories
  │           │               │
  ▼           ▼               ▼
teletypegames.org   my.example   other.example

The reference store is ttg-batocera-store.

"<store name>" ▸ "Update <store name>"
        │
        ├─ GET /api/software                        the whole catalog
        ├─ keep platforms this box can run          c64 → c64, tic80 → tic80
        ├─ pick the newest non-dev release          that carries the right asset
        ├─ GET /api/download?path=<asset>           .prg / .tic into the store's folder
        ├─ GET <software.imageUrl>                  box art
        ├─ write gamelist.xml                       title, desc, author, image
        └─ write es_systems_<id>.cfg                the store's menu entry

Python 3 standard library only — Batocera ships python3 and no pip.

The store's own menu entry

Everything the store installs lives in one folder of its own, /userdata/roms/<subfolder>, with a directory per platform inside it. An es_systems_<id>.cfg in EmulationStation's user config declares one ES system per platform, all carrying the same <group>, and ES turns a group into a single carousel entry holding a folder per system:

main carousel:  …  ▸  Teletype Games  ▸  ┌ Commodore 64
                                         ├ TIC-80
                                         ├ Ports
                                         └ Update Teletype Games

Nothing about emulators is hardcoded here. The engine reads the box's own /usr/share/emulationstation/es_systems.cfg and copies the launch command, the extensions, the platform, the theme and the emulator list off the system it is borrowing — a file Batocera generated for this particular image, so it lists exactly the emulators the box has. The one thing that is rewritten is %SYSTEM%: ES replaces it with the name of the system being launched, which for us would be ttg-c64, a name configgen has never heard of. The box's own system name goes in instead, and a game of ours is launched exactly as the box would launch it, with the emulator the user configured for that system.

Five details worth knowing, all of them EmulationStation's, and the middle two were learned the hard way:

• The menu entry is declared, not implied. ES would invent the group parent by itself, but then its theme folder would be the group's name, which no theme has — and HideUniqueGroups (on by default) dissolves a group that has a single system in it unless a system by that name exists. The declared parent fixes both: it carries a <theme> (ports by default, since every theme has that one) and keeps the entry in place on a box that only has c64 games.
• A system with no games of its own is dropped, not just hidden: loadSystem logs System "..." has no games! Ignoring it. and throws it away. So the menu entry cannot be an empty shelf — it is the store/ folder, and the updater in it is the game that keeps it alive. Which is also why the Update … item sits directly in the entry rather than in a subfolder.
• The entry must not be the directory above the systems. ES marks a directory it has enumerated with a <dir>/* entry in its file cache, and adds that mark before reading the contents; from then on any path under it that is not itself cached answers "does not exist" (FileSystemUtil.cpp, getCacheEntry). Systems load in a thread pool, so an entry sitting above the systems loses that race for a random couple of them, and ES drops those with System "..." path does not exist ! — a different couple on every start. store/ is their sibling, so there is no race to lose.
• An empty system does not show. Only the platforms that actually have something installed become folders, so the entry is never full of dead ends.
• A restart is needed for a new system. The engine restarts ES itself after a sync that added one.

The store id names the entry. If the box already has a system by that name — a store called nes, say — the entry is called <id>-store instead, and the log says so: an overlay whose <name> matches an existing system would modify that system rather than add one.

For the old behaviour — games inside the box's own systems, under a subfolder, merged into the box's gamelists — set emulationstation.menu.mode to merge.

Installing a store

The installer takes the store as a parameter, so it works with any config:

curl -fsSL https://git.teletypegames.org/stores/warp-engine-batocera-store/raw/branch/master/install.sh |
  STORE_CONFIG=https://git.example.org/tools/my-store/raw/branch/master/config.json sh

From a checkout, with a local config:

scp -r warp-engine-batocera-store root@batocera:/tmp/
ssh root@batocera 'STORE_CONFIG=/tmp/my-config.json /tmp/warp-engine-batocera-store/install.sh'

It reads store.id from the config, installs the engine and the config into /userdata/system/batocera-store/<id>/, writes a <id>-store launcher, and runs the first sync — which is what creates the menu entry. Restart EmulationStation (batocera-es-swissknife --restart) to see it.

It installs two files: store.py from this repository and warpstore.py from the shared core. They have to sit next to each other — the engine imports the core from its own directory.

Installer knobs, all environment variables: STORE_CONFIG (required), BATOCERA_STORE_ROOT, BATOCERA_PORTS_DIR, BATOCERA_PORT_NAME, ENGINE_RAW_BASE, WARPSTORE_RAW_BASE, and WARPSTORE_SRC to install a local warpstore.py instead of downloading it.

Upgrading a store that was installed before the menu entry

Engine 4.0 is the layout change, so the first sync after the upgrade moves the store off the box's own systems. It does that by removing the old install and downloading again, not by carrying files across: the ROMs, box art and port payloads go, our nodes come out of the box's gamelists, the empty folders are swept, the Ports entry is taken away — and then the sync fills the store's own folder. The catalog is small enough that this is minutes, and there is no half-moved state to end up in.

What is kept: your gamelists (only our nodes go — play counts and favourites of your own games are untouched), and the gamelist.xml.<id>-backup copies, which are listed at the end of the run for you to delete or keep.

-n shows the whole thing without doing any of it. To stay on the old layout, set emulationstation.menu.mode to merge before the first sync.

Writing a store repository

Four small files:

my-batocera-store/
├── config.json     the store: URL, name, subfolder, platform mapping
├── install.sh      a wrapper that hands that config to the engine installer
├── uninstall.sh    the same for the engine's uninstaller
└── README.md

install.sh is small enough to quote in full:

#!/bin/bash
set -euo pipefail
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/stores/warp-engine-batocera-store/raw/branch/master}"
export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-batocera-store/raw/branch/master/config.json}"
curl -fsSL "$ENGINE_RAW_BASE/install.sh" | sh

uninstall.sh is the same shape, except that it passes STORE_ID — no config fetch is needed to take a store away, and the id is what names its home:

export STORE_ID="${STORE_ID:-example}"
curl -fsSL "$ENGINE_RAW_BASE/uninstall.sh" | sh

Pick a store.id and a paths.subfolder nobody else uses — they are what keeps two stores on the same box out of each other's way, and the id is what names the menu entry and its es_systems file.

Use

On the device. "<store name>" ▸ "Update <store name>". It downloads anything new, then restarts EmulationStation so the games show up. ES gives a launched script no console, so the output goes to store.log in the store home.

Over SSH, through the launcher the installer wrote:

S=/userdata/system/batocera-store/example-store

$S list                 # compatible catalog entries; * installed, ^ update available
$S sync                 # download everything new, refresh gamelists and the menu
$S sync blessingofra    # just one title (never prunes)
$S -n sync              # dry run
$S remove c64:demo      # uninstall (bare name works too)
$S purge                # uninstall everything this store installed
$S config               # effective configuration, and where the menu entry lives

Global flags go before the subcommand: $S --roms-root /tmp/roms sync.

Uninstalling

ssh root@batocera 'curl -fsSL https://git.teletypegames.org/stores/warp-engine-batocera-store/raw/branch/master/uninstall.sh | sh'

or ./uninstall.sh from a checkout on the box. It removes the games first, through the engine — purge — and only then the launcher, any Ports entry and the store home: the shell script has no way of knowing which ROMs, box art, port payloads, gamelist entries and ES systems were the store's, and the engine's state.json does. If the engine cannot finish — an unmounted ROMs root, say — nothing else is touched, so you are never left with the files but not the engine that knows about them.

Environment variable	Meaning
STORE_ID	which store to remove; needed only when several are installed
STORE_CONFIG	alternative to STORE_ID — the id is read out of the config
DRY_RUN	1 to print what would go and remove nothing
KEEP_HOME	1 to keep the store home (state, catalog cache, log) for a reinstall
BATOCERA_STORE_ROOT, BATOCERA_PORTS_DIR, BATOCERA_PORT_NAME	as for the installer

In the store's own layout an uninstall is total: the store's ROM folder, its gamelists and its es_systems_<id>.cfg all go, and the box is left as it was. In merge mode there is more to be careful about, and these stay behind:

• ROMs you put in the store's subfolder yourself. Directories are removed only when empty, so one stray file keeps its directory. ports/.data itself also stays: another store may still keep a payload there.
• your gamelists. They belong to the box: our <game> and <folder> nodes go, the rest — your own games, their play counts and favourites — is written back untouched. A gamelist we created and that is left empty is deleted, since it was never yours.
• gamelist.xml.<store id>-backup, listed at the end of the run: your gamelists as we first found them. Deleting them is your call.

A dry run cannot predict which directories will end up empty, so it lists the files but not the directory removals.

To take every store off a machine — both engines, the shared core, the launchers — there is one script for that in warpstore:

curl -fsSL https://git.teletypegames.org/engines/warpstore/raw/branch/master/uninstall.sh | sh

Where things land

/userdata/system/batocera-store/
├── example-store                       launcher for the store below
└── example/
    ├── store.py, warpstore.py          the engine and its shared core
    ├── config.json                     the store that selected it
    └── state.json, catalog.json, store.log

/userdata/system/configs/emulationstation/es_systems_example.cfg   the menu entry

/userdata/roms/example/                 the store's own ROM folder
├── c64/
│   ├── blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
│   ├── images/blessingofra.png, …
│   └── gamelist.xml
├── ports/
│   ├── <name>.sh                       the launchers ES lists
│   ├── images/<name>.png
│   ├── .data/<name>/                   the unpacked programs
│   └── gamelist.xml
└── store/                              the menu entry itself
    ├── update.sh                       the sync, as the menu starts it
    └── gamelist.xml

The box's own system folders are not touched at all, which is what makes a prune safe: nothing outside the store's folder can be reached, and inside it only what state.json says we installed is ever deleted — a ROM you drop in there yourself keeps its file and its gamelist entry.

In merge mode the same content goes into <roms_root>/<system>/<subfolder>/ and our nodes are merged into the box's gamelist.xml. The first time one is touched it is copied to gamelist.xml.<store id>-backup; on every merge only <game>/<folder> nodes under that store's subfolder are rewritten, so your own entries keep their play counts, favourites and scraped media, and two stores can share one gamelist.

Either way, the tags EmulationStation owns — favorite, playcount, lastplayed, rating — are carried over onto the node we rewrite. A favourite stays a favourite across a sync.

Configuration

config.json in the store home, template in config.example.json:

Key	Default	Meaning
store.id	warp	slug: names the store home, the log prefix, the menu entry and its es_systems file
store.name	WarpEngine Store	display name of the menu entry
store.base_url	—	WarpEngine host
store.api.catalog	/api/software	catalog endpoint, if the engine is mounted elsewhere
store.api.download	/api/download	download endpoint
paths.roms_root	/userdata/roms	where systems live
paths.subfolder	warp	the store's own folder inside roms_root (in merge mode: its subfolder inside each system)
emulationstation.menu.mode	system	system: the store gets its own menu entry. merge: install into the box's own systems
emulationstation.menu.name	null → store.name	label of the menu entry
emulationstation.menu.theme	ports	theme folder the entry borrows its logo from
emulationstation.menu.updater	true	an Update … entry inside the menu that runs the sync
emulationstation.menu.labels	{}	per-system folder labels; a system not listed keeps the box's own <fullname>
emulationstation.ports_entry	null → false in system mode, true in merge mode	also write a Ports entry that runs the sync
emulationstation.folder_name	WarpEngine Store	merge mode only: display name of our folder inside the box's gamelist; null hides it
emulationstation.restart	true	restart EmulationStation after a change
emulationstation.config_dir	null → /userdata/system/configs/emulationstation	where ES keeps its user config
catalog.statuses	["released", "archived"]	catalog status values to install
catalog.owner_id	null	restrict to one publisher (/api/software?owner_id=)
catalog.only / catalog.exclude	[]	software-name allow / deny lists
platforms	c64, tic80	platform → system, asset kind, extension, enabled; install picks ROM or Port, and the kind may be per-architecture
behavior.prune	true	remove games that left the catalog or the filters
behavior.timeout	30	HTTP timeout, seconds
behavior.insecure	false	skip TLS verification (self-hosted test instances)

BATOCERA_ES_CONFIG_DIR and BATOCERA_ES_SYSTEMS_CFG override the two EmulationStation paths from the environment, which is what makes a staging root testable off a real box.

Platform mapping

A catalog platform is installable when its release asset is a file a system this box has can boot directly:

Catalog platform	Asset kind	Batocera system	Extension
c64	cartridge	c64 (VICE)	.prg
tic80	cartridge	tic80	.tic

"Has" means declared in the box's own es_systems.cfg — an emulator that got built into this image. (A ROM folder is a poorer test: it can be there with nothing behind it. The folders are the fallback when the file cannot be read.)

The other WarpEngine platforms (ebitengine, love, godot, bevy, phaser) ship html and per-OS zip archives, not a ROM a Batocera system launches, so they are not mapped by default. Adding one is a config edit:

"platforms": { "godot": { "system": "godot", "kind": "linux_x64", "ext": ".zip", "enabled": true } }

Native games as Ports

A cartridge is booted by an emulator, so it goes to the folder of the system that boots it. A native build is a program: Batocera runs Linux, and its Ports system takes .sh launchers. Set install to port and point system at ports:

"platforms": {
  "bevy": {
    "system": "ports",
    "install": "port",
    "kind": { "x86_64": "linux_x64", "aarch64": "linux_arm64" },
    "ext": ".zip",
    "enabled": true
  }
}

The zip is unpacked and a launcher written:

/userdata/roms/<subfolder>/ports/<name>.sh          the launcher ES lists
/userdata/roms/<subfolder>/ports/images/<name>.png  box art
/userdata/roms/<subfolder>/ports/.data/<name>/      the unpacked program

The payload sits under .data, whose leading dot is what hides it from EmulationStation — only the .sh next to it is meant to be listed. system: "ports" still names where the launcher definition comes from: our own ports system copies the box's, which is sh, so the launcher is simply run.

The launcher cds into the directory holding the executable before running it: a game loads its assets/ relative to the working directory. Python's zip extraction drops the executable bit, so it is restored from the archive's recorded mode.

install defaults to rom, so a config that does not mention it behaves exactly as before.

Per-architecture assets

Batocera runs on x86_64 mini-PCs, on the Raspberry Pi and on the ARM handhelds alike, and a native binary only starts on the architecture it was built for — an x86_64 build installs on a Pi and then does nothing. A cartridge is data for an emulator, so it is architecture-independent; a native build is not.

Where the right asset depends on the machine, kind (and ext, if it differs) can be a map instead of a string. * is the fallback:

"platforms": {
  "bevy": {
    "system": "bevy",
    "kind": { "x86_64": "linux_x64", "aarch64": "linux_arm64" },
    "ext": ".zip",
    "enabled": true
  }
}

The engine reports the architecture it detected — x86_64, aarch64, armhf, x86 — at the top of list, and config prints it as detected_arch. With no entry for the running machine and no *, the title is skipped and says so: skipped bevydemo: bevy has no asset kind for aarch64.

A platform is also skipped when the box does not have its system — list reports that as skipped <name>: this box has no '<system>' system.

How a sync stays safe next to EmulationStation

EmulationStation keeps gamelists in memory and writes them back when it exits, so a merge done while it runs can be clobbered. Launched from the menu (sync-from-es), the engine therefore downloads first, then hands the gamelist merge to a detached apply-gamelists --wait-pid <es-pid> child that waits for the old ES process to die before writing. From SSH, with no ES running, the merge happens inline.

The es_systems_<id>.cfg needs none of that: ES reads it and never writes it, so it is written straight away — it just does not take effect until ES restarts.

Re-running a sync is idempotent: files already present are left alone, a release bump (1.1 → 2.0.0) deletes the old asset before fetching the new one, and a gamelist or es_systems file whose content has not changed is not rewritten at all. GET /api/download is used rather than /file/, so downloads count towards the catalog's statistics.

State

state.json records what this store installed, keyed <system>:<name> so the same software name on two systems stays two entries. layout says which directory layout those records belong to, which is how the upgrade to the store's own folder knows it has to run once. A version: 1 state file written by the pre-split ttg-store is migrated to the current key format on first run.