From 3b72cf81cce7c55f9957e96099710fbe82254052 Mon Sep 17 00:00:00 2001 From: Zsolt Tasnadi Date: Sun, 23 Aug 2026 12:57:25 +0200 Subject: [PATCH] Give the store its own EmulationStation menu entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The games were scattered across the box's own systems — a folder inside c64, another inside tic80, the launchers inside Ports, and the sync a fourth entry somewhere else. There was no one place where the store appeared, and living in the box's folders is what forced the gamelist merging, the backups and every "never step outside your subfolder" check. Now the store gets a folder of its own, /userdata/roms/, with a directory per platform, and an es_systems_.cfg overlay declares one ES system per platform under a shared — which EmulationStation draws as a single carousel entry holding a folder per system. The gamelists are ours, the box's systems are untouched, and the sync is startable from inside the entry: Store -> "Update ". Nothing about emulators is hardcoded. The launch command, the extensions, the platform, the theme and the emulator list are copied off the box's own es_systems.cfg — a file Batocera generated for this image, so it lists exactly what got built into it. %SYSTEM% is resolved to the source system name on the way, since ES would otherwise substitute ttg-c64, which configgen has never heard of. The old behaviour is emulationstation.menu.mode: "merge", and switching either way migrates: the layout the files are in is recorded in state.json, and a mismatch removes the old one and downloads again rather than shuffling files half-way. purge follows that record too, so an uninstall after an upgrade whose first sync never ran still finds the games where they actually are. Every path now goes through a Layout, so the two modes cannot drift apart. Also: a gamelist or es_systems file whose content has not changed is not rewritten at all, and the tags ES owns — favourite, playcount, lastplayed — are carried over onto the nodes we rewrite, which merge mode used to lose. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 234 +++++++++--- config.example.json | 11 +- install.sh | 37 +- store.py | 869 +++++++++++++++++++++++++++++++++++++++----- uninstall.sh | 8 +- 5 files changed, 987 insertions(+), 172 deletions(-) diff --git a/README.md b/README.md index f54ed9a..5d6dd5a 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ # warp-engine-batocera-store — a Batocera store engine for WarpEngine sites Pulls games from a [WarpEngine](https://git.teletypegames.org/tools/warp_engine) -catalog straight into a Batocera box's ROM folders, with EmulationStation -metadata and box art. Triggered from the **Ports** menu on the device, or from -the CLI over SSH. +catalog straight onto a Batocera box, with EmulationStation metadata and box art, +and gives the store **its own entry in the EmulationStation menu**. Started from +that entry on the device, or from the CLI over SSH. This repository is the engine only. It knows the WarpEngine API but nothing about any particular site: the host, the store's name and where its games land @@ -31,18 +31,65 @@ The reference store is [`ttg-batocera-store`](https://git.teletypegames.org/stores/ttg-batocera-store). ``` -Ports ▸ "" +"" ▸ Store ▸ "Update " │ ├─ GET /api/software the whole catalog ├─ keep platforms this box can run c64 → c64, tic80 → tic80 ├─ pick the newest non-dev release that carries the right asset - ├─ GET /api/download?path= .prg / .tic into the ROM folder + ├─ GET /api/download?path= .prg / .tic into the store's folder ├─ GET box art - └─ merge gamelist.xml title, desc, author, image + ├─ write gamelist.xml title, desc, author, image + └─ write es_systems_.cfg the store's menu entry ``` Python 3 standard library only — Batocera ships python3 and no pip. +## The store's own menu entry + +Everything the store installs lives in one folder of its own, +`/userdata/roms/`, with a directory per platform inside it. An +`es_systems_.cfg` in EmulationStation's user config declares one ES system +per platform, all carrying the same ``, and ES turns a group into a single +carousel entry holding a folder per system: + +``` +main carousel: … ▸ Teletype Games ▸ ┌ Commodore 64 + ├ TIC-80 + ├ Ports + └ Store → "Update Teletype Games" +``` + +Nothing about emulators is hardcoded here. The engine reads the box's own +`/usr/share/emulationstation/es_systems.cfg` and copies the launch command, the +extensions, the platform, the theme and the emulator list off the system it is +borrowing — a file Batocera generated for this particular image, so it lists +exactly the emulators the box has. The one thing that is rewritten is `%SYSTEM%`: +ES replaces it with the name of the system being launched, which for us would be +`ttg-c64`, a name configgen has never heard of. The box's own system name goes in +instead, and a game of ours is launched exactly as the box would launch it, with +the emulator the user configured for that system. + +Three details worth knowing, all of them EmulationStation's: + +- **The menu entry is declared, not implied.** ES would invent the group parent + by itself, but then its theme folder would be the group's name, which no theme + has — and `HideUniqueGroups` (on by default) dissolves a group that has a + single system in it unless a system by that name exists. The declared parent + fixes both: it carries a `` (`ports` by default, since every theme has + that one) and keeps the entry in place on a box that only has c64 games. +- **An empty system does not show.** Only the platforms that actually have + something installed become folders, so the entry is never full of dead ends. +- **A restart is needed for a new system.** The engine restarts ES itself after a + sync that added one. + +The store id names the entry. If the box already has a system by that name — a +store called `nes`, say — the entry is called `-store` instead, and the log +says so: an overlay whose `` matches an existing system would *modify that +system* rather than add one. + +For the old behaviour — games inside the box's own systems, under a subfolder, +merged into the box's gamelists — set `emulationstation.menu.mode` to `merge`. + ## Installing a store The installer takes the store as a parameter, so it works with any config: @@ -60,9 +107,9 @@ ssh root@batocera 'STORE_CONFIG=/tmp/my-config.json /tmp/warp-engine-batocera-st ``` It reads `store.id` from the config, installs the engine and the config into -`/userdata/system/batocera-store//`, writes a `-store` launcher and a -Ports entry, and runs the first sync. Restart EmulationStation -(`batocera-es-swissknife --restart`) to see the games. +`/userdata/system/batocera-store//`, writes a `-store` launcher, and runs +the first sync — which is what creates the menu entry. Restart EmulationStation +(`batocera-es-swissknife --restart`) to see it. It installs two files: `store.py` from this repository and `warpstore.py` from the [shared core](https://git.teletypegames.org/engines/warpstore). They have to @@ -73,6 +120,23 @@ Installer knobs, all environment variables: `STORE_CONFIG` (required), `ENGINE_RAW_BASE`, `WARPSTORE_RAW_BASE`, and `WARPSTORE_SRC` to install a local `warpstore.py` instead of downloading it. +## Upgrading a store that was installed before the menu entry + +Engine 4.0 is the layout change, so the first sync after the upgrade moves the +store off the box's own systems. It does that by **removing the old install and +downloading again**, not by carrying files across: the ROMs, box art and port +payloads go, our nodes come out of the box's gamelists, the empty folders are +swept, the Ports entry is taken away — and then the sync fills the store's own +folder. The catalog is small enough that this is minutes, and there is no +half-moved state to end up in. + +What is kept: your gamelists (only our nodes go — play counts and favourites of +your own games are untouched), and the `gamelist.xml.-backup` copies, which +are listed at the end of the run for you to delete or keep. + +`-n` shows the whole thing without doing any of it. To stay on the old layout, +set `emulationstation.menu.mode` to `merge` before the first sync. + ## Writing a store repository Four small files: @@ -92,7 +156,6 @@ my-batocera-store/ set -euo pipefail ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/stores/warp-engine-batocera-store/raw/branch/master}" export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-batocera-store/raw/branch/master/config.json}" -export BATOCERA_PORT_NAME="My Store" curl -fsSL "$ENGINE_RAW_BASE/install.sh" | sh ``` @@ -105,13 +168,15 @@ curl -fsSL "$ENGINE_RAW_BASE/uninstall.sh" | sh ``` Pick a `store.id` and a `paths.subfolder` nobody else uses — they are what keeps -two stores on the same box from deleting each other's games. +two stores on the same box out of each other's way, and the id is what names the +menu entry and its `es_systems` file. ## Use -**On the device.** *Ports ▸ "\"*. The script downloads anything -new, then restarts EmulationStation so the games show up. Ports scripts get no -console in Batocera, so the output goes to `store.log` in the store home. +**On the device.** *"\" ▸ Store ▸ "Update \"*. It +downloads anything new, then restarts EmulationStation so the games show up. ES +gives a launched script no console, so the output goes to `store.log` in the +store home. **Over SSH**, through the launcher the installer wrote: @@ -119,12 +184,12 @@ console in Batocera, so the output goes to `store.log` in the store home. S=/userdata/system/batocera-store/example-store $S list # compatible catalog entries; * installed, ^ update available -$S sync # download everything new, refresh gamelists +$S sync # download everything new, refresh gamelists and the menu $S sync blessingofra # just one title (never prunes) $S -n sync # dry run $S remove c64:demo # uninstall (bare name works too) $S purge # uninstall everything this store installed -$S config # effective configuration +$S config # effective configuration, and where the menu entry lives ``` Global flags go **before** the subcommand: `$S --roms-root /tmp/roms sync`. @@ -136,12 +201,12 @@ ssh root@batocera 'curl -fsSL https://git.teletypegames.org/stores/warp-engine-b ``` or `./uninstall.sh` from a checkout on the box. It removes the games first, -through the engine — `purge` — and only then the Ports entry, the launcher and the -store home: the shell script has no way of knowing which ROMs, box art, Ports -payloads and gamelist entries were the store's, and the engine's `state.json` -does. If the engine cannot finish — an unmounted ROMs root, say — nothing else is -touched, so you are never left with the files but not the engine that knows about -them. +through the engine — `purge` — and only then the launcher, any Ports entry and +the store home: the shell script has no way of knowing which ROMs, box art, port +payloads, gamelist entries and ES systems were the store's, and the engine's +`state.json` does. If the engine cannot finish — an unmounted ROMs root, say — +nothing else is touched, so you are never left with the files but not the engine +that knows about them. | Environment variable | Meaning | |---|---| @@ -151,11 +216,19 @@ them. | `KEEP_HOME` | `1` to keep the store home (state, catalog cache, log) for a reinstall | | `BATOCERA_STORE_ROOT`, `BATOCERA_PORTS_DIR`, `BATOCERA_PORT_NAME` | as for the installer | -What it deliberately leaves behind: +In the store's own layout an uninstall is total: the store's ROM folder, its +gamelists and its `es_systems_.cfg` all go, and the box is left as it was. +In `merge` mode there is more to be careful about, and these stay behind: -- **ROMs you put in the store's subfolder yourself.** Directories are removed only when empty, so one stray file keeps its directory. `ports/.data` itself also stays: another store may still keep a payload there. -- **your gamelists.** They belong to the box: our `` and `` nodes go, the rest — your own games, their play counts and favourites — is written back untouched. A gamelist we *created* and that is left empty is deleted, since it was never yours. -- **`gamelist.xml.-backup`**, listed at the end of the run: your gamelists as we first found them. Deleting them is your call. +- **ROMs you put in the store's subfolder yourself.** Directories are removed + only when empty, so one stray file keeps its directory. `ports/.data` itself + also stays: another store may still keep a payload there. +- **your gamelists.** They belong to the box: our `` and `` nodes + go, the rest — your own games, their play counts and favourites — is written + back untouched. A gamelist we *created* and that is left empty is deleted, + since it was never yours. +- **`gamelist.xml.-backup`**, listed at the end of the run: your + gamelists as we first found them. Deleting them is your call. A dry run cannot predict which directories will end up empty, so it lists the files but not the directory removals. @@ -178,18 +251,38 @@ curl -fsSL https://git.teletypegames.org/engines/warpstore/raw/branch/master/uni ├── config.json the store that selected it └── state.json, catalog.json, store.log -/userdata/roms/c64/example/ blessingofra-2.0.0.prg, rabbit-1.0.0.prg, … -/userdata/roms/c64/example/images/ blessingofra.png, … -/userdata/roms/c64/gamelist.xml our entries merged in -/userdata/roms/ports/Example Store.sh +/userdata/system/configs/emulationstation/es_systems_example.cfg the menu entry + +/userdata/roms/example/ the store's own ROM folder +├── c64/ +│ ├── blessingofra-2.0.0.prg, rabbit-1.0.0.prg, … +│ ├── images/blessingofra.png, … +│ └── gamelist.xml +├── ports/ +│ ├── .sh the launchers ES lists +│ ├── images/.png +│ ├── .data// the unpacked programs +│ └── gamelist.xml +└── store/ + ├── update.sh the sync, as the menu starts it + └── gamelist.xml ``` -Everything a store installs lives under its own `paths.subfolder`, so a prune -can never reach ROMs you put there yourself — or another store's games. The -first time a `gamelist.xml` is touched it is copied to -`gamelist.xml.-backup`; on every merge only ``/`` nodes -under that store's subfolder are rewritten, so your own entries keep their play -counts, favourites and scraped media, and two stores can share one gamelist. +The box's own system folders are not touched at all, which is what makes a prune +safe: nothing outside the store's folder can be reached, and inside it only what +`state.json` says we installed is ever deleted — a ROM you drop in there +yourself keeps its file *and* its gamelist entry. + +In `merge` mode the same content goes into `///` +and our nodes are merged into the box's `gamelist.xml`. The first time one is +touched it is copied to `gamelist.xml.-backup`; on every merge only +``/`` nodes under that store's subfolder are rewritten, so your own +entries keep their play counts, favourites and scraped media, and two stores can +share one gamelist. + +Either way, the tags EmulationStation owns — `favorite`, `playcount`, +`lastplayed`, `rating` — are carried over onto the node we rewrite. A favourite +stays a favourite across a sync. ## Configuration @@ -197,15 +290,22 @@ counts, favourites and scraped media, and two stores can share one gamelist. | Key | Default | Meaning | | --- | --- | --- | -| `store.id` | `warp` | slug: names the store home, log prefix and gamelist backup | -| `store.name` | `WarpEngine Store` | display name for the Ports entry | +| `store.id` | `warp` | slug: names the store home, the log prefix, the menu entry and its `es_systems` file | +| `store.name` | `WarpEngine Store` | display name of the menu entry | | `store.base_url` | — | WarpEngine host | | `store.api.catalog` | `/api/software` | catalog endpoint, if the engine is mounted elsewhere | | `store.api.download` | `/api/download` | download endpoint | | `paths.roms_root` | `/userdata/roms` | where systems live | -| `paths.subfolder` | `warp` | this store's subfolder inside each system | -| `emulationstation.folder_name` | `WarpEngine Store` | display name of that folder in ES | +| `paths.subfolder` | `warp` | the store's own folder inside `roms_root` (in `merge` mode: its subfolder inside each system) | +| `emulationstation.menu.mode` | `system` | `system`: the store gets its own menu entry. `merge`: install into the box's own systems | +| `emulationstation.menu.name` | `null` → `store.name` | label of the menu entry | +| `emulationstation.menu.theme` | `ports` | theme folder the entry borrows its logo from | +| `emulationstation.menu.updater` | `true` | an *Update …* entry inside the menu that runs the sync | +| `emulationstation.menu.labels` | `{}` | per-system folder labels; a system not listed keeps the box's own `` | +| `emulationstation.ports_entry` | `null` → `false` in `system` mode, `true` in `merge` mode | also write a Ports entry that runs the sync | +| `emulationstation.folder_name` | `WarpEngine Store` | `merge` mode only: display name of our folder inside the box's gamelist; `null` hides it | | `emulationstation.restart` | `true` | restart EmulationStation after a change | +| `emulationstation.config_dir` | `null` → `/userdata/system/configs/emulationstation` | where ES keeps its user config | | `catalog.statuses` | `["released", "archived"]` | catalog `status` values to install | | `catalog.owner_id` | `null` | restrict to one publisher (`/api/software?owner_id=`) | | `catalog.only` / `catalog.exclude` | `[]` | software-name allow / deny lists | @@ -214,16 +314,24 @@ counts, favourites and scraped media, and two stores can share one gamelist. | `behavior.timeout` | `30` | HTTP timeout, seconds | | `behavior.insecure` | `false` | skip TLS verification (self-hosted test instances) | +`BATOCERA_ES_CONFIG_DIR` and `BATOCERA_ES_SYSTEMS_CFG` override the two +EmulationStation paths from the environment, which is what makes a staging root +testable off a real box. + ### Platform mapping -A catalog platform is installable when its release asset is a file the Batocera -system can boot directly: +A catalog platform is installable when its release asset is a file a system this +box has can boot directly: | Catalog platform | Asset kind | Batocera system | Extension | | --- | --- | --- | --- | | `c64` | `cartridge` | `c64` (VICE) | `.prg` | | `tic80` | `cartridge` | `tic80` | `.tic` | +"Has" means declared in the box's own `es_systems.cfg` — an emulator that got +built into this image. (A ROM folder is a poorer test: it can be there with +nothing behind it. The folders are the fallback when the file cannot be read.) + The other WarpEngine platforms (`ebitengine`, `love`, `godot`, `bevy`, `phaser`) ship `html` and per-OS zip archives, not a ROM a Batocera system launches, so they are not mapped by default. Adding one is a config edit: @@ -234,9 +342,10 @@ launches, so they are not mapped by default. Adding one is a config edit: ### Native games as Ports -A cartridge is booted by an emulator, so it goes into that system's ROM folder. -A native build is a program: Batocera runs Linux, and its **Ports** system takes -`.sh` launchers. Set `install` to `port` and point `system` at `ports`: +A cartridge is booted by an emulator, so it goes to the folder of the system that +boots it. A native build is a program: Batocera runs Linux, and its **Ports** +system takes `.sh` launchers. Set `install` to `port` and point `system` at +`ports`: ```json "platforms": { @@ -253,15 +362,15 @@ A native build is a program: Batocera runs Linux, and its **Ports** system takes The zip is unpacked and a launcher written: ``` -/userdata/roms/ports//.sh the launcher ES lists -/userdata/roms/ports//images/.png box art -/userdata/roms/ports/.data/// the unpacked program +/userdata/roms//ports/.sh the launcher ES lists +/userdata/roms//ports/images/.png box art +/userdata/roms//ports/.data// the unpacked program ``` The payload sits under `.data`, whose leading dot is what hides it from -EmulationStation — the launchers themselves are in a normal subfolder, scanned -like any other system's. Both paths carry the store's subfolder, so an uninstall -or a prune can only ever reach this store's own files. +EmulationStation — only the `.sh` next to it is meant to be listed. `system: +"ports"` still names where the launcher *definition* comes from: our own ports +system copies the box's, which is `sh`, so the launcher is simply run. The launcher `cd`s into the directory holding the executable before running it: a game loads its `assets/` relative to the working directory. Python's zip @@ -297,26 +406,31 @@ The engine reports the architecture it detected — `x86_64`, `aarch64`, `armhf` entry for the running machine and no `*`, the title is skipped and says so: `skipped bevydemo: bevy has no asset kind for aarch64`. -A platform is also skipped when the box has no ROM folder for its system — -`list` reports that as `skipped : no '' ROM folder`. +A platform is also skipped when the box does not have its system — +`list` reports that as `skipped : this box has no '' system`. ## How a sync stays safe next to EmulationStation EmulationStation keeps gamelists in memory and writes them back when it exits, -so a merge done while it runs can be clobbered. Launched from the Ports menu +so a merge done while it runs can be clobbered. Launched from the menu (`sync-from-es`), the engine therefore downloads first, then hands the gamelist merge to a detached `apply-gamelists --wait-pid ` child that waits for the old ES process to die before writing. From SSH, with no ES running, the merge happens inline. -Re-running a sync is idempotent: files already present are left alone, and a -release bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new -one. `GET /api/download` is used rather than `/file/`, so downloads count +The `es_systems_.cfg` needs none of that: ES reads it and never writes it, +so it is written straight away — it just does not take effect until ES restarts. + +Re-running a sync is idempotent: files already present are left alone, a release +bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new one, and a +gamelist or `es_systems` file whose content has not changed is not rewritten at +all. `GET /api/download` is used rather than `/file/`, so downloads count towards the catalog's statistics. ## State `state.json` records what this store installed, keyed `:` so the -same software name on two systems stays two entries. A `version: 1` state file -written by the pre-split `ttg-store` is migrated to that key format on first -run. +same software name on two systems stays two entries. `layout` says which +directory layout those records belong to, which is how the upgrade to the store's +own folder knows it has to run once. A `version: 1` state file written by the +pre-split `ttg-store` is migrated to the current key format on first run. diff --git a/config.example.json b/config.example.json index a96da69..536450e 100644 --- a/config.example.json +++ b/config.example.json @@ -13,8 +13,17 @@ "subfolder": "example" }, "emulationstation": { + "menu": { + "mode": "system", + "name": null, + "theme": "ports", + "updater": true, + "labels": {} + }, + "ports_entry": null, "folder_name": "Example Store", - "restart": true + "restart": true, + "config_dir": null }, "catalog": { "statuses": [ diff --git a/install.sh b/install.sh index 797bb28..04567f7 100755 --- a/install.sh +++ b/install.sh @@ -45,17 +45,26 @@ trap 'rm -f "$TMP_CFG"' EXIT say "reading store config from $STORE_CONFIG" fetch "$STORE_CONFIG" "$TMP_CFG" +# The store id names its home; the mode decides whether a Ports entry is wanted +# at all, since a store with its own ES menu entry has an updater inside it. STORE_META="$(python3 -c ' import json, sys -cfg = json.load(open(sys.argv[1])).get("store") or {} -sid = cfg.get("id") or "" +cfg = json.load(open(sys.argv[1])) +store = cfg.get("store") or {} +sid = store.get("id") or "" if not sid or "/" in sid: sys.exit("store.id is required and must not contain a slash") -print(sid, cfg.get("name") or sid) +es = cfg.get("emulationstation") or {} +mode = ((es.get("menu") or {}).get("mode") or "system").lower() +port = es.get("ports_entry") +port = (mode == "merge") if port is None else bool(port) +print(sid, "1" if port else "0", store.get("name") or sid) ' "$TMP_CFG")" || die "cannot read $STORE_CONFIG" STORE_ID="${STORE_META%% *}" -STORE_NAME="${STORE_META#* }" +STORE_REST="${STORE_META#* }" +WANT_PORTS_ENTRY="${STORE_REST%% *}" +STORE_NAME="${STORE_REST#* }" STORE_HOME="$STORE_ROOT/$STORE_ID" PORT_NAME="${BATOCERA_PORT_NAME:-$STORE_NAME}" @@ -99,8 +108,11 @@ EOF chmod 0755 "$LAUNCHER" # The Ports entry: EmulationStation gives launched scripts no console, so send -# everything to a log file the user can read over SSH. -cat > "$PORTS_DIR/$PORT_NAME.sh" < "$PORTS_DIR/$PORT_NAME.sh" <>"$STORE_HOME/store.log" 2>&1 @@ -108,17 +120,24 @@ echo "=== \$(date -Iseconds) sync started ===" "$LAUNCHER" sync-from-es echo "=== \$(date -Iseconds) sync finished (exit \$?) ===" EOF -chmod 0755 "$PORTS_DIR/$PORT_NAME.sh" -say "ports entry: $PORTS_DIR/$PORT_NAME.sh" + chmod 0755 "$PORTS_DIR/$PORT_NAME.sh" + say "ports entry: $PORTS_DIR/$PORT_NAME.sh" +fi say "running the first sync" "$LAUNCHER" --verbose sync --no-restart +if [ "$WANT_PORTS_ENTRY" = "1" ]; then + MENU_LINE=" Menu: Ports -> \"$PORT_NAME\" (re-run any time to pull new releases)" +else + MENU_LINE=" Menu: \"$STORE_NAME\" -> Store -> \"Update $STORE_NAME\" (re-run any time to pull new releases)" +fi + cat < "$PORT_NAME" (re-run any time to pull new releases) +$MENU_LINE CLI: $LAUNCHER list|sync|remove Config: $STORE_HOME/config.json Log: $STORE_HOME/store.log diff --git a/store.py b/store.py index 6134683..344a16c 100755 --- a/store.py +++ b/store.py @@ -2,9 +2,19 @@ """store.py — the Batocera adapter of the WarpEngine store engine. Reads a WarpEngine catalog (`GET /api/software`), keeps the entries whose -platform maps to a system this Batocera box actually has, downloads the matching -release asset into that system's ROM folder and writes EmulationStation -metadata (title, description, author, box art) into its gamelist.xml. +platform maps to a system this Batocera box can run, downloads the matching +release asset and writes EmulationStation metadata (title, description, author, +box art) into a gamelist.xml. + +Where the games land is the store's choice, `emulationstation.menu.mode`: + + system the store gets its own EmulationStation menu entry. Its games live in + `///`, and an `es_systems_.cfg` + declares one ES system per platform, all grouped under the store's + name — so the box shows one entry holding a folder per platform. + merge the games go into the box's own systems, under `//`, + and are merged into the box's gamelists. What the engine did before + the menu entry existed. The engine knows nothing about any particular site: which host to talk to, what the store is called and where its games land all come from `config.json`. A @@ -40,7 +50,13 @@ except ImportError: "(https://git.teletypegames.org/engines/warpstore)") from warpstore import debug, die, log -VERSION = "3.1.0" +VERSION = "4.0.0" + +# EmulationStation's own files. The user config dir is where our es_systems +# overlay goes; the shipped es_systems.cfg is what we read a system's launch +# command out of, so that nothing about emulators is hardcoded here. +ES_CONFIG_DIR = "/userdata/system/configs/emulationstation" +ES_SYSTEMS_CFG = "/usr/share/emulationstation/es_systems.cfg" # Every store keeps its code, config, state and log in one directory, so a box # can carry several stores side by side without them treading on each other. @@ -65,16 +81,41 @@ DEFAULT_CONFIG = { }, "paths": { "roms_root": "/userdata/roms", - # Everything we install lives under this subfolder of the system's ROM - # dir, so a prune can never touch ROMs the user put there themselves — - # nor games installed by another store. + # Everything we install lives under this subfolder. In "system" mode it + # is the store's own ROM folder, `//`; in + # "merge" mode it is a subfolder of the box's own system folders, + # `//`. Either way nothing outside it is + # ever touched, so a prune cannot reach the user's own ROMs — nor + # another store's games. "subfolder": "warp", }, "emulationstation": { - # Display name of our subfolder in ES; null hides the folder node. + "menu": { + # "system": the store gets its own ES menu entry (see the module + # docstring). "merge": install into the box's own systems. + "mode": "system", + # Label of that menu entry; null means store.name. + "name": None, + # Theme folder the entry borrows its logo from. "ports" exists in + # every Batocera theme; a name no theme knows would show up blank. + "theme": "ports", + # An "Update " entry inside the menu, so the sync can be + # started from the same place the games are. + "updater": True, + # Per-system folder labels inside the entry; a system that is not + # listed keeps the fullname the box's own es_systems.cfg gives it. + "labels": {}, + }, + # A Ports entry that runs the sync. Redundant once the store has its own + # menu entry with an updater in it, so "system" mode defaults to false. + "ports_entry": None, + # Display name of our subfolder in the box's gamelists ("merge" mode + # only); null hides the folder node. "folder_name": "WarpEngine Store", # Restart EmulationStation after a sync that changed something. "restart": True, + # Where ES keeps its user config; null means the Batocera default. + "config_dir": None, }, "catalog": { # Catalog `status` values worth installing. "development" is left out. @@ -108,6 +149,181 @@ def roms_root(cfg): return cfg["paths"]["roms_root"] +def menu(cfg): + return cfg["emulationstation"]["menu"] + + +def es_config_dir(cfg): + return (os.environ.get("BATOCERA_ES_CONFIG_DIR") + or cfg["emulationstation"].get("config_dir") + or ES_CONFIG_DIR) + + +def es_systems_cfg(cfg): + return os.environ.get("BATOCERA_ES_SYSTEMS_CFG") or ES_SYSTEMS_CFG + + +def wants_ports_entry(cfg): + """A Ports entry for the sync — off by default once the store has a menu.""" + want = cfg["emulationstation"].get("ports_entry") + if want is None: + return isinstance(layout(cfg), MergeLayout) + return bool(want) + + +# -------------------------------------------------------------------------- +# where the files go +# -------------------------------------------------------------------------- + + +class Layout: + """Where this store's files live, and what it is allowed to delete. + + Two answers, one per `emulationstation.menu.mode`, and every path in the + engine goes through one of them: the install, the removal, the gamelist and + the empty-directory sweep all ask the layout rather than build paths of + their own. + """ + + def __init__(self, cfg): + self.cfg = cfg + + def base(self, system): + """The directory this system's content lives in.""" + raise NotImplementedError + + def rel_rom(self, game): + return game["asset"] + + def rel_image(self, game, ext): + return f"images/{game['name']}{ext}" + + def rel_launcher(self, game): + return f"{game['name']}.sh" + + def rel_data(self, game): + # The leading dot is what hides the unpacked program from ES; only the + # `.sh` launcher next to it is meant to be listed as a game. + return f".data/{game['name']}" + + def gamelist(self, system): + return os.path.join(self.base(system), "gamelist.xml") + + def owns(self, rel): + """May we delete this path (relative to `base`)? The prune's safety net. + + Everything under the store's own folder is ours; what is refused is a + path that would climb out of it, however it is spelled. + """ + if os.path.isabs(rel): + return False + normalized = os.path.normpath(rel) + return normalized != ".." and not normalized.startswith(".." + os.sep) + + def claims(self, rel, wanted, base): + """Is this existing gamelist node ours to rewrite? (see merge_gamelist)""" + raise NotImplementedError + + def purge_dirs(self, system): + """Directories to try removing once the games are gone, deepest first.""" + return [] + + # Only the box's own gamelists are worth backing up, and only ours is worth + # deleting when it ends up empty. + backup_gamelists = True + folder_node = False + + +class MergeLayout(Layout): + """`//` — inside the box's own systems. + + Shared ground: the ROMs sit next to the user's, the gamelist is the box's, + and another store may keep its own games in the same folders. Hence the + subfolder prefix on everything and the `owns` check before every delete. + """ + + backup_gamelists = True + folder_node = True + + def base(self, system): + return os.path.join(roms_root(self.cfg), system) + + def rel_rom(self, game): + return f"{subfolder(self.cfg)}/{game['asset']}" + + def rel_image(self, game, ext): + return f"{subfolder(self.cfg)}/images/{game['name']}{ext}" + + def rel_launcher(self, game): + return f"{subfolder(self.cfg)}/{game['name']}.sh" + + def rel_data(self, game): + return f".data/{subfolder(self.cfg)}/{game['name']}" + + def owns(self, rel): + prefix = subfolder(self.cfg) + "/" + return rel.startswith(prefix) or rel.startswith(f".data/{prefix}") + + def claims(self, rel, wanted, base): + return rel.startswith(subfolder(self.cfg) + "/") or rel == subfolder(self.cfg) + + def purge_dirs(self, system): + base = self.base(system) + return [os.path.join(base, subfolder(self.cfg), "images"), + os.path.join(base, subfolder(self.cfg)), + # `.data` itself stays: it is shared ground, another store may + # still be keeping a payload there. + os.path.join(base, ".data", subfolder(self.cfg))] + + +class SystemLayout(Layout): + """`//` — the store's own ROM folder. + + Every file under it is ours, including the gamelists, so there is nothing to + back up and nothing to merge around. What keeps the user's own files safe + here is that we only ever delete what `state.json` says we installed. + """ + + backup_gamelists = False + folder_node = False + + def store_root(self): + return os.path.join(roms_root(self.cfg), subfolder(self.cfg)) + + def base(self, system): + return os.path.join(self.store_root(), system) + + def claims(self, rel, wanted, base): + if not rel or rel in wanted: + return True + # A leftover of ours points at a file that is not there any more; a ROM + # the user added themselves does exist, and keeps its entry. + return not os.path.exists(os.path.join(base, rel)) + + def purge_dirs(self, system): + base = self.base(system) + return [os.path.join(base, "images"), os.path.join(base, ".data"), base] + + +def layout_name(lay): + return "merge" if isinstance(lay, MergeLayout) else "system" + + +def layout_named(cfg, name): + return MergeLayout(cfg) if name == "merge" else SystemLayout(cfg) + + +_LAYOUT = None + + +def layout(cfg): + global _LAYOUT + if _LAYOUT is None or _LAYOUT.cfg is not cfg: + mode = (menu(cfg).get("mode") or "system").lower() + _LAYOUT = MergeLayout(cfg) if mode == "merge" else SystemLayout(cfg) + return _LAYOUT + + # -------------------------------------------------------------------------- # catalog -> Batocera # -------------------------------------------------------------------------- @@ -124,7 +340,7 @@ def accept_for_batocera(available_systems): def accept(spec, sw): system = spec["system"] if system not in available_systems: - raise ws.Skip(f"no '{system}' ROM folder on this box") + raise ws.Skip(f"this box has no '{system}' system") return system, {"system": system, "install": spec.get("install", "rom")} return accept @@ -147,35 +363,45 @@ def es_date(iso): def system_dir(cfg, system): - return os.path.join(roms_root(cfg), system) + return layout(cfg).base(system) + + +def box_systems(cfg): + """The box's own system definitions, by name — or None if unreadable. + + This one file answers both questions the engine has about the machine: which + systems it can run, and how it launches them. A system is in there because + an emulator for it was built into this particular Batocera image, which is a + truer compatibility test than the presence of a ROM folder. + """ + path = es_systems_cfg(cfg) + if not os.path.isfile(path): + debug(f"no {path} — falling back to the ROM folders") + return None + try: + root = ET.parse(path).getroot() + except ET.ParseError as exc: + log(f"warning: {path} is not valid XML ({exc}) — falling back to the ROM folders") + return None + nodes = {} + for node in root.findall("system"): + name = (node.findtext("name") or "").strip() + if name: + nodes[name] = node + return nodes or None def detect_systems(cfg): - """Systems this box has a ROM folder for — our compatibility check.""" + """Systems this box can run.""" + nodes = box_systems(cfg) + if nodes: + return set(nodes) root = roms_root(cfg) if not os.path.isdir(root): return set() return {d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d))} -def rel_rom_path(cfg, game): - return f"{subfolder(cfg)}/{game['asset']}" - - -def rel_image_path(cfg, game, ext): - return f"{subfolder(cfg)}/images/{game['name']}{ext}" - - -def rel_port_launcher(cfg, game): - return f"{subfolder(cfg)}/{game['name']}.sh" - - -def rel_port_data(cfg, game): - # A `.data` pont-prefixe rejti el az ES elol; a store almappaja alatta - # tartja kulon az egyes store-ok payloadjat. - return f".data/{subfolder(cfg)}/{game['name']}" - - def find_executable(root, name): """A kicsomagolt fa fo binarisa. @@ -202,9 +428,10 @@ def find_executable(root, name): def install_port(cfg, game, dry_run=False): """A zip asset kicsomagolasa + Ports indito. Visszaad (launcher, data) relativ utakat.""" - base = system_dir(cfg, game["system"]) - launcher_rel = rel_port_launcher(cfg, game) - data_rel = rel_port_data(cfg, game) + lay = layout(cfg) + base = lay.base(game["system"]) + launcher_rel = lay.rel_launcher(game) + data_rel = lay.rel_data(game) launcher_abs = os.path.join(base, launcher_rel) data_abs = os.path.join(base, data_rel) @@ -254,14 +481,15 @@ def install_port(cfg, game, dry_run=False): def install_game(cfg, game, state_entry, dry_run=False): """Download the asset + box art if missing. Returns (record, changed).""" - base = system_dir(cfg, game["system"]) + lay = layout(cfg) + base = lay.base(game["system"]) changed = False data_rel = None if game.get("install") == "port": # A port ket dologbol all: egy indito .sh es a kicsomagolt payload. - rom_rel = rel_port_launcher(cfg, game) - data_rel = rel_port_data(cfg, game) + rom_rel = lay.rel_launcher(game) + data_rel = lay.rel_data(game) installed = (os.path.isfile(os.path.join(base, rom_rel)) and os.path.isdir(os.path.join(base, data_rel)) and state_entry and state_entry.get("asset") == game["asset"]) @@ -271,7 +499,7 @@ def install_game(cfg, game, state_entry, dry_run=False): rom_rel, data_rel = install_port(cfg, game, dry_run=dry_run) changed = True else: - rom_rel = rel_rom_path(cfg, game) + rom_rel = lay.rel_rom(game) rom_abs = os.path.join(base, rom_rel) if os.path.isfile(rom_abs) and os.path.getsize(rom_abs) > 0: debug(f"{game['name']}: {game['asset']} already present") @@ -290,7 +518,7 @@ def install_game(cfg, game, state_entry, dry_run=False): if not wanted: if dry_run: log(f"would fetch box art for {game['name']}") - image_rel = rel_image_path(cfg, game, ".png") + image_rel = lay.rel_image(game, ".png") else: image_rel = fetch_image(cfg, game, base) changed = changed or bool(image_rel) @@ -298,7 +526,7 @@ def install_game(cfg, game, state_entry, dry_run=False): # Box art disappeared from the catalog: drop the file we cached. if image_rel and not dry_run: stale = os.path.join(base, image_rel) - if image_rel.startswith(subfolder(cfg) + "/") and os.path.isfile(stale): + if lay.owns(image_rel) and os.path.isfile(stale): os.unlink(stale) changed = True image_rel = None @@ -316,21 +544,25 @@ def fetch_image(cfg, game, base): body, ext = ws.download_image(cfg, game) if body is None: return None - rel = rel_image_path(cfg, game, ext) + rel = layout(cfg).rel_image(game, ext) ws.write_atomic(os.path.join(base, rel), body) debug(f"{game['name']}: box art -> {rel}") return rel -def remove_game(cfg, record, dry_run=False): - base = system_dir(cfg, record["system"]) +def remove_game(cfg, record, dry_run=False, lay=None): + """Delete one installed game's files. `lay` is for the migration, which has + to remove what a *previous* layout installed.""" + lay = lay or layout(cfg) + base = lay.base(record["system"]) for rel in (record.get("rom"), record.get("image")): if not rel: continue path = os.path.join(base, rel) - # Never step outside our own subfolder — another store may own it. - if not rel.startswith(subfolder(cfg) + "/"): - log(f"warning: refusing to delete {path} (outside {subfolder(cfg)}/)") + # Never step outside what this layout says is ours — in merge mode the + # folder is shared with the user and possibly with another store. + if not lay.owns(rel): + log(f"warning: refusing to delete {path} (not ours)") continue if os.path.isfile(path): if dry_run: @@ -343,8 +575,8 @@ def remove_game(cfg, record, dry_run=False): data_rel = record.get("data") if data_rel: data_abs = os.path.join(base, data_rel) - if not data_rel.startswith(f".data/{subfolder(cfg)}/"): - log(f"warning: refusing to delete {data_abs} (outside .data/{subfolder(cfg)}/)") + if not lay.owns(data_rel): + log(f"warning: refusing to delete {data_abs} (not ours)") elif os.path.isdir(data_abs): if dry_run: log(f"would remove {data_abs}") @@ -359,7 +591,7 @@ def remove_game(cfg, record, dry_run=False): def gamelist_path(cfg, system): - return os.path.join(system_dir(cfg, system), "gamelist.xml") + return layout(cfg).gamelist(system) def normalize_path(text): @@ -367,18 +599,37 @@ def normalize_path(text): return text[2:] if text.startswith("./") else text -def merge_gamelist(cfg, system, records, dry_run=False, backup=True): - """Rewrite only the / nodes under our subfolder. +# The tags we write ourselves, from the catalog. Everything else in a +# node was put there by EmulationStation — play counts, favourites, the last +# time it was launched — and has to survive a rewrite. +OUR_TAGS = ("path", "name", "desc", "image", "thumbnail", "developer", "publisher", + "releasedate") - Everything else in the file — the user's own scraped ROMs, their play - counts, favourites, and whatever another store installed under its own - subfolder — is parsed and written back untouched. + +def merge_gamelist(cfg, system, records, dry_run=False, backup=True, lay=None): + """Rewrite our / nodes, and only ours. + + In merge mode "ours" means everything under the store's subfolder: the rest + of the file is the box's — the user's own scraped ROMs, their play counts and + favourites, and whatever another store installed under its own subfolder. + + In system mode the file is in the store's own folder, so the rule is + different: a node is ours if we are writing that game now, or if the file it + points at is gone (a leftover of ours). A ROM the user dropped into the + folder themselves keeps its entry. + + Either way, the tags EmulationStation owns are carried over onto the new + node — a favourite stays a favourite across a sync. `backup=False` is for the uninstall: by then any copy worth having was made on the first sync, and backing up again would only preserve our own entries. """ - path = gamelist_path(cfg, system) - prefix = subfolder(cfg) + "/" + lay = lay or layout(cfg) + path = lay.gamelist(system) + base = lay.base(system) + wanted = {record["rom"]: record for record in records} + if not records and not os.path.isfile(path): + return if os.path.isfile(path): try: @@ -388,18 +639,23 @@ def merge_gamelist(cfg, system, records, dry_run=False, backup=True): root = ET.Element("gameList") else: backup_path = f"{path}.{cfg['store']['id']}-backup" - if backup and not os.path.exists(backup_path) and not dry_run: + if backup and lay.backup_gamelists and not os.path.exists(backup_path) and not dry_run: shutil.copy2(path, backup_path) else: root = ET.Element("gameList") + # What ES added to the nodes we are about to rewrite, kept aside by path. + carried = {} for node in list(root): target = normalize_path(node.findtext("path")) - if target.startswith(prefix) or target == subfolder(cfg): - root.remove(node) + if not lay.claims(target, wanted, base): + continue + if target in wanted: + carried[target] = [child for child in node if child.tag not in OUR_TAGS] + root.remove(node) folder_name = cfg["emulationstation"].get("folder_name") - if records and folder_name: + if records and lay.folder_node and folder_name: folder = ET.SubElement(root, "folder") ET.SubElement(folder, "path").text = "./" + subfolder(cfg) ET.SubElement(folder, "name").text = folder_name @@ -419,9 +675,16 @@ def merge_gamelist(cfg, system, records, dry_run=False, backup=True): released = record.get("releasedate") or es_date(record.get("created_at")) if released: ET.SubElement(game, "releasedate").text = released + for child in carried.get(record["rom"], []): + game.append(child) indent(root) blob = b'\n' + ET.tostring(root, encoding="utf-8") + if os.path.isfile(path): + with open(path, "rb") as fh: + if fh.read() == blob: + debug(f"{path} is up to date") + return if dry_run: log(f"would write {path} ({len(records)} entries)") return @@ -444,6 +707,358 @@ def indent(elem, level=0): elem.tail = pad +# -------------------------------------------------------------------------- +# the store's own menu entry +# -------------------------------------------------------------------------- + +# The folder that holds the updater. Not a catalog platform, so it can never +# collide with one. +STORE_SYSTEM = "store" + + +def es_parent_name(cfg, box=None): + """The ES system name of the menu entry itself. + + The store id — unless the box already has a system by that name. An overlay + whose `` matches an existing system *modifies that system* instead of + adding one, and a store that called itself `nes` must not rewrite the box's + NES. + """ + name = cfg["store"]["id"] + if box and name in box: + name = f"{name}-store" + log(f"'{cfg['store']['id']}' is already a system on this box — " + f"the menu entry is called '{name}'") + return name + + +def source_system(system): + """The box system one of our systems borrows its launcher from.""" + return "ports" if system == STORE_SYSTEM else system + + +def system_label(cfg, system, src): + labels = menu(cfg).get("labels") or {} + if system in labels: + return labels[system] + if system == STORE_SYSTEM: + return "Store" + fullname = (src.findtext("fullname") or "").strip() if src is not None else "" + return fullname or system + + +def configured_exts(cfg, system): + """The extensions our own config expects for a system — the last resort when + the box's system definition has none to copy.""" + exts = set() + for spec in (cfg.get("platforms") or {}).values(): + if spec.get("system") != system: + continue + ext = spec.get("ext") + for value in (ext.values() if isinstance(ext, dict) else [ext]): + if value: + exts.add(value) + return " ".join(sorted(exts)) + + +def text_node(parent, tag, value): + node = ET.SubElement(parent, tag) + node.text = value + return node + + +def derive_child(cfg, parent, system, src): + """One `` of ours, borrowing everything about emulators from the box. + + `%SYSTEM%` is resolved here rather than left to EmulationStation: ES replaces + it with the name of the system being launched (`FileData.cpp`), which for us + would be `ttg-c64` — a name configgen has never heard of. The box's own + system name goes in instead, so a game of ours is launched exactly as the + box would launch it, with the emulator the user configured for that system. + """ + src_name = source_system(system) + node = ET.Element("system") + text_node(node, "name", f"{parent}-{system}") + text_node(node, "fullname", system_label(cfg, system, src)) + text_node(node, "path", layout(cfg).base(system)) + text_node(node, "extension", + (src.findtext("extension") or "").strip() or configured_exts(cfg, system)) + text_node(node, "command", (src.findtext("command") or "").replace("%SYSTEM%", src_name)) + for tag in ("platform", "theme", "manufacturer", "release", "hardware"): + value = (src.findtext(tag) or "").strip() + if value: + text_node(node, tag, value) + # This is what puts the system inside the store's menu entry instead of + # giving it one of its own. + text_node(node, "group", parent) + emulators = src.find("emulators") + if emulators is not None: + node.append(emulators) + return node + + +def build_es_systems(cfg, systems, box): + """The `` for the store: the menu entry, and a system under it + for every folder that has something in it.""" + lay = layout(cfg) + parent = es_parent_name(cfg, box) + root = ET.Element("systemList") + + # The menu entry is declared rather than left to ES to invent, for two + # reasons: it is the only way to give it a `` (a group ES creates + # itself looks for a theme folder named after the group, which no theme + # has), and with `HideUniqueGroups` on — the default — a group with a single + # system in it is dissolved unless a system by that name exists. + entry = ET.SubElement(root, "system") + text_node(entry, "name", parent) + text_node(entry, "fullname", menu(cfg).get("name") or cfg["store"]["name"]) + text_node(entry, "path", lay.store_root()) + # ES drops a system that has no extension or no command, so both are here — + # an extension nothing on earth has, and a command that does nothing. This + # entry is a shelf for the systems below, it never holds a game itself. + text_node(entry, "extension", f".{parent}-none") + text_node(entry, "command", "true") + text_node(entry, "platform", "pc") + text_node(entry, "theme", menu(cfg).get("theme") or "ports") + + for system in sorted(systems): + src = box.get(source_system(system)) + if src is None: + log(f"warning: no '{source_system(system)}' system on this box — " + f"{system} is not added to the menu") + continue + root.append(derive_child(cfg, parent, system, src)) + return root + + +def es_systems_path(cfg): + return os.path.join(es_config_dir(cfg), f"es_systems_{cfg['store']['id']}.cfg") + + +def write_es_systems(cfg, systems, dry_run=False): + """Declare our systems to EmulationStation. True if the file changed. + + ES writes this file never, so unlike a gamelist it can be written while ES + is running. What it cannot do is take effect without a restart. + """ + box = box_systems(cfg) + if not box: + log(f"warning: cannot read {es_systems_cfg(cfg)} — the store's menu entry was " + "not written. The games are installed; ES will list them once it can.") + return False + + root = build_es_systems(cfg, systems, box) + indent(root) + blob = b'\n' + ET.tostring(root, encoding="utf-8") + path = es_systems_path(cfg) + if os.path.isfile(path): + with open(path, "rb") as fh: + if fh.read() == blob: + debug(f"{path} is up to date") + return False + if dry_run: + log(f"would write {path} ({len(root) - 1} systems)") + return False + os.makedirs(os.path.dirname(path), exist_ok=True) + ws.write_atomic(path, blob) + log(f"EmulationStation systems: {path} ({len(root) - 1} systems)") + return True + + +def remove_es_systems(cfg, dry_run=False): + path = es_systems_path(cfg) + if not os.path.isfile(path): + return + if dry_run: + log(f"would remove {path}") + return + os.unlink(path) + log(f"removed {path}") + + +def updater_script(cfg): + """The updater, as the ES menu launches it. + + A `.sh` in a folder of ours, so it is a game as far as ES is concerned. It + calls this same engine — not the installer's launcher — so it keeps working + whatever the store home is called. + """ + return ( + "#!/bin/bash\n" + f"# {cfg['store']['name']} — pulls new releases from the catalog.\n" + f"exec >>{shlex.quote(os.path.join(ws.HOME, 'store.log'))} 2>&1\n" + 'echo "=== $(date -Iseconds) sync started ==="\n' + f"BATOCERA_STORE_HOME={shlex.quote(ws.HOME)} " + f"{shlex.quote(sys.executable)} {shlex.quote(os.path.abspath(__file__))} " + f"--config {shlex.quote(ws.CONFIG_PATH)} sync-from-es\n" + 'echo "=== $(date -Iseconds) sync finished (exit $?) ==="\n' + ).encode("utf-8") + + +def updater_records(cfg): + label = menu(cfg).get("name") or cfg["store"]["name"] + return [{ + "system": STORE_SYSTEM, + "rom": "update.sh", + "title": f"Update {label}", + "desc": "Downloads everything new from the catalog and refreshes this menu.", + }] + + +def write_updater(cfg, dry_run=False): + path = os.path.join(layout(cfg).base(STORE_SYSTEM), "update.sh") + blob = updater_script(cfg) + if os.path.isfile(path): + with open(path, "rb") as fh: + if fh.read() == blob: + return + if dry_run: + log(f"would write {path}") + return + ws.write_atomic(path, blob) + os.chmod(path, 0o755) + log(f"updater: {path}") + + +def apply_menu(cfg, systems, dry_run=False, gamelists=True): + """Write everything the store's menu entry is made of. + + Returns True if EmulationStation has to restart to notice. `gamelists=False` + is for a sync launched from inside ES, which defers every gamelist write to + the child that waits for ES to exit. + """ + lay = layout(cfg) + if not isinstance(lay, SystemLayout): + return False + + wanted = set(systems) + updater = bool(menu(cfg).get("updater")) + if updater: + wanted.add(STORE_SYSTEM) + # ES refuses a system whose path does not exist, so the folders go first. + for system in sorted(wanted): + if not dry_run: + os.makedirs(lay.base(system), exist_ok=True) + + changed = write_es_systems(cfg, wanted, dry_run=dry_run) + if updater: + write_updater(cfg, dry_run=dry_run) + if gamelists: + merge_gamelist(cfg, STORE_SYSTEM, updater_records(cfg), dry_run=dry_run) + return changed + + +# -------------------------------------------------------------------------- +# migration from the merge layout +# -------------------------------------------------------------------------- + + +def ports_entries(cfg): + """The Ports scripts that run this store, found by looking inside them. + + The name cannot be derived from the config — a store repository may set + `BATOCERA_PORT_NAME` to anything — but whatever it is called, the script + names this store's home. Same trick as `uninstall.sh`. + """ + ports_dir = os.path.join(roms_root(cfg), "ports") + if not os.path.isdir(ports_dir): + return [] + found = [] + for name in sorted(os.listdir(ports_dir)): + if not name.endswith(".sh"): + continue + path = os.path.join(ports_dir, name) + try: + with open(path, "r", errors="replace") as fh: + if ws.HOME and ws.HOME in fh.read(): + found.append(path) + except OSError: + continue + return found + + +def remove_ports_entry(cfg, dry_run=False): + for path in ports_entries(cfg): + if dry_run: + log(f"would remove the Ports entry {path}") + else: + os.unlink(path) + log(f"removed the Ports entry {path}") + + +def migrate_layout(cfg, dry_run=False): + """Take the store off the box when `menu.mode` changed under it. + + The games move by being removed and downloaded again, not by being carried + across: the catalog is small, and a half-moved install would be worse than a + slightly longer sync. What this does is exactly what an uninstall does — + ROMs, box art, port payloads, gamelist nodes, the empty folders — so the old + layout is gone completely, and then the sync fills the new one. + + A state file with no `layout` in it was written by an engine that only had + the merge layout, so that is what it is assumed to be. + """ + want = layout_name(layout(cfg)) + state = ws.load_state() + have = state.get("layout") or "merge" + if have == want: + return False + + installed = state["installed"] + if not installed: + if not dry_run: + state["layout"] = want + ws.save_state(state) + return False + + log(f"the store's layout changed ({have} -> {want}) — removing the " + f"{len(installed)} installed games from the old one, they will be " + "downloaded again") + old = layout_named(cfg, have) + systems = set() + for key, record in list(installed.items()): + remove_game(cfg, record, dry_run=dry_run, lay=old) + systems.add(ws.record_scope(record)) + if not dry_run: + del installed[key] + + if have == "system": + # The updater and the menu entry belong to the layout we are leaving. + systems.add(STORE_SYSTEM) + updater = os.path.join(old.base(STORE_SYSTEM), "update.sh") + if os.path.isfile(updater): + if dry_run: + log(f"would remove {updater}") + else: + os.unlink(updater) + remove_es_systems(cfg, dry_run=dry_run) + + for system in sorted(systems): + merge_gamelist(cfg, system, [], dry_run=dry_run, backup=False, lay=old) + drop_empty_gamelist(cfg, system, dry_run=dry_run, lay=old) + + dirs = [] + for system in sorted(systems): + dirs += old.purge_dirs(system) + if have == "system": + dirs.append(old.store_root()) + ws.prune_empty_dirs(dirs, root=roms_root(cfg), dry_run=dry_run) + + if not wants_ports_entry(cfg): + remove_ports_entry(cfg, dry_run=dry_run) + + leftovers = gamelist_backups(cfg, systems, lay=old) + if leftovers: + log("left in place — copies of your gamelists as we first found them:") + for path in leftovers: + log(f" {path}") + + if not dry_run: + state["layout"] = want + ws.save_state(state) + return True + + # -------------------------------------------------------------------------- # EmulationStation # -------------------------------------------------------------------------- @@ -500,11 +1115,15 @@ def perform_sync(cfg, names=None, dry_run=False): Returns (state, changed, touched_systems). Gamelists are left alone — the caller decides when it is safe to write them. """ + migrate_layout(cfg, dry_run=dry_run) + systems = detect_systems(cfg) if not systems: - # An unmounted or wrong ROMs root would otherwise look like "no system - # is compatible any more" and prune everything we ever installed. - die(f"no system folders under {roms_root(cfg)} — is it mounted?") + # An unreadable es_systems.cfg with an unmounted or wrong ROMs root + # behind it would otherwise look like "no system is compatible any more" + # and prune everything we ever installed. + die(f"cannot tell which systems this box has: no {es_systems_cfg(cfg)}, " + f"and no system folders under {roms_root(cfg)} — is it mounted?") catalog = ws.fetch_catalog(cfg, use_cache=True) games, skipped = select_games(cfg, catalog, systems) @@ -555,18 +1174,23 @@ def perform_sync(cfg, names=None, dry_run=False): def cmd_sync(cfg, args): state, changed, touched = perform_sync(cfg, args.name, dry_run=args.dry_run) + grouped = ws.records_by_scope(state["installed"]) + # The menu entry is written even when nothing was downloaded: a config that + # renamed it, or a first sync that found nothing to install, still has to + # leave the box with a way in. + menu_changed = apply_menu(cfg, grouped, dry_run=args.dry_run) + if args.dry_run: log("dry run — nothing written") return 0 if not changed and not args.force: log("already up to date") - return 0 + else: + for system in sorted(touched | set(grouped)): + merge_gamelist(cfg, system, grouped.get(system, [])) - grouped = ws.records_by_scope(state["installed"]) - for system in sorted(touched | set(grouped)): - merge_gamelist(cfg, system, grouped.get(system, [])) - - if cfg["emulationstation"].get("restart") and not args.no_restart and es_pid(): + if (cfg["emulationstation"].get("restart") and not args.no_restart + and (changed or menu_changed) and es_pid()): restart_es() return 0 @@ -579,15 +1203,19 @@ def cmd_sync_from_es(cfg, args): child that waits for the old ES process to die first. """ state, changed, _ = perform_sync(cfg) + grouped = ws.records_by_scope(state["installed"]) pid = es_pid() - + # The es_systems overlay and the updater script are ours alone — ES never + # writes them back — so they can be written now. Only the gamelists wait. + menu_changed = apply_menu(cfg, grouped, gamelists=not pid) if not changed: log("already up to date") - if not pid or not changed: - # Nothing racing us (or nothing new) — write the gamelists right here. - if changed: - for system, records in ws.records_by_scope(state["installed"]).items(): - merge_gamelist(cfg, system, records) + + if not pid: + # Nothing racing us — write the gamelists right here. A file whose + # content has not changed is not rewritten, so this is cheap. + for system, records in grouped.items(): + merge_gamelist(cfg, system, records) return 0 subprocess.Popen( @@ -595,7 +1223,8 @@ def cmd_sync_from_es(cfg, args): "--config", ws.CONFIG_PATH, "apply-gamelists", "--wait-pid", str(pid)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True, ) - if cfg["emulationstation"].get("restart") and not args.no_restart: + if (cfg["emulationstation"].get("restart") and not args.no_restart + and (changed or menu_changed)): restart_es() return 0 @@ -608,9 +1237,11 @@ def cmd_apply_gamelists(cfg, args): grouped = ws.records_by_scope(state["installed"]) if not grouped: log("nothing installed") - return 0 for system, records in grouped.items(): merge_gamelist(cfg, system, records, dry_run=args.dry_run) + # The updater's own gamelist waited for ES to exit along with the rest. + if isinstance(layout(cfg), SystemLayout) and menu(cfg).get("updater"): + merge_gamelist(cfg, STORE_SYSTEM, updater_records(cfg), dry_run=args.dry_run) return 0 @@ -619,8 +1250,12 @@ def cmd_list(cfg, args): catalog = ws.fetch_catalog(cfg, use_cache=True) host = ws.host() log(f"architecture: {host['arch']}") + state = ws.load_state() + if state["installed"] and (state.get("layout") or "merge") != layout_name(layout(cfg)): + log(f"the next sync will move these games to the '{layout_name(layout(cfg))}' " + f"layout — they are removed from the old one and downloaded again") games, skipped = select_games(cfg, catalog, systems, host) - installed = ws.load_state()["installed"] + installed = state["installed"] if not games: log("no compatible games in the catalog") @@ -663,15 +1298,20 @@ def cmd_remove(cfg, args): return 0 -def drop_empty_gamelist(cfg, system, dry_run=False): +def drop_empty_gamelist(cfg, system, dry_run=False, lay=None): """Delete a gamelist that is empty and that we created. - No `.-backup` next to it means we never found a gamelist there — it is - ours alone, and with our nodes gone there is nothing in it. Leaving the empty - shell behind would mean the uninstall did not quite put the box back. + In merge mode, no `.-backup` next to it means we never found a gamelist + there — it is ours alone, and with our nodes gone there is nothing in it. In + system mode the file is in our own folder, so an empty one is always ours. + Leaving the empty shell behind would mean the uninstall did not quite put the + box back. """ - path = gamelist_path(cfg, system) - if not os.path.isfile(path) or os.path.exists(f"{path}.{cfg['store']['id']}-backup"): + lay = lay or layout(cfg) + path = lay.gamelist(system) + if not os.path.isfile(path): + return + if lay.backup_gamelists and os.path.exists(f"{path}.{cfg['store']['id']}-backup"): return try: root = ET.parse(path).getroot() @@ -686,9 +1326,10 @@ def drop_empty_gamelist(cfg, system, dry_run=False): log(f"removed the now-empty {path}") -def gamelist_backups(cfg, systems): +def gamelist_backups(cfg, systems, lay=None): """The `gamelist.xml.-backup` copies we made, if any are still around.""" - paths = [f"{gamelist_path(cfg, system)}.{cfg['store']['id']}-backup" for system in systems] + lay = lay or layout(cfg) + paths = [f"{lay.gamelist(system)}.{cfg['store']['id']}-backup" for system in systems] return sorted(p for p in paths if os.path.isfile(p)) @@ -700,38 +1341,58 @@ def cmd_purge(cfg, args): what `uninstall.sh` runs before deleting the store itself: the shell script has no way of knowing which files were ours. - The gamelists are merged, not deleted: they belong to the box, and after our - nodes are gone the rest of the user's library is still in them. + In merge mode the gamelists are merged, not deleted: they belong to the box, + and after our nodes are gone the rest of the user's library is still in them. + In system mode they are ours, and go with the folders — along with the + es_systems overlay, which is what the store's menu entry is made of. """ state = ws.load_state() installed = state["installed"] if not installed: log("nothing is installed by this store") + # Which layout the files are actually in is what `state.json` says, not what + # the config asks for today: an upgrade whose first sync never ran still has + # its games in the old places, and an uninstall has to find them there. + recorded = state.get("layout") or ("merge" if installed else layout_name(layout(cfg))) + lay = layout_named(cfg, recorded) + systems = set() for key, record in list(installed.items()): - remove_game(cfg, record, dry_run=args.dry_run) + remove_game(cfg, record, dry_run=args.dry_run, lay=lay) systems.add(ws.record_scope(record)) if not args.dry_run: del installed[key] if not args.dry_run: ws.save_state(state) + # The menu entry and its updater are not catalog games, so no state record + # points at them. Both are removed whichever layout is recorded — a leftover + # es_systems file would leave the box with a menu entry and nothing in it. + remove_es_systems(cfg, dry_run=args.dry_run) + store_layout = SystemLayout(cfg) + updater = os.path.join(store_layout.base(STORE_SYSTEM), "update.sh") + if os.path.isfile(updater): + if args.dry_run: + log(f"would remove {updater}") + else: + os.unlink(updater) for system in sorted(systems): - merge_gamelist(cfg, system, [], dry_run=args.dry_run, backup=False) - drop_empty_gamelist(cfg, system, dry_run=args.dry_run) + merge_gamelist(cfg, system, [], dry_run=args.dry_run, backup=False, lay=lay) + drop_empty_gamelist(cfg, system, dry_run=args.dry_run, lay=lay) dirs = [] for system in sorted(systems): - base = system_dir(cfg, system) - dirs += [os.path.join(base, subfolder(cfg), "images"), - os.path.join(base, subfolder(cfg)), - # `.data` itself stays: it is shared ground, another store may - # still be keeping a payload there. - os.path.join(base, ".data", subfolder(cfg))] + dirs += lay.purge_dirs(system) + # The store's own folder is the system layout's, whatever the records say. + if os.path.isdir(store_layout.store_root()): + merge_gamelist(cfg, STORE_SYSTEM, [], dry_run=args.dry_run, backup=False, + lay=store_layout) + drop_empty_gamelist(cfg, STORE_SYSTEM, dry_run=args.dry_run, lay=store_layout) + dirs += store_layout.purge_dirs(STORE_SYSTEM) + [store_layout.store_root()] ws.prune_empty_dirs(dirs, root=roms_root(cfg), dry_run=args.dry_run) - leftovers = gamelist_backups(cfg, systems) + leftovers = gamelist_backups(cfg, systems, lay=lay) if leftovers: log("left in place — these are copies of your gamelists as we first found them:") for path in leftovers: @@ -753,8 +1414,17 @@ def cmd_config(cfg, args): log(f"wrote default config to {args.config}") return 0 # A felismert architektura nem a configbol jon, de az asset-valasztast - # eldonti, ezert itt is lathatonak kell lennie. - print(json.dumps({"detected_arch": ws.machine(), **cfg}, indent=2, ensure_ascii=False)) + # eldonti, ezert itt is lathatonak kell lennie. Ugyanez all a menupontra: + # a neve es a helye a configbol szarmazik, de nem all benne. + lay = layout(cfg) + extra = {"detected_arch": ws.machine(), "install_dir": lay.base("")} + if isinstance(lay, SystemLayout): + extra["es_menu"] = { + "entry": es_parent_name(cfg, box_systems(cfg)), + "file": es_systems_path(cfg), + "root": lay.store_root(), + } + print(json.dumps({**extra, **cfg}, indent=2, ensure_ascii=False)) return 0 @@ -818,6 +1488,9 @@ def main(argv=None): overrides={"store.base_url": args.base_url, "paths.roms_root": args.roms_root}, required=("paths.subfolder",), ) + mode = (menu(cfg).get("mode") or "system").lower() + if mode not in ("system", "merge"): + die(f"config: emulationstation.menu.mode is '{mode}' — expected 'system' or 'merge'") os.makedirs(ws.HOME, exist_ok=True) return args.func(cfg, args) diff --git a/uninstall.sh b/uninstall.sh index 715cfb9..e220b50 100755 --- a/uninstall.sh +++ b/uninstall.sh @@ -2,9 +2,9 @@ # Remove a WarpEngine store from a Batocera box. # # The engine does the removing, not this script: it is the only thing that knows -# which ROMs, box art, Ports payloads and gamelist entries were the store's — so -# it runs `purge` first, and only then do the launcher, the Ports entry and the -# store home go. +# which ROMs, box art, Ports payloads, gamelist entries and ES systems were the +# store's — so it runs `purge` first, and only then do the launcher, the Ports +# entry and the store home go. # # ./uninstall.sh # the only store installed, or say which one # STORE_ID=example ./uninstall.sh @@ -127,5 +127,5 @@ if [ "$DRY_RUN" = "1" ]; then say "dry run — nothing was removed." else say "done. EmulationStation was restarted if it was running; if not, the games are" - say "already gone from the gamelists." + say "already gone from the gamelists, and so is the store's own menu entry." fi