diff --git a/README.md b/README.md index f54ed9a..5d6dd5a 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ # warp-engine-batocera-store — a Batocera store engine for WarpEngine sites Pulls games from a [WarpEngine](https://git.teletypegames.org/tools/warp_engine) -catalog straight into a Batocera box's ROM folders, with EmulationStation -metadata and box art. Triggered from the **Ports** menu on the device, or from -the CLI over SSH. +catalog straight onto a Batocera box, with EmulationStation metadata and box art, +and gives the store **its own entry in the EmulationStation menu**. Started from +that entry on the device, or from the CLI over SSH. This repository is the engine only. It knows the WarpEngine API but nothing about any particular site: the host, the store's name and where its games land @@ -31,18 +31,65 @@ The reference store is [`ttg-batocera-store`](https://git.teletypegames.org/stores/ttg-batocera-store). ``` -Ports ▸ "" +"" ▸ Store ▸ "Update " │ ├─ GET /api/software the whole catalog ├─ keep platforms this box can run c64 → c64, tic80 → tic80 ├─ pick the newest non-dev release that carries the right asset - ├─ GET /api/download?path= .prg / .tic into the ROM folder + ├─ GET /api/download?path= .prg / .tic into the store's folder ├─ GET box art - └─ merge gamelist.xml title, desc, author, image + ├─ write gamelist.xml title, desc, author, image + └─ write es_systems_.cfg the store's menu entry ``` Python 3 standard library only — Batocera ships python3 and no pip. +## The store's own menu entry + +Everything the store installs lives in one folder of its own, +`/userdata/roms/`, with a directory per platform inside it. An +`es_systems_.cfg` in EmulationStation's user config declares one ES system +per platform, all carrying the same ``, and ES turns a group into a single +carousel entry holding a folder per system: + +``` +main carousel: … ▸ Teletype Games ▸ ┌ Commodore 64 + ├ TIC-80 + ├ Ports + └ Store → "Update Teletype Games" +``` + +Nothing about emulators is hardcoded here. The engine reads the box's own +`/usr/share/emulationstation/es_systems.cfg` and copies the launch command, the +extensions, the platform, the theme and the emulator list off the system it is +borrowing — a file Batocera generated for this particular image, so it lists +exactly the emulators the box has. The one thing that is rewritten is `%SYSTEM%`: +ES replaces it with the name of the system being launched, which for us would be +`ttg-c64`, a name configgen has never heard of. The box's own system name goes in +instead, and a game of ours is launched exactly as the box would launch it, with +the emulator the user configured for that system. + +Three details worth knowing, all of them EmulationStation's: + +- **The menu entry is declared, not implied.** ES would invent the group parent + by itself, but then its theme folder would be the group's name, which no theme + has — and `HideUniqueGroups` (on by default) dissolves a group that has a + single system in it unless a system by that name exists. The declared parent + fixes both: it carries a `` (`ports` by default, since every theme has + that one) and keeps the entry in place on a box that only has c64 games. +- **An empty system does not show.** Only the platforms that actually have + something installed become folders, so the entry is never full of dead ends. +- **A restart is needed for a new system.** The engine restarts ES itself after a + sync that added one. + +The store id names the entry. If the box already has a system by that name — a +store called `nes`, say — the entry is called `-store` instead, and the log +says so: an overlay whose `` matches an existing system would *modify that +system* rather than add one. + +For the old behaviour — games inside the box's own systems, under a subfolder, +merged into the box's gamelists — set `emulationstation.menu.mode` to `merge`. + ## Installing a store The installer takes the store as a parameter, so it works with any config: @@ -60,9 +107,9 @@ ssh root@batocera 'STORE_CONFIG=/tmp/my-config.json /tmp/warp-engine-batocera-st ``` It reads `store.id` from the config, installs the engine and the config into -`/userdata/system/batocera-store//`, writes a `-store` launcher and a -Ports entry, and runs the first sync. Restart EmulationStation -(`batocera-es-swissknife --restart`) to see the games. +`/userdata/system/batocera-store//`, writes a `-store` launcher, and runs +the first sync — which is what creates the menu entry. Restart EmulationStation +(`batocera-es-swissknife --restart`) to see it. It installs two files: `store.py` from this repository and `warpstore.py` from the [shared core](https://git.teletypegames.org/engines/warpstore). They have to @@ -73,6 +120,23 @@ Installer knobs, all environment variables: `STORE_CONFIG` (required), `ENGINE_RAW_BASE`, `WARPSTORE_RAW_BASE`, and `WARPSTORE_SRC` to install a local `warpstore.py` instead of downloading it. +## Upgrading a store that was installed before the menu entry + +Engine 4.0 is the layout change, so the first sync after the upgrade moves the +store off the box's own systems. It does that by **removing the old install and +downloading again**, not by carrying files across: the ROMs, box art and port +payloads go, our nodes come out of the box's gamelists, the empty folders are +swept, the Ports entry is taken away — and then the sync fills the store's own +folder. The catalog is small enough that this is minutes, and there is no +half-moved state to end up in. + +What is kept: your gamelists (only our nodes go — play counts and favourites of +your own games are untouched), and the `gamelist.xml.-backup` copies, which +are listed at the end of the run for you to delete or keep. + +`-n` shows the whole thing without doing any of it. To stay on the old layout, +set `emulationstation.menu.mode` to `merge` before the first sync. + ## Writing a store repository Four small files: @@ -92,7 +156,6 @@ my-batocera-store/ set -euo pipefail ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/stores/warp-engine-batocera-store/raw/branch/master}" export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-batocera-store/raw/branch/master/config.json}" -export BATOCERA_PORT_NAME="My Store" curl -fsSL "$ENGINE_RAW_BASE/install.sh" | sh ``` @@ -105,13 +168,15 @@ curl -fsSL "$ENGINE_RAW_BASE/uninstall.sh" | sh ``` Pick a `store.id` and a `paths.subfolder` nobody else uses — they are what keeps -two stores on the same box from deleting each other's games. +two stores on the same box out of each other's way, and the id is what names the +menu entry and its `es_systems` file. ## Use -**On the device.** *Ports ▸ "\"*. The script downloads anything -new, then restarts EmulationStation so the games show up. Ports scripts get no -console in Batocera, so the output goes to `store.log` in the store home. +**On the device.** *"\" ▸ Store ▸ "Update \"*. It +downloads anything new, then restarts EmulationStation so the games show up. ES +gives a launched script no console, so the output goes to `store.log` in the +store home. **Over SSH**, through the launcher the installer wrote: @@ -119,12 +184,12 @@ console in Batocera, so the output goes to `store.log` in the store home. S=/userdata/system/batocera-store/example-store $S list # compatible catalog entries; * installed, ^ update available -$S sync # download everything new, refresh gamelists +$S sync # download everything new, refresh gamelists and the menu $S sync blessingofra # just one title (never prunes) $S -n sync # dry run $S remove c64:demo # uninstall (bare name works too) $S purge # uninstall everything this store installed -$S config # effective configuration +$S config # effective configuration, and where the menu entry lives ``` Global flags go **before** the subcommand: `$S --roms-root /tmp/roms sync`. @@ -136,12 +201,12 @@ ssh root@batocera 'curl -fsSL https://git.teletypegames.org/stores/warp-engine-b ``` or `./uninstall.sh` from a checkout on the box. It removes the games first, -through the engine — `purge` — and only then the Ports entry, the launcher and the -store home: the shell script has no way of knowing which ROMs, box art, Ports -payloads and gamelist entries were the store's, and the engine's `state.json` -does. If the engine cannot finish — an unmounted ROMs root, say — nothing else is -touched, so you are never left with the files but not the engine that knows about -them. +through the engine — `purge` — and only then the launcher, any Ports entry and +the store home: the shell script has no way of knowing which ROMs, box art, port +payloads, gamelist entries and ES systems were the store's, and the engine's +`state.json` does. If the engine cannot finish — an unmounted ROMs root, say — +nothing else is touched, so you are never left with the files but not the engine +that knows about them. | Environment variable | Meaning | |---|---| @@ -151,11 +216,19 @@ them. | `KEEP_HOME` | `1` to keep the store home (state, catalog cache, log) for a reinstall | | `BATOCERA_STORE_ROOT`, `BATOCERA_PORTS_DIR`, `BATOCERA_PORT_NAME` | as for the installer | -What it deliberately leaves behind: +In the store's own layout an uninstall is total: the store's ROM folder, its +gamelists and its `es_systems_.cfg` all go, and the box is left as it was. +In `merge` mode there is more to be careful about, and these stay behind: -- **ROMs you put in the store's subfolder yourself.** Directories are removed only when empty, so one stray file keeps its directory. `ports/.data` itself also stays: another store may still keep a payload there. -- **your gamelists.** They belong to the box: our `` and `` nodes go, the rest — your own games, their play counts and favourites — is written back untouched. A gamelist we *created* and that is left empty is deleted, since it was never yours. -- **`gamelist.xml.-backup`**, listed at the end of the run: your gamelists as we first found them. Deleting them is your call. +- **ROMs you put in the store's subfolder yourself.** Directories are removed + only when empty, so one stray file keeps its directory. `ports/.data` itself + also stays: another store may still keep a payload there. +- **your gamelists.** They belong to the box: our `` and `` nodes + go, the rest — your own games, their play counts and favourites — is written + back untouched. A gamelist we *created* and that is left empty is deleted, + since it was never yours. +- **`gamelist.xml.-backup`**, listed at the end of the run: your + gamelists as we first found them. Deleting them is your call. A dry run cannot predict which directories will end up empty, so it lists the files but not the directory removals. @@ -178,18 +251,38 @@ curl -fsSL https://git.teletypegames.org/engines/warpstore/raw/branch/master/uni ├── config.json the store that selected it └── state.json, catalog.json, store.log -/userdata/roms/c64/example/ blessingofra-2.0.0.prg, rabbit-1.0.0.prg, … -/userdata/roms/c64/example/images/ blessingofra.png, … -/userdata/roms/c64/gamelist.xml our entries merged in -/userdata/roms/ports/Example Store.sh +/userdata/system/configs/emulationstation/es_systems_example.cfg the menu entry + +/userdata/roms/example/ the store's own ROM folder +├── c64/ +│ ├── blessingofra-2.0.0.prg, rabbit-1.0.0.prg, … +│ ├── images/blessingofra.png, … +│ └── gamelist.xml +├── ports/ +│ ├── .sh the launchers ES lists +│ ├── images/.png +│ ├── .data// the unpacked programs +│ └── gamelist.xml +└── store/ + ├── update.sh the sync, as the menu starts it + └── gamelist.xml ``` -Everything a store installs lives under its own `paths.subfolder`, so a prune -can never reach ROMs you put there yourself — or another store's games. The -first time a `gamelist.xml` is touched it is copied to -`gamelist.xml.-backup`; on every merge only ``/`` nodes -under that store's subfolder are rewritten, so your own entries keep their play -counts, favourites and scraped media, and two stores can share one gamelist. +The box's own system folders are not touched at all, which is what makes a prune +safe: nothing outside the store's folder can be reached, and inside it only what +`state.json` says we installed is ever deleted — a ROM you drop in there +yourself keeps its file *and* its gamelist entry. + +In `merge` mode the same content goes into `///` +and our nodes are merged into the box's `gamelist.xml`. The first time one is +touched it is copied to `gamelist.xml.-backup`; on every merge only +``/`` nodes under that store's subfolder are rewritten, so your own +entries keep their play counts, favourites and scraped media, and two stores can +share one gamelist. + +Either way, the tags EmulationStation owns — `favorite`, `playcount`, +`lastplayed`, `rating` — are carried over onto the node we rewrite. A favourite +stays a favourite across a sync. ## Configuration @@ -197,15 +290,22 @@ counts, favourites and scraped media, and two stores can share one gamelist. | Key | Default | Meaning | | --- | --- | --- | -| `store.id` | `warp` | slug: names the store home, log prefix and gamelist backup | -| `store.name` | `WarpEngine Store` | display name for the Ports entry | +| `store.id` | `warp` | slug: names the store home, the log prefix, the menu entry and its `es_systems` file | +| `store.name` | `WarpEngine Store` | display name of the menu entry | | `store.base_url` | — | WarpEngine host | | `store.api.catalog` | `/api/software` | catalog endpoint, if the engine is mounted elsewhere | | `store.api.download` | `/api/download` | download endpoint | | `paths.roms_root` | `/userdata/roms` | where systems live | -| `paths.subfolder` | `warp` | this store's subfolder inside each system | -| `emulationstation.folder_name` | `WarpEngine Store` | display name of that folder in ES | +| `paths.subfolder` | `warp` | the store's own folder inside `roms_root` (in `merge` mode: its subfolder inside each system) | +| `emulationstation.menu.mode` | `system` | `system`: the store gets its own menu entry. `merge`: install into the box's own systems | +| `emulationstation.menu.name` | `null` → `store.name` | label of the menu entry | +| `emulationstation.menu.theme` | `ports` | theme folder the entry borrows its logo from | +| `emulationstation.menu.updater` | `true` | an *Update …* entry inside the menu that runs the sync | +| `emulationstation.menu.labels` | `{}` | per-system folder labels; a system not listed keeps the box's own `` | +| `emulationstation.ports_entry` | `null` → `false` in `system` mode, `true` in `merge` mode | also write a Ports entry that runs the sync | +| `emulationstation.folder_name` | `WarpEngine Store` | `merge` mode only: display name of our folder inside the box's gamelist; `null` hides it | | `emulationstation.restart` | `true` | restart EmulationStation after a change | +| `emulationstation.config_dir` | `null` → `/userdata/system/configs/emulationstation` | where ES keeps its user config | | `catalog.statuses` | `["released", "archived"]` | catalog `status` values to install | | `catalog.owner_id` | `null` | restrict to one publisher (`/api/software?owner_id=`) | | `catalog.only` / `catalog.exclude` | `[]` | software-name allow / deny lists | @@ -214,16 +314,24 @@ counts, favourites and scraped media, and two stores can share one gamelist. | `behavior.timeout` | `30` | HTTP timeout, seconds | | `behavior.insecure` | `false` | skip TLS verification (self-hosted test instances) | +`BATOCERA_ES_CONFIG_DIR` and `BATOCERA_ES_SYSTEMS_CFG` override the two +EmulationStation paths from the environment, which is what makes a staging root +testable off a real box. + ### Platform mapping -A catalog platform is installable when its release asset is a file the Batocera -system can boot directly: +A catalog platform is installable when its release asset is a file a system this +box has can boot directly: | Catalog platform | Asset kind | Batocera system | Extension | | --- | --- | --- | --- | | `c64` | `cartridge` | `c64` (VICE) | `.prg` | | `tic80` | `cartridge` | `tic80` | `.tic` | +"Has" means declared in the box's own `es_systems.cfg` — an emulator that got +built into this image. (A ROM folder is a poorer test: it can be there with +nothing behind it. The folders are the fallback when the file cannot be read.) + The other WarpEngine platforms (`ebitengine`, `love`, `godot`, `bevy`, `phaser`) ship `html` and per-OS zip archives, not a ROM a Batocera system launches, so they are not mapped by default. Adding one is a config edit: @@ -234,9 +342,10 @@ launches, so they are not mapped by default. Adding one is a config edit: ### Native games as Ports -A cartridge is booted by an emulator, so it goes into that system's ROM folder. -A native build is a program: Batocera runs Linux, and its **Ports** system takes -`.sh` launchers. Set `install` to `port` and point `system` at `ports`: +A cartridge is booted by an emulator, so it goes to the folder of the system that +boots it. A native build is a program: Batocera runs Linux, and its **Ports** +system takes `.sh` launchers. Set `install` to `port` and point `system` at +`ports`: ```json "platforms": { @@ -253,15 +362,15 @@ A native build is a program: Batocera runs Linux, and its **Ports** system takes The zip is unpacked and a launcher written: ``` -/userdata/roms/ports//.sh the launcher ES lists -/userdata/roms/ports//images/.png box art -/userdata/roms/ports/.data/// the unpacked program +/userdata/roms//ports/.sh the launcher ES lists +/userdata/roms//ports/images/.png box art +/userdata/roms//ports/.data// the unpacked program ``` The payload sits under `.data`, whose leading dot is what hides it from -EmulationStation — the launchers themselves are in a normal subfolder, scanned -like any other system's. Both paths carry the store's subfolder, so an uninstall -or a prune can only ever reach this store's own files. +EmulationStation — only the `.sh` next to it is meant to be listed. `system: +"ports"` still names where the launcher *definition* comes from: our own ports +system copies the box's, which is `sh`, so the launcher is simply run. The launcher `cd`s into the directory holding the executable before running it: a game loads its `assets/` relative to the working directory. Python's zip @@ -297,26 +406,31 @@ The engine reports the architecture it detected — `x86_64`, `aarch64`, `armhf` entry for the running machine and no `*`, the title is skipped and says so: `skipped bevydemo: bevy has no asset kind for aarch64`. -A platform is also skipped when the box has no ROM folder for its system — -`list` reports that as `skipped : no '' ROM folder`. +A platform is also skipped when the box does not have its system — +`list` reports that as `skipped : this box has no '' system`. ## How a sync stays safe next to EmulationStation EmulationStation keeps gamelists in memory and writes them back when it exits, -so a merge done while it runs can be clobbered. Launched from the Ports menu +so a merge done while it runs can be clobbered. Launched from the menu (`sync-from-es`), the engine therefore downloads first, then hands the gamelist merge to a detached `apply-gamelists --wait-pid ` child that waits for the old ES process to die before writing. From SSH, with no ES running, the merge happens inline. -Re-running a sync is idempotent: files already present are left alone, and a -release bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new -one. `GET /api/download` is used rather than `/file/`, so downloads count +The `es_systems_.cfg` needs none of that: ES reads it and never writes it, +so it is written straight away — it just does not take effect until ES restarts. + +Re-running a sync is idempotent: files already present are left alone, a release +bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new one, and a +gamelist or `es_systems` file whose content has not changed is not rewritten at +all. `GET /api/download` is used rather than `/file/`, so downloads count towards the catalog's statistics. ## State `state.json` records what this store installed, keyed `:` so the -same software name on two systems stays two entries. A `version: 1` state file -written by the pre-split `ttg-store` is migrated to that key format on first -run. +same software name on two systems stays two entries. `layout` says which +directory layout those records belong to, which is how the upgrade to the store's +own folder knows it has to run once. A `version: 1` state file written by the +pre-split `ttg-store` is migrated to the current key format on first run. diff --git a/config.example.json b/config.example.json index a96da69..536450e 100644 --- a/config.example.json +++ b/config.example.json @@ -13,8 +13,17 @@ "subfolder": "example" }, "emulationstation": { + "menu": { + "mode": "system", + "name": null, + "theme": "ports", + "updater": true, + "labels": {} + }, + "ports_entry": null, "folder_name": "Example Store", - "restart": true + "restart": true, + "config_dir": null }, "catalog": { "statuses": [ diff --git a/install.sh b/install.sh index 797bb28..04567f7 100755 --- a/install.sh +++ b/install.sh @@ -45,17 +45,26 @@ trap 'rm -f "$TMP_CFG"' EXIT say "reading store config from $STORE_CONFIG" fetch "$STORE_CONFIG" "$TMP_CFG" +# The store id names its home; the mode decides whether a Ports entry is wanted +# at all, since a store with its own ES menu entry has an updater inside it. STORE_META="$(python3 -c ' import json, sys -cfg = json.load(open(sys.argv[1])).get("store") or {} -sid = cfg.get("id") or "" +cfg = json.load(open(sys.argv[1])) +store = cfg.get("store") or {} +sid = store.get("id") or "" if not sid or "/" in sid: sys.exit("store.id is required and must not contain a slash") -print(sid, cfg.get("name") or sid) +es = cfg.get("emulationstation") or {} +mode = ((es.get("menu") or {}).get("mode") or "system").lower() +port = es.get("ports_entry") +port = (mode == "merge") if port is None else bool(port) +print(sid, "1" if port else "0", store.get("name") or sid) ' "$TMP_CFG")" || die "cannot read $STORE_CONFIG" STORE_ID="${STORE_META%% *}" -STORE_NAME="${STORE_META#* }" +STORE_REST="${STORE_META#* }" +WANT_PORTS_ENTRY="${STORE_REST%% *}" +STORE_NAME="${STORE_REST#* }" STORE_HOME="$STORE_ROOT/$STORE_ID" PORT_NAME="${BATOCERA_PORT_NAME:-$STORE_NAME}" @@ -99,8 +108,11 @@ EOF chmod 0755 "$LAUNCHER" # The Ports entry: EmulationStation gives launched scripts no console, so send -# everything to a log file the user can read over SSH. -cat > "$PORTS_DIR/$PORT_NAME.sh" < "$PORTS_DIR/$PORT_NAME.sh" <>"$STORE_HOME/store.log" 2>&1 @@ -108,17 +120,24 @@ echo "=== \$(date -Iseconds) sync started ===" "$LAUNCHER" sync-from-es echo "=== \$(date -Iseconds) sync finished (exit \$?) ===" EOF -chmod 0755 "$PORTS_DIR/$PORT_NAME.sh" -say "ports entry: $PORTS_DIR/$PORT_NAME.sh" + chmod 0755 "$PORTS_DIR/$PORT_NAME.sh" + say "ports entry: $PORTS_DIR/$PORT_NAME.sh" +fi say "running the first sync" "$LAUNCHER" --verbose sync --no-restart +if [ "$WANT_PORTS_ENTRY" = "1" ]; then + MENU_LINE=" Menu: Ports -> \"$PORT_NAME\" (re-run any time to pull new releases)" +else + MENU_LINE=" Menu: \"$STORE_NAME\" -> Store -> \"Update $STORE_NAME\" (re-run any time to pull new releases)" +fi + cat < "$PORT_NAME" (re-run any time to pull new releases) +$MENU_LINE CLI: $LAUNCHER list|sync|remove Config: $STORE_HOME/config.json Log: $STORE_HOME/store.log diff --git a/store.py b/store.py index 6134683..344a16c 100755 --- a/store.py +++ b/store.py @@ -2,9 +2,19 @@ """store.py — the Batocera adapter of the WarpEngine store engine. Reads a WarpEngine catalog (`GET /api/software`), keeps the entries whose -platform maps to a system this Batocera box actually has, downloads the matching -release asset into that system's ROM folder and writes EmulationStation -metadata (title, description, author, box art) into its gamelist.xml. +platform maps to a system this Batocera box can run, downloads the matching +release asset and writes EmulationStation metadata (title, description, author, +box art) into a gamelist.xml. + +Where the games land is the store's choice, `emulationstation.menu.mode`: + + system the store gets its own EmulationStation menu entry. Its games live in + `///`, and an `es_systems_.cfg` + declares one ES system per platform, all grouped under the store's + name — so the box shows one entry holding a folder per platform. + merge the games go into the box's own systems, under `//`, + and are merged into the box's gamelists. What the engine did before + the menu entry existed. The engine knows nothing about any particular site: which host to talk to, what the store is called and where its games land all come from `config.json`. A @@ -40,7 +50,13 @@ except ImportError: "(https://git.teletypegames.org/engines/warpstore)") from warpstore import debug, die, log -VERSION = "3.1.0" +VERSION = "4.0.0" + +# EmulationStation's own files. The user config dir is where our es_systems +# overlay goes; the shipped es_systems.cfg is what we read a system's launch +# command out of, so that nothing about emulators is hardcoded here. +ES_CONFIG_DIR = "/userdata/system/configs/emulationstation" +ES_SYSTEMS_CFG = "/usr/share/emulationstation/es_systems.cfg" # Every store keeps its code, config, state and log in one directory, so a box # can carry several stores side by side without them treading on each other. @@ -65,16 +81,41 @@ DEFAULT_CONFIG = { }, "paths": { "roms_root": "/userdata/roms", - # Everything we install lives under this subfolder of the system's ROM - # dir, so a prune can never touch ROMs the user put there themselves — - # nor games installed by another store. + # Everything we install lives under this subfolder. In "system" mode it + # is the store's own ROM folder, `//`; in + # "merge" mode it is a subfolder of the box's own system folders, + # `//`. Either way nothing outside it is + # ever touched, so a prune cannot reach the user's own ROMs — nor + # another store's games. "subfolder": "warp", }, "emulationstation": { - # Display name of our subfolder in ES; null hides the folder node. + "menu": { + # "system": the store gets its own ES menu entry (see the module + # docstring). "merge": install into the box's own systems. + "mode": "system", + # Label of that menu entry; null means store.name. + "name": None, + # Theme folder the entry borrows its logo from. "ports" exists in + # every Batocera theme; a name no theme knows would show up blank. + "theme": "ports", + # An "Update " entry inside the menu, so the sync can be + # started from the same place the games are. + "updater": True, + # Per-system folder labels inside the entry; a system that is not + # listed keeps the fullname the box's own es_systems.cfg gives it. + "labels": {}, + }, + # A Ports entry that runs the sync. Redundant once the store has its own + # menu entry with an updater in it, so "system" mode defaults to false. + "ports_entry": None, + # Display name of our subfolder in the box's gamelists ("merge" mode + # only); null hides the folder node. "folder_name": "WarpEngine Store", # Restart EmulationStation after a sync that changed something. "restart": True, + # Where ES keeps its user config; null means the Batocera default. + "config_dir": None, }, "catalog": { # Catalog `status` values worth installing. "development" is left out. @@ -108,6 +149,181 @@ def roms_root(cfg): return cfg["paths"]["roms_root"] +def menu(cfg): + return cfg["emulationstation"]["menu"] + + +def es_config_dir(cfg): + return (os.environ.get("BATOCERA_ES_CONFIG_DIR") + or cfg["emulationstation"].get("config_dir") + or ES_CONFIG_DIR) + + +def es_systems_cfg(cfg): + return os.environ.get("BATOCERA_ES_SYSTEMS_CFG") or ES_SYSTEMS_CFG + + +def wants_ports_entry(cfg): + """A Ports entry for the sync — off by default once the store has a menu.""" + want = cfg["emulationstation"].get("ports_entry") + if want is None: + return isinstance(layout(cfg), MergeLayout) + return bool(want) + + +# -------------------------------------------------------------------------- +# where the files go +# -------------------------------------------------------------------------- + + +class Layout: + """Where this store's files live, and what it is allowed to delete. + + Two answers, one per `emulationstation.menu.mode`, and every path in the + engine goes through one of them: the install, the removal, the gamelist and + the empty-directory sweep all ask the layout rather than build paths of + their own. + """ + + def __init__(self, cfg): + self.cfg = cfg + + def base(self, system): + """The directory this system's content lives in.""" + raise NotImplementedError + + def rel_rom(self, game): + return game["asset"] + + def rel_image(self, game, ext): + return f"images/{game['name']}{ext}" + + def rel_launcher(self, game): + return f"{game['name']}.sh" + + def rel_data(self, game): + # The leading dot is what hides the unpacked program from ES; only the + # `.sh` launcher next to it is meant to be listed as a game. + return f".data/{game['name']}" + + def gamelist(self, system): + return os.path.join(self.base(system), "gamelist.xml") + + def owns(self, rel): + """May we delete this path (relative to `base`)? The prune's safety net. + + Everything under the store's own folder is ours; what is refused is a + path that would climb out of it, however it is spelled. + """ + if os.path.isabs(rel): + return False + normalized = os.path.normpath(rel) + return normalized != ".." and not normalized.startswith(".." + os.sep) + + def claims(self, rel, wanted, base): + """Is this existing gamelist node ours to rewrite? (see merge_gamelist)""" + raise NotImplementedError + + def purge_dirs(self, system): + """Directories to try removing once the games are gone, deepest first.""" + return [] + + # Only the box's own gamelists are worth backing up, and only ours is worth + # deleting when it ends up empty. + backup_gamelists = True + folder_node = False + + +class MergeLayout(Layout): + """`//` — inside the box's own systems. + + Shared ground: the ROMs sit next to the user's, the gamelist is the box's, + and another store may keep its own games in the same folders. Hence the + subfolder prefix on everything and the `owns` check before every delete. + """ + + backup_gamelists = True + folder_node = True + + def base(self, system): + return os.path.join(roms_root(self.cfg), system) + + def rel_rom(self, game): + return f"{subfolder(self.cfg)}/{game['asset']}" + + def rel_image(self, game, ext): + return f"{subfolder(self.cfg)}/images/{game['name']}{ext}" + + def rel_launcher(self, game): + return f"{subfolder(self.cfg)}/{game['name']}.sh" + + def rel_data(self, game): + return f".data/{subfolder(self.cfg)}/{game['name']}" + + def owns(self, rel): + prefix = subfolder(self.cfg) + "/" + return rel.startswith(prefix) or rel.startswith(f".data/{prefix}") + + def claims(self, rel, wanted, base): + return rel.startswith(subfolder(self.cfg) + "/") or rel == subfolder(self.cfg) + + def purge_dirs(self, system): + base = self.base(system) + return [os.path.join(base, subfolder(self.cfg), "images"), + os.path.join(base, subfolder(self.cfg)), + # `.data` itself stays: it is shared ground, another store may + # still be keeping a payload there. + os.path.join(base, ".data", subfolder(self.cfg))] + + +class SystemLayout(Layout): + """`//` — the store's own ROM folder. + + Every file under it is ours, including the gamelists, so there is nothing to + back up and nothing to merge around. What keeps the user's own files safe + here is that we only ever delete what `state.json` says we installed. + """ + + backup_gamelists = False + folder_node = False + + def store_root(self): + return os.path.join(roms_root(self.cfg), subfolder(self.cfg)) + + def base(self, system): + return os.path.join(self.store_root(), system) + + def claims(self, rel, wanted, base): + if not rel or rel in wanted: + return True + # A leftover of ours points at a file that is not there any more; a ROM + # the user added themselves does exist, and keeps its entry. + return not os.path.exists(os.path.join(base, rel)) + + def purge_dirs(self, system): + base = self.base(system) + return [os.path.join(base, "images"), os.path.join(base, ".data"), base] + + +def layout_name(lay): + return "merge" if isinstance(lay, MergeLayout) else "system" + + +def layout_named(cfg, name): + return MergeLayout(cfg) if name == "merge" else SystemLayout(cfg) + + +_LAYOUT = None + + +def layout(cfg): + global _LAYOUT + if _LAYOUT is None or _LAYOUT.cfg is not cfg: + mode = (menu(cfg).get("mode") or "system").lower() + _LAYOUT = MergeLayout(cfg) if mode == "merge" else SystemLayout(cfg) + return _LAYOUT + + # -------------------------------------------------------------------------- # catalog -> Batocera # -------------------------------------------------------------------------- @@ -124,7 +340,7 @@ def accept_for_batocera(available_systems): def accept(spec, sw): system = spec["system"] if system not in available_systems: - raise ws.Skip(f"no '{system}' ROM folder on this box") + raise ws.Skip(f"this box has no '{system}' system") return system, {"system": system, "install": spec.get("install", "rom")} return accept @@ -147,35 +363,45 @@ def es_date(iso): def system_dir(cfg, system): - return os.path.join(roms_root(cfg), system) + return layout(cfg).base(system) + + +def box_systems(cfg): + """The box's own system definitions, by name — or None if unreadable. + + This one file answers both questions the engine has about the machine: which + systems it can run, and how it launches them. A system is in there because + an emulator for it was built into this particular Batocera image, which is a + truer compatibility test than the presence of a ROM folder. + """ + path = es_systems_cfg(cfg) + if not os.path.isfile(path): + debug(f"no {path} — falling back to the ROM folders") + return None + try: + root = ET.parse(path).getroot() + except ET.ParseError as exc: + log(f"warning: {path} is not valid XML ({exc}) — falling back to the ROM folders") + return None + nodes = {} + for node in root.findall("system"): + name = (node.findtext("name") or "").strip() + if name: + nodes[name] = node + return nodes or None def detect_systems(cfg): - """Systems this box has a ROM folder for — our compatibility check.""" + """Systems this box can run.""" + nodes = box_systems(cfg) + if nodes: + return set(nodes) root = roms_root(cfg) if not os.path.isdir(root): return set() return {d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d))} -def rel_rom_path(cfg, game): - return f"{subfolder(cfg)}/{game['asset']}" - - -def rel_image_path(cfg, game, ext): - return f"{subfolder(cfg)}/images/{game['name']}{ext}" - - -def rel_port_launcher(cfg, game): - return f"{subfolder(cfg)}/{game['name']}.sh" - - -def rel_port_data(cfg, game): - # A `.data` pont-prefixe rejti el az ES elol; a store almappaja alatta - # tartja kulon az egyes store-ok payloadjat. - return f".data/{subfolder(cfg)}/{game['name']}" - - def find_executable(root, name): """A kicsomagolt fa fo binarisa. @@ -202,9 +428,10 @@ def find_executable(root, name): def install_port(cfg, game, dry_run=False): """A zip asset kicsomagolasa + Ports indito. Visszaad (launcher, data) relativ utakat.""" - base = system_dir(cfg, game["system"]) - launcher_rel = rel_port_launcher(cfg, game) - data_rel = rel_port_data(cfg, game) + lay = layout(cfg) + base = lay.base(game["system"]) + launcher_rel = lay.rel_launcher(game) + data_rel = lay.rel_data(game) launcher_abs = os.path.join(base, launcher_rel) data_abs = os.path.join(base, data_rel) @@ -254,14 +481,15 @@ def install_port(cfg, game, dry_run=False): def install_game(cfg, game, state_entry, dry_run=False): """Download the asset + box art if missing. Returns (record, changed).""" - base = system_dir(cfg, game["system"]) + lay = layout(cfg) + base = lay.base(game["system"]) changed = False data_rel = None if game.get("install") == "port": # A port ket dologbol all: egy indito .sh es a kicsomagolt payload. - rom_rel = rel_port_launcher(cfg, game) - data_rel = rel_port_data(cfg, game) + rom_rel = lay.rel_launcher(game) + data_rel = lay.rel_data(game) installed = (os.path.isfile(os.path.join(base, rom_rel)) and os.path.isdir(os.path.join(base, data_rel)) and state_entry and state_entry.get("asset") == game["asset"]) @@ -271,7 +499,7 @@ def install_game(cfg, game, state_entry, dry_run=False): rom_rel, data_rel = install_port(cfg, game, dry_run=dry_run) changed = True else: - rom_rel = rel_rom_path(cfg, game) + rom_rel = lay.rel_rom(game) rom_abs = os.path.join(base, rom_rel) if os.path.isfile(rom_abs) and os.path.getsize(rom_abs) > 0: debug(f"{game['name']}: {game['asset']} already present") @@ -290,7 +518,7 @@ def install_game(cfg, game, state_entry, dry_run=False): if not wanted: if dry_run: log(f"would fetch box art for {game['name']}") - image_rel = rel_image_path(cfg, game, ".png") + image_rel = lay.rel_image(game, ".png") else: image_rel = fetch_image(cfg, game, base) changed = changed or bool(image_rel) @@ -298,7 +526,7 @@ def install_game(cfg, game, state_entry, dry_run=False): # Box art disappeared from the catalog: drop the file we cached. if image_rel and not dry_run: stale = os.path.join(base, image_rel) - if image_rel.startswith(subfolder(cfg) + "/") and os.path.isfile(stale): + if lay.owns(image_rel) and os.path.isfile(stale): os.unlink(stale) changed = True image_rel = None @@ -316,21 +544,25 @@ def fetch_image(cfg, game, base): body, ext = ws.download_image(cfg, game) if body is None: return None - rel = rel_image_path(cfg, game, ext) + rel = layout(cfg).rel_image(game, ext) ws.write_atomic(os.path.join(base, rel), body) debug(f"{game['name']}: box art -> {rel}") return rel -def remove_game(cfg, record, dry_run=False): - base = system_dir(cfg, record["system"]) +def remove_game(cfg, record, dry_run=False, lay=None): + """Delete one installed game's files. `lay` is for the migration, which has + to remove what a *previous* layout installed.""" + lay = lay or layout(cfg) + base = lay.base(record["system"]) for rel in (record.get("rom"), record.get("image")): if not rel: continue path = os.path.join(base, rel) - # Never step outside our own subfolder — another store may own it. - if not rel.startswith(subfolder(cfg) + "/"): - log(f"warning: refusing to delete {path} (outside {subfolder(cfg)}/)") + # Never step outside what this layout says is ours — in merge mode the + # folder is shared with the user and possibly with another store. + if not lay.owns(rel): + log(f"warning: refusing to delete {path} (not ours)") continue if os.path.isfile(path): if dry_run: @@ -343,8 +575,8 @@ def remove_game(cfg, record, dry_run=False): data_rel = record.get("data") if data_rel: data_abs = os.path.join(base, data_rel) - if not data_rel.startswith(f".data/{subfolder(cfg)}/"): - log(f"warning: refusing to delete {data_abs} (outside .data/{subfolder(cfg)}/)") + if not lay.owns(data_rel): + log(f"warning: refusing to delete {data_abs} (not ours)") elif os.path.isdir(data_abs): if dry_run: log(f"would remove {data_abs}") @@ -359,7 +591,7 @@ def remove_game(cfg, record, dry_run=False): def gamelist_path(cfg, system): - return os.path.join(system_dir(cfg, system), "gamelist.xml") + return layout(cfg).gamelist(system) def normalize_path(text): @@ -367,18 +599,37 @@ def normalize_path(text): return text[2:] if text.startswith("./") else text -def merge_gamelist(cfg, system, records, dry_run=False, backup=True): - """Rewrite only the / nodes under our subfolder. +# The tags we write ourselves, from the catalog. Everything else in a +# node was put there by EmulationStation — play counts, favourites, the last +# time it was launched — and has to survive a rewrite. +OUR_TAGS = ("path", "name", "desc", "image", "thumbnail", "developer", "publisher", + "releasedate") - Everything else in the file — the user's own scraped ROMs, their play - counts, favourites, and whatever another store installed under its own - subfolder — is parsed and written back untouched. + +def merge_gamelist(cfg, system, records, dry_run=False, backup=True, lay=None): + """Rewrite our / nodes, and only ours. + + In merge mode "ours" means everything under the store's subfolder: the rest + of the file is the box's — the user's own scraped ROMs, their play counts and + favourites, and whatever another store installed under its own subfolder. + + In system mode the file is in the store's own folder, so the rule is + different: a node is ours if we are writing that game now, or if the file it + points at is gone (a leftover of ours). A ROM the user dropped into the + folder themselves keeps its entry. + + Either way, the tags EmulationStation owns are carried over onto the new + node — a favourite stays a favourite across a sync. `backup=False` is for the uninstall: by then any copy worth having was made on the first sync, and backing up again would only preserve our own entries. """ - path = gamelist_path(cfg, system) - prefix = subfolder(cfg) + "/" + lay = lay or layout(cfg) + path = lay.gamelist(system) + base = lay.base(system) + wanted = {record["rom"]: record for record in records} + if not records and not os.path.isfile(path): + return if os.path.isfile(path): try: @@ -388,18 +639,23 @@ def merge_gamelist(cfg, system, records, dry_run=False, backup=True): root = ET.Element("gameList") else: backup_path = f"{path}.{cfg['store']['id']}-backup" - if backup and not os.path.exists(backup_path) and not dry_run: + if backup and lay.backup_gamelists and not os.path.exists(backup_path) and not dry_run: shutil.copy2(path, backup_path) else: root = ET.Element("gameList") + # What ES added to the nodes we are about to rewrite, kept aside by path. + carried = {} for node in list(root): target = normalize_path(node.findtext("path")) - if target.startswith(prefix) or target == subfolder(cfg): - root.remove(node) + if not lay.claims(target, wanted, base): + continue + if target in wanted: + carried[target] = [child for child in node if child.tag not in OUR_TAGS] + root.remove(node) folder_name = cfg["emulationstation"].get("folder_name") - if records and folder_name: + if records and lay.folder_node and folder_name: folder = ET.SubElement(root, "folder") ET.SubElement(folder, "path").text = "./" + subfolder(cfg) ET.SubElement(folder, "name").text = folder_name @@ -419,9 +675,16 @@ def merge_gamelist(cfg, system, records, dry_run=False, backup=True): released = record.get("releasedate") or es_date(record.get("created_at")) if released: ET.SubElement(game, "releasedate").text = released + for child in carried.get(record["rom"], []): + game.append(child) indent(root) blob = b'\n' + ET.tostring(root, encoding="utf-8") + if os.path.isfile(path): + with open(path, "rb") as fh: + if fh.read() == blob: + debug(f"{path} is up to date") + return if dry_run: log(f"would write {path} ({len(records)} entries)") return @@ -444,6 +707,358 @@ def indent(elem, level=0): elem.tail = pad +# -------------------------------------------------------------------------- +# the store's own menu entry +# -------------------------------------------------------------------------- + +# The folder that holds the updater. Not a catalog platform, so it can never +# collide with one. +STORE_SYSTEM = "store" + + +def es_parent_name(cfg, box=None): + """The ES system name of the menu entry itself. + + The store id — unless the box already has a system by that name. An overlay + whose `` matches an existing system *modifies that system* instead of + adding one, and a store that called itself `nes` must not rewrite the box's + NES. + """ + name = cfg["store"]["id"] + if box and name in box: + name = f"{name}-store" + log(f"'{cfg['store']['id']}' is already a system on this box — " + f"the menu entry is called '{name}'") + return name + + +def source_system(system): + """The box system one of our systems borrows its launcher from.""" + return "ports" if system == STORE_SYSTEM else system + + +def system_label(cfg, system, src): + labels = menu(cfg).get("labels") or {} + if system in labels: + return labels[system] + if system == STORE_SYSTEM: + return "Store" + fullname = (src.findtext("fullname") or "").strip() if src is not None else "" + return fullname or system + + +def configured_exts(cfg, system): + """The extensions our own config expects for a system — the last resort when + the box's system definition has none to copy.""" + exts = set() + for spec in (cfg.get("platforms") or {}).values(): + if spec.get("system") != system: + continue + ext = spec.get("ext") + for value in (ext.values() if isinstance(ext, dict) else [ext]): + if value: + exts.add(value) + return " ".join(sorted(exts)) + + +def text_node(parent, tag, value): + node = ET.SubElement(parent, tag) + node.text = value + return node + + +def derive_child(cfg, parent, system, src): + """One `` of ours, borrowing everything about emulators from the box. + + `%SYSTEM%` is resolved here rather than left to EmulationStation: ES replaces + it with the name of the system being launched (`FileData.cpp`), which for us + would be `ttg-c64` — a name configgen has never heard of. The box's own + system name goes in instead, so a game of ours is launched exactly as the + box would launch it, with the emulator the user configured for that system. + """ + src_name = source_system(system) + node = ET.Element("system") + text_node(node, "name", f"{parent}-{system}") + text_node(node, "fullname", system_label(cfg, system, src)) + text_node(node, "path", layout(cfg).base(system)) + text_node(node, "extension", + (src.findtext("extension") or "").strip() or configured_exts(cfg, system)) + text_node(node, "command", (src.findtext("command") or "").replace("%SYSTEM%", src_name)) + for tag in ("platform", "theme", "manufacturer", "release", "hardware"): + value = (src.findtext(tag) or "").strip() + if value: + text_node(node, tag, value) + # This is what puts the system inside the store's menu entry instead of + # giving it one of its own. + text_node(node, "group", parent) + emulators = src.find("emulators") + if emulators is not None: + node.append(emulators) + return node + + +def build_es_systems(cfg, systems, box): + """The `` for the store: the menu entry, and a system under it + for every folder that has something in it.""" + lay = layout(cfg) + parent = es_parent_name(cfg, box) + root = ET.Element("systemList") + + # The menu entry is declared rather than left to ES to invent, for two + # reasons: it is the only way to give it a `` (a group ES creates + # itself looks for a theme folder named after the group, which no theme + # has), and with `HideUniqueGroups` on — the default — a group with a single + # system in it is dissolved unless a system by that name exists. + entry = ET.SubElement(root, "system") + text_node(entry, "name", parent) + text_node(entry, "fullname", menu(cfg).get("name") or cfg["store"]["name"]) + text_node(entry, "path", lay.store_root()) + # ES drops a system that has no extension or no command, so both are here — + # an extension nothing on earth has, and a command that does nothing. This + # entry is a shelf for the systems below, it never holds a game itself. + text_node(entry, "extension", f".{parent}-none") + text_node(entry, "command", "true") + text_node(entry, "platform", "pc") + text_node(entry, "theme", menu(cfg).get("theme") or "ports") + + for system in sorted(systems): + src = box.get(source_system(system)) + if src is None: + log(f"warning: no '{source_system(system)}' system on this box — " + f"{system} is not added to the menu") + continue + root.append(derive_child(cfg, parent, system, src)) + return root + + +def es_systems_path(cfg): + return os.path.join(es_config_dir(cfg), f"es_systems_{cfg['store']['id']}.cfg") + + +def write_es_systems(cfg, systems, dry_run=False): + """Declare our systems to EmulationStation. True if the file changed. + + ES writes this file never, so unlike a gamelist it can be written while ES + is running. What it cannot do is take effect without a restart. + """ + box = box_systems(cfg) + if not box: + log(f"warning: cannot read {es_systems_cfg(cfg)} — the store's menu entry was " + "not written. The games are installed; ES will list them once it can.") + return False + + root = build_es_systems(cfg, systems, box) + indent(root) + blob = b'\n' + ET.tostring(root, encoding="utf-8") + path = es_systems_path(cfg) + if os.path.isfile(path): + with open(path, "rb") as fh: + if fh.read() == blob: + debug(f"{path} is up to date") + return False + if dry_run: + log(f"would write {path} ({len(root) - 1} systems)") + return False + os.makedirs(os.path.dirname(path), exist_ok=True) + ws.write_atomic(path, blob) + log(f"EmulationStation systems: {path} ({len(root) - 1} systems)") + return True + + +def remove_es_systems(cfg, dry_run=False): + path = es_systems_path(cfg) + if not os.path.isfile(path): + return + if dry_run: + log(f"would remove {path}") + return + os.unlink(path) + log(f"removed {path}") + + +def updater_script(cfg): + """The updater, as the ES menu launches it. + + A `.sh` in a folder of ours, so it is a game as far as ES is concerned. It + calls this same engine — not the installer's launcher — so it keeps working + whatever the store home is called. + """ + return ( + "#!/bin/bash\n" + f"# {cfg['store']['name']} — pulls new releases from the catalog.\n" + f"exec >>{shlex.quote(os.path.join(ws.HOME, 'store.log'))} 2>&1\n" + 'echo "=== $(date -Iseconds) sync started ==="\n' + f"BATOCERA_STORE_HOME={shlex.quote(ws.HOME)} " + f"{shlex.quote(sys.executable)} {shlex.quote(os.path.abspath(__file__))} " + f"--config {shlex.quote(ws.CONFIG_PATH)} sync-from-es\n" + 'echo "=== $(date -Iseconds) sync finished (exit $?) ==="\n' + ).encode("utf-8") + + +def updater_records(cfg): + label = menu(cfg).get("name") or cfg["store"]["name"] + return [{ + "system": STORE_SYSTEM, + "rom": "update.sh", + "title": f"Update {label}", + "desc": "Downloads everything new from the catalog and refreshes this menu.", + }] + + +def write_updater(cfg, dry_run=False): + path = os.path.join(layout(cfg).base(STORE_SYSTEM), "update.sh") + blob = updater_script(cfg) + if os.path.isfile(path): + with open(path, "rb") as fh: + if fh.read() == blob: + return + if dry_run: + log(f"would write {path}") + return + ws.write_atomic(path, blob) + os.chmod(path, 0o755) + log(f"updater: {path}") + + +def apply_menu(cfg, systems, dry_run=False, gamelists=True): + """Write everything the store's menu entry is made of. + + Returns True if EmulationStation has to restart to notice. `gamelists=False` + is for a sync launched from inside ES, which defers every gamelist write to + the child that waits for ES to exit. + """ + lay = layout(cfg) + if not isinstance(lay, SystemLayout): + return False + + wanted = set(systems) + updater = bool(menu(cfg).get("updater")) + if updater: + wanted.add(STORE_SYSTEM) + # ES refuses a system whose path does not exist, so the folders go first. + for system in sorted(wanted): + if not dry_run: + os.makedirs(lay.base(system), exist_ok=True) + + changed = write_es_systems(cfg, wanted, dry_run=dry_run) + if updater: + write_updater(cfg, dry_run=dry_run) + if gamelists: + merge_gamelist(cfg, STORE_SYSTEM, updater_records(cfg), dry_run=dry_run) + return changed + + +# -------------------------------------------------------------------------- +# migration from the merge layout +# -------------------------------------------------------------------------- + + +def ports_entries(cfg): + """The Ports scripts that run this store, found by looking inside them. + + The name cannot be derived from the config — a store repository may set + `BATOCERA_PORT_NAME` to anything — but whatever it is called, the script + names this store's home. Same trick as `uninstall.sh`. + """ + ports_dir = os.path.join(roms_root(cfg), "ports") + if not os.path.isdir(ports_dir): + return [] + found = [] + for name in sorted(os.listdir(ports_dir)): + if not name.endswith(".sh"): + continue + path = os.path.join(ports_dir, name) + try: + with open(path, "r", errors="replace") as fh: + if ws.HOME and ws.HOME in fh.read(): + found.append(path) + except OSError: + continue + return found + + +def remove_ports_entry(cfg, dry_run=False): + for path in ports_entries(cfg): + if dry_run: + log(f"would remove the Ports entry {path}") + else: + os.unlink(path) + log(f"removed the Ports entry {path}") + + +def migrate_layout(cfg, dry_run=False): + """Take the store off the box when `menu.mode` changed under it. + + The games move by being removed and downloaded again, not by being carried + across: the catalog is small, and a half-moved install would be worse than a + slightly longer sync. What this does is exactly what an uninstall does — + ROMs, box art, port payloads, gamelist nodes, the empty folders — so the old + layout is gone completely, and then the sync fills the new one. + + A state file with no `layout` in it was written by an engine that only had + the merge layout, so that is what it is assumed to be. + """ + want = layout_name(layout(cfg)) + state = ws.load_state() + have = state.get("layout") or "merge" + if have == want: + return False + + installed = state["installed"] + if not installed: + if not dry_run: + state["layout"] = want + ws.save_state(state) + return False + + log(f"the store's layout changed ({have} -> {want}) — removing the " + f"{len(installed)} installed games from the old one, they will be " + "downloaded again") + old = layout_named(cfg, have) + systems = set() + for key, record in list(installed.items()): + remove_game(cfg, record, dry_run=dry_run, lay=old) + systems.add(ws.record_scope(record)) + if not dry_run: + del installed[key] + + if have == "system": + # The updater and the menu entry belong to the layout we are leaving. + systems.add(STORE_SYSTEM) + updater = os.path.join(old.base(STORE_SYSTEM), "update.sh") + if os.path.isfile(updater): + if dry_run: + log(f"would remove {updater}") + else: + os.unlink(updater) + remove_es_systems(cfg, dry_run=dry_run) + + for system in sorted(systems): + merge_gamelist(cfg, system, [], dry_run=dry_run, backup=False, lay=old) + drop_empty_gamelist(cfg, system, dry_run=dry_run, lay=old) + + dirs = [] + for system in sorted(systems): + dirs += old.purge_dirs(system) + if have == "system": + dirs.append(old.store_root()) + ws.prune_empty_dirs(dirs, root=roms_root(cfg), dry_run=dry_run) + + if not wants_ports_entry(cfg): + remove_ports_entry(cfg, dry_run=dry_run) + + leftovers = gamelist_backups(cfg, systems, lay=old) + if leftovers: + log("left in place — copies of your gamelists as we first found them:") + for path in leftovers: + log(f" {path}") + + if not dry_run: + state["layout"] = want + ws.save_state(state) + return True + + # -------------------------------------------------------------------------- # EmulationStation # -------------------------------------------------------------------------- @@ -500,11 +1115,15 @@ def perform_sync(cfg, names=None, dry_run=False): Returns (state, changed, touched_systems). Gamelists are left alone — the caller decides when it is safe to write them. """ + migrate_layout(cfg, dry_run=dry_run) + systems = detect_systems(cfg) if not systems: - # An unmounted or wrong ROMs root would otherwise look like "no system - # is compatible any more" and prune everything we ever installed. - die(f"no system folders under {roms_root(cfg)} — is it mounted?") + # An unreadable es_systems.cfg with an unmounted or wrong ROMs root + # behind it would otherwise look like "no system is compatible any more" + # and prune everything we ever installed. + die(f"cannot tell which systems this box has: no {es_systems_cfg(cfg)}, " + f"and no system folders under {roms_root(cfg)} — is it mounted?") catalog = ws.fetch_catalog(cfg, use_cache=True) games, skipped = select_games(cfg, catalog, systems) @@ -555,18 +1174,23 @@ def perform_sync(cfg, names=None, dry_run=False): def cmd_sync(cfg, args): state, changed, touched = perform_sync(cfg, args.name, dry_run=args.dry_run) + grouped = ws.records_by_scope(state["installed"]) + # The menu entry is written even when nothing was downloaded: a config that + # renamed it, or a first sync that found nothing to install, still has to + # leave the box with a way in. + menu_changed = apply_menu(cfg, grouped, dry_run=args.dry_run) + if args.dry_run: log("dry run — nothing written") return 0 if not changed and not args.force: log("already up to date") - return 0 + else: + for system in sorted(touched | set(grouped)): + merge_gamelist(cfg, system, grouped.get(system, [])) - grouped = ws.records_by_scope(state["installed"]) - for system in sorted(touched | set(grouped)): - merge_gamelist(cfg, system, grouped.get(system, [])) - - if cfg["emulationstation"].get("restart") and not args.no_restart and es_pid(): + if (cfg["emulationstation"].get("restart") and not args.no_restart + and (changed or menu_changed) and es_pid()): restart_es() return 0 @@ -579,15 +1203,19 @@ def cmd_sync_from_es(cfg, args): child that waits for the old ES process to die first. """ state, changed, _ = perform_sync(cfg) + grouped = ws.records_by_scope(state["installed"]) pid = es_pid() - + # The es_systems overlay and the updater script are ours alone — ES never + # writes them back — so they can be written now. Only the gamelists wait. + menu_changed = apply_menu(cfg, grouped, gamelists=not pid) if not changed: log("already up to date") - if not pid or not changed: - # Nothing racing us (or nothing new) — write the gamelists right here. - if changed: - for system, records in ws.records_by_scope(state["installed"]).items(): - merge_gamelist(cfg, system, records) + + if not pid: + # Nothing racing us — write the gamelists right here. A file whose + # content has not changed is not rewritten, so this is cheap. + for system, records in grouped.items(): + merge_gamelist(cfg, system, records) return 0 subprocess.Popen( @@ -595,7 +1223,8 @@ def cmd_sync_from_es(cfg, args): "--config", ws.CONFIG_PATH, "apply-gamelists", "--wait-pid", str(pid)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True, ) - if cfg["emulationstation"].get("restart") and not args.no_restart: + if (cfg["emulationstation"].get("restart") and not args.no_restart + and (changed or menu_changed)): restart_es() return 0 @@ -608,9 +1237,11 @@ def cmd_apply_gamelists(cfg, args): grouped = ws.records_by_scope(state["installed"]) if not grouped: log("nothing installed") - return 0 for system, records in grouped.items(): merge_gamelist(cfg, system, records, dry_run=args.dry_run) + # The updater's own gamelist waited for ES to exit along with the rest. + if isinstance(layout(cfg), SystemLayout) and menu(cfg).get("updater"): + merge_gamelist(cfg, STORE_SYSTEM, updater_records(cfg), dry_run=args.dry_run) return 0 @@ -619,8 +1250,12 @@ def cmd_list(cfg, args): catalog = ws.fetch_catalog(cfg, use_cache=True) host = ws.host() log(f"architecture: {host['arch']}") + state = ws.load_state() + if state["installed"] and (state.get("layout") or "merge") != layout_name(layout(cfg)): + log(f"the next sync will move these games to the '{layout_name(layout(cfg))}' " + f"layout — they are removed from the old one and downloaded again") games, skipped = select_games(cfg, catalog, systems, host) - installed = ws.load_state()["installed"] + installed = state["installed"] if not games: log("no compatible games in the catalog") @@ -663,15 +1298,20 @@ def cmd_remove(cfg, args): return 0 -def drop_empty_gamelist(cfg, system, dry_run=False): +def drop_empty_gamelist(cfg, system, dry_run=False, lay=None): """Delete a gamelist that is empty and that we created. - No `.-backup` next to it means we never found a gamelist there — it is - ours alone, and with our nodes gone there is nothing in it. Leaving the empty - shell behind would mean the uninstall did not quite put the box back. + In merge mode, no `.-backup` next to it means we never found a gamelist + there — it is ours alone, and with our nodes gone there is nothing in it. In + system mode the file is in our own folder, so an empty one is always ours. + Leaving the empty shell behind would mean the uninstall did not quite put the + box back. """ - path = gamelist_path(cfg, system) - if not os.path.isfile(path) or os.path.exists(f"{path}.{cfg['store']['id']}-backup"): + lay = lay or layout(cfg) + path = lay.gamelist(system) + if not os.path.isfile(path): + return + if lay.backup_gamelists and os.path.exists(f"{path}.{cfg['store']['id']}-backup"): return try: root = ET.parse(path).getroot() @@ -686,9 +1326,10 @@ def drop_empty_gamelist(cfg, system, dry_run=False): log(f"removed the now-empty {path}") -def gamelist_backups(cfg, systems): +def gamelist_backups(cfg, systems, lay=None): """The `gamelist.xml.-backup` copies we made, if any are still around.""" - paths = [f"{gamelist_path(cfg, system)}.{cfg['store']['id']}-backup" for system in systems] + lay = lay or layout(cfg) + paths = [f"{lay.gamelist(system)}.{cfg['store']['id']}-backup" for system in systems] return sorted(p for p in paths if os.path.isfile(p)) @@ -700,38 +1341,58 @@ def cmd_purge(cfg, args): what `uninstall.sh` runs before deleting the store itself: the shell script has no way of knowing which files were ours. - The gamelists are merged, not deleted: they belong to the box, and after our - nodes are gone the rest of the user's library is still in them. + In merge mode the gamelists are merged, not deleted: they belong to the box, + and after our nodes are gone the rest of the user's library is still in them. + In system mode they are ours, and go with the folders — along with the + es_systems overlay, which is what the store's menu entry is made of. """ state = ws.load_state() installed = state["installed"] if not installed: log("nothing is installed by this store") + # Which layout the files are actually in is what `state.json` says, not what + # the config asks for today: an upgrade whose first sync never ran still has + # its games in the old places, and an uninstall has to find them there. + recorded = state.get("layout") or ("merge" if installed else layout_name(layout(cfg))) + lay = layout_named(cfg, recorded) + systems = set() for key, record in list(installed.items()): - remove_game(cfg, record, dry_run=args.dry_run) + remove_game(cfg, record, dry_run=args.dry_run, lay=lay) systems.add(ws.record_scope(record)) if not args.dry_run: del installed[key] if not args.dry_run: ws.save_state(state) + # The menu entry and its updater are not catalog games, so no state record + # points at them. Both are removed whichever layout is recorded — a leftover + # es_systems file would leave the box with a menu entry and nothing in it. + remove_es_systems(cfg, dry_run=args.dry_run) + store_layout = SystemLayout(cfg) + updater = os.path.join(store_layout.base(STORE_SYSTEM), "update.sh") + if os.path.isfile(updater): + if args.dry_run: + log(f"would remove {updater}") + else: + os.unlink(updater) for system in sorted(systems): - merge_gamelist(cfg, system, [], dry_run=args.dry_run, backup=False) - drop_empty_gamelist(cfg, system, dry_run=args.dry_run) + merge_gamelist(cfg, system, [], dry_run=args.dry_run, backup=False, lay=lay) + drop_empty_gamelist(cfg, system, dry_run=args.dry_run, lay=lay) dirs = [] for system in sorted(systems): - base = system_dir(cfg, system) - dirs += [os.path.join(base, subfolder(cfg), "images"), - os.path.join(base, subfolder(cfg)), - # `.data` itself stays: it is shared ground, another store may - # still be keeping a payload there. - os.path.join(base, ".data", subfolder(cfg))] + dirs += lay.purge_dirs(system) + # The store's own folder is the system layout's, whatever the records say. + if os.path.isdir(store_layout.store_root()): + merge_gamelist(cfg, STORE_SYSTEM, [], dry_run=args.dry_run, backup=False, + lay=store_layout) + drop_empty_gamelist(cfg, STORE_SYSTEM, dry_run=args.dry_run, lay=store_layout) + dirs += store_layout.purge_dirs(STORE_SYSTEM) + [store_layout.store_root()] ws.prune_empty_dirs(dirs, root=roms_root(cfg), dry_run=args.dry_run) - leftovers = gamelist_backups(cfg, systems) + leftovers = gamelist_backups(cfg, systems, lay=lay) if leftovers: log("left in place — these are copies of your gamelists as we first found them:") for path in leftovers: @@ -753,8 +1414,17 @@ def cmd_config(cfg, args): log(f"wrote default config to {args.config}") return 0 # A felismert architektura nem a configbol jon, de az asset-valasztast - # eldonti, ezert itt is lathatonak kell lennie. - print(json.dumps({"detected_arch": ws.machine(), **cfg}, indent=2, ensure_ascii=False)) + # eldonti, ezert itt is lathatonak kell lennie. Ugyanez all a menupontra: + # a neve es a helye a configbol szarmazik, de nem all benne. + lay = layout(cfg) + extra = {"detected_arch": ws.machine(), "install_dir": lay.base("")} + if isinstance(lay, SystemLayout): + extra["es_menu"] = { + "entry": es_parent_name(cfg, box_systems(cfg)), + "file": es_systems_path(cfg), + "root": lay.store_root(), + } + print(json.dumps({**extra, **cfg}, indent=2, ensure_ascii=False)) return 0 @@ -818,6 +1488,9 @@ def main(argv=None): overrides={"store.base_url": args.base_url, "paths.roms_root": args.roms_root}, required=("paths.subfolder",), ) + mode = (menu(cfg).get("mode") or "system").lower() + if mode not in ("system", "merge"): + die(f"config: emulationstation.menu.mode is '{mode}' — expected 'system' or 'merge'") os.makedirs(ws.HOME, exist_ok=True) return args.func(cfg, args) diff --git a/uninstall.sh b/uninstall.sh index 715cfb9..e220b50 100755 --- a/uninstall.sh +++ b/uninstall.sh @@ -2,9 +2,9 @@ # Remove a WarpEngine store from a Batocera box. # # The engine does the removing, not this script: it is the only thing that knows -# which ROMs, box art, Ports payloads and gamelist entries were the store's — so -# it runs `purge` first, and only then do the launcher, the Ports entry and the -# store home go. +# which ROMs, box art, Ports payloads, gamelist entries and ES systems were the +# store's — so it runs `purge` first, and only then do the launcher, the Ports +# entry and the store home go. # # ./uninstall.sh # the only store installed, or say which one # STORE_ID=example ./uninstall.sh @@ -127,5 +127,5 @@ if [ "$DRY_RUN" = "1" ]; then say "dry run — nothing was removed." else say "done. EmulationStation was restarted if it was running; if not, the games are" - say "already gone from the gamelists." + say "already gone from the gamelists, and so is the store's own menu entry." fi