general store infra
This commit is contained in:
@@ -1,98 +1,158 @@
|
|||||||
# ttg-store — Teletype Games catalog client for Batocera
|
# warp-engine-batocera-store — a Batocera store engine for WarpEngine sites
|
||||||
|
|
||||||
Pulls games from the WarpEngine catalog on <https://teletypegames.org> straight
|
Pulls games from a [WarpEngine](https://git.teletypegames.org/tools/warp_engine)
|
||||||
into a Batocera box's ROM folders, with EmulationStation metadata and box art.
|
catalog straight into a Batocera box's ROM folders, with EmulationStation
|
||||||
Triggered from the **Ports** menu on the device, or from the CLI over SSH.
|
metadata and box art. Triggered from the **Ports** menu on the device, or from
|
||||||
|
the CLI over SSH.
|
||||||
|
|
||||||
|
This repository is the engine only. It knows the WarpEngine API but nothing
|
||||||
|
about any particular site: the host, the store's name and where its games land
|
||||||
|
all come from a `config.json` that lives in a separate **store repository**.
|
||||||
|
|
||||||
```
|
```
|
||||||
Ports ▸ "Teletype Games Store"
|
warp-engine-batocera-store the engine — store.py, install.sh
|
||||||
|
▲
|
||||||
|
│ config.json
|
||||||
|
│
|
||||||
|
┌─────┴─────┬───────────────┐
|
||||||
|
│ │ │
|
||||||
|
ttg-… my-… other-… store repositories
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
teletypegames.org my.example other.example
|
||||||
|
```
|
||||||
|
|
||||||
|
The reference store is
|
||||||
|
[`ttg-batocera-store`](https://git.teletypegames.org/tools/ttg-batocera-store).
|
||||||
|
|
||||||
|
```
|
||||||
|
Ports ▸ "<store name>"
|
||||||
│
|
│
|
||||||
├─ GET /api/software the whole catalog
|
├─ GET /api/software the whole catalog
|
||||||
├─ keep platforms this box can run c64 → c64, tic80 → tic80
|
├─ keep platforms this box can run c64 → c64, tic80 → tic80
|
||||||
├─ pick the newest non-dev release that carries the right asset
|
├─ pick the newest non-dev release that carries the right asset
|
||||||
├─ GET /api/download?path=<asset> .prg / .tic into the ROM folder
|
├─ GET /api/download?path=<asset> .prg / .tic into the ROM folder
|
||||||
├─ GET /api/image/<id> box art
|
├─ GET <software.imageUrl> box art
|
||||||
└─ merge gamelist.xml title, desc, author, image
|
└─ merge gamelist.xml title, desc, author, image
|
||||||
```
|
```
|
||||||
|
|
||||||
Python 3 standard library only — Batocera ships python3 and no pip.
|
Python 3 standard library only — Batocera ships python3 and no pip.
|
||||||
|
|
||||||
Repository: `https://git.teletypegames.org/tools/batocera-store`.
|
## Installing a store
|
||||||
The catalog side lives in `teletypegames` (`libs/ruby/warp_engine`).
|
|
||||||
|
|
||||||
## Install
|
The installer takes the store as a parameter, so it works with any config:
|
||||||
|
|
||||||
From a checkout of this repo:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
scp -r batocera-store root@batocera:/tmp/
|
STORE_CONFIG=https://git.example.org/tools/my-store/raw/branch/master/config.json \
|
||||||
ssh root@batocera /tmp/batocera-store/install.sh
|
bash <(curl -fsSL https://git.teletypegames.org/tools/warp-engine-batocera-store/raw/branch/master/install.sh)
|
||||||
```
|
```
|
||||||
|
|
||||||
Or straight onto the device:
|
From a checkout, with a local config:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -fsSL https://git.teletypegames.org/tools/batocera-store/raw/branch/master/install.sh | bash
|
scp -r warp-engine-batocera-store root@batocera:/tmp/
|
||||||
|
ssh root@batocera 'STORE_CONFIG=/tmp/my-config.json /tmp/warp-engine-batocera-store/install.sh'
|
||||||
```
|
```
|
||||||
|
|
||||||
The installer drops `ttg-store` in `/userdata/system/ttg-store/`, writes a
|
It reads `store.id` from the config, installs the engine and the config into
|
||||||
default config, creates the Ports entry and runs the first sync. Restart
|
`/userdata/system/batocera-store/<id>/`, writes a `<id>-store` launcher and a
|
||||||
EmulationStation (`batocera-es-swissknife --restart`) to see the games.
|
Ports entry, and runs the first sync. Restart EmulationStation
|
||||||
|
(`batocera-es-swissknife --restart`) to see the games.
|
||||||
|
|
||||||
|
Installer knobs, all environment variables: `STORE_CONFIG` (required),
|
||||||
|
`BATOCERA_STORE_ROOT`, `BATOCERA_PORTS_DIR`, `BATOCERA_PORT_NAME`,
|
||||||
|
`ENGINE_RAW_BASE`.
|
||||||
|
|
||||||
|
## Writing a store repository
|
||||||
|
|
||||||
|
Three files:
|
||||||
|
|
||||||
|
```
|
||||||
|
my-batocera-store/
|
||||||
|
├── config.json the store: URL, name, subfolder, platform mapping
|
||||||
|
├── install.sh a wrapper that hands that config to the engine installer
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
`install.sh` is small enough to quote in full:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/tools/warp-engine-batocera-store/raw/branch/master}"
|
||||||
|
export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-batocera-store/raw/branch/master/config.json}"
|
||||||
|
export BATOCERA_PORT_NAME="My Store"
|
||||||
|
curl -fsSL "$ENGINE_RAW_BASE/install.sh" | bash
|
||||||
|
```
|
||||||
|
|
||||||
|
Pick a `store.id` and a `paths.subfolder` nobody else uses — they are what keeps
|
||||||
|
two stores on the same box from deleting each other's games.
|
||||||
|
|
||||||
## Use
|
## Use
|
||||||
|
|
||||||
**On the device.** *Ports ▸ Teletype Games Store*. The script downloads
|
**On the device.** *Ports ▸ "\<store name\>"*. The script downloads anything
|
||||||
anything new, then restarts EmulationStation so the games show up. Ports
|
new, then restarts EmulationStation so the games show up. Ports scripts get no
|
||||||
scripts get no console in Batocera, so the output goes to
|
console in Batocera, so the output goes to `store.log` in the store home.
|
||||||
`/userdata/system/ttg-store/ttg-store.log`.
|
|
||||||
|
|
||||||
**Over SSH.**
|
**Over SSH**, through the launcher the installer wrote:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
ttg-store list # compatible catalog entries; * installed, ^ update available
|
S=/userdata/system/batocera-store/example-store
|
||||||
ttg-store sync # download everything new, refresh gamelists
|
|
||||||
ttg-store sync blessingofra # just one title (never prunes)
|
$S list # compatible catalog entries; * installed, ^ update available
|
||||||
ttg-store -n sync # dry run
|
$S sync # download everything new, refresh gamelists
|
||||||
ttg-store remove c64demo # uninstall
|
$S sync blessingofra # just one title (never prunes)
|
||||||
ttg-store config # effective configuration
|
$S -n sync # dry run
|
||||||
|
$S remove c64:demo # uninstall (bare name works too)
|
||||||
|
$S config # effective configuration
|
||||||
```
|
```
|
||||||
|
|
||||||
Global flags go **before** the subcommand: `ttg-store --roms-root /tmp/roms sync`.
|
Global flags go **before** the subcommand: `$S --roms-root /tmp/roms sync`.
|
||||||
|
|
||||||
## Where things land
|
## Where things land
|
||||||
|
|
||||||
```
|
```
|
||||||
/userdata/system/ttg-store/ ttg-store, config.json, state.json, catalog.json, ttg-store.log
|
/userdata/system/batocera-store/
|
||||||
/userdata/roms/c64/teletypegames/ blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
|
├── example-store launcher for the store below
|
||||||
/userdata/roms/c64/teletypegames/images/ blessingofra.png, …
|
└── example/
|
||||||
|
├── store.py, config.json the engine and the store that selected it
|
||||||
|
└── state.json, catalog.json, store.log
|
||||||
|
|
||||||
|
/userdata/roms/c64/example/ blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
|
||||||
|
/userdata/roms/c64/example/images/ blessingofra.png, …
|
||||||
/userdata/roms/c64/gamelist.xml our entries merged in
|
/userdata/roms/c64/gamelist.xml our entries merged in
|
||||||
/userdata/roms/ports/Teletype Games Store.sh
|
/userdata/roms/ports/Example Store.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
Everything installed lives under the `teletypegames/` subfolder of a system, so
|
Everything a store installs lives under its own `paths.subfolder`, so a prune
|
||||||
a prune can never reach ROMs you put there yourself. The first time a
|
can never reach ROMs you put there yourself — or another store's games. The
|
||||||
`gamelist.xml` is touched it is copied to `gamelist.xml.ttg-backup`; on every
|
first time a `gamelist.xml` is touched it is copied to
|
||||||
merge only `<game>`/`<folder>` nodes under `teletypegames/` are rewritten —
|
`gamelist.xml.<store id>-backup`; on every merge only `<game>`/`<folder>` nodes
|
||||||
your own entries keep their play counts, favourites and scraped media.
|
under that store's subfolder are rewritten, so your own entries keep their play
|
||||||
|
counts, favourites and scraped media, and two stores can share one gamelist.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
`/userdata/system/ttg-store/config.json`, written by `ttg-store config --write`:
|
`config.json` in the store home, template in `config.example.json`:
|
||||||
|
|
||||||
| Key | Default | Meaning |
|
| Key | Default | Meaning |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `base_url` | `https://teletypegames.org` | WarpEngine host |
|
| `store.id` | `warp` | slug: names the store home, log prefix and gamelist backup |
|
||||||
| `roms_root` | `/userdata/roms` | where systems live |
|
| `store.name` | `WarpEngine Store` | display name for the Ports entry |
|
||||||
| `subfolder` | `teletypegames` | our subfolder inside each system |
|
| `store.base_url` | — | WarpEngine host |
|
||||||
| `folder_name` | `Teletype Games` | display name of that folder in ES |
|
| `store.api.catalog` | `/api/software` | catalog endpoint, if the engine is mounted elsewhere |
|
||||||
| `statuses` | `["released", "archived"]` | catalog `status` values to install |
|
| `store.api.download` | `/api/download` | download endpoint |
|
||||||
| `owner_id` | `null` | restrict to one publisher (`/api/software?owner_id=`) |
|
| `paths.roms_root` | `/userdata/roms` | where systems live |
|
||||||
| `only` / `exclude` | `[]` | software-name allow / deny lists |
|
| `paths.subfolder` | `warp` | this store's subfolder inside each system |
|
||||||
|
| `emulationstation.folder_name` | `WarpEngine Store` | display name of that folder in ES |
|
||||||
|
| `emulationstation.restart` | `true` | restart EmulationStation after a change |
|
||||||
|
| `catalog.statuses` | `["released", "archived"]` | catalog `status` values to install |
|
||||||
|
| `catalog.owner_id` | `null` | restrict to one publisher (`/api/software?owner_id=`) |
|
||||||
|
| `catalog.only` / `catalog.exclude` | `[]` | software-name allow / deny lists |
|
||||||
| `platforms` | c64, tic80 | platform → system, asset kind, extension, enabled |
|
| `platforms` | c64, tic80 | platform → system, asset kind, extension, enabled |
|
||||||
| `prune` | `true` | remove games that left the catalog or the filters |
|
| `behavior.prune` | `true` | remove games that left the catalog or the filters |
|
||||||
| `restart_es` | `true` | restart EmulationStation after a change |
|
| `behavior.timeout` | `30` | HTTP timeout, seconds |
|
||||||
| `timeout` | `30` | HTTP timeout, seconds |
|
| `behavior.insecure` | `false` | skip TLS verification (self-hosted test instances) |
|
||||||
| `insecure` | `false` | skip TLS verification (self-hosted test instances) |
|
|
||||||
|
|
||||||
### Platform mapping
|
### Platform mapping
|
||||||
|
|
||||||
@@ -106,20 +166,20 @@ system can boot directly:
|
|||||||
|
|
||||||
The other WarpEngine platforms (`ebitengine`, `love`, `godot`, `bevy`,
|
The other WarpEngine platforms (`ebitengine`, `love`, `godot`, `bevy`,
|
||||||
`phaser`) ship `html` and per-OS zip archives, not a ROM a Batocera system
|
`phaser`) ship `html` and per-OS zip archives, not a ROM a Batocera system
|
||||||
launches, so they are not mapped. Adding one is a config edit:
|
launches, so they are not mapped by default. Adding one is a config edit:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
"platforms": { "godot": { "system": "godot", "kind": "linux_x64", "ext": ".zip", "enabled": true } }
|
"platforms": { "godot": { "system": "godot", "kind": "linux_x64", "ext": ".zip", "enabled": true } }
|
||||||
```
|
```
|
||||||
|
|
||||||
A platform is also skipped when the box has no ROM folder for its system —
|
A platform is also skipped when the box has no ROM folder for its system —
|
||||||
`ttg-store list` reports that as `skipped <name>: no '<system>' ROM folder`.
|
`list` reports that as `skipped <name>: no '<system>' ROM folder`.
|
||||||
|
|
||||||
## How a sync stays safe next to EmulationStation
|
## How a sync stays safe next to EmulationStation
|
||||||
|
|
||||||
EmulationStation keeps gamelists in memory and writes them back when it exits,
|
EmulationStation keeps gamelists in memory and writes them back when it exits,
|
||||||
so a merge done while it runs can be clobbered. Launched from the Ports menu
|
so a merge done while it runs can be clobbered. Launched from the Ports menu
|
||||||
(`sync-from-es`), the tool therefore downloads first, then hands the gamelist
|
(`sync-from-es`), the engine therefore downloads first, then hands the gamelist
|
||||||
merge to a detached `apply-gamelists --wait-pid <es-pid>` child that waits for
|
merge to a detached `apply-gamelists --wait-pid <es-pid>` child that waits for
|
||||||
the old ES process to die before writing. From SSH, with no ES running, the
|
the old ES process to die before writing. From SSH, with no ES running, the
|
||||||
merge happens inline.
|
merge happens inline.
|
||||||
@@ -128,3 +188,10 @@ Re-running a sync is idempotent: files already present are left alone, and a
|
|||||||
release bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new
|
release bump (`1.1` → `2.0.0`) deletes the old asset before fetching the new
|
||||||
one. `GET /api/download` is used rather than `/file/`, so downloads count
|
one. `GET /api/download` is used rather than `/file/`, so downloads count
|
||||||
towards the catalog's statistics.
|
towards the catalog's statistics.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
`state.json` records what this store installed, keyed `<system>:<name>` so the
|
||||||
|
same software name on two systems stays two entries. A `version: 1` state file
|
||||||
|
written by the pre-split `ttg-store` is migrated to that key format on first
|
||||||
|
run.
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
{
|
||||||
|
"store": {
|
||||||
|
"id": "example",
|
||||||
|
"name": "Example Store",
|
||||||
|
"base_url": "https://example.org",
|
||||||
|
"api": {
|
||||||
|
"catalog": "/api/software",
|
||||||
|
"download": "/api/download"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
"paths": {
|
||||||
|
"roms_root": "/userdata/roms",
|
||||||
|
"subfolder": "example"
|
||||||
|
},
|
||||||
|
|
||||||
|
"emulationstation": {
|
||||||
|
"folder_name": "Example Store",
|
||||||
|
"restart": true
|
||||||
|
},
|
||||||
|
|
||||||
|
"catalog": {
|
||||||
|
"statuses": ["released", "archived"],
|
||||||
|
"owner_id": null,
|
||||||
|
"only": [],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
|
||||||
|
"platforms": {
|
||||||
|
"c64": { "system": "c64", "kind": "cartridge", "ext": ".prg", "enabled": true },
|
||||||
|
"tic80": { "system": "tic80", "kind": "cartridge", "ext": ".tic", "enabled": true }
|
||||||
|
},
|
||||||
|
|
||||||
|
"behavior": {
|
||||||
|
"prune": true,
|
||||||
|
"timeout": 30,
|
||||||
|
"insecure": false
|
||||||
|
}
|
||||||
|
}
|
||||||
+79
-33
@@ -1,70 +1,116 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Install ttg-store on a Batocera box.
|
# Install a WarpEngine store on a Batocera box.
|
||||||
#
|
#
|
||||||
# scp -r batocera-store root@batocera:/tmp/ && ssh root@batocera /tmp/batocera-store/install.sh
|
# This installer belongs to the engine, so it takes the store as a parameter:
|
||||||
|
# point STORE_CONFIG at a config.json — a local path or a URL — and it sets up
|
||||||
|
# that store. A store repository's own install.sh is a three-line wrapper that
|
||||||
|
# supplies its config; see README.
|
||||||
#
|
#
|
||||||
# or, straight from the forge:
|
# STORE_CONFIG=./config.json ./install.sh
|
||||||
#
|
# STORE_CONFIG=https://git.example.org/tools/my-store/raw/branch/master/config.json \
|
||||||
# curl -fsSL https://git.teletypegames.org/tools/batocera-store/raw/branch/master/install.sh | bash
|
# bash <(curl -fsSL https://git.teletypegames.org/tools/warp-engine-batocera-store/raw/branch/master/install.sh)
|
||||||
#
|
#
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
HOME_DIR="${TTG_STORE_HOME:-/userdata/system/ttg-store}"
|
STORE_ROOT="${BATOCERA_STORE_ROOT:-/userdata/system/batocera-store}"
|
||||||
PORTS_DIR="${TTG_PORTS_DIR:-/userdata/roms/ports}"
|
PORTS_DIR="${BATOCERA_PORTS_DIR:-/userdata/roms/ports}"
|
||||||
PORT_NAME="${TTG_PORT_NAME:-Teletype Games Store}"
|
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/tools/warp-engine-batocera-store/raw/branch/master}"
|
||||||
RAW_BASE="${TTG_RAW_BASE:-https://git.teletypegames.org/tools/batocera-store/raw/branch/master}"
|
STORE_CONFIG="${STORE_CONFIG:-}"
|
||||||
SRC_DIR="$(cd "$(dirname "$(readlink -f "$0")")" && pwd)"
|
# Only set when run as a file; piped from curl "$0" is `bash`, and then the
|
||||||
|
# engine has to come from the forge rather than from whatever the cwd holds.
|
||||||
|
SRC_DIR=""
|
||||||
|
if [ -f "$0" ]; then
|
||||||
|
SRC_DIR="$(cd "$(dirname "$(readlink -f "$0")")" && pwd)"
|
||||||
|
fi
|
||||||
|
|
||||||
say() { echo "[install] $*"; }
|
say() { echo "[install] $*"; }
|
||||||
|
die() { echo "[install] error: $*" >&2; exit 1; }
|
||||||
|
|
||||||
command -v python3 >/dev/null 2>&1 || { echo "python3 is required" >&2; exit 1; }
|
command -v python3 >/dev/null 2>&1 || die "python3 is required"
|
||||||
|
[ -n "$STORE_CONFIG" ] || die "STORE_CONFIG is required (path or URL of a store config.json)"
|
||||||
[ -d /userdata ] || say "warning: /userdata not found — this does not look like a Batocera box"
|
[ -d /userdata ] || say "warning: /userdata not found — this does not look like a Batocera box"
|
||||||
|
|
||||||
mkdir -p "$HOME_DIR" "$PORTS_DIR"
|
fetch() { # fetch <source> <dest>; source may be a local path or a URL
|
||||||
|
case "$1" in
|
||||||
|
http://*|https://*) curl -fsSL "$1" -o "$2" ;;
|
||||||
|
*) [ -f "$1" ] || die "no such file: $1"; cp "$1" "$2" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
if [ -f "$SRC_DIR/ttg-store" ]; then
|
# Stage the config first: the store id in it decides where everything lands.
|
||||||
say "installing ttg-store from $SRC_DIR"
|
TMP_CFG="$(mktemp)"
|
||||||
install -m 0755 "$SRC_DIR/ttg-store" "$HOME_DIR/ttg-store"
|
trap 'rm -f "$TMP_CFG"' EXIT
|
||||||
|
say "reading store config from $STORE_CONFIG"
|
||||||
|
fetch "$STORE_CONFIG" "$TMP_CFG"
|
||||||
|
|
||||||
|
STORE_META="$(python3 -c '
|
||||||
|
import json, sys
|
||||||
|
cfg = json.load(open(sys.argv[1])).get("store") or {}
|
||||||
|
sid = cfg.get("id") or ""
|
||||||
|
if not sid or "/" in sid:
|
||||||
|
sys.exit("store.id is required and must not contain a slash")
|
||||||
|
print(sid, cfg.get("name") or sid)
|
||||||
|
' "$TMP_CFG")" || die "cannot read $STORE_CONFIG"
|
||||||
|
|
||||||
|
STORE_ID="${STORE_META%% *}"
|
||||||
|
STORE_NAME="${STORE_META#* }"
|
||||||
|
|
||||||
|
STORE_HOME="$STORE_ROOT/$STORE_ID"
|
||||||
|
PORT_NAME="${BATOCERA_PORT_NAME:-$STORE_NAME}"
|
||||||
|
LAUNCHER="$STORE_ROOT/$STORE_ID-store"
|
||||||
|
|
||||||
|
mkdir -p "$STORE_HOME" "$PORTS_DIR"
|
||||||
|
|
||||||
|
if [ -n "$SRC_DIR" ] && [ -f "$SRC_DIR/store.py" ]; then
|
||||||
|
say "installing the engine from $SRC_DIR"
|
||||||
|
install -m 0755 "$SRC_DIR/store.py" "$STORE_HOME/store.py"
|
||||||
else
|
else
|
||||||
say "downloading ttg-store from $RAW_BASE"
|
say "downloading the engine from $ENGINE_RAW_BASE"
|
||||||
curl -fsSL "$RAW_BASE/ttg-store" -o "$HOME_DIR/ttg-store"
|
curl -fsSL "$ENGINE_RAW_BASE/store.py" -o "$STORE_HOME/store.py"
|
||||||
chmod 0755 "$HOME_DIR/ttg-store"
|
chmod 0755 "$STORE_HOME/store.py"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ttg-store reads TTG_STORE_HOME for its config/state/cache locations.
|
# The config is the store's identity, so it is always refreshed; state.json,
|
||||||
export TTG_STORE_HOME="$HOME_DIR"
|
# catalog.json and the log stay put, which makes re-running the upgrade path.
|
||||||
|
if [ -f "$STORE_HOME/config.json" ]; then
|
||||||
if [ -f "$HOME_DIR/config.json" ]; then
|
say "updating $STORE_HOME/config.json (previous copy kept as config.json.bak)"
|
||||||
say "keeping existing $HOME_DIR/config.json"
|
cp "$STORE_HOME/config.json" "$STORE_HOME/config.json.bak"
|
||||||
else
|
|
||||||
"$HOME_DIR/ttg-store" config --write
|
|
||||||
fi
|
fi
|
||||||
|
install -m 0644 "$TMP_CFG" "$STORE_HOME/config.json"
|
||||||
|
|
||||||
|
# A short launcher so the CLI is one path, not a python invocation.
|
||||||
|
cat > "$LAUNCHER" <<EOF
|
||||||
|
#!/bin/bash
|
||||||
|
# $STORE_NAME — CLI wrapper around the shared batocera-store engine.
|
||||||
|
export BATOCERA_STORE_HOME="$STORE_HOME"
|
||||||
|
exec "$STORE_HOME/store.py" --config "$STORE_HOME/config.json" "\$@"
|
||||||
|
EOF
|
||||||
|
chmod 0755 "$LAUNCHER"
|
||||||
|
|
||||||
# The Ports entry: EmulationStation gives launched scripts no console, so send
|
# The Ports entry: EmulationStation gives launched scripts no console, so send
|
||||||
# everything to a log file the user can read over SSH.
|
# everything to a log file the user can read over SSH.
|
||||||
cat > "$PORTS_DIR/$PORT_NAME.sh" <<EOF
|
cat > "$PORTS_DIR/$PORT_NAME.sh" <<EOF
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Teletype Games Store — syncs the catalog into this box's ROM folders.
|
# $STORE_NAME — syncs the catalog into this box's ROM folders.
|
||||||
export TTG_STORE_HOME="$HOME_DIR"
|
exec >>"$STORE_HOME/store.log" 2>&1
|
||||||
exec >>"$HOME_DIR/ttg-store.log" 2>&1
|
|
||||||
echo "=== \$(date -Iseconds) sync started ==="
|
echo "=== \$(date -Iseconds) sync started ==="
|
||||||
"$HOME_DIR/ttg-store" sync-from-es
|
"$LAUNCHER" sync-from-es
|
||||||
echo "=== \$(date -Iseconds) sync finished (exit \$?) ==="
|
echo "=== \$(date -Iseconds) sync finished (exit \$?) ==="
|
||||||
EOF
|
EOF
|
||||||
chmod 0755 "$PORTS_DIR/$PORT_NAME.sh"
|
chmod 0755 "$PORTS_DIR/$PORT_NAME.sh"
|
||||||
say "ports entry: $PORTS_DIR/$PORT_NAME.sh"
|
say "ports entry: $PORTS_DIR/$PORT_NAME.sh"
|
||||||
|
|
||||||
say "running the first sync"
|
say "running the first sync"
|
||||||
"$HOME_DIR/ttg-store" --verbose sync --no-restart
|
"$LAUNCHER" --verbose sync --no-restart
|
||||||
|
|
||||||
cat <<EOF
|
cat <<EOF
|
||||||
|
|
||||||
[install] done.
|
[install] done.
|
||||||
|
|
||||||
Menu: Ports -> "$PORT_NAME" (re-run any time to pull new releases)
|
Menu: Ports -> "$PORT_NAME" (re-run any time to pull new releases)
|
||||||
CLI: $HOME_DIR/ttg-store list|sync|remove
|
CLI: $LAUNCHER list|sync|remove
|
||||||
Config: $HOME_DIR/config.json
|
Config: $STORE_HOME/config.json
|
||||||
Log: $HOME_DIR/ttg-store.log
|
Log: $STORE_HOME/store.log
|
||||||
|
|
||||||
Restart EmulationStation to see the games: batocera-es-swissknife --restart
|
Restart EmulationStation to see the games: batocera-es-swissknife --restart
|
||||||
EOF
|
EOF
|
||||||
|
|||||||
+212
-92
@@ -1,11 +1,15 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""ttg-store — Teletype Games catalog client for Batocera.
|
"""store.py — WarpEngine catalog client for Batocera.
|
||||||
|
|
||||||
Reads the WarpEngine catalog API (`GET /api/software`), keeps the entries whose
|
Reads a WarpEngine catalog API (`GET /api/software`), keeps the entries whose
|
||||||
platform maps to a system this Batocera box actually has, downloads the matching
|
platform maps to a system this Batocera box actually has, downloads the matching
|
||||||
release asset into that system's ROM folder and writes EmulationStation
|
release asset into that system's ROM folder and writes EmulationStation
|
||||||
metadata (title, description, author, box art) into its gamelist.xml.
|
metadata (title, description, author, box art) into its gamelist.xml.
|
||||||
|
|
||||||
|
The engine knows nothing about any particular site: which host to talk to, what
|
||||||
|
the store is called and where its games land all come from `config.json`. A
|
||||||
|
store repository is therefore just that config plus an installer — see README.
|
||||||
|
|
||||||
Standard library only — Batocera ships python3 but no pip packages.
|
Standard library only — Batocera ships python3 but no pip packages.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -22,23 +26,43 @@ import urllib.parse
|
|||||||
import urllib.request
|
import urllib.request
|
||||||
import xml.etree.ElementTree as ET
|
import xml.etree.ElementTree as ET
|
||||||
|
|
||||||
VERSION = "1.0.0"
|
VERSION = "2.0.0"
|
||||||
USER_AGENT = f"ttg-store/{VERSION} (batocera)"
|
STATE_VERSION = 2
|
||||||
STATE_VERSION = 1
|
|
||||||
|
|
||||||
HOME = os.environ.get("TTG_STORE_HOME", "/userdata/system/ttg-store")
|
# Every store keeps its code, config, state and log in one directory, so a box
|
||||||
CONFIG_PATH = os.path.join(HOME, "config.json")
|
# can carry several stores side by side without them treading on each other.
|
||||||
STATE_PATH = os.path.join(HOME, "state.json")
|
# The installer exports BATOCERA_STORE_HOME; run straight from a checkout the
|
||||||
CATALOG_CACHE = os.path.join(HOME, "catalog.json")
|
# script's own directory is the store home.
|
||||||
|
DEFAULT_HOME = os.environ.get("BATOCERA_STORE_HOME") or os.path.dirname(os.path.abspath(__file__))
|
||||||
|
|
||||||
DEFAULT_CONFIG = {
|
DEFAULT_CONFIG = {
|
||||||
# WarpEngine host serving /api/software, /api/download and /api/image.
|
"store": {
|
||||||
"base_url": "https://teletypegames.org",
|
# Short slug: names the state, the log prefix and the gamelist backup.
|
||||||
|
"id": "warp",
|
||||||
|
# Human-readable, used for the Ports entry and the ES folder label.
|
||||||
|
"name": "WarpEngine Store",
|
||||||
|
# The WarpEngine host serving the catalog.
|
||||||
|
"base_url": "https://example.org",
|
||||||
|
# Endpoint paths, in case the engine is mounted somewhere other than /.
|
||||||
|
"api": {
|
||||||
|
"catalog": "/api/software",
|
||||||
|
"download": "/api/download",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"paths": {
|
||||||
"roms_root": "/userdata/roms",
|
"roms_root": "/userdata/roms",
|
||||||
# Everything we install lives under this subfolder of the system's ROM dir,
|
# Everything we install lives under this subfolder of the system's ROM
|
||||||
# so a prune can never touch ROMs the user put there themselves.
|
# dir, so a prune can never touch ROMs the user put there themselves —
|
||||||
"subfolder": "teletypegames",
|
# nor games installed by another store.
|
||||||
"folder_name": "Teletype Games",
|
"subfolder": "warp",
|
||||||
|
},
|
||||||
|
"emulationstation": {
|
||||||
|
# Display name of our subfolder in ES; null hides the folder node.
|
||||||
|
"folder_name": "WarpEngine Store",
|
||||||
|
# Restart EmulationStation after a sync that changed something.
|
||||||
|
"restart": True,
|
||||||
|
},
|
||||||
|
"catalog": {
|
||||||
# Catalog `status` values worth installing. "development" is left out.
|
# Catalog `status` values worth installing. "development" is left out.
|
||||||
"statuses": ["released", "archived"],
|
"statuses": ["released", "archived"],
|
||||||
# Restrict to one publisher (WarpEngine owner_id), null = whole catalog.
|
# Restrict to one publisher (WarpEngine owner_id), null = whole catalog.
|
||||||
@@ -46,17 +70,18 @@ DEFAULT_CONFIG = {
|
|||||||
# Software `name` allow/deny lists; empty allow list means "everything".
|
# Software `name` allow/deny lists; empty allow list means "everything".
|
||||||
"only": [],
|
"only": [],
|
||||||
"exclude": [],
|
"exclude": [],
|
||||||
|
},
|
||||||
# WarpEngine platform -> Batocera system + which release asset to pull.
|
# WarpEngine platform -> Batocera system + which release asset to pull.
|
||||||
"platforms": {
|
"platforms": {
|
||||||
"c64": {"system": "c64", "kind": "cartridge", "ext": ".prg", "enabled": True},
|
"c64": {"system": "c64", "kind": "cartridge", "ext": ".prg", "enabled": True},
|
||||||
"tic80": {"system": "tic80", "kind": "cartridge", "ext": ".tic", "enabled": True},
|
"tic80": {"system": "tic80", "kind": "cartridge", "ext": ".tic", "enabled": True},
|
||||||
},
|
},
|
||||||
|
"behavior": {
|
||||||
# Drop installed games that fell out of the catalog or the filters.
|
# Drop installed games that fell out of the catalog or the filters.
|
||||||
"prune": True,
|
"prune": True,
|
||||||
# Restart EmulationStation after a sync that changed something.
|
|
||||||
"restart_es": True,
|
|
||||||
"timeout": 30,
|
"timeout": 30,
|
||||||
"insecure": False,
|
"insecure": False,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -65,10 +90,26 @@ DEFAULT_CONFIG = {
|
|||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
VERBOSE = False
|
VERBOSE = False
|
||||||
|
# Log prefix and temp-file prefix; replaced with the store id once configured.
|
||||||
|
TAG = "batocera-store"
|
||||||
|
|
||||||
|
HOME = DEFAULT_HOME
|
||||||
|
CONFIG_PATH = os.path.join(HOME, "config.json")
|
||||||
|
STATE_PATH = os.path.join(HOME, "state.json")
|
||||||
|
CATALOG_CACHE = os.path.join(HOME, "catalog.json")
|
||||||
|
|
||||||
|
|
||||||
|
def set_home(config_path):
|
||||||
|
"""Anchor state and cache next to the config file that selected this store."""
|
||||||
|
global HOME, CONFIG_PATH, STATE_PATH, CATALOG_CACHE
|
||||||
|
CONFIG_PATH = os.path.abspath(config_path)
|
||||||
|
HOME = os.path.dirname(CONFIG_PATH)
|
||||||
|
STATE_PATH = os.path.join(HOME, "state.json")
|
||||||
|
CATALOG_CACHE = os.path.join(HOME, "catalog.json")
|
||||||
|
|
||||||
|
|
||||||
def log(msg):
|
def log(msg):
|
||||||
print(f"[ttg-store] {msg}", flush=True)
|
print(f"[{TAG}] {msg}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
def debug(msg):
|
def debug(msg):
|
||||||
@@ -77,7 +118,7 @@ def debug(msg):
|
|||||||
|
|
||||||
|
|
||||||
def die(msg, code=1):
|
def die(msg, code=1):
|
||||||
print(f"[ttg-store] error: {msg}", file=sys.stderr, flush=True)
|
print(f"[{TAG}] error: {msg}", file=sys.stderr, flush=True)
|
||||||
sys.exit(code)
|
sys.exit(code)
|
||||||
|
|
||||||
|
|
||||||
@@ -100,7 +141,7 @@ def write_atomic(path, blob):
|
|||||||
"""Write bytes to `path` via a temp file in the same dir, then rename."""
|
"""Write bytes to `path` via a temp file in the same dir, then rename."""
|
||||||
directory = os.path.dirname(path) or "."
|
directory = os.path.dirname(path) or "."
|
||||||
os.makedirs(directory, exist_ok=True)
|
os.makedirs(directory, exist_ok=True)
|
||||||
fd, tmp = tempfile.mkstemp(dir=directory, prefix=".ttg-", suffix=".tmp")
|
fd, tmp = tempfile.mkstemp(dir=directory, prefix=f".{TAG}-", suffix=".tmp")
|
||||||
try:
|
try:
|
||||||
with os.fdopen(fd, "wb") as fh:
|
with os.fdopen(fd, "wb") as fh:
|
||||||
fh.write(blob)
|
fh.write(blob)
|
||||||
@@ -109,33 +150,75 @@ def write_atomic(path, blob):
|
|||||||
if os.path.exists(tmp):
|
if os.path.exists(tmp):
|
||||||
os.unlink(tmp)
|
os.unlink(tmp)
|
||||||
raise
|
raise
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def deep_merge(base, overlay):
|
||||||
|
"""Recursively merge `overlay` into a copy of `base`; dicts merge, rest wins."""
|
||||||
|
merged = dict(base)
|
||||||
|
for key, value in (overlay or {}).items():
|
||||||
|
if isinstance(value, dict) and isinstance(merged.get(key), dict):
|
||||||
|
merged[key] = deep_merge(merged[key], value)
|
||||||
|
else:
|
||||||
|
merged[key] = value
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
def load_config(path, overrides=None):
|
def load_config(path, overrides=None):
|
||||||
cfg = json.loads(json.dumps(DEFAULT_CONFIG)) # deep copy
|
global TAG
|
||||||
|
|
||||||
user = load_json(path, default=None)
|
user = load_json(path, default=None)
|
||||||
if user:
|
if user is None:
|
||||||
platforms = user.pop("platforms", None)
|
log(f"warning: no config at {path} — falling back to the built-in defaults")
|
||||||
cfg.update(user)
|
cfg = deep_merge(json.loads(json.dumps(DEFAULT_CONFIG)), user or {})
|
||||||
if platforms:
|
|
||||||
for name, spec in platforms.items():
|
|
||||||
merged = dict(cfg["platforms"].get(name, {}))
|
|
||||||
merged.update(spec)
|
|
||||||
cfg["platforms"][name] = merged
|
|
||||||
for key, value in (overrides or {}).items():
|
for key, value in (overrides or {}).items():
|
||||||
if value is not None:
|
if value is None:
|
||||||
cfg[key] = value
|
continue
|
||||||
cfg["base_url"] = cfg["base_url"].rstrip("/")
|
section, _, leaf = key.partition(".")
|
||||||
|
cfg[section][leaf] = value
|
||||||
|
|
||||||
|
cfg["store"]["base_url"] = str(cfg["store"]["base_url"]).rstrip("/")
|
||||||
|
if not cfg["store"].get("id"):
|
||||||
|
die("config: store.id is required")
|
||||||
|
if not cfg["paths"].get("subfolder"):
|
||||||
|
die("config: paths.subfolder is required — it is what keeps stores apart")
|
||||||
|
TAG = f"{cfg['store']['id']}-store"
|
||||||
return cfg
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
# Shorthands for the values the code below reaches for constantly.
|
||||||
|
def base_url(cfg):
|
||||||
|
return cfg["store"]["base_url"]
|
||||||
|
|
||||||
|
|
||||||
|
def subfolder(cfg):
|
||||||
|
return cfg["paths"]["subfolder"]
|
||||||
|
|
||||||
|
|
||||||
|
def roms_root(cfg):
|
||||||
|
return cfg["paths"]["roms_root"]
|
||||||
|
|
||||||
|
|
||||||
|
def api_url(cfg, endpoint, params=None):
|
||||||
|
path = cfg["store"]["api"][endpoint]
|
||||||
|
url = base_url(cfg) + "/" + path.lstrip("/")
|
||||||
|
if params:
|
||||||
|
url += "?" + urllib.parse.urlencode(params)
|
||||||
|
return url
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
# HTTP
|
# HTTP
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def user_agent(cfg):
|
||||||
|
return f"batocera-store/{VERSION} ({cfg['store']['id']})"
|
||||||
|
|
||||||
|
|
||||||
def _opener(cfg):
|
def _opener(cfg):
|
||||||
if cfg.get("insecure"):
|
if cfg["behavior"].get("insecure"):
|
||||||
import ssl
|
import ssl
|
||||||
|
|
||||||
ctx = ssl.create_default_context()
|
ctx = ssl.create_default_context()
|
||||||
@@ -147,20 +230,20 @@ def _opener(cfg):
|
|||||||
|
|
||||||
def http_get(cfg, url):
|
def http_get(cfg, url):
|
||||||
"""Return (body_bytes, content_type)."""
|
"""Return (body_bytes, content_type)."""
|
||||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
req = urllib.request.Request(url, headers={"User-Agent": user_agent(cfg)})
|
||||||
with _opener(cfg).open(req, timeout=cfg["timeout"]) as resp:
|
with _opener(cfg).open(req, timeout=cfg["behavior"]["timeout"]) as resp:
|
||||||
return resp.read(), resp.headers.get("Content-Type", "")
|
return resp.read(), resp.headers.get("Content-Type", "")
|
||||||
|
|
||||||
|
|
||||||
def http_download(cfg, url, dest):
|
def http_download(cfg, url, dest):
|
||||||
"""Stream `url` into `dest` atomically. Returns bytes written."""
|
"""Stream `url` into `dest` atomically. Returns bytes written."""
|
||||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
req = urllib.request.Request(url, headers={"User-Agent": user_agent(cfg)})
|
||||||
directory = os.path.dirname(dest) or "."
|
directory = os.path.dirname(dest) or "."
|
||||||
os.makedirs(directory, exist_ok=True)
|
os.makedirs(directory, exist_ok=True)
|
||||||
fd, tmp = tempfile.mkstemp(dir=directory, prefix=".ttg-", suffix=".part")
|
fd, tmp = tempfile.mkstemp(dir=directory, prefix=f".{TAG}-", suffix=".part")
|
||||||
written = 0
|
written = 0
|
||||||
try:
|
try:
|
||||||
with _opener(cfg).open(req, timeout=cfg["timeout"]) as resp, os.fdopen(fd, "wb") as out:
|
with _opener(cfg).open(req, timeout=cfg["behavior"]["timeout"]) as resp, os.fdopen(fd, "wb") as out:
|
||||||
while True:
|
while True:
|
||||||
chunk = resp.read(64 * 1024)
|
chunk = resp.read(64 * 1024)
|
||||||
if not chunk:
|
if not chunk:
|
||||||
@@ -183,9 +266,8 @@ def http_download(cfg, url, dest):
|
|||||||
|
|
||||||
|
|
||||||
def fetch_catalog(cfg, use_cache=False):
|
def fetch_catalog(cfg, use_cache=False):
|
||||||
url = f"{cfg['base_url']}/api/software"
|
owner = cfg["catalog"].get("owner_id")
|
||||||
if cfg.get("owner_id") is not None:
|
url = api_url(cfg, "catalog", {"owner_id": owner} if owner is not None else None)
|
||||||
url += "?" + urllib.parse.urlencode({"owner_id": cfg["owner_id"]})
|
|
||||||
try:
|
try:
|
||||||
body, _ = http_get(cfg, url)
|
body, _ = http_get(cfg, url)
|
||||||
data = json.loads(body.decode("utf-8"))
|
data = json.loads(body.decode("utf-8"))
|
||||||
@@ -237,9 +319,9 @@ def pick_release(entry, kind, ext):
|
|||||||
|
|
||||||
def select_games(cfg, catalog, available_systems):
|
def select_games(cfg, catalog, available_systems):
|
||||||
"""Catalog -> list of installable game dicts, plus a list of skip reasons."""
|
"""Catalog -> list of installable game dicts, plus a list of skip reasons."""
|
||||||
statuses = {s.lower() for s in cfg.get("statuses") or []}
|
statuses = {s.lower() for s in cfg["catalog"].get("statuses") or []}
|
||||||
only = {n.lower() for n in cfg.get("only") or []}
|
only = {n.lower() for n in cfg["catalog"].get("only") or []}
|
||||||
exclude = {n.lower() for n in cfg.get("exclude") or []}
|
exclude = {n.lower() for n in cfg["catalog"].get("exclude") or []}
|
||||||
|
|
||||||
games, skipped = [], []
|
games, skipped = [], []
|
||||||
for entry in catalog.get("softwares") or []:
|
for entry in catalog.get("softwares") or []:
|
||||||
@@ -297,23 +379,23 @@ def es_date(iso):
|
|||||||
|
|
||||||
|
|
||||||
def system_dir(cfg, system):
|
def system_dir(cfg, system):
|
||||||
return os.path.join(cfg["roms_root"], system)
|
return os.path.join(roms_root(cfg), system)
|
||||||
|
|
||||||
|
|
||||||
def detect_systems(cfg):
|
def detect_systems(cfg):
|
||||||
"""Systems this box has a ROM folder for — our compatibility check."""
|
"""Systems this box has a ROM folder for — our compatibility check."""
|
||||||
root = cfg["roms_root"]
|
root = roms_root(cfg)
|
||||||
if not os.path.isdir(root):
|
if not os.path.isdir(root):
|
||||||
return set()
|
return set()
|
||||||
return {d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d))}
|
return {d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d))}
|
||||||
|
|
||||||
|
|
||||||
def rel_rom_path(cfg, game):
|
def rel_rom_path(cfg, game):
|
||||||
return f"{cfg['subfolder']}/{game['asset']}"
|
return f"{subfolder(cfg)}/{game['asset']}"
|
||||||
|
|
||||||
|
|
||||||
def rel_image_path(cfg, game, ext):
|
def rel_image_path(cfg, game, ext):
|
||||||
return f"{cfg['subfolder']}/images/{game['name']}{ext}"
|
return f"{subfolder(cfg)}/images/{game['name']}{ext}"
|
||||||
|
|
||||||
|
|
||||||
def install_game(cfg, game, state_entry, dry_run=False):
|
def install_game(cfg, game, state_entry, dry_run=False):
|
||||||
@@ -326,7 +408,7 @@ def install_game(cfg, game, state_entry, dry_run=False):
|
|||||||
if os.path.isfile(rom_abs) and os.path.getsize(rom_abs) > 0:
|
if os.path.isfile(rom_abs) and os.path.getsize(rom_abs) > 0:
|
||||||
debug(f"{game['name']}: {game['asset']} already present")
|
debug(f"{game['name']}: {game['asset']} already present")
|
||||||
else:
|
else:
|
||||||
url = f"{cfg['base_url']}/api/download?" + urllib.parse.urlencode({"path": game["asset"]})
|
url = api_url(cfg, "download", {"path": game["asset"]})
|
||||||
if dry_run:
|
if dry_run:
|
||||||
log(f"would download {game['name']} {game['version']} -> {rom_abs}")
|
log(f"would download {game['name']} {game['version']} -> {rom_abs}")
|
||||||
else:
|
else:
|
||||||
@@ -348,7 +430,7 @@ def install_game(cfg, game, state_entry, dry_run=False):
|
|||||||
# Box art disappeared from the catalog: drop the file we cached.
|
# Box art disappeared from the catalog: drop the file we cached.
|
||||||
if image_rel and not dry_run:
|
if image_rel and not dry_run:
|
||||||
stale = os.path.join(base, image_rel)
|
stale = os.path.join(base, image_rel)
|
||||||
if image_rel.startswith(cfg["subfolder"] + "/") and os.path.isfile(stale):
|
if image_rel.startswith(subfolder(cfg) + "/") and os.path.isfile(stale):
|
||||||
os.unlink(stale)
|
os.unlink(stale)
|
||||||
changed = True
|
changed = True
|
||||||
image_rel = None
|
image_rel = None
|
||||||
@@ -359,10 +441,15 @@ def install_game(cfg, game, state_entry, dry_run=False):
|
|||||||
return record, changed
|
return record, changed
|
||||||
|
|
||||||
|
|
||||||
|
def image_url(cfg, game):
|
||||||
|
"""The catalog gives a server-relative `imageUrl`; absolute ones pass through."""
|
||||||
|
url = game["image_url"]
|
||||||
|
return url if url.startswith(("http://", "https://")) else base_url(cfg) + url
|
||||||
|
|
||||||
|
|
||||||
def fetch_image(cfg, game, base):
|
def fetch_image(cfg, game, base):
|
||||||
url = cfg["base_url"] + game["image_url"]
|
|
||||||
try:
|
try:
|
||||||
body, ctype = http_get(cfg, url)
|
body, ctype = http_get(cfg, image_url(cfg, game))
|
||||||
except (urllib.error.URLError, OSError) as exc:
|
except (urllib.error.URLError, OSError) as exc:
|
||||||
log(f"warning: box art for {game['name']} failed: {exc}")
|
log(f"warning: box art for {game['name']} failed: {exc}")
|
||||||
return None
|
return None
|
||||||
@@ -381,9 +468,9 @@ def remove_game(cfg, record, dry_run=False):
|
|||||||
if not rel:
|
if not rel:
|
||||||
continue
|
continue
|
||||||
path = os.path.join(base, rel)
|
path = os.path.join(base, rel)
|
||||||
# Never step outside our own subfolder.
|
# Never step outside our own subfolder — another store may own it.
|
||||||
if not rel.startswith(cfg["subfolder"] + "/"):
|
if not rel.startswith(subfolder(cfg) + "/"):
|
||||||
log(f"warning: refusing to delete {path} (outside {cfg['subfolder']}/)")
|
log(f"warning: refusing to delete {path} (outside {subfolder(cfg)}/)")
|
||||||
continue
|
continue
|
||||||
if os.path.isfile(path):
|
if os.path.isfile(path):
|
||||||
if dry_run:
|
if dry_run:
|
||||||
@@ -411,10 +498,11 @@ def merge_gamelist(cfg, system, records, dry_run=False):
|
|||||||
"""Rewrite only the <game>/<folder> nodes under our subfolder.
|
"""Rewrite only the <game>/<folder> nodes under our subfolder.
|
||||||
|
|
||||||
Everything else in the file — the user's own scraped ROMs, their play
|
Everything else in the file — the user's own scraped ROMs, their play
|
||||||
counts, favourites — is parsed and written back untouched.
|
counts, favourites, and whatever another store installed under its own
|
||||||
|
subfolder — is parsed and written back untouched.
|
||||||
"""
|
"""
|
||||||
path = gamelist_path(cfg, system)
|
path = gamelist_path(cfg, system)
|
||||||
prefix = cfg["subfolder"] + "/"
|
prefix = subfolder(cfg) + "/"
|
||||||
|
|
||||||
if os.path.isfile(path):
|
if os.path.isfile(path):
|
||||||
try:
|
try:
|
||||||
@@ -423,7 +511,7 @@ def merge_gamelist(cfg, system, records, dry_run=False):
|
|||||||
log(f"warning: {path} is not valid XML ({exc}) — starting a fresh gamelist")
|
log(f"warning: {path} is not valid XML ({exc}) — starting a fresh gamelist")
|
||||||
root = ET.Element("gameList")
|
root = ET.Element("gameList")
|
||||||
else:
|
else:
|
||||||
backup = path + ".ttg-backup"
|
backup = f"{path}.{cfg['store']['id']}-backup"
|
||||||
if not os.path.exists(backup) and not dry_run:
|
if not os.path.exists(backup) and not dry_run:
|
||||||
shutil.copy2(path, backup)
|
shutil.copy2(path, backup)
|
||||||
else:
|
else:
|
||||||
@@ -431,13 +519,14 @@ def merge_gamelist(cfg, system, records, dry_run=False):
|
|||||||
|
|
||||||
for node in list(root):
|
for node in list(root):
|
||||||
target = normalize_path(node.findtext("path"))
|
target = normalize_path(node.findtext("path"))
|
||||||
if target.startswith(prefix) or target == cfg["subfolder"]:
|
if target.startswith(prefix) or target == subfolder(cfg):
|
||||||
root.remove(node)
|
root.remove(node)
|
||||||
|
|
||||||
if records and cfg.get("folder_name"):
|
folder_name = cfg["emulationstation"].get("folder_name")
|
||||||
|
if records and folder_name:
|
||||||
folder = ET.SubElement(root, "folder")
|
folder = ET.SubElement(root, "folder")
|
||||||
ET.SubElement(folder, "path").text = "./" + cfg["subfolder"]
|
ET.SubElement(folder, "path").text = "./" + subfolder(cfg)
|
||||||
ET.SubElement(folder, "name").text = cfg["folder_name"]
|
ET.SubElement(folder, "name").text = folder_name
|
||||||
|
|
||||||
for record in sorted(records, key=lambda r: r["title"].lower()):
|
for record in sorted(records, key=lambda r: r["title"].lower()):
|
||||||
game = ET.SubElement(root, "game")
|
game = ET.SubElement(root, "game")
|
||||||
@@ -528,14 +617,32 @@ def restart_es():
|
|||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def game_key(record):
|
||||||
|
"""Two systems may both carry a game called `foo` — key on both."""
|
||||||
|
return f"{record['system']}:{record['name']}"
|
||||||
|
|
||||||
|
|
||||||
def load_state():
|
def load_state():
|
||||||
state = load_json(STATE_PATH, default=None) or {}
|
state = load_json(STATE_PATH, default=None) or {}
|
||||||
if state.get("version") != STATE_VERSION:
|
version = state.get("version")
|
||||||
|
if version == 1:
|
||||||
|
state = migrate_state_v1(state)
|
||||||
|
elif version != STATE_VERSION:
|
||||||
state = {"version": STATE_VERSION, "installed": {}}
|
state = {"version": STATE_VERSION, "installed": {}}
|
||||||
state.setdefault("installed", {})
|
state.setdefault("installed", {})
|
||||||
return state
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def migrate_state_v1(state):
|
||||||
|
"""v1 keyed `installed` by bare software name; v2 keys by `system:name`."""
|
||||||
|
installed = {}
|
||||||
|
for record in (state.get("installed") or {}).values():
|
||||||
|
if record.get("system") and record.get("name"):
|
||||||
|
installed[game_key(record)] = record
|
||||||
|
log(f"migrated {len(installed)} state entries to the per-system key format")
|
||||||
|
return {"version": STATE_VERSION, "installed": installed}
|
||||||
|
|
||||||
|
|
||||||
def save_state(state):
|
def save_state(state):
|
||||||
state["version"] = STATE_VERSION
|
state["version"] = STATE_VERSION
|
||||||
write_json(STATE_PATH, state)
|
write_json(STATE_PATH, state)
|
||||||
@@ -548,6 +655,15 @@ def records_by_system(installed):
|
|||||||
return grouped
|
return grouped
|
||||||
|
|
||||||
|
|
||||||
|
def match_keys(installed, names):
|
||||||
|
"""Resolve user-typed `name` or `system:name` arguments to state keys."""
|
||||||
|
wanted = {n.lower() for n in names}
|
||||||
|
return [
|
||||||
|
key for key, record in installed.items()
|
||||||
|
if key.lower() in wanted or record["name"].lower() in wanted
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
# commands
|
# commands
|
||||||
# --------------------------------------------------------------------------
|
# --------------------------------------------------------------------------
|
||||||
@@ -563,7 +679,7 @@ def perform_sync(cfg, names=None, dry_run=False):
|
|||||||
if not systems:
|
if not systems:
|
||||||
# An unmounted or wrong ROMs root would otherwise look like "no system
|
# An unmounted or wrong ROMs root would otherwise look like "no system
|
||||||
# is compatible any more" and prune everything we ever installed.
|
# is compatible any more" and prune everything we ever installed.
|
||||||
die(f"no system folders under {cfg['roms_root']} — is it mounted?")
|
die(f"no system folders under {roms_root(cfg)} — is it mounted?")
|
||||||
|
|
||||||
catalog = fetch_catalog(cfg, use_cache=True)
|
catalog = fetch_catalog(cfg, use_cache=True)
|
||||||
games, skipped = select_games(cfg, catalog, systems)
|
games, skipped = select_games(cfg, catalog, systems)
|
||||||
@@ -572,14 +688,15 @@ def perform_sync(cfg, names=None, dry_run=False):
|
|||||||
|
|
||||||
if names:
|
if names:
|
||||||
wanted = {n.lower() for n in names}
|
wanted = {n.lower() for n in names}
|
||||||
games = [g for g in games if g["name"].lower() in wanted]
|
games = [g for g in games if g["name"].lower() in wanted or game_key(g).lower() in wanted]
|
||||||
|
|
||||||
state = load_state()
|
state = load_state()
|
||||||
installed = state["installed"]
|
installed = state["installed"]
|
||||||
touched, changed = set(), False
|
touched, changed = set(), False
|
||||||
|
|
||||||
for game in games:
|
for game in games:
|
||||||
previous = installed.get(game["name"])
|
key = game_key(game)
|
||||||
|
previous = installed.get(key)
|
||||||
if previous and previous.get("asset") != game["asset"]:
|
if previous and previous.get("asset") != game["asset"]:
|
||||||
log(f"{game['name']}: {previous.get('version')} -> {game['version']}, removing old files")
|
log(f"{game['name']}: {previous.get('version')} -> {game['version']}, removing old files")
|
||||||
remove_game(cfg, previous, dry_run=dry_run)
|
remove_game(cfg, previous, dry_run=dry_run)
|
||||||
@@ -590,21 +707,21 @@ def perform_sync(cfg, names=None, dry_run=False):
|
|||||||
except (urllib.error.URLError, OSError) as exc:
|
except (urllib.error.URLError, OSError) as exc:
|
||||||
log(f"warning: {game['name']} failed: {exc}")
|
log(f"warning: {game['name']} failed: {exc}")
|
||||||
continue
|
continue
|
||||||
if did or installed.get(game["name"]) != record:
|
if did or installed.get(key) != record:
|
||||||
changed = True
|
changed = True
|
||||||
installed[game["name"]] = record
|
installed[key] = record
|
||||||
touched.add(record["system"])
|
touched.add(record["system"])
|
||||||
|
|
||||||
# A name-limited sync is not a full picture of the catalog, so never prune.
|
# A name-limited sync is not a full picture of the catalog, so never prune.
|
||||||
if cfg.get("prune") and not names:
|
if cfg["behavior"].get("prune") and not names:
|
||||||
keep = {g["name"] for g in games}
|
keep = {game_key(g) for g in games}
|
||||||
for name in list(installed):
|
for key in list(installed):
|
||||||
if name in keep:
|
if key in keep:
|
||||||
continue
|
continue
|
||||||
log(f"pruning {name} (no longer in the catalog or filtered out)")
|
log(f"pruning {key} (no longer in the catalog or filtered out)")
|
||||||
remove_game(cfg, installed[name], dry_run=dry_run)
|
remove_game(cfg, installed[key], dry_run=dry_run)
|
||||||
touched.add(installed[name]["system"])
|
touched.add(installed[key]["system"])
|
||||||
del installed[name]
|
del installed[key]
|
||||||
changed = True
|
changed = True
|
||||||
|
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
@@ -625,7 +742,7 @@ def cmd_sync(cfg, args):
|
|||||||
for system in sorted(touched | set(grouped)):
|
for system in sorted(touched | set(grouped)):
|
||||||
merge_gamelist(cfg, system, grouped.get(system, []))
|
merge_gamelist(cfg, system, grouped.get(system, []))
|
||||||
|
|
||||||
if cfg.get("restart_es") and not args.no_restart and es_pid():
|
if cfg["emulationstation"].get("restart") and not args.no_restart and es_pid():
|
||||||
restart_es()
|
restart_es()
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
@@ -651,10 +768,10 @@ def cmd_sync_from_es(cfg, args):
|
|||||||
|
|
||||||
subprocess.Popen(
|
subprocess.Popen(
|
||||||
[sys.executable, os.path.abspath(__file__),
|
[sys.executable, os.path.abspath(__file__),
|
||||||
"--config", args.config, "apply-gamelists", "--wait-pid", str(pid)],
|
"--config", CONFIG_PATH, "apply-gamelists", "--wait-pid", str(pid)],
|
||||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True,
|
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True,
|
||||||
)
|
)
|
||||||
if cfg.get("restart_es") and not args.no_restart:
|
if cfg["emulationstation"].get("restart") and not args.no_restart:
|
||||||
restart_es()
|
restart_es()
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
@@ -682,7 +799,7 @@ def cmd_list(cfg, args):
|
|||||||
if not games:
|
if not games:
|
||||||
log("no compatible games in the catalog")
|
log("no compatible games in the catalog")
|
||||||
for game in sorted(games, key=lambda g: (g["system"], g["title"].lower())):
|
for game in sorted(games, key=lambda g: (g["system"], g["title"].lower())):
|
||||||
record = installed.get(game["name"])
|
record = installed.get(game_key(game))
|
||||||
if not record:
|
if not record:
|
||||||
mark = " "
|
mark = " "
|
||||||
elif record.get("asset") == game["asset"]:
|
elif record.get("asset") == game["asset"]:
|
||||||
@@ -701,15 +818,16 @@ def cmd_remove(cfg, args):
|
|||||||
state = load_state()
|
state = load_state()
|
||||||
installed = state["installed"]
|
installed = state["installed"]
|
||||||
touched = set()
|
touched = set()
|
||||||
for name in args.name:
|
keys = match_keys(installed, args.name)
|
||||||
record = installed.get(name)
|
if not keys:
|
||||||
if not record:
|
log(f"not installed: {', '.join(args.name)}")
|
||||||
log(f"{name} is not installed")
|
return 0
|
||||||
continue
|
for key in keys:
|
||||||
|
record = installed[key]
|
||||||
remove_game(cfg, record, dry_run=args.dry_run)
|
remove_game(cfg, record, dry_run=args.dry_run)
|
||||||
touched.add(record["system"])
|
touched.add(record["system"])
|
||||||
if not args.dry_run:
|
if not args.dry_run:
|
||||||
del installed[name]
|
del installed[key]
|
||||||
if args.dry_run:
|
if args.dry_run:
|
||||||
return 0
|
return 0
|
||||||
save_state(state)
|
save_state(state)
|
||||||
@@ -737,10 +855,11 @@ def main(argv=None):
|
|||||||
global VERBOSE
|
global VERBOSE
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
prog="ttg-store",
|
prog=os.path.basename(sys.argv[0]) or "store.py",
|
||||||
description="Install Teletype Games catalog releases into Batocera.",
|
description="Install WarpEngine catalog releases into Batocera.",
|
||||||
)
|
)
|
||||||
parser.add_argument("--config", default=CONFIG_PATH, help=f"config file (default: {CONFIG_PATH})")
|
parser.add_argument("--config", default=CONFIG_PATH,
|
||||||
|
help=f"store config file (default: {CONFIG_PATH})")
|
||||||
parser.add_argument("--base-url", help="override the catalog base URL")
|
parser.add_argument("--base-url", help="override the catalog base URL")
|
||||||
parser.add_argument("--roms-root", help="override the ROMs root (default: /userdata/roms)")
|
parser.add_argument("--roms-root", help="override the ROMs root (default: /userdata/roms)")
|
||||||
parser.add_argument("-n", "--dry-run", action="store_true", help="report actions, change nothing")
|
parser.add_argument("-n", "--dry-run", action="store_true", help="report actions, change nothing")
|
||||||
@@ -765,12 +884,12 @@ def main(argv=None):
|
|||||||
p_list = sub.add_parser("list", help="show compatible catalog entries")
|
p_list = sub.add_parser("list", help="show compatible catalog entries")
|
||||||
p_list.set_defaults(func=cmd_list)
|
p_list.set_defaults(func=cmd_list)
|
||||||
|
|
||||||
p_rm = sub.add_parser("remove", help="uninstall games")
|
p_rm = sub.add_parser("remove", help="uninstall games (name or system:name)")
|
||||||
p_rm.add_argument("name", nargs="+")
|
p_rm.add_argument("name", nargs="+")
|
||||||
p_rm.set_defaults(func=cmd_remove)
|
p_rm.set_defaults(func=cmd_remove)
|
||||||
|
|
||||||
p_cfg = sub.add_parser("config", help="print the effective config")
|
p_cfg = sub.add_parser("config", help="print the effective config")
|
||||||
p_cfg.add_argument("--write", action="store_true", help="write the default config file")
|
p_cfg.add_argument("--write", action="store_true", help="write a template config file")
|
||||||
p_cfg.add_argument("--force", action="store_true")
|
p_cfg.add_argument("--force", action="store_true")
|
||||||
p_cfg.set_defaults(func=cmd_config)
|
p_cfg.set_defaults(func=cmd_config)
|
||||||
|
|
||||||
@@ -780,7 +899,8 @@ def main(argv=None):
|
|||||||
return 2
|
return 2
|
||||||
|
|
||||||
VERBOSE = args.verbose
|
VERBOSE = args.verbose
|
||||||
cfg = load_config(args.config, {"base_url": args.base_url, "roms_root": args.roms_root})
|
set_home(args.config)
|
||||||
|
cfg = load_config(args.config, {"store.base_url": args.base_url, "paths.roms_root": args.roms_root})
|
||||||
os.makedirs(HOME, exist_ok=True)
|
os.makedirs(HOME, exist_ok=True)
|
||||||
return args.func(cfg, args)
|
return args.func(cfg, args)
|
||||||
|
|
||||||
Reference in New Issue
Block a user