stores/warp-engine-retroarch-store
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

A RetroArch store engine for WarpEngine catalogs

Browse Source 
The Batocera store reaches one kind of machine. RetroArch reads the same
cartridges on desktop Linux, Windows and macOS, on Android, on the Steam
Deck and on the console ports — the same seven titles, a far larger
audience — so this writes RetroArch's own library format instead:
one .lpl playlist per platform, with box art in the thumbnail folders that
playlist's name points at.

Three things it does differently, each because RetroArch is not one
machine:

- It reads retroarch.cfg for its directories. On the macOS install this
  was developed against, playlist_directory is ~/Documents/RetroArch/
  playlists while cores and thumbnails are under ~/Library/Application
  Support/RetroArch — an engine assuming <base>/playlists writes into the
  void. The leading ':' of a portable install is expanded too.
- It can write for a machine it is not running on (`export`). Android runs
  RetroArch but cannot run Python, so the only way to serve it is to
  render the tree here and copy it over; the same mechanism handles an SD
  card that will be mounted elsewhere.
- It refuses to write while RetroArch is running, because RetroArch holds
  its playlists in memory and writes them back on exit. The Batocera
  engine defers the write instead; here there is no Ports menu launching
  us, so refusing is both simpler and safer.

Decisions worth recording. One playlist per platform, because
default_core_path is a header field — that is what makes every entry start
without a core prompt; the entries themselves stay on DETECT so the file
survives being moved to a machine whose cores live elsewhere. A missing
core is a note, not an error, or the store would never run for anyone the
first time. The playlist name carries the store's name, so we never write
into RetroArch's own Commodore - 64.lpl and never collide with the
official thumbnail packs — and an entry the user added to our playlist
survives a rewrite. Thumbnails must be PNG and one of our covers is a
GIF, so the engine shells out to sips/magick/convert/ffmpeg, and simply
goes without the image if the machine has none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
mr.zero
2026-08-17 18:00:19 +02:00
co-authored by Claude Opus 5
commit e6a55f56a2
5 changed files with 1500 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+2
View File
@@ -0,0 +1,2 @@
__pycache__/
*.pyc
README.md
+331
View File
@@ -0,0 +1,331 @@
# warp-engine-retroarch-store — a RetroArch store engine for WarpEngine sites
Pulls games from a [WarpEngine](https://git.teletypegames.org/engines/warp_engine)
catalog into RetroArch's own library: one `.lpl` playlist per platform, with box
art in the thumbnail folders that playlist's name points at.
This repository is the engine only. It knows the WarpEngine API but nothing about
any particular site: the host, the store's name and where its games land all come
from a `config.json` that lives in a separate **store repository**. The shared
part — catalog, releases, state, HTTP — is
[`warpstore`](https://git.teletypegames.org/engines/warpstore), which this engine
has in common with the [Batocera engine](https://git.teletypegames.org/tools/warp-engine-batocera-store).
```
warpstore the shared core
▲
warp-engine-retroarch-store this engine — retroarch_store.py, install.sh
▲
│ config.json
ttg-retroarch-store store repositories
▼
teletypegames.org
```
The reference store is
[`ttg-retroarch-store`](https://git.teletypegames.org/tools/ttg-retroarch-store).
```
<store>-retroarch-store sync
│
├─ GET /api/software the whole catalog
├─ keep platforms a libretro core boots c64 → VICE, tic80 → TIC-80
├─ pick the newest non-dev release that carries the cartridge
├─ GET /api/download?path=<asset> .prg / .tic into the content folder
├─ GET <software.imageUrl> box art, converted to PNG
└─ write <store> - <label>.lpl playlist + thumbnails
```
Python 3 standard library only.
## Why this is not the Batocera store
RetroArch is not one machine. The same playlist format is read on desktop Linux,
Windows and macOS, on Android, on the Steam Deck and on the console ports — so
the engine assumes nothing about where things are:
- **It reads `retroarch.cfg`.** The playlist folder is not necessarily inside the
RetroArch directory: on the macOS install this was developed against,
`playlist_directory` is `~/Documents/RetroArch/playlists` while the cores and
thumbnails are under `~/Library/Application Support/RetroArch`. An engine that
assumed `<base>/playlists` would write into the void. `~` is expanded, and so is
the leading `:` a portable install uses to mean its own directory.
- **It can write for a machine it is not running on** (`export`). Android has no
Python, so the only way to serve it is to render the tree elsewhere and copy it
over. See *Another machine* below.
- **It never edits `retroarch.cfg`.** Nothing the user configured is touched.
What it does *not* do, compared with Batocera: a `.lpl` entry has no description
or developer field, so the catalog's `desc` and `author` are lost — RetroArch
takes that from its own databases, which are a binary format outside this
engine's scope. And RetroArch cannot launch scripts, so there is no in-app
trigger: the sync runs from a shell or from a scheduler.
## Installing a store
The installer takes the store as a parameter, so it works with any config:
```sh
STORE_CONFIG=https://git.example.org/tools/my-store/raw/branch/master/config.json \
bash <(curl -fsSL https://git.teletypegames.org/tools/warp-engine-retroarch-store/raw/branch/master/install.sh)
```
From a checkout, with a local config:
```sh
STORE_CONFIG=./my-config.json ./install.sh
```
It reads `store.id` from the config, installs the engine, the shared core and the
config into `~/.local/share/warp-engine-store/<id>/`, writes a
`<id>-retroarch-store` launcher into `~/.local/bin`, prints the paths it resolved
and runs the first sync.
| Environment variable | Meaning |
|---|---|
| `STORE_CONFIG` | **required** — path or URL of the store's `config.json` |
| `STORE_ROOT` | where stores live (default `${XDG_DATA_HOME:-~/.local/share}/warp-engine-store`) |
| `BIN_DIR` | where the launcher goes (default `~/.local/bin`) |
| `RETROARCH_DIR` | RetroArch's directory, if it is somewhere unusual |
| `STORE_SKIP_SYNC` | `1` to install without syncing |
| `ENGINE_RAW_BASE` | where to fetch `retroarch_store.py` from |
| `WARPSTORE_RAW_BASE` | where to fetch `warpstore.py` from |
| `WARPSTORE_SRC` | a local `warpstore.py` to install instead of downloading it |
## Use
```sh
S=~/.local/bin/example-retroarch-store
$S paths # resolved directories and whether the cores are there
$S list # compatible catalog entries; * installed, ^ update available
$S sync # download everything new, refresh the playlists
$S sync blessingofra # just one title (never prunes)
$S -n sync # dry run
$S remove c64:c64demo # uninstall (bare name works too)
$S config # effective configuration
```
Global flags go **before** the subcommand: `$S --retroarch-dir /mnt/ra sync`.
**Restart RetroArch after a sync.** It builds its menu at startup, so a new
playlist does not appear in a running instance.
**A sync refuses to run while RetroArch is up.** RetroArch holds its playlists in
memory and writes them back when it exits — favourites, last played, sort order —
so writing underneath it can be undone. Close it, or pass `--force` if you know
what you are doing (`behavior.refuse_while_running: false` to stop asking).
## Where things land
```
~/.local/share/warp-engine-store/example/
├── retroarch_store.py, warpstore.py, config.json
└── state.json, catalog.json
<content_root>/example/c64/ blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
<playlists_dir>/Example Store - Commodore 64.lpl
<thumbnails_dir>/Example Store - Commodore 64/Named_Boxarts/Blessing of Ra.png
/Named_Titles/… /Named_Snaps/…
```
`content_root` defaults to `<retroarch_dir>/content`; the other two come from
`retroarch.cfg`. Run `paths` to see what was resolved and why.
Everything lands under the store's own `paths.subfolder`, which is what a prune
is allowed to touch — never content you put there yourself, never another
store's. The playlist file carries the store's name, so it is ours; but an entry
*you* added to it survives a rewrite, because only items pointing inside the
store's content folder are replaced. The first time a playlist is touched it is
copied to `<name>.lpl.<store id>-backup`.
## The playlist
```json
{
"version": "1.5",
"default_core_path": "/…/cores/vice_x64_libretro.dylib",
"default_core_name": "VICE x64",
"label_display_mode": 0,
"right_thumbnail_mode": 0,
"left_thumbnail_mode": 0,
"thumbnail_match_mode": 0,
"sort_mode": 0,
"items": [
{
"path": "/…/content/example/c64/blessingofra-2.0.0.prg",
"entry_slot": -1,
"label": "Blessing of Ra",
"core_path": "DETECT",
"core_name": "DETECT",
"crc32": "4253CFC6|crc",
"db_name": "Example Store - Commodore 64.lpl"
}
]
}
```
- **One playlist per platform**, because `default_core_path` is a *header* field:
that is the only way every entry starts with one click and no core prompt. The
entries themselves stay on `DETECT`, so the file still works on a machine whose
cores live somewhere else — the worst case is one prompt, not a dead entry.
- **The name is the namespace.** `{store} - {label}` means we never write into
RetroArch's own `Commodore - 64.lpl` and never collide with the official
thumbnail packs. It is also the `db_name`, which is how RetroArch finds the
thumbnails: `<thumbnails_dir>/<db_name without .lpl>/Named_*/`.
- **The display fields are all `0`** — whatever the user has set globally. A store
has no business overriding how someone likes their library displayed.
- **`crc32` is real**, computed from the downloaded file, which lets RetroArch tie
saves and thumbnails to the content. Turn it off with `playlist.write_crc32`.
### A missing core is a note, not an error
If the core named for a platform is not in `libretro_directory`, the playlist is
still written: `default_core_path` stays empty and the entries stay on `DETECT`,
so RetroArch asks once which core to use. `list` and `paths` say so, with the fix:
```
c64: core not found: vice_x64_libretro.dylib — entries stay on DETECT
(Online Updater ▸ Core Downloader ▸ VICE x64)
```
A store that refused to run until the user had installed cores would never run
for anyone the first time.
### Box art must be PNG
RetroArch looks for `<label>.png` and nothing else. The catalog serves whatever
was uploaded — one of ours is a GIF — and the standard library cannot re-encode an
image, so the engine shells out to whatever the machine has: `sips` (built into
macOS), `magick`, `convert` or `ffmpeg`, first one found. With none of them the
entry simply has no box art, which beats a failed sync.
The cover is downloaded once and placed in all three `Named_*` folders (hardlinked
where the filesystem allows), so whichever thumbnail type the user has selected,
there is an image for it. `playlist.thumbnail_kinds` narrows that.
## Another machine: `export`
`sync` installs into this machine's RetroArch. `export` renders the whole store
into a canonical tree meant for somewhere else — the only way to serve Android,
which runs RetroArch but cannot run Python.
```sh
# Android over adb
$S export /tmp/ra-example \
--target-prefix /storage/emulated/0/RetroArch \
--target-libretro-dir /data/data/com.retroarch.aarch64/cores \
--core-suffix _android.so
adb push /tmp/ra-example/. /storage/emulated/0/RetroArch/
# An SD card mounted here, but at /mnt/sdcard on the handheld
$S export /Volumes/SDCARD/RetroArch --target-prefix /mnt/sdcard/RetroArch
```
The tree is always `<dir>/playlists`, `<dir>/thumbnails`, `<dir>/content`.
`--target-prefix` is what the playlists say the content is called on the machine
that will run it; without it, the export directory's own path is used (right when
the export lands where it will stay). Target paths are written the way the target
spells them — POSIX unless the prefix looks like Windows.
`export` is **stateless on purpose**: it is a complete picture of the catalog, not
an incremental update, and it must not disturb the state of a local install living
in the same store home. `--target-libretro-dir` is taken on trust, since the other
machine cannot be inspected; leave it out and the entries stay on `DETECT`.
**Not verified on a device yet.** The Android paths above are the documented
defaults, not measurements — read the device's own `retroarch.cfg` if a push does
not show up, and whether scoped storage lets RetroArch read pushed content is the
open question.
## Writing a store repository
Three files:
```
my-retroarch-store/
├── config.json the store: URL, name, subfolder, platform → core mapping
├── install.sh a wrapper that hands that config to the engine installer
└── README.md
```
`install.sh` in full:
```sh
#!/bin/bash
set -euo pipefail
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/tools/warp-engine-retroarch-store/raw/branch/master}"
export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-retroarch-store/raw/branch/master/config.json}"
curl -fsSL "$ENGINE_RAW_BASE/install.sh" | bash
```
Pick a `store.id` and a `paths.subfolder` nobody else uses — they are what keeps
two stores on one machine from deleting each other's games.
## Configuration
`config.json` in the store home, template in `config.example.json`
(`config --write` regenerates it):
| Key | Default | Meaning |
| --- | --- | --- |
| `store.id` | `warp` | slug: names the store home, log prefix and backups |
| `store.name` | `WarpEngine Store` | goes into the playlist name — what the user sees |
| `store.base_url` | — | WarpEngine host |
| `store.api.catalog` / `.download` | `/api/software`, `/api/download` | endpoints, if the engine is mounted elsewhere |
| `paths.retroarch_dir` | `null` | RetroArch's directory; null = env, then the per-OS candidates |
| `paths.playlists_dir` | `null` | null = from `retroarch.cfg`, then `<retroarch_dir>/playlists` |
| `paths.thumbnails_dir` | `null` | as above, `thumbnails` |
| `paths.libretro_dir` | `null` | as above, `cores` |
| `paths.content_root` | `null` | null = `<retroarch_dir>/content` |
| `paths.subfolder` | `warp` | this store's folder inside it — the prune boundary |
| `paths.target_prefix` | `null` | default for `export --target-prefix` |
| `playlist.name_template` | `{store} - {label}` | `store`, `label`, `platform` are substituted |
| `playlist.write_crc32` | `true` | compute the content CRC32 for each entry |
| `playlist.thumbnail_kinds` | all three `Named_*` | which thumbnail folders to fill |
| `catalog.statuses` | `["released", "archived"]` | catalog `status` values to install |
| `catalog.owner_id` | `null` | restrict to one publisher |
| `catalog.only` / `catalog.exclude` | `[]` | software-name allow / deny lists |
| `platforms` | c64, tic80 | platform → label, asset kind, extension, core, enabled |
| `behavior.prune` | `true` | remove games that left the catalog or the filters |
| `behavior.timeout` | `30` | HTTP timeout, seconds |
| `behavior.insecure` | `false` | skip TLS verification (self-hosted test instances) |
| `behavior.refuse_while_running` | `true` | do not write while RetroArch is running |
| `behavior.convert_images` | `true` | convert non-PNG box art with an external tool |
### Platform mapping
A catalog platform is installable when a libretro core boots its asset directly:
| Catalog platform | Asset kind | Core | Extension |
| --- | --- | --- | --- |
| `c64` | `cartridge` | `vice_x64_libretro` (VICE x64) | `.prg` |
| `tic80` | `cartridge` | `tic80_libretro` (TIC-80) | `.tic` |
Both cores are built for every relevant target — macOS x86_64/arm64, Linux
x86_64/aarch64/armhf, Windows x86_64, Android arm64-v8a/armeabi-v7a — so the
catalogue's cartridges run everywhere RetroArch does. Adding a platform is a
config edit:
```json
"platforms": {
"nes": { "label": "NES", "kind": "cartridge", "ext": ".nes",
"core": { "file": "fceumm_libretro", "name": "FCEUmm" }, "enabled": true }
}
```
`kind` and `ext` may be per-host maps rather than strings — `{"windows": …, "*": …}`,
see `warpstore.resolve_for_host` — but for a cartridge they never need to be: it
is data for an emulator, identical on every machine.
The other WarpEngine platforms (`ebitengine`, `love`, `godot`, `bevy`, `phaser`)
ship web builds and per-OS native archives. No libretro core runs those, so this
engine cannot serve them; on Batocera they install as Ports instead.
## State
`state.json` records what this store installed, keyed `<platform>:<name>` so the
same software name on two platforms stays two entries. Re-running a sync is
idempotent: files already present are left alone, and a release bump
(`1.0` → `2.0.0`) deletes the old asset, its thumbnails and its playlist entry
before fetching the new one.
config.example.json
+67
View File
@@ -0,0 +1,67 @@
{
"store": {
"id": "warp",
"name": "WarpEngine Store",
"base_url": "https://example.org",
"api": {
"catalog": "/api/software",
"download": "/api/download"
}
},
"paths": {
"retroarch_dir": null,
"playlists_dir": null,
"thumbnails_dir": null,
"libretro_dir": null,
"content_root": null,
"subfolder": "warp",
"target_prefix": null
},
"playlist": {
"name_template": "{store} - {label}",
"write_crc32": true,
"thumbnail_kinds": [
"Named_Boxarts",
"Named_Titles",
"Named_Snaps"
]
},
"catalog": {
"statuses": [
"released",
"archived"
],
"owner_id": null,
"only": [],
"exclude": []
},
"platforms": {
"c64": {
"label": "Commodore 64",
"kind": "cartridge",
"ext": ".prg",
"core": {
"file": "vice_x64_libretro",
"name": "VICE x64"
},
"enabled": true
},
"tic80": {
"label": "TIC-80",
"kind": "cartridge",
"ext": ".tic",
"core": {
"file": "tic80_libretro",
"name": "TIC-80"
},
"enabled": true
}
},
"behavior": {
"prune": true,
"timeout": 30,
"insecure": false,
"refuse_while_running": true,
"convert_images": true
}
}
install.sh
Executable
+123
View File
@@ -0,0 +1,123 @@
#!/bin/bash
# Install a WarpEngine store into a RetroArch installation.
#
# This installer belongs to the engine, so it takes the store as a parameter:
# point STORE_CONFIG at a config.json — a local path or a URL — and it sets up
# that store. A store repository's own install.sh is a short wrapper that
# supplies its config; see README.
#
# STORE_CONFIG=./config.json ./install.sh
# STORE_CONFIG=https://git.example.org/tools/my-store/raw/branch/master/config.json \
# bash <(curl -fsSL https://git.teletypegames.org/tools/warp-engine-retroarch-store/raw/branch/master/install.sh)
#
set -euo pipefail
STORE_ROOT="${STORE_ROOT:-${XDG_DATA_HOME:-$HOME/.local/share}/warp-engine-store}"
BIN_DIR="${BIN_DIR:-$HOME/.local/bin}"
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/tools/warp-engine-retroarch-store/raw/branch/master}"
# The shared core lives in its own repository, because both store engines use it.
WARPSTORE_RAW_BASE="${WARPSTORE_RAW_BASE:-https://git.teletypegames.org/engines/warpstore/raw/branch/master}"
STORE_CONFIG="${STORE_CONFIG:-}"
# Only set when run as a file; piped from curl "$0" is `bash`, and then the
# engine has to come from the forge rather than from whatever the cwd holds.
SRC_DIR=""
if [ -f "$0" ]; then
SRC_DIR="$(cd "$(dirname "$0")" && pwd)"
fi
say() { echo "[install] $*"; }
die() { echo "[install] error: $*" >&2; exit 1; }
command -v python3 >/dev/null 2>&1 || die "python3 is required"
command -v curl >/dev/null 2>&1 || die "curl is required"
[ -n "$STORE_CONFIG" ] || die "STORE_CONFIG is required (path or URL of a store config.json)"
fetch() { # fetch <source> <dest>; source may be a local path or a URL
case "$1" in
http://*|https://*) curl -fsSL "$1" -o "$2" ;;
*) [ -f "$1" ] || die "no such file: $1"; cp "$1" "$2" ;;
esac
}
# Stage the config first: the store id in it decides where everything lands.
TMP_CFG="$(mktemp)"
trap 'rm -f "$TMP_CFG"' EXIT
say "reading store config from $STORE_CONFIG"
fetch "$STORE_CONFIG" "$TMP_CFG"
STORE_META="$(python3 -c '
import json, sys
cfg = json.load(open(sys.argv[1])).get("store") or {}
sid = cfg.get("id") or ""
if not sid or "/" in sid:
sys.exit("store.id is required and must not contain a slash")
print(sid, cfg.get("name") or sid)
' "$TMP_CFG")" || die "cannot read $STORE_CONFIG"
STORE_ID="${STORE_META%% *}"
STORE_NAME="${STORE_META#* }"
STORE_HOME="$STORE_ROOT/$STORE_ID"
LAUNCHER="$BIN_DIR/$STORE_ID-retroarch-store"
mkdir -p "$STORE_HOME" "$BIN_DIR"
install_file() { # install_file <name> <raw base> [local override]
if [ -n "${3:-}" ] && [ -f "${3:-}" ]; then
say "installing $1 from ${3}"
install -m 0755 "$3" "$STORE_HOME/$1"
elif [ -n "$SRC_DIR" ] && [ -f "$SRC_DIR/$1" ]; then
say "installing $1 from $SRC_DIR"
install -m 0755 "$SRC_DIR/$1" "$STORE_HOME/$1"
else
say "downloading $1 from $2"
curl -fsSL "$2/$1" -o "$STORE_HOME/$1"
chmod 0755 "$STORE_HOME/$1"
fi
}
install_file retroarch_store.py "$ENGINE_RAW_BASE"
install_file warpstore.py "$WARPSTORE_RAW_BASE" "${WARPSTORE_SRC:-}"
# The config is the store's identity, so it is always refreshed; state.json,
# catalog.json and the log stay put, which makes re-running the upgrade path.
if [ -f "$STORE_HOME/config.json" ]; then
say "updating $STORE_HOME/config.json (previous copy kept as config.json.bak)"
cp "$STORE_HOME/config.json" "$STORE_HOME/config.json.bak"
fi
install -m 0644 "$TMP_CFG" "$STORE_HOME/config.json"
# A short launcher so the CLI is one path, not a python invocation.
cat > "$LAUNCHER" <<EOF
#!/bin/bash
# $STORE_NAME — CLI wrapper around the shared RetroArch store engine.
export RETROARCH_STORE_HOME="$STORE_HOME"
exec "$STORE_HOME/retroarch_store.py" --config "$STORE_HOME/config.json" "\$@"
EOF
chmod 0755 "$LAUNCHER"
say "resolved paths:"
"$LAUNCHER" paths 2>&1 | sed 's/^/ /' || true
if [ "${STORE_SKIP_SYNC:-0}" != "1" ]; then
say "running the first sync"
"$LAUNCHER" --verbose sync || say "the first sync did not finish — see the message above"
fi
case ":$PATH:" in
*":$BIN_DIR:"*) ;;
*) say "note: $BIN_DIR is not on your PATH" ;;
esac
cat <<EOF
[install] done.
CLI: $LAUNCHER list|sync|remove|export|paths|config
Config: $STORE_HOME/config.json
Restart RetroArch to see the playlists — it builds its menu at startup.
Keep it up to date by running the sync on a schedule, e.g. in crontab:
0 * * * * $LAUNCHER sync >>$STORE_HOME/store.log 2>&1
EOF
retroarch_store.py
Executable
+977
View File
@@ -0,0 +1,977 @@
#!/usr/bin/env python3
"""retroarch_store.py — the RetroArch adapter of the WarpEngine store engine.
Reads a WarpEngine catalog (`GET /api/software`), downloads the cartridge assets
a libretro core can boot, and writes RetroArch's own library format: one `.lpl`
playlist per platform, with box art in the thumbnail folders that playlist's name
points at.
RetroArch is not one machine. The same playlist format is read on desktop Linux,
Windows, macOS, Android, the Steam Deck and the console ports — so the engine
never assumes where anything is: it reads `retroarch.cfg` for the four
directories it needs, and it can write a tree for a *different* machine than the
one it runs on (`export`), which is how Android is served at all.
Everything that is not specific to RetroArch — the catalog, release selection,
host matching, state, HTTP — lives in `warpstore.py`, the module this engine
shares with the other store engines. The installer puts the two side by side.
Standard library only.
"""
import argparse
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import urllib.error
import zlib
# The shared core sits next to this script (the installer puts it there); make
# sure that is where we look, however the script was invoked.
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
try:
import warpstore as ws
except ImportError:
sys.exit("error: warpstore.py is missing next to retroarch_store.py — reinstall the store "
"engine (https://git.teletypegames.org/engines/warpstore)")
from warpstore import debug, die, log
VERSION = "1.0.0"
# The playlist format version RetroArch writes today. Bump only after checking
# what a current RetroArch produces — the field is how it decides how to read
# the rest of the file.
PLAYLIST_VERSION = "1.5"
# Each store keeps its config, state, cache and log in one directory. The
# installer exports RETROARCH_STORE_HOME; run from a checkout, the script's own
# directory is the home.
DEFAULT_HOME = os.environ.get("RETROARCH_STORE_HOME") or os.path.dirname(os.path.abspath(__file__))
DEFAULT_CONFIG_PATH = os.path.join(DEFAULT_HOME, "config.json")
DEFAULT_CONFIG = {
"store": {
# Short slug: names the store home, the log prefix and the backups.
"id": "warp",
# Human-readable: goes into the playlist name, so it is what the user
# sees in RetroArch's main menu.
"name": "WarpEngine Store",
"base_url": "https://example.org",
"api": {
"catalog": "/api/software",
"download": "/api/download",
},
},
"paths": {
# All four null by default, meaning "work it out": the environment, then
# retroarch.cfg, then the per-OS candidates. Set one to pin it.
"retroarch_dir": None,
"playlists_dir": None,
"thumbnails_dir": None,
"libretro_dir": None,
# Where the cartridges land. Null = <retroarch_dir>/content.
"content_root": None,
# Our own folder inside it: what a prune is allowed to touch, and what
# keeps two stores on one machine out of each other's files.
"subfolder": "warp",
# Default for `export --target-prefix`: how the exported tree will be
# spelled on the machine that will actually run it.
"target_prefix": None,
},
"playlist": {
# `store` and `label` come from the config, `platform` from the catalog.
"name_template": "{store} - {label}",
# A real CRC32 helps RetroArch tie saves and thumbnails to content.
"write_crc32": True,
# One download, three folders: whichever thumbnail type the user has
# selected globally, there is an image for it. We do not touch their
# setting.
"thumbnail_kinds": ["Named_Boxarts", "Named_Titles", "Named_Snaps"],
},
"catalog": {
"statuses": ["released", "archived"],
"owner_id": None,
"only": [],
"exclude": [],
},
# WarpEngine platform -> playlist label, asset to pull, core that boots it.
"platforms": {
"c64": {
"label": "Commodore 64",
"kind": "cartridge",
"ext": ".prg",
"core": {"file": "vice_x64_libretro", "name": "VICE x64"},
"enabled": True,
},
"tic80": {
"label": "TIC-80",
"kind": "cartridge",
"ext": ".tic",
"core": {"file": "tic80_libretro", "name": "TIC-80"},
"enabled": True,
},
},
"behavior": {
"prune": True,
"timeout": 30,
"insecure": False,
# RetroArch writes its playlists back when it exits, so a sync while it
# runs can be undone. Refuse rather than lose the work.
"refuse_while_running": True,
# Box art must be PNG; convert with whatever the machine has if not.
"convert_images": True,
},
}
# What a libretro core file is called, per host. Android is the odd one out, and
# it is also the host we can only ever write for from somewhere else.
CORE_SUFFIX = {"darwin": ".dylib", "linux": ".so", "windows": ".dll", "android": "_android.so"}
# Where RetroArch keeps its configuration, in the order worth trying. Only used
# when nothing else says — retroarch.cfg itself is always preferred over these.
RETROARCH_DIRS = {
"darwin": ["~/Library/Application Support/RetroArch"],
"linux": [
"~/.config/retroarch",
"~/.var/app/org.libretro.RetroArch/config/retroarch", # flatpak (Steam Deck)
"~/snap/retroarch/current/.config/retroarch",
],
"windows": ["$APPDATA/RetroArch", "C:/RetroArch"],
"android": ["/storage/emulated/0/RetroArch", "/sdcard/RetroArch"],
}
# retroarch.cfg is not always in the RetroArch directory itself: a macOS install
# keeps it under config/.
CFG_NAMES = ["retroarch.cfg", "config/retroarch.cfg"]
# RetroArch replaces these characters when it turns a playlist label into a
# thumbnail file name, so we have to do the same or the image is never found.
THUMB_UNSAFE = '&*/:`<>?\\|'
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
# Tried in order; the first one the machine actually has wins.
CONVERTERS = [
("sips", ["sips", "-s", "format", "png", "{src}", "--out", "{dst}"]),
("magick", ["magick", "{src}", "{dst}"]),
("convert", ["convert", "{src}", "{dst}"]),
("ffmpeg", ["ffmpeg", "-y", "-loglevel", "error", "-i", "{src}", "{dst}"]),
]
def subfolder(cfg):
return cfg["paths"]["subfolder"]
# --------------------------------------------------------------------------
# retroarch.cfg
# --------------------------------------------------------------------------
def parse_retroarch_cfg(path):
"""RetroArch's config: `key = "value"` lines, `#` comments."""
values = {}
try:
with open(path, "r", encoding="utf-8", errors="replace") as fh:
for line in fh:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
values[key.strip()] = value.strip().strip('"')
except OSError as exc:
debug(f"cannot read {path}: {exc}")
return values
def find_retroarch_cfg(base):
for name in CFG_NAMES:
path = os.path.join(base, name)
if os.path.isfile(path):
return path
return None
def resolve_ra_path(value, base):
"""Expand a directory as RetroArch spells it.
`~` is the home directory; a leading `:` means "relative to RetroArch's own
base directory", which is how a portable install refers to itself.
"""
value = (value or "").strip()
if not value:
return ""
if value.startswith(":"):
rest = value[1:].lstrip("\\/")
return os.path.abspath(os.path.join(base, rest)) if rest else os.path.abspath(base)
expanded = os.path.expanduser(os.path.expandvars(value))
return os.path.abspath(expanded) if os.path.isabs(expanded) else os.path.abspath(os.path.join(base, expanded))
def find_retroarch_dir(cfg, override=None):
"""RetroArch's directory: the flag, the config, the environment, then a guess."""
for value, source in ((override, "--retroarch-dir"),
(cfg["paths"].get("retroarch_dir"), "config"),
(os.environ.get("RETROARCH_DIR"), "RETROARCH_DIR")):
if value:
path = os.path.abspath(os.path.expanduser(os.path.expandvars(value)))
if not os.path.isdir(path):
die(f"{source} points at {path}, which is not a directory")
return path, source
candidates = RETROARCH_DIRS.get(ws.host_os(), [])
for candidate in candidates:
path = os.path.abspath(os.path.expanduser(os.path.expandvars(candidate)))
if os.path.isdir(path):
return path, "found"
die("cannot find RetroArch's directory. Tried:\n " + "\n ".join(candidates)
+ "\nPoint at it with --retroarch-dir, RETROARCH_DIR or paths.retroarch_dir.")
# --------------------------------------------------------------------------
# where we write, and how the target spells it
# --------------------------------------------------------------------------
def target_join(base, *parts):
"""Join a path the way the *target* machine writes them, not this one.
An export written on a Mac for an Android phone must contain POSIX paths;
only a Windows-looking base gets backslashes.
"""
sep = "\\" if ("\\" in base or re.match(r"^[A-Za-z]:", base)) else "/"
joined = base.rstrip("/\\")
for part in parts:
joined += sep + str(part).strip("/\\")
return joined
def target_within(path, root):
"""Containment test in *target* space, where os.sep may not apply."""
p = (path or "").replace("\\", "/")
r = (root or "").replace("\\", "/").rstrip("/")
return bool(r) and (p == r or p.startswith(r + "/"))
class Tree:
"""Where files go locally, and how the files we write name them.
For a local install the two coincide. For an export they do not: the content
lands in a staging directory here, while the playlists have to name it as it
will be on the machine that runs it. Keeping the two apart in one object is
what makes Android — where no Python can run — reachable at all.
"""
def __init__(self, playlists_dir, thumbnails_dir, content_root, target_content_root,
libretro_dir, target_libretro_dir, core_suffix, sources):
self.playlists_dir = playlists_dir
self.thumbnails_dir = thumbnails_dir
self.content_root = content_root
self.target_content_root = target_content_root
self.libretro_dir = libretro_dir
self.target_libretro_dir = target_libretro_dir
self.core_suffix = core_suffix
self.sources = sources
def content_dir(self, cfg, platform):
"""Local directory holding one platform's cartridges."""
return os.path.join(self.content_root, subfolder(cfg), platform)
def target_content_dir(self, cfg, platform):
return target_join(self.target_content_root, subfolder(cfg), platform)
def owned_root(self, cfg):
"""The only subtree this store may ever delete from."""
return os.path.join(self.content_root, subfolder(cfg))
def local_tree(cfg, args):
"""The tree of the RetroArch installed on this machine."""
base, base_source = find_retroarch_dir(cfg, getattr(args, "retroarch_dir", None))
cfg_path = find_retroarch_cfg(base)
racfg = parse_retroarch_cfg(cfg_path) if cfg_path else {}
sources = {"retroarch_dir": f"{base} ({base_source})",
"retroarch.cfg": cfg_path or "not found — using defaults"}
def pick(config_key, ra_key, default_name):
override = cfg["paths"].get(config_key)
if override:
path = os.path.abspath(os.path.expanduser(os.path.expandvars(override)))
sources[config_key] = f"{path} (config)"
return path
if racfg.get(ra_key):
path = resolve_ra_path(racfg[ra_key], base)
sources[config_key] = f"{path} (retroarch.cfg {ra_key})"
return path
path = os.path.join(base, default_name)
sources[config_key] = f"{path} (default)"
return path
playlists = pick("playlists_dir", "playlist_directory", "playlists")
thumbnails = pick("thumbnails_dir", "thumbnails_directory", "thumbnails")
libretro = pick("libretro_dir", "libretro_directory", "cores")
content = pick("content_root", "", "content")
suffix = getattr(args, "core_suffix", None) or CORE_SUFFIX.get(ws.host_os(), ".so")
sources["core_suffix"] = suffix
return Tree(playlists, thumbnails, content, content, libretro, None, suffix, sources)
def export_tree(cfg, args):
"""A canonical RetroArch-shaped tree for a machine that is not this one."""
root = os.path.abspath(os.path.expanduser(args.directory))
target = args.target_prefix or cfg["paths"].get("target_prefix") or root
suffix = args.core_suffix or CORE_SUFFIX.get(ws.host_os(), ".so")
sources = {
"export root": root,
"target prefix": target,
"core_suffix": suffix,
"target libretro dir": args.target_libretro_dir or "unknown — entries stay on DETECT",
}
return Tree(
playlists_dir=os.path.join(root, "playlists"),
thumbnails_dir=os.path.join(root, "thumbnails"),
content_root=os.path.join(root, "content"),
target_content_root=target_join(target, "content"),
libretro_dir=None,
target_libretro_dir=args.target_libretro_dir,
core_suffix=suffix,
sources=sources,
)
# --------------------------------------------------------------------------
# names
# --------------------------------------------------------------------------
def playlist_name(cfg, platform, spec):
"""What the user sees in RetroArch's menu, and the name everything hangs off.
It is also the `db_name` and the thumbnail folder, so it carries the store's
name: we never write into RetroArch's own `Commodore - 64.lpl`, and we never
collide with the official thumbnail packs.
"""
return cfg["playlist"]["name_template"].format(
store=cfg["store"]["name"], label=spec.get("label") or platform, platform=platform,
).strip()
def playlist_file_name(name):
return re.sub(r'[\\/:*?"<>|]', "_", name) + ".lpl"
def thumb_file_name(label):
return "".join("_" if c in THUMB_UNSAFE else c for c in label) + ".png"
# --------------------------------------------------------------------------
# catalog -> RetroArch
# --------------------------------------------------------------------------
def accept_for_retroarch(cfg):
"""The adapter's veto: a title is installable if a core is named for it.
The scope is the catalog platform, one per playlist — not the playlist name,
so renaming the store does not re-key everything that is installed.
"""
def accept(spec, sw):
platform = sw.get("platform")
if not (spec.get("core") or {}).get("file"):
raise ws.Skip(f"no core configured for {platform}")
return platform, {"playlist": playlist_name(cfg, platform, spec)}
return accept
def select_games(cfg, catalog, host_info=None):
return ws.select_games(cfg, catalog, accept_for_retroarch(cfg), host_info)
def resolve_core(cfg, tree, platform):
"""(core_path, core_name, note) for a platform's playlist header.
A missing core is a note, never an error: the playlist still works, the user
is asked which core to use once, and `Online Updater ▸ Core Downloader`
fixes it for good. A store that refuses to run without cores installed would
never run for anyone the first time.
"""
spec = cfg["platforms"][platform]
core = spec.get("core") or {}
name = core.get("name") or ""
file_name = (core.get("file") or "") + tree.core_suffix
if tree.target_libretro_dir:
# An export: we cannot check the other machine, so take the user's word.
return target_join(tree.target_libretro_dir, file_name), name, ""
if tree.libretro_dir:
path = os.path.join(tree.libretro_dir, file_name)
if os.path.isfile(path):
return path, name, ""
return "", name, (f"core not found: {file_name} — entries stay on DETECT "
f"(Online Updater ▸ Core Downloader ▸ {name})")
return "", name, f"core location unknown ({file_name}) — entries stay on DETECT"
# --------------------------------------------------------------------------
# box art
# --------------------------------------------------------------------------
def to_png(blob, ext, allow_convert=True):
"""RetroArch thumbnails must be PNG. Convert if the machine can, else None.
The catalog serves whatever was uploaded — today one cover is a GIF — and
the standard library cannot re-encode an image, so this leans on whatever
converter the host happens to have. Failing that, the entry simply has no
box art, which is far better than a broken sync.
"""
if blob[:8] == PNG_MAGIC:
return blob
if not allow_convert:
return None
for name, template in CONVERTERS:
if not shutil.which(name):
continue
tmp = tempfile.mkdtemp(prefix=f".{ws.TAG}-img-")
try:
src = os.path.join(tmp, "in" + (ext or ".img"))
dst = os.path.join(tmp, "out.png")
with open(src, "wb") as fh:
fh.write(blob)
cmd = [part.format(src=src, dst=dst) for part in template]
result = subprocess.run(cmd, capture_output=True, timeout=60)
if result.returncode == 0 and os.path.isfile(dst) and os.path.getsize(dst) > 0:
with open(dst, "rb") as fh:
converted = fh.read()
debug(f"converted box art to PNG with {name}")
return converted
debug(f"{name} could not convert the box art: {result.returncode}")
except (OSError, subprocess.SubprocessError) as exc:
debug(f"{name} failed: {exc}")
finally:
shutil.rmtree(tmp, ignore_errors=True)
return None
def thumb_paths(cfg, tree, record):
"""The local thumbnail files for one entry, one per configured kind."""
playlist = record["playlist"]
name = thumb_file_name(record["title"])
return [os.path.join(tree.thumbnails_dir, playlist, kind, name)
for kind in cfg["playlist"]["thumbnail_kinds"]]
def install_thumbnails(cfg, tree, record, dry_run=False):
"""Fetch the cover once and put it in every thumbnail folder that wants it."""
wanted = thumb_paths(cfg, tree, record)
if all(os.path.isfile(p) and os.path.getsize(p) > 0 for p in wanted):
debug(f"{record['name']}: box art already in place")
return wanted, False
if dry_run:
log(f"would fetch box art for {record['name']} ({len(wanted)} folders)")
return wanted, True
blob, ext = ws.download_image(cfg, record)
if blob is None:
return [], False
png = to_png(blob, ext, allow_convert=cfg["behavior"].get("convert_images", True))
if png is None:
log(f"warning: {record['name']}: box art is {ext or 'not PNG'} and no converter "
f"(sips/magick/convert/ffmpeg) is available — skipping the thumbnail")
return [], False
first = None
for path in wanted:
if first is None:
ws.write_atomic(path, png)
first = path
continue
# Same bytes three times: link where the filesystem allows it.
os.makedirs(os.path.dirname(path), exist_ok=True)
if os.path.exists(path):
os.unlink(path)
try:
os.link(first, path)
except OSError:
shutil.copy2(first, path)
debug(f"{record['name']}: box art -> {len(wanted)} thumbnail folders")
return wanted, True
# --------------------------------------------------------------------------
# installation
# --------------------------------------------------------------------------
def crc32_of(path):
"""RetroArch stores the content's CRC32 as `XXXXXXXX|crc`."""
checksum = 0
with open(path, "rb") as fh:
while True:
chunk = fh.read(256 * 1024)
if not chunk:
break
checksum = zlib.crc32(chunk, checksum)
return format(checksum & 0xFFFFFFFF, "08X")
def install_game(cfg, tree, game, state_entry, dry_run=False):
"""Download the cartridge and its cover. Returns (record, changed)."""
platform = game["platform"]
local = os.path.join(tree.content_dir(cfg, platform), game["asset"])
changed = False
if os.path.isfile(local) and os.path.getsize(local) > 0:
debug(f"{game['name']}: {game['asset']} already present")
elif dry_run:
log(f"would download {game['name']} {game['version']} -> {local}")
changed = True
else:
size = ws.http_download(cfg, ws.download_url(cfg, game["asset"]), local)
log(f"installed {game['name']} {game['version']} ({size} bytes) -> {local}")
changed = True
record = dict(game)
record["content"] = local
record["target"] = target_join(tree.target_content_dir(cfg, platform), game["asset"])
record["crc32"] = ""
if cfg["playlist"].get("write_crc32") and os.path.isfile(local):
record["crc32"] = crc32_of(local)
thumbs = (state_entry or {}).get("thumbs") or []
if game.get("image_url"):
thumbs, did = install_thumbnails(cfg, tree, record, dry_run=dry_run)
changed = changed or did
elif thumbs and not dry_run:
# The cover disappeared from the catalog: drop what we cached.
remove_thumbnails(tree, thumbs, dry_run=dry_run)
thumbs, changed = [], True
record["thumbs"] = thumbs
return record, changed
def remove_thumbnails(tree, thumbs, dry_run=False):
for path in thumbs or []:
if not ws.within(path, tree.thumbnails_dir):
log(f"warning: refusing to delete {path} (outside {tree.thumbnails_dir})")
continue
if os.path.isfile(path):
if dry_run:
log(f"would remove {path}")
else:
os.unlink(path)
debug(f"removed {path}")
def remove_game(cfg, tree, record, dry_run=False):
"""Delete one entry's files — never anything outside what this store owns."""
content = record.get("content")
owned = tree.owned_root(cfg)
if content:
if not ws.within(content, owned):
log(f"warning: refusing to delete {content} (outside {owned})")
elif os.path.isfile(content):
if dry_run:
log(f"would remove {content}")
else:
os.unlink(content)
debug(f"removed {content}")
remove_thumbnails(tree, record.get("thumbs"), dry_run=dry_run)
# --------------------------------------------------------------------------
# the playlist
# --------------------------------------------------------------------------
def playlist_path(cfg, tree, platform):
spec = cfg["platforms"][platform]
return os.path.join(tree.playlists_dir, playlist_file_name(playlist_name(cfg, platform, spec)))
def read_playlist(path):
data = ws.load_json(path, default=None)
if not isinstance(data, dict) or not isinstance(data.get("items"), list):
if os.path.isfile(path):
log(f"warning: {path} is not a playlist we can read — starting a fresh one")
return None
return data
def write_playlist(cfg, tree, platform, records, dry_run=False):
"""Write one platform's playlist, keeping anything the user added to it.
The file carries the store's name, so it is ours — but a user may still have
added an entry of their own to it, and that survives: only items pointing
inside this store's content folder are rewritten.
"""
spec = cfg["platforms"].get(platform)
if not spec:
debug(f"no platform config for {platform} — not writing a playlist")
return
path = playlist_path(cfg, tree, platform)
name = os.path.basename(path)
ours = tree.target_content_dir(cfg, platform)
existing = read_playlist(path)
foreign = [i for i in (existing or {}).get("items", [])
if not target_within(i.get("path"), ours)]
if existing and not dry_run:
backup = f"{path}.{cfg['store']['id']}-backup"
if not os.path.exists(backup):
os.makedirs(os.path.dirname(backup), exist_ok=True)
shutil.copy2(path, backup)
if not records and not foreign:
if os.path.isfile(path):
if dry_run:
log(f"would remove the now-empty playlist {path}")
else:
os.unlink(path)
log(f"removed the now-empty playlist {path}")
return
core_path, core_name, note = resolve_core(cfg, tree, platform)
if note:
log(f"{playlist_name(cfg, platform, spec)}: {note}")
items = list(foreign)
for record in sorted(records, key=lambda r: r["title"].lower()):
items.append({
"path": record["target"],
"entry_slot": -1,
"label": record["title"],
# The header's default core runs these; DETECT keeps the file
# portable to a machine whose cores live somewhere else.
"core_path": "DETECT",
"core_name": "DETECT",
"crc32": f"{record['crc32']}|crc" if record.get("crc32") else "",
"db_name": name,
})
playlist = {
"version": PLAYLIST_VERSION,
"default_core_path": core_path,
"default_core_name": core_name,
# Zero everywhere means "whatever the user has set globally" — a store
# has no business overriding how someone likes their library displayed.
"label_display_mode": 0,
"right_thumbnail_mode": 0,
"left_thumbnail_mode": 0,
"thumbnail_match_mode": 0,
"sort_mode": 0,
"items": items,
}
if dry_run:
log(f"would write {path} ({len(records)} of ours, {len(foreign)} kept)")
return
ws.write_json(path, playlist)
log(f"playlist updated: {path} ({len(records)} entries" +
(f", {len(foreign)} kept" if foreign else "") + ")")
# --------------------------------------------------------------------------
# RetroArch itself
# --------------------------------------------------------------------------
def retroarch_running():
"""True if a RetroArch process is up — it would write our playlist back."""
if ws.host_os() == "windows":
try:
out = subprocess.run(["tasklist"], capture_output=True, text=True, timeout=15).stdout
return "retroarch.exe" in out.lower()
except (OSError, subprocess.SubprocessError):
debug("cannot tell whether RetroArch is running")
return False
if not shutil.which("pgrep"):
debug("no pgrep — cannot tell whether RetroArch is running")
return False
for name in ("retroarch", "RetroArch", "org.libretro.RetroArch"):
try:
if subprocess.run(["pgrep", "-x", name], capture_output=True, timeout=15).returncode == 0:
return True
except (OSError, subprocess.SubprocessError):
debug("cannot tell whether RetroArch is running")
return False
return False
def guard_running(cfg, args):
if not cfg["behavior"].get("refuse_while_running") or getattr(args, "force", False):
return
if retroarch_running():
die("RetroArch is running: it holds its playlists in memory and writes them back on "
"exit, so a sync now can be undone. Close it and try again (or --force).")
# --------------------------------------------------------------------------
# commands
# --------------------------------------------------------------------------
def install_all(cfg, tree, games, installed, dry_run=False, prune=False):
"""Install every game, prune what left the catalog. Returns (changed, scopes)."""
touched, changed = set(), False
for game in games:
key = ws.game_key(game)
previous = installed.get(key)
if previous and previous.get("asset") != game["asset"]:
log(f"{game['name']}: {previous.get('version')} -> {game['version']}, removing old files")
remove_game(cfg, tree, previous, dry_run=dry_run)
previous = None
changed = True
try:
record, did = install_game(cfg, tree, game, previous, dry_run=dry_run)
except (urllib.error.URLError, OSError) as exc:
log(f"warning: {game['name']} failed: {exc}")
continue
if did or installed.get(key) != record:
changed = True
installed[key] = record
touched.add(record["scope"])
if prune:
keep = {ws.game_key(g) for g in games}
for key in list(installed):
if key in keep:
continue
log(f"pruning {key} (no longer in the catalog or filtered out)")
remove_game(cfg, tree, installed[key], dry_run=dry_run)
touched.add(ws.record_scope(installed[key]))
del installed[key]
changed = True
return changed, touched
def cmd_sync(cfg, args):
guard_running(cfg, args)
tree = local_tree(cfg, args)
catalog = ws.fetch_catalog(cfg, use_cache=True)
games, skipped = select_games(cfg, catalog)
for reason in skipped:
log(f"skipped {reason}")
if args.name:
games = ws.limit_to_names(games, args.name)
state = ws.load_state()
installed = state["installed"]
# A name-limited sync is not a full picture of the catalog, so never prune.
changed, touched = install_all(cfg, tree, games, installed, dry_run=args.dry_run,
prune=cfg["behavior"].get("prune") and not args.name)
if args.dry_run:
grouped = ws.records_by_scope(installed)
for platform in sorted(touched | set(grouped)):
write_playlist(cfg, tree, platform, grouped.get(platform, []), dry_run=True)
log("dry run — nothing written")
return 0
ws.save_state(state)
if not changed and not args.force:
log("already up to date")
return 0
grouped = ws.records_by_scope(installed)
for platform in sorted(touched | set(grouped)):
write_playlist(cfg, tree, platform, grouped.get(platform, []))
log("restart RetroArch to see the playlist — it builds the menu at startup")
return 0
def cmd_export(cfg, args):
"""Render the whole store into a tree meant for another machine.
Stateless on purpose: an export is a complete picture of the catalog, not an
incremental update to this machine's installation, and it must not disturb
the state of a local install living in the same store home.
"""
tree = export_tree(cfg, args)
catalog = ws.fetch_catalog(cfg, use_cache=True)
games, skipped = select_games(cfg, catalog)
for reason in skipped:
log(f"skipped {reason}")
if args.name:
games = ws.limit_to_names(games, args.name)
log(f"exporting {len(games)} titles to {tree.sources['export root']}")
log(f"playlists will point at {tree.target_content_root}")
records = {}
install_all(cfg, tree, games, records, dry_run=args.dry_run)
grouped = ws.records_by_scope(records)
for platform, group in sorted(grouped.items()):
write_playlist(cfg, tree, platform, group, dry_run=args.dry_run)
if not args.dry_run:
log("copy the tree onto the device, e.g.: adb push "
f"{os.path.abspath(args.directory)}/. {tree.sources['target prefix']}/")
return 0
def cmd_list(cfg, args):
tree = local_tree(cfg, args)
catalog = ws.fetch_catalog(cfg, use_cache=True)
host = ws.host()
log(f"host: {host['os']}/{host['arch']}, core suffix {tree.core_suffix}")
games, skipped = select_games(cfg, catalog, host)
installed = ws.load_state()["installed"]
if not games:
log("no compatible games in the catalog")
for game in sorted(games, key=lambda g: (g["scope"], g["title"].lower())):
record = installed.get(ws.game_key(game))
if not record:
mark = " "
elif record.get("asset") == game["asset"]:
mark = "* "
else:
mark = "^ "
print(f"{mark}{game['platform']:<8} {game['name']:<18} {game['version']:<10} {game['title']}")
if games:
print("\n * installed ^ update available")
for reason in skipped:
log(f"skipped {reason}")
for platform in sorted({g["scope"] for g in games}):
_, core_name, note = resolve_core(cfg, tree, platform)
log(f"{platform}: {note or f'core ready ({core_name})'}")
return 0
def cmd_remove(cfg, args):
guard_running(cfg, args)
tree = local_tree(cfg, args)
state = ws.load_state()
installed = state["installed"]
keys = ws.match_keys(installed, args.name)
if not keys:
log(f"not installed: {', '.join(args.name)}")
return 0
touched = set()
for key in keys:
record = installed[key]
remove_game(cfg, tree, record, dry_run=args.dry_run)
touched.add(ws.record_scope(record))
if not args.dry_run:
del installed[key]
if args.dry_run:
return 0
ws.save_state(state)
grouped = ws.records_by_scope(installed)
for platform in touched:
write_playlist(cfg, tree, platform, grouped.get(platform, []))
return 0
def cmd_paths(cfg, args):
"""Where this machine's RetroArch keeps things, and where each answer came from.
The paths are not in the config — they are worked out from the machine — and
they decide everything the store does, so they are worth being able to look
at on their own. This is the first thing to run when a sync went somewhere
unexpected.
"""
tree = local_tree(cfg, args)
host = ws.host()
print(f"detected: os={host['os']} arch={host['arch']} core_suffix={tree.core_suffix}")
for key in ("retroarch_dir", "retroarch.cfg", "playlists_dir", "thumbnails_dir",
"libretro_dir", "content_root"):
print(f" {key:<16} {tree.sources.get(key, '-')}")
print(f" {'store folder':<16} {tree.owned_root(cfg)}")
for platform in sorted(cfg["platforms"]):
if not cfg["platforms"][platform].get("enabled", True):
continue
spec = cfg["platforms"][platform]
_, core_name, note = resolve_core(cfg, tree, platform)
print(f" {playlist_file_name(playlist_name(cfg, platform, spec))}")
print(f" {'core':<14} {note or f'ready ({core_name})'}")
return 0
def cmd_config(cfg, args):
if args.write:
if os.path.exists(args.config) and not args.force:
die(f"{args.config} already exists (use --force to overwrite)")
ws.write_json(args.config, DEFAULT_CONFIG)
log(f"wrote default config to {args.config}")
return 0
print(json.dumps(cfg, indent=2, ensure_ascii=False))
return 0
# --------------------------------------------------------------------------
def main(argv=None):
parser = argparse.ArgumentParser(
prog=os.path.basename(sys.argv[0]) or "retroarch_store.py",
description="Install WarpEngine catalog releases into RetroArch playlists.",
)
parser.add_argument("--config", default=DEFAULT_CONFIG_PATH,
help=f"store config file (default: {DEFAULT_CONFIG_PATH})")
parser.add_argument("--base-url", help="override the catalog base URL")
parser.add_argument("--retroarch-dir", help="override RetroArch's directory")
parser.add_argument("--core-suffix", help="libretro core file suffix (default: this host's)")
parser.add_argument("-n", "--dry-run", action="store_true", help="report actions, change nothing")
parser.add_argument("-v", "--verbose", action="store_true")
parser.add_argument("--version", action="version",
version=f"{VERSION} (warpstore {ws.VERSION})")
sub = parser.add_subparsers(dest="command")
p_sync = sub.add_parser("sync", help="download new/updated games and refresh playlists")
p_sync.add_argument("name", nargs="*", help="limit to these software names")
p_sync.add_argument("--force", action="store_true",
help="write even when RetroArch is running or nothing changed")
p_sync.set_defaults(func=cmd_sync)
p_exp = sub.add_parser("export", help="render the store for another machine (Android, an SD card)")
p_exp.add_argument("directory", help="staging directory to write the RetroArch tree into")
p_exp.add_argument("name", nargs="*", help="limit to these software names")
p_exp.add_argument("--target-prefix", help="how that tree will be spelled on the target machine")
p_exp.add_argument("--target-libretro-dir", help="the target's cores folder, if you know it")
# Also accepted here, where an export for another OS makes it the natural
# thing to reach for. SUPPRESS keeps the global flag's value when omitted.
p_exp.add_argument("--core-suffix", default=argparse.SUPPRESS,
help="libretro core file suffix on the target (e.g. _android.so)")
p_exp.set_defaults(func=cmd_export)
p_list = sub.add_parser("list", help="show compatible catalog entries and core status")
p_list.set_defaults(func=cmd_list)
p_rm = sub.add_parser("remove", help="uninstall games (name or platform:name)")
p_rm.add_argument("name", nargs="+")
p_rm.add_argument("--force", action="store_true", help="write even when RetroArch is running")
p_rm.set_defaults(func=cmd_remove)
p_paths = sub.add_parser("paths", help="show the resolved RetroArch directories and core status")
p_paths.set_defaults(func=cmd_paths)
p_cfg = sub.add_parser("config", help="print the effective config")
p_cfg.add_argument("--write", action="store_true", help="write a template config file")
p_cfg.add_argument("--force", action="store_true")
p_cfg.set_defaults(func=cmd_config)
args = parser.parse_args(argv)
if not args.command:
parser.print_help()
return 2
ws.set_verbose(args.verbose)
ws.init(args.config)
cfg = ws.load_config(
args.config, DEFAULT_CONFIG,
overrides={"store.base_url": args.base_url},
required=("paths.subfolder",),
)
os.makedirs(ws.HOME, exist_ok=True)
return args.func(cfg, args)
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
sys.exit(130)