warp-engine-retroarch-store/README.md

warp-engine-retroarch-store — a RetroArch store engine for WarpEngine sites

Pulls games from a WarpEngine catalog into RetroArch's own library: one .lpl playlist per platform, with box art in the thumbnail folders that playlist's name points at.

This repository is the engine only. It knows the WarpEngine API but nothing about any particular site: the host, the store's name and where its games land all come from a config.json that lives in a separate store repository. The shared part — catalog, releases, state, HTTP — is warpstore, which this engine has in common with the Batocera engine.

warpstore                     the shared core
    ▲
warp-engine-retroarch-store   this engine — retroarch_store.py, install.sh
    ▲
    │ config.json
ttg-retroarch-store           store repositories
    ▼
teletypegames.org

The reference store is ttg-retroarch-store.

<store>-retroarch-store sync
        │
        ├─ GET /api/software                      the whole catalog
        ├─ keep platforms a libretro core boots   c64 → VICE, tic80 → TIC-80
        ├─ pick the newest non-dev release        that carries the cartridge
        ├─ GET /api/download?path=<asset>         .prg / .tic into the content folder
        ├─ GET <software.imageUrl>                box art, converted to PNG
        └─ write <store> - <label>.lpl            playlist + thumbnails

Python 3 standard library only.

Why this is not the Batocera store

RetroArch is not one machine. The same playlist format is read on desktop Linux, Windows and macOS, on Android, on the Steam Deck and on the console ports — so the engine assumes nothing about where things are:

• It reads retroarch.cfg. The playlist folder is not necessarily inside the RetroArch directory: on the macOS install this was developed against, playlist_directory is ~/Documents/RetroArch/playlists while the cores and thumbnails are under ~/Library/Application Support/RetroArch. An engine that assumed <base>/playlists would write into the void. ~ is expanded, and so is the leading : a portable install uses to mean its own directory.
• It can write for a machine it is not running on (export). Android has no Python, so the only way to serve it is to render the tree elsewhere and copy it over. See Another machine below.
• It never edits retroarch.cfg. Nothing the user configured is touched.

What it does not do, compared with Batocera: a .lpl entry has no description or developer field, so the catalog's desc and author are lost — RetroArch takes that from its own databases, which are a binary format outside this engine's scope. And RetroArch cannot launch scripts, so there is no in-app trigger: the sync runs from a shell or from a scheduler.

Installing a store

The installer takes the store as a parameter, so it works with any config:

STORE_CONFIG=https://git.example.org/tools/my-store/raw/branch/master/config.json \
  bash <(curl -fsSL https://git.teletypegames.org/tools/warp-engine-retroarch-store/raw/branch/master/install.sh)

From a checkout, with a local config:

STORE_CONFIG=./my-config.json ./install.sh

It reads store.id from the config, installs the engine, the shared core and the config into ~/.local/share/warp-engine-store/<id>/, writes a <id>-retroarch-store launcher into ~/.local/bin, prints the paths it resolved and runs the first sync.

Environment variable	Meaning
STORE_CONFIG	required — path or URL of the store's config.json
STORE_ROOT	where stores live (default ${XDG_DATA_HOME:-~/.local/share}/warp-engine-store)
BIN_DIR	where the launcher goes (default ~/.local/bin)
RETROARCH_DIR	RetroArch's directory, if it is somewhere unusual
STORE_SKIP_SYNC	1 to install without syncing
ENGINE_RAW_BASE	where to fetch retroarch_store.py from
WARPSTORE_RAW_BASE	where to fetch warpstore.py from
WARPSTORE_SRC	a local warpstore.py to install instead of downloading it

Use

S=~/.local/bin/example-retroarch-store

$S paths                # resolved directories and whether the cores are there
$S list                 # compatible catalog entries; * installed, ^ update available
$S sync                 # download everything new, refresh the playlists
$S sync blessingofra    # just one title (never prunes)
$S -n sync              # dry run
$S remove c64:c64demo   # uninstall (bare name works too)
$S config               # effective configuration

Global flags go before the subcommand: $S --retroarch-dir /mnt/ra sync.

Restart RetroArch after a sync. It builds its menu at startup, so a new playlist does not appear in a running instance.

A sync refuses to run while RetroArch is up. RetroArch holds its playlists in memory and writes them back when it exits — favourites, last played, sort order — so writing underneath it can be undone. Close it, or pass --force if you know what you are doing (behavior.refuse_while_running: false to stop asking).

Where things land

~/.local/share/warp-engine-store/example/
├── retroarch_store.py, warpstore.py, config.json
└── state.json, catalog.json

<content_root>/example/c64/           blessingofra-2.0.0.prg, rabbit-1.0.0.prg, …
<playlists_dir>/Example Store - Commodore 64.lpl
<thumbnails_dir>/Example Store - Commodore 64/Named_Boxarts/Blessing of Ra.png
                                             /Named_Titles/…  /Named_Snaps/…

content_root defaults to <retroarch_dir>/content; the other two come from retroarch.cfg. Run paths to see what was resolved and why.

Everything lands under the store's own paths.subfolder, which is what a prune is allowed to touch — never content you put there yourself, never another store's. The playlist file carries the store's name, so it is ours; but an entry you added to it survives a rewrite, because only items pointing inside the store's content folder are replaced. The first time a playlist is touched it is copied to <name>.lpl.<store id>-backup.

The playlist

{
  "version": "1.5",
  "default_core_path": "/…/cores/vice_x64_libretro.dylib",
  "default_core_name": "VICE x64",
  "label_display_mode": 0,
  "right_thumbnail_mode": 0,
  "left_thumbnail_mode": 0,
  "thumbnail_match_mode": 0,
  "sort_mode": 0,
  "items": [
    {
      "path": "/…/content/example/c64/blessingofra-2.0.0.prg",
      "entry_slot": -1,
      "label": "Blessing of Ra",
      "core_path": "DETECT",
      "core_name": "DETECT",
      "crc32": "4253CFC6|crc",
      "db_name": "Example Store - Commodore 64.lpl"
    }
  ]
}

• One playlist per platform, because default_core_path is a header field: that is the only way every entry starts with one click and no core prompt. The entries themselves stay on DETECT, so the file still works on a machine whose cores live somewhere else — the worst case is one prompt, not a dead entry.
• The name is the namespace. {store} - {label} means we never write into RetroArch's own Commodore - 64.lpl and never collide with the official thumbnail packs. It is also the db_name, which is how RetroArch finds the thumbnails: <thumbnails_dir>/<db_name without .lpl>/Named_*/.
• The display fields are all 0 — whatever the user has set globally. A store has no business overriding how someone likes their library displayed.
• crc32 is real, computed from the downloaded file, which lets RetroArch tie saves and thumbnails to the content. Turn it off with playlist.write_crc32.

A missing core is a note, not an error

If the core named for a platform is not in libretro_directory, the playlist is still written: default_core_path stays empty and the entries stay on DETECT, so RetroArch asks once which core to use. list and paths say so, with the fix:

c64: core not found: vice_x64_libretro.dylib — entries stay on DETECT
     (Online Updater ▸ Core Downloader ▸ VICE x64)

A store that refused to run until the user had installed cores would never run for anyone the first time.

Box art must be PNG

RetroArch looks for <label>.png and nothing else. The catalog serves whatever was uploaded — one of ours is a GIF — and the standard library cannot re-encode an image, so the engine shells out to whatever the machine has: sips (built into macOS), magick, convert or ffmpeg, first one found. With none of them the entry simply has no box art, which beats a failed sync.

The cover is downloaded once and placed in all three Named_* folders (hardlinked where the filesystem allows), so whichever thumbnail type the user has selected, there is an image for it. playlist.thumbnail_kinds narrows that.

Another machine: export

sync installs into this machine's RetroArch. export renders the whole store into a canonical tree meant for somewhere else — the only way to serve Android, which runs RetroArch but cannot run Python.

# Android over adb
$S export /tmp/ra-example \
    --target-prefix /storage/emulated/0/RetroArch \
    --target-libretro-dir /data/data/com.retroarch.aarch64/cores \
    --core-suffix _android.so
adb push /tmp/ra-example/. /storage/emulated/0/RetroArch/

# An SD card mounted here, but at /mnt/sdcard on the handheld
$S export /Volumes/SDCARD/RetroArch --target-prefix /mnt/sdcard/RetroArch

The tree is always <dir>/playlists, <dir>/thumbnails, <dir>/content. --target-prefix is what the playlists say the content is called on the machine that will run it; without it, the export directory's own path is used (right when the export lands where it will stay). Target paths are written the way the target spells them — POSIX unless the prefix looks like Windows.

export is stateless on purpose: it is a complete picture of the catalog, not an incremental update, and it must not disturb the state of a local install living in the same store home. --target-libretro-dir is taken on trust, since the other machine cannot be inspected; leave it out and the entries stay on DETECT.

Not verified on a device yet. The Android paths above are the documented defaults, not measurements — read the device's own retroarch.cfg if a push does not show up, and whether scoped storage lets RetroArch read pushed content is the open question.

Writing a store repository

Three files:

my-retroarch-store/
├── config.json     the store: URL, name, subfolder, platform → core mapping
├── install.sh      a wrapper that hands that config to the engine installer
└── README.md

install.sh in full:

#!/bin/bash
set -euo pipefail
ENGINE_RAW_BASE="${ENGINE_RAW_BASE:-https://git.teletypegames.org/tools/warp-engine-retroarch-store/raw/branch/master}"
export STORE_CONFIG="${STORE_CONFIG:-https://git.example.org/tools/my-retroarch-store/raw/branch/master/config.json}"
curl -fsSL "$ENGINE_RAW_BASE/install.sh" | bash

Pick a store.id and a paths.subfolder nobody else uses — they are what keeps two stores on one machine from deleting each other's games.

Configuration

config.json in the store home, template in config.example.json (config --write regenerates it):

Key	Default	Meaning
store.id	warp	slug: names the store home, log prefix and backups
store.name	WarpEngine Store	goes into the playlist name — what the user sees
store.base_url	—	WarpEngine host
store.api.catalog / .download	/api/software, /api/download	endpoints, if the engine is mounted elsewhere
paths.retroarch_dir	null	RetroArch's directory; null = env, then the per-OS candidates
paths.playlists_dir	null	null = from retroarch.cfg, then <retroarch_dir>/playlists
paths.thumbnails_dir	null	as above, thumbnails
paths.libretro_dir	null	as above, cores
paths.content_root	null	null = <retroarch_dir>/content
paths.subfolder	warp	this store's folder inside it — the prune boundary
paths.target_prefix	null	default for export --target-prefix
playlist.name_template	{store} - {label}	store, label, platform are substituted
playlist.write_crc32	true	compute the content CRC32 for each entry
playlist.thumbnail_kinds	all three Named_*	which thumbnail folders to fill
catalog.statuses	["released", "archived"]	catalog status values to install
catalog.owner_id	null	restrict to one publisher
catalog.only / catalog.exclude	[]	software-name allow / deny lists
platforms	c64, tic80	platform → label, asset kind, extension, core, enabled
behavior.prune	true	remove games that left the catalog or the filters
behavior.timeout	30	HTTP timeout, seconds
behavior.insecure	false	skip TLS verification (self-hosted test instances)
behavior.refuse_while_running	true	do not write while RetroArch is running
behavior.convert_images	true	convert non-PNG box art with an external tool

Platform mapping

A catalog platform is installable when a libretro core boots its asset directly:

Catalog platform	Asset kind	Core	Extension
c64	cartridge	vice_x64_libretro (VICE x64)	.prg
tic80	cartridge	tic80_libretro (TIC-80)	.tic

Both cores are built for every relevant target — macOS x86_64/arm64, Linux x86_64/aarch64/armhf, Windows x86_64, Android arm64-v8a/armeabi-v7a — so the catalogue's cartridges run everywhere RetroArch does. Adding a platform is a config edit:

"platforms": {
  "nes": { "label": "NES", "kind": "cartridge", "ext": ".nes",
           "core": { "file": "fceumm_libretro", "name": "FCEUmm" }, "enabled": true }
}

kind and ext may be per-host maps rather than strings — {"windows": …, "*": …}, see warpstore.resolve_for_host — but for a cartridge they never need to be: it is data for an emulator, identical on every machine.

The other WarpEngine platforms (ebitengine, love, godot, bevy, phaser) ship web builds and per-OS native archives. No libretro core runs those, so this engine cannot serve them; on Batocera they install as Ports instead.

State

state.json records what this store installed, keyed <platform>:<name> so the same software name on two platforms stays two entries. Re-running a sync is idempotent: files already present are left alone, and a release bump (1.0 → 2.0.0) deletes the old asset, its thumbnails and its playlist entry before fetching the new one.