Operations Guide
Day-to-day server operations, monitoring, and maintenance.
Server Access
Section titled “Server Access”ssh srv1.rondosudo -u rondo-sync -H bashcd /srv/rondo-syncThe production hostname checked by the sync scripts is srv1-rondo-club.
The rondo-sync account owns the checkout, scheduler, database files and
pipeline locks. Viscerno remains on the previous VPS. GlitchTip runs on
yolo1 under its own operations setup.
Checking Sync Status
Section titled “Checking Sync Status”The dashboard reports domain counts for People runs. People created, People updated, and People unchanged come from the Rondo Club submission result. Download and preparation volumes are shown as Processed in the run’s step table, so they do not look like newly created people.
View and Edit Sync Schedules
Section titled “View and Edit Sync Schedules”crontab -lView Recent Logs
Section titled “View Recent Logs”# List recent log filesls -lt logs/cron/ | head -20
# View latest people sync logcat logs/cron/$(ls -t logs/cron/ | grep sync-people | head -1)
# Tail a running synctail -f logs/cron/sync-people-*.logLogging after terminal disconnects
Section titled “Logging after terminal disconnects”lib/logger.js writes each message to its timestamped file before attempting terminal output. If stdout or stderr reports EPIPE, the shared logger stops writing to that stream and continues writing to disk. This covers both asynchronous stream errors and synchronous write failures. Other terminal errors still fail visibly.
Each process keeps one error handler per terminal stream, shared across logger instances and retained after an individual logger closes. File-write failures still emit a warning through stderr when it is available. This protects logging when an output pipe closes; it does not make a process immune to SSH hangups or replace the normal server wrapper.
test/logger.test.js covers healthy output, loss of either or both terminal streams, synchronous EPIPE, shared handlers and non-EPIPE failures.
Check if a Sync is Running
Section titled “Check if a Sync is Running”# Check for running Node processesps aux | grep "node sync-"
# Check lock filesls -la .sync-*.lockManual Sync Runs
Section titled “Manual Sync Runs”Individual Pipelines
Section titled “Individual Pipelines”scripts/sync.sh people # Members, parents, photos → Laposta + Rondo Clubscripts/sync.sh nikki # Nikki contributions → Rondo Clubscripts/sync.sh freescout # Rondo Club members → FreeScout customersscripts/sync.sh teams # Teams + work history → Rondo Clubscripts/sync.sh twelve --scheduled # Hourly due check; source login only inside club windowsscripts/sync.sh functions # Commissies + free fields (recent updates)scripts/sync.sh functions --all # Commissies + free fields (all members)scripts/sync.sh discipline # Discipline cases → Rondo Clubscripts/sync.sh all # All pipelines sequentiallySingle Member Sync
Section titled “Single Member Sync”Sync a specific member by KNVB ID:
node pipelines/sync-individual.js KNVB123456 --verbose # Full syncnode pipelines/sync-individual.js KNVB123456 --dry-run --verbose # Preview onlynode pipelines/sync-individual.js KNVB123456 --fetch --verbose # Fetch fresh data from Sportlink firstnode pipelines/sync-individual.js --search "Jan Jansen" # Search by nameDirect Script Execution
Section titled “Direct Script Execution”For debugging individual steps:
node steps/download-data-from-sportlink.js --verbose # Download onlynode steps/prepare-laposta-members.js --verbose # Prepare Laposta datanode steps/submit-laposta-list.js --verbose # Submit to Lapostanode steps/submit-rondo-club-sync.js --verbose # Submit to Rondo Club (includes birthdate)node steps/download-photos-from-api.js --verbose # Download photosnode steps/upload-photos-to-rondo-club.js --verbose # Upload photosDatabase Inspection
Section titled “Database Inspection”Member Lookups
Section titled “Member Lookups”# Find member in Sportlink datanode tools/show-sportlink-member.js member@example.com
# Find member in Laposta trackingnode tools/show-laposta-member.js member@example.comnode tools/show-laposta-member.js member@example.com 2 # Specific list
# Show pending Laposta changesnode tools/show-laposta-changes.js # Changes onlynode tools/show-laposta-changes.js --all # All members with diffsDirect SQLite Queries
Section titled “Direct SQLite Queries”# Member countsqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_members"
# Members needing sync (hash mismatch)sqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_members WHERE last_synced_hash IS NULL OR last_synced_hash != source_hash"
# Photo state distributionsqlite3 data/rondo-sync.sqlite "SELECT photo_state, COUNT(*) FROM rondo_club_members GROUP BY photo_state"
# Team countsqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_teams WHERE rondo_club_id IS NOT NULL"
# Commissie countsqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_commissies WHERE rondo_club_id IS NOT NULL"
# Recent Sportlink downloadssqlite3 data/laposta-sync.sqlite "SELECT id, created_at FROM sportlink_runs ORDER BY id DESC LIMIT 5"
# Nikki contributions with outstanding balancenode tools/show-nikki-contributions.js --outstanding
# FreeScout customer countsqlite3 data/freescout-sync.sqlite "SELECT COUNT(*) FROM freescout_customers WHERE freescout_id IS NOT NULL"Deploying Updates
Section titled “Deploying Updates”# On local machine: commit and pushgit add . && git commit -m "description" && git push
# On srv1: fast-forward the checkout as the sync accountssh srv1.rondo "sudo -u rondo-sync -H git -C /srv/rondo-sync pull --ff-only"If dependencies changed, run npm ci from the checkout as rondo-sync.
Data Validation
Section titled “Data Validation”Verify Rondo Club ID Mappings
Section titled “Verify Rondo Club ID Mappings”Checks that all tracked rondo_club_id values still point to valid WordPress posts:
node tools/verify-rondo-club-data.js --verbose # Report onlynode tools/verify-rondo-club-data.js --fix --verbose # Fix invalid IDsRepopulate Missing Rondo Club IDs
Section titled “Repopulate Missing Rondo Club IDs”If IDs were lost (e.g., database restored from backup):
node tools/repopulate-rondo-club-ids.js --dry-run --verbose # Previewnode tools/repopulate-rondo-club-ids.js --verbose # ApplyValidate Photo Consistency
Section titled “Validate Photo Consistency”Check that photo files on disk match database state:
node tools/check-photo-consistency.js --verbose # Reportnode tools/check-photo-consistency.js --fix # Fix mismatchesCleanup Operations
Section titled “Cleanup Operations”Remove Duplicate Members
Section titled “Remove Duplicate Members”node tools/delete-duplicates.js --verbose # Dry runnode tools/delete-duplicates.js --apply # Delete duplicatesMerge Duplicate Person Records
Section titled “Merge Duplicate Person Records”When a parent and member record exist for the same person:
node tools/merge-duplicate-person.js --parent=123 --member=456Remove Orphaned Data
Section titled “Remove Orphaned Data”# Orphaned relationshipsnode tools/cleanup-orphan-relationships.js --verbosenode tools/cleanup-orphan-relationships.js --fix
# Duplicate relationshipsnode tools/cleanup-duplicate-relationships.jsDeduplicate Laposta Lists
Section titled “Deduplicate Laposta Lists”node tools/dedupe-laposta-list.js # Dry run, all listsnode tools/dedupe-laposta-list.js --apply # Delete duplicatesnode tools/dedupe-laposta-list.js 2 --apply # List 2 onlyLog Management
Section titled “Log Management”Logs accumulate in logs/cron/. Clean up old logs periodically:
# Delete logs older than 30 daysfind logs/cron/ -name "*.log" -mtime +30 -delete
# Check disk usagedu -sh logs/srv1 backups
Section titled “srv1 backups”Encrypted off-server snapshots
Section titled “Encrypted off-server snapshots”The Mac mini’s club.rondo.srv1-backup launch agent pulls an export at 00:30,
06:30, 12:30 and 18:30 in the mini’s Europe/Amsterdam timezone. It also runs
at login. The backup SSH key can request an export and read completed exports
via restricted rsync. The restic repository and password stay on the mini side
of the connection.
The export covers production and demo WordPress files, deployed themes and
uploads, Sync files and data, and server configuration. Each WordPress MySQL
database uses a separate transaction-consistent dump. Sync SQLite snapshots
use the online backup API to include committed WAL data and pass
PRAGMA quick_check. Each database is consistent at its own snapshot time.
Caches, logs, dependencies, old theme releases and migration archives are
excluded. srv1 keeps its three latest complete exports.
The mini verifies the archive’s SHA-256 before writing the encrypted snapshot
and removes the incoming plaintext copy when the run ends. It retains all
snapshots for two days, then 14 daily, eight weekly and twelve monthly restore
points. It checks restic repository metadata each run and reads stored data
weekly. Check launchd and ~/Library/Application Support/Rondo Backup/status.json
for the last successful run. A failed run records the failure locally.
Hourly local database snapshots
Section titled “Hourly local database snapshots”On srv1, rondo-prod-db-backup.timer runs ten minutes past every hour in
Europe/Amsterdam and catches up after downtime. It backs up only rondo_prod
and keeps seven days of root-only, unencrypted snapshots under
/var/lib/rondo-backup/hourly-mysql/rondo_prod/. Each timestamped directory
contains database.sql.gz and a checksum and table-count manifest.
The service verifies the entire gzip stream and publishes the new directory
atomically before pruning expired snapshots. Check the timer and the
rondo-prod-db-backup.service journal for results.
Restore verification
Section titled “Restore verification”Restore a real restic snapshot into an isolated directory and check the archive checksum. Verify SQLite integrity, import both off-server MySQL dumps into separate disposable databases, and check the WordPress URLs and table counts. Test an hourly snapshot by importing its decompressed SQL into a disposable database. Keep cron, mail and sync writes disabled during recovery checks. See the source runbook for ownership, ACLs and extended attributes, package inventory and replacement-host steps. srv1 backup and recovery runbook
Monitoring Checklist
Section titled “Monitoring Checklist”GlitchTip collects filtered step errors, failed/partial runs, crashes and dashboard server errors. /health/sync checks aggregate schedule coverage: completed partial runs count as active, while their errors remain available for investigation. Keep external heartbeat monitoring for VPS outages. Private local logs retain the original messages that outgoing events omit.
Sync errors are grouped by pipeline, step and a safe error_code; unknown causes retain default stack grouping. Use the local logs for the original message. RunTracker also emits one performance transaction per completed run/step with full duration, a shared trace ID and aggregate created/updated/skipped/failed counts. No per-member spans or detail payloads leave the tracker. GLITCHTIP_PERFORMANCE=false disables these measurements while retaining errors; see performance measurements.
GlitchTip’s daily backup timer stores private local archives under /opt/glitchtip/backups, including credentials and the DuckDB performance cold-storage volume as cold-storage.tar. Restore cold storage alongside the database and uploads. Check systemctl status rondo-glitchtip-backup.timer and disk usage on yolo1. These archives stay on that host. See GlitchTip hosting and backups.
Daily checks:
- People sync email reports arrive at the configured cadence
- No errors in latest sync report
Weekly checks:
- Team sync ran at its configured slot (check
logs/cron/sync-teams-*) - Full functions sync ran at its configured slot (check
logs/cron/sync-functions-*) - Discipline sync ran at its configured slot (check
logs/cron/sync-discipline-*) - Disk usage reasonable (
du -sh /srv/rondo-sync/)
Monthly checks:
- Log files cleaned up (
find logs/cron/ -name "*.log" -mtime +30 | wc -l) - Photo directory size reasonable (
du -sh photos/) - Database sizes stable (
ls -lh data/*.sqlite)
Editable schedule operations
Section titled “Editable schedule operations”Use Beheer → Sync schedules to inspect saved timing, next runs and enable switches. See Editable Sync Schedules before installing or migrating cron. crontab -l shows the minute dispatcher and separate Twelve check, while data/sync-schedules.json holds the ten pipeline cadences. Run server commands as rondo-sync; root-owned locks can prevent scheduled work.
Inspect logs/cron/scheduler.log when a due job is not dispatched. The dispatcher does not catch up missed slots after downtime. Dashboard overdue checks and watchdog stale budgets follow the saved schedule; disabled schedules also suppress automatic healing. Cadence changes reset previous-slot checks until the first eligible slot. Weekly/monthly checklists must follow the current saved timing rather than assume the initial default days.