Skip to content

Operations Guide

Day-to-day server operations, monitoring, and maintenance.

Terminal window
ssh srv1.rondo
sudo -u rondo-sync -H bash
cd /srv/rondo-sync

The production hostname checked by the sync scripts is srv1-rondo-club. The rondo-sync account owns the checkout, scheduler, database files and pipeline locks. Viscerno remains on the previous VPS. GlitchTip runs on yolo1 under its own operations setup.

The dashboard reports domain counts for People runs. People created, People updated, and People unchanged come from the Rondo Club submission result. Download and preparation volumes are shown as Processed in the run’s step table, so they do not look like newly created people.

Terminal window
crontab -l
Terminal window
# List recent log files
ls -lt logs/cron/ | head -20
# View latest people sync log
cat logs/cron/$(ls -t logs/cron/ | grep sync-people | head -1)
# Tail a running sync
tail -f logs/cron/sync-people-*.log

lib/logger.js writes each message to its timestamped file before attempting terminal output. If stdout or stderr reports EPIPE, the shared logger stops writing to that stream and continues writing to disk. This covers both asynchronous stream errors and synchronous write failures. Other terminal errors still fail visibly.

Each process keeps one error handler per terminal stream, shared across logger instances and retained after an individual logger closes. File-write failures still emit a warning through stderr when it is available. This protects logging when an output pipe closes; it does not make a process immune to SSH hangups or replace the normal server wrapper.

test/logger.test.js covers healthy output, loss of either or both terminal streams, synchronous EPIPE, shared handlers and non-EPIPE failures.

Terminal window
# Check for running Node processes
ps aux | grep "node sync-"
# Check lock files
ls -la .sync-*.lock
Terminal window
scripts/sync.sh people # Members, parents, photos → Laposta + Rondo Club
scripts/sync.sh nikki # Nikki contributions → Rondo Club
scripts/sync.sh freescout # Rondo Club members → FreeScout customers
scripts/sync.sh teams # Teams + work history → Rondo Club
scripts/sync.sh twelve --scheduled # Hourly due check; source login only inside club windows
scripts/sync.sh functions # Commissies + free fields (recent updates)
scripts/sync.sh functions --all # Commissies + free fields (all members)
scripts/sync.sh discipline # Discipline cases → Rondo Club
scripts/sync.sh all # All pipelines sequentially

Sync a specific member by KNVB ID:

Terminal window
node pipelines/sync-individual.js KNVB123456 --verbose # Full sync
node pipelines/sync-individual.js KNVB123456 --dry-run --verbose # Preview only
node pipelines/sync-individual.js KNVB123456 --fetch --verbose # Fetch fresh data from Sportlink first
node pipelines/sync-individual.js --search "Jan Jansen" # Search by name

For debugging individual steps:

Terminal window
node steps/download-data-from-sportlink.js --verbose # Download only
node steps/prepare-laposta-members.js --verbose # Prepare Laposta data
node steps/submit-laposta-list.js --verbose # Submit to Laposta
node steps/submit-rondo-club-sync.js --verbose # Submit to Rondo Club (includes birthdate)
node steps/download-photos-from-api.js --verbose # Download photos
node steps/upload-photos-to-rondo-club.js --verbose # Upload photos
Terminal window
# Find member in Sportlink data
node tools/show-sportlink-member.js member@example.com
# Find member in Laposta tracking
node tools/show-laposta-member.js member@example.com
node tools/show-laposta-member.js member@example.com 2 # Specific list
# Show pending Laposta changes
node tools/show-laposta-changes.js # Changes only
node tools/show-laposta-changes.js --all # All members with diffs
Terminal window
# Member count
sqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_members"
# Members needing sync (hash mismatch)
sqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_members WHERE last_synced_hash IS NULL OR last_synced_hash != source_hash"
# Photo state distribution
sqlite3 data/rondo-sync.sqlite "SELECT photo_state, COUNT(*) FROM rondo_club_members GROUP BY photo_state"
# Team count
sqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_teams WHERE rondo_club_id IS NOT NULL"
# Commissie count
sqlite3 data/rondo-sync.sqlite "SELECT COUNT(*) FROM rondo_club_commissies WHERE rondo_club_id IS NOT NULL"
# Recent Sportlink downloads
sqlite3 data/laposta-sync.sqlite "SELECT id, created_at FROM sportlink_runs ORDER BY id DESC LIMIT 5"
# Nikki contributions with outstanding balance
node tools/show-nikki-contributions.js --outstanding
# FreeScout customer count
sqlite3 data/freescout-sync.sqlite "SELECT COUNT(*) FROM freescout_customers WHERE freescout_id IS NOT NULL"
Terminal window
# On local machine: commit and push
git add . && git commit -m "description" && git push
# On srv1: fast-forward the checkout as the sync account
ssh srv1.rondo "sudo -u rondo-sync -H git -C /srv/rondo-sync pull --ff-only"

If dependencies changed, run npm ci from the checkout as rondo-sync.

Checks that all tracked rondo_club_id values still point to valid WordPress posts:

Terminal window
node tools/verify-rondo-club-data.js --verbose # Report only
node tools/verify-rondo-club-data.js --fix --verbose # Fix invalid IDs

If IDs were lost (e.g., database restored from backup):

Terminal window
node tools/repopulate-rondo-club-ids.js --dry-run --verbose # Preview
node tools/repopulate-rondo-club-ids.js --verbose # Apply

Check that photo files on disk match database state:

Terminal window
node tools/check-photo-consistency.js --verbose # Report
node tools/check-photo-consistency.js --fix # Fix mismatches
Terminal window
node tools/delete-duplicates.js --verbose # Dry run
node tools/delete-duplicates.js --apply # Delete duplicates

When a parent and member record exist for the same person:

Terminal window
node tools/merge-duplicate-person.js --parent=123 --member=456
Terminal window
# Orphaned relationships
node tools/cleanup-orphan-relationships.js --verbose
node tools/cleanup-orphan-relationships.js --fix
# Duplicate relationships
node tools/cleanup-duplicate-relationships.js
Terminal window
node tools/dedupe-laposta-list.js # Dry run, all lists
node tools/dedupe-laposta-list.js --apply # Delete duplicates
node tools/dedupe-laposta-list.js 2 --apply # List 2 only

Logs accumulate in logs/cron/. Clean up old logs periodically:

Terminal window
# Delete logs older than 30 days
find logs/cron/ -name "*.log" -mtime +30 -delete
# Check disk usage
du -sh logs/

The Mac mini’s club.rondo.srv1-backup launch agent pulls an export at 00:30, 06:30, 12:30 and 18:30 in the mini’s Europe/Amsterdam timezone. It also runs at login. The backup SSH key can request an export and read completed exports via restricted rsync. The restic repository and password stay on the mini side of the connection.

The export covers production and demo WordPress files, deployed themes and uploads, Sync files and data, and server configuration. Each WordPress MySQL database uses a separate transaction-consistent dump. Sync SQLite snapshots use the online backup API to include committed WAL data and pass PRAGMA quick_check. Each database is consistent at its own snapshot time. Caches, logs, dependencies, old theme releases and migration archives are excluded. srv1 keeps its three latest complete exports.

The mini verifies the archive’s SHA-256 before writing the encrypted snapshot and removes the incoming plaintext copy when the run ends. It retains all snapshots for two days, then 14 daily, eight weekly and twelve monthly restore points. It checks restic repository metadata each run and reads stored data weekly. Check launchd and ~/Library/Application Support/Rondo Backup/status.json for the last successful run. A failed run records the failure locally.

On srv1, rondo-prod-db-backup.timer runs ten minutes past every hour in Europe/Amsterdam and catches up after downtime. It backs up only rondo_prod and keeps seven days of root-only, unencrypted snapshots under /var/lib/rondo-backup/hourly-mysql/rondo_prod/. Each timestamped directory contains database.sql.gz and a checksum and table-count manifest. The service verifies the entire gzip stream and publishes the new directory atomically before pruning expired snapshots. Check the timer and the rondo-prod-db-backup.service journal for results.

Restore a real restic snapshot into an isolated directory and check the archive checksum. Verify SQLite integrity, import both off-server MySQL dumps into separate disposable databases, and check the WordPress URLs and table counts. Test an hourly snapshot by importing its decompressed SQL into a disposable database. Keep cron, mail and sync writes disabled during recovery checks. See the source runbook for ownership, ACLs and extended attributes, package inventory and replacement-host steps. srv1 backup and recovery runbook

GlitchTip collects filtered step errors, failed/partial runs, crashes and dashboard server errors. /health/sync checks aggregate schedule coverage: completed partial runs count as active, while their errors remain available for investigation. Keep external heartbeat monitoring for VPS outages. Private local logs retain the original messages that outgoing events omit.

Sync errors are grouped by pipeline, step and a safe error_code; unknown causes retain default stack grouping. Use the local logs for the original message. RunTracker also emits one performance transaction per completed run/step with full duration, a shared trace ID and aggregate created/updated/skipped/failed counts. No per-member spans or detail payloads leave the tracker. GLITCHTIP_PERFORMANCE=false disables these measurements while retaining errors; see performance measurements.

GlitchTip’s daily backup timer stores private local archives under /opt/glitchtip/backups, including credentials and the DuckDB performance cold-storage volume as cold-storage.tar. Restore cold storage alongside the database and uploads. Check systemctl status rondo-glitchtip-backup.timer and disk usage on yolo1. These archives stay on that host. See GlitchTip hosting and backups.

Daily checks:

  • People sync email reports arrive at the configured cadence
  • No errors in latest sync report

Weekly checks:

  • Team sync ran at its configured slot (check logs/cron/sync-teams-*)
  • Full functions sync ran at its configured slot (check logs/cron/sync-functions-*)
  • Discipline sync ran at its configured slot (check logs/cron/sync-discipline-*)
  • Disk usage reasonable (du -sh /srv/rondo-sync/)

Monthly checks:

  • Log files cleaned up (find logs/cron/ -name "*.log" -mtime +30 | wc -l)
  • Photo directory size reasonable (du -sh photos/)
  • Database sizes stable (ls -lh data/*.sqlite)

Use Beheer → Sync schedules to inspect saved timing, next runs and enable switches. See Editable Sync Schedules before installing or migrating cron. crontab -l shows the minute dispatcher and separate Twelve check, while data/sync-schedules.json holds the ten pipeline cadences. Run server commands as rondo-sync; root-owned locks can prevent scheduled work.

Inspect logs/cron/scheduler.log when a due job is not dispatched. The dispatcher does not catch up missed slots after downtime. Dashboard overdue checks and watchdog stale budgets follow the saved schedule; disabled schedules also suppress automatic healing. Cadence changes reset previous-slot checks until the first eligible slot. Weekly/monthly checklists must follow the current saved timing rather than assume the initial default days.