Worker Pipeline

The worker exists to keep slow work out of the API process. It is an RQ consumer that pulls a job off Redis, runs it to completion, and updates the backup’s status as it goes. It shares the backend’s Docker image and virtualenv, so every parser and database model is already on hand.

The indexing job does the same thing for every backup: mark it busy, clear any previous rows, then loop over the artifact registry, parsing each SQLite database it finds and writing normalized rows plus search entries.

Running the worker

The worker is plain RQ on the default queue. Both of these start one:

rq worker default --url redis://localhost:6379/0
# or the console script from pyproject.toml:
apple-juicer-worker

It reads the Redis DSN from APPLE_JUICER_REDIS__URL. Run it inside the same virtualenv as the backend so the parsers and SQLAlchemy models resolve.

The job, step by step

The backend enqueues index_backup_job, the sync wrapper that runs the async job under asyncio.run. Enqueuing the coroutine directly was the original bug that made indexing silently do nothing; the wrapper is what RQ actually calls. Once a worker picks it up, it:

1. Sets the backup’s status to INDEXING.
2. Truncates any artifact rows left from a previous run, so a re-index is always clean.
3. Loops over the artifact registry. For each registered type whose database is present, it parses the SQLite file and writes normalized rows plus entries in ArtifactSearchIndex.
4. Sets the status to INDEXED and commits.

Adding a new artifact type means registering one ArtifactSpec. The worker picks it up on the next run with no other changes — there is no per-type code in the job itself. See Data Storage and the Directory Layout for where a spec lives.

What each parser reads

Every module in parsers/ reads one SQLite database copied out of the backup and returns records the registry ingests:

Parser	Source database	Produces
parsers/photos.py	Photos.sqlite	Photo assets
parsers/messages.py	chat.db	iMessage/SMS conversations, messages, attachments
parsers/whatsapp.py	ChatStorage.sqlite	WhatsApp chats, messages, attachments
parsers/notes.py	NoteStore.sqlite	Notes
parsers/calendar.py	Calendar.sqlitedb	Calendars and events
parsers/contacts.py	AddressBook.sqlitedb	Contacts
parsers/calls.py	CallHistory.storedata	Call records
parsers/safari.py	History.db	Safari visits
parsers/locations.py	routined caches	Significant locations
parsers/voicemail.py	voicemail.db	Voicemails

A parser whose source database is missing returns nothing and the loop moves on, so a backup that never used WhatsApp simply indexes without it. Two sources are flaky by nature: the routined location cache is often excluded from backups, and Safari’s History.db domain shifts between iOS versions. Both parsers are written defensively and will leave their views empty rather than fail the job.

Errors, retries, and scaling

A fatal error bubbles up to RQ. Pass --max-retries or --retry-interval when enqueuing if you want automatic retries; otherwise re-running the job is safe because it truncates first. Logs go to the worker logger, which Docker Compose surfaces in the worker container output.

To index faster, run more workers — add replicas in docker-compose.yml or use docker compose up --scale worker=3. Every job targets a single backup, so workers never collide on the default queue.