type
concept
created
Tue Apr 07 2026 02:00:00 GMT+0200 (Central European Summer Time)
updated
Tue Apr 07 2026 02:00:00 GMT+0200 (Central European Summer Time)
sources
raw/notes/activeContext, raw/notes/progress, raw/articles/DISCOVERIES
tags
celery beat tasks scheduling email newsletters expiry

Celery Beat Tasks

abstract
The marketplace runs Celery worker and beat processes via PM2, handling scheduled tasks including email polling (every 5 minutes for ingestion), newsletter delivery, exclusivity window expiry, and pre-production item transitions.

Architecture

Scheduled Tasks

Email Polling for Ingestion (Every 5 Minutes)

Task: apps.ingestion.tasks.poll_ingestion_email

The ingestion pipeline supports receiving surplus lists via email. The beat task polls the ingestion email inbox every 5 minutes, looking for:

When found, it extracts attachments, creates IngestionBatch records, and triggers the processing pipeline (parse -> validate -> commit/reject).

Auto-Commit Trusted Batches

Task: apps.ingestion.tasks.auto_commit_trusted_batch

For mills marked as trusted (established relationship, consistent data quality), validated ingestion batches can be auto-committed without manual review. This task checks for validated batches from trusted mills and commits them, creating SurplusItems and triggering the matching algorithm.

Batch Processing

Task: apps.ingestion.tasks.process_ingestion_batch

Triggered when a new ingestion batch is created (either via email polling or manual upload). Processes the file through the full pipeline: file_processor (hash, validate) -> parser (column resolution, unit conversion) -> validator (field validation, dedup).

Newsletter Generation

Task: Newsletter generation and delivery tasks

Per-buyer personalized newsletters are generated based on the buyer's newsletter_frequency setting (daily, weekly, or on_match). The newsletter includes visibility-filtered surplus matches. Delivery tracking covers: generated -> sent -> delivered -> opened -> clicked (or bounced/failed).

Exclusivity Window Expiry

The exclusivity mechanism grants buyers 48-hour windows to negotiate on surplus items. When a window expires without a deal, Celery beat triggers the release, transitioning the item back to available status and optionally trying the next interested buyer (up to 3 grants per item).

Pre-Production Auto-Transition

Pre-production items have a future available_from date. Celery beat checks for items whose date has arrived and transitions them from pre_production to available status, making them visible to the matching algorithm and newsletter generation.

PM2 Process Configuration

The Celery processes are managed alongside the main API server:

PM2 Name Process Notes
marketplace-api gunicorn on port 8910 Main API server
marketplace-celery-worker Celery worker Processes async tasks
marketplace-celery-beat Celery beat Triggers scheduled tasks

All three processes are configured for auto-restart and log rotation via PM2.

Task Flow Diagram

Email Inbox
    │ (every 5 min)
    ▼
poll_ingestion_email
    │
    ▼
process_ingestion_batch
    │
    ├── parse → validate
    │
    ▼
auto_commit_trusted_batch (if trusted mill)
    │
    ▼
SurplusItems created → matching triggered → newsletters generated

Sources

Related