How do I pass the API key securely to cron?

Set it in the crontab itself (stored in /var/spool/cron/, readable only by root) or source a secrets file at the top of the script: source /etc/ctf-secrets. Never hardcode it in a script committed to version control.

What if the conversion server is down during the cron run?

The retry loop with exponential backoff handles transient outages. If all retries fail, the file is not added to the done log, so it will be retried on the next night's run automatically.

Can I run this more frequently than nightly?

Yes. Change the cron schedule (e.g., */15 * * * * for every 15 minutes). The done log ensures already-converted files are skipped on every run. For sub-minute frequency, consider inotifywait instead — see the folder-monitoring guide.

How do I handle files added during the conversion run?

The script iterates the directory at job start. Files added mid-run are not processed until the next cron fire. This is usually fine for nightly jobs. For real-time needs, inotifywait is more appropriate.

What format does the API auto-detect source from?

Source format is detected from the file extension. A file named report.docx is treated as DOCX. You only need to specify the target format in the API call.

Cron Job to Convert Files Nightly: Bash Script + Log Rotation

A recurring conversion job is one of the most common automation requests: nightly export of reports to PDF, daily normalization of uploaded images to WebP, weekly archival of logs to compressed formats. The pattern is simple but the details matter — proper logging, failure alerting, and idempotency are what separate a script that works from one you trust.

TL;DR

Three components: a bash script that converts files and exits non-zero on any failure, a logrotate config that keeps logs manageable, and a crontab entry that captures both stdout and stderr. Alerting is handled by checking exit status after the job runs.

# crontab -e
0 2 * * * /opt/scripts/nightly-convert.sh >> /var/log/ctf-nightly.log 2>&1 || /opt/scripts/alert.sh "nightly-convert failed"

The use case

You have a drop directory — /data/incoming/ — where upstream processes deposit files throughout the day. Every night at 2am, you want everything converted to a target format and moved to /data/converted/. Files that have already been converted should be skipped (idempotent). The job should log what it did, and if anything fails, you should know before morning standup.

Common real-world versions of this pattern:

DOCX → PDF for a legal team's nightly document archive
PNG screenshots → WebP for a build pipeline's asset optimization step
Raw audio recordings → MP3 for podcast episode processing
Uploaded CSVs → JSON for a data warehouse ingestion job

The ChangeThisFile API handles 690 routes — source format is auto-detected from filename, so you only need to specify the target.

Working bash script

#!/usr/bin/env bash
set -euo pipefail

# ---- config ----------------------------------------------------------------
API_KEY="${CTF_API_KEY:?CTF_API_KEY not set}"
INPUT_DIR="${CTF_INPUT_DIR:-/data/incoming}"
OUTPUT_DIR="${CTF_OUTPUT_DIR:-/data/converted}"
TARGET_FORMAT="${CTF_TARGET_FORMAT:-pdf}"
API_URL="https://changethisfile.com/v1/convert"
DONE_LOG="${CTF_DONE_LOG:-/var/lib/ctf/converted.log}"
# ---------------------------------------------------------------------------

log() { echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*"; }

mkdir -p "$OUTPUT_DIR" "$(dirname "$DONE_LOG")"
touch "$DONE_LOG"

converted=0
skipped=0
failed=0

for f in "$INPUT_DIR"/*; do
  [[ -f "$f" ]] || continue
  filename=$(basename "$f")
  stem="${filename%.*}"
  out="$OUTPUT_DIR/${stem}.${TARGET_FORMAT}"

  # Idempotency: skip files we already converted
  if grep -qF "$filename" "$DONE_LOG"; then
    log "SKIP $filename (already converted)"
    ((skipped++)) || true
    continue
  fi

  log "CONVERT $filename -> ${stem}.${TARGET_FORMAT}"

  http_status=$(curl -sf \
    -w "%{http_code}" \
    -o "$out" \
    -H "Authorization: Bearer $API_KEY" \
    -F "file=@$f" \
    -F "target=$TARGET_FORMAT" \
    "$API_URL"
  ) || { log "ERROR curl failed for $filename"; ((failed++)) || true; continue; }

  if [[ "$http_status" == "200" ]]; then
    echo "$filename" >> "$DONE_LOG"
    log "OK $filename (HTTP $http_status, $(stat -c%s "$out") bytes out)"
    ((converted++)) || true
  else
    log "ERROR HTTP $http_status for $filename"
    rm -f "$out"  # remove partial output
    ((failed++)) || true
  fi
done

log "DONE: converted=$converted skipped=$skipped failed=$failed"

# Exit non-zero if any conversions failed — cron will catch this
[[ $failed -eq 0 ]] || exit 1

Save to /opt/scripts/nightly-convert.sh and chmod +x it. Set CTF_API_KEY in the cron environment or a sourced file.

Error handling and retries

The script above fails fast on any curl error (-sf exits non-zero on HTTP errors and network failures). But for transient network issues, a retry wrapper is better than a hard fail:

convert_with_retry() {
  local file="$1" out="$2" target="$3"
  local attempts=3
  local delay=10

  for i in $(seq 1 $attempts); do
    http_status=$(curl -sf \
      --max-time 120 \
      --retry 0 \
      -w "%{http_code}" \
      -o "$out" \
      -H "Authorization: Bearer $API_KEY" \
      -F "file=@$file" \
      -F "target=$target" \
      "$API_URL"
    ) && [[ "$http_status" == "200" ]] && return 0

    log "RETRY $i/$attempts for $(basename "$file") (status: $http_status)"
    sleep "$delay"
    delay=$((delay * 2))  # exponential backoff
  done

  return 1
}

Use --max-time 120 rather than curl's default (infinite). Large files — video, dense PDFs — can take 60-90 seconds to convert server-side. Without a timeout, a hung conversion blocks the entire job indefinitely.

For the alert script, a simple curl to a Slack webhook covers most needs:

#!/usr/bin/env bash
# /opt/scripts/alert.sh
set -euo pipefail
SLACK_WEBHOOK="${CTF_SLACK_WEBHOOK:?}"
curl -sS -X POST "$SLACK_WEBHOOK" \
  -H 'Content-Type: application/json' \
  -d "{\"text\": \":rotating_light: $1 on $(hostname) at $(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"

Scheduling with cron and logrotate

The full crontab entry:

# /etc/cron.d/nightly-convert or: crontab -e
SHELL=/bin/bash
CTF_API_KEY=ctf_sk_your_key_here
CTF_INPUT_DIR=/data/incoming
CTF_OUTPUT_DIR=/data/converted
CTF_TARGET_FORMAT=pdf
CTF_DONE_LOG=/var/lib/ctf/converted.log
CTF_SLACK_WEBHOOK=https://hooks.slack.com/services/T.../B.../...

# Run at 2am, append stdout+stderr to log, alert on failure
0 2 * * * root /opt/scripts/nightly-convert.sh >> /var/log/ctf-nightly.log 2>&1 || /opt/scripts/alert.sh "nightly-convert failed"

Log rotation via logrotate keeps disk usage bounded:

# /etc/logrotate.d/ctf-nightly
/var/log/ctf-nightly.log {
  daily
  rotate 30
  compress
  delaycompress
  missingok
  notifempty
  create 0640 root adm
}

With daily and rotate 30, you keep 30 days of compressed logs. Each compressed day is typically a few KB for a nightly job — negligible disk cost.

Test your logrotate config without waiting: logrotate -f /etc/logrotate.d/ctf-nightly

Production tips

Lock against concurrent runs. If conversion takes longer than 24 hours (unlikely but possible with large batches), a second cron run will start. Use flock: wrap the script call with flock -n /var/lock/ctf-nightly.lock /opt/scripts/nightly-convert.sh.
Track the done log in a separate filesystem from the output. If the output disk fills up, you want the done log to remain intact so you don't re-convert everything on the next run.
Use MAILTO for low-cost alerting. Set MAILTO=ops@yourcompany.com in the crontab — cron will email you the script's output on any non-zero exit, no Slack webhook required for basic setups.
Free tier covers 1,000 conversions/month. For a nightly job processing under 33 files/day on average, that's enough. If you're near the limit, paid plans start at $29/mo for 10K conversions.
Verify the output before marking done. Check that the output file exists and has non-zero size before appending to the done log. A successful HTTP 200 response doesn't guarantee the output wrote correctly to disk.

A nightly conversion cron job needs four things to be production-ready: structured logging, idempotency (done log), failure alerting (MAILTO or webhook), and log rotation. The pattern above handles all four in under 80 lines of bash. Get a free API key to run it today — 1,000 conversions/month at no cost.

Cron Job for Nightly File Conversion: Bash Script Pattern