Introduce a dedicated `audit` log channel (daily rotation, configurable retention via
LOG_AUDIT_DAYS) and a small `auditLog()` / `auditLogWebhookFailure()` helper used to
record state-changing API operations and webhook events.
Instrumented:
- API mutation endpoints (create / update / delete / start / stop / restart) across
applications, services, databases (incl. backups, env vars, storage), servers,
projects + environments, scheduled tasks, private keys, GitHub apps, cloud provider
tokens, Hetzner server provisioning, instance enable/disable.
- Webhook signature verification outcomes for GitHub, GitLab, Bitbucket, Gitea and
Stripe, plus the Sentinel push endpoint.
- Authentication and authorization outcomes via the global exception handler and
the `ApiAbility` middleware (unauthenticated, ability-denied, policy-denied).
The helper is wrapped in try/catch so logging failures never affect the request
path. Successful operations log at `info`; suspicious/denied requests log at
`warning`. Operators wanting a failures-only feed can set `LOG_AUDIT_LEVEL=warning`.
Includes a feature test suite covering the helper, the webhook providers and the
new auth/authorization log paths.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extract the shouldRunNow() method from ScheduledJobManager and ServerManagerJob into
a reusable shouldRunCronNow() helper function. This centralizes cron scheduling logic
and enables consistent deduplication behavior across all scheduled job types.
- Create shouldRunCronNow() helper in bootstrap/helpers/shared.php with timezone
and dedup support
- Refactor ScheduledJobManager and ServerManagerJob to use the shared helper
- Add ScheduledJobDiagnostics command for inspecting cache state and scheduling
decisions across all scheduled jobs
- Simplify shouldRunNow tests to directly test the helper function
- Add DockerCleanupJob test for error handling and execution tracking
- Increase scheduled log retention from 1 to 7 days
- Add retry configuration to CoolifyTask (3 tries, 600s timeout)
- Add retry configuration to ScheduledTaskJob (3 tries, configurable timeout)
- Add retry configuration to DatabaseBackupJob (2 tries)
- Implement exponential backoff for all jobs (30s, 60s, 120s intervals)
- Add failed() handlers with comprehensive error logging to scheduled-errors channel
- Add execution tracking: started_at, retry_count, duration (decimal), error_details
- Add configurable timeout field to scheduled tasks (60-3600s, default 300s)
- Update UI to include timeout configuration in task creation/editing forms
- Increase ScheduledJobManager lock expiration from 60s to 90s for high-load environments
- Implement safe queue cleanup with restart vs runtime modes
- Restart mode: aggressive cleanup (marks all processing jobs as failed)
- Runtime mode: conservative cleanup (only marks jobs >12h as failed, skips deployments)
- Add cleanup:redis --restart flag for system startup
- Integrate cleanup into Dev.php init() for development environment
- Increase scheduled-errors log retention from 7 to 14 days
- Create comprehensive test suite (unit and feature tests)
- Add TESTING_GUIDE.md with manual testing instructions
Fixes issues with jobs failing after single attempt and "attempted too many times" errors
