From 372ea268103b77cb47a648e63df392d1122672f1 Mon Sep 17 00:00:00 2001 From: Andras Bacsai <5845193+andrasbacsai@users.noreply.github.com> Date: Mon, 10 Nov 2025 11:42:18 +0100 Subject: [PATCH] chore: remove outdated testing guide for scheduled tasks --- TESTING_GUIDE.md | 235 ----------------------------------------------- 1 file changed, 235 deletions(-) delete mode 100644 TESTING_GUIDE.md diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md deleted file mode 100644 index 91a79cd62..000000000 --- a/TESTING_GUIDE.md +++ /dev/null @@ -1,235 +0,0 @@ -# Testing Guide: Scheduled Tasks Improvements - -## Overview -This guide covers testing all the improvements made to the scheduled tasks system, including retry logic, timeout handling, and error logging. - -## Jobs Modified - -1. **CoolifyTask** - Infrastructure job for SSH operations (3 retries, 600s timeout) -2. **ScheduledTaskJob** - Scheduled container commands (3 retries, configurable timeout) -3. **DatabaseBackupJob** - Database backups (2 retries, existing timeout) - ---- - -## Quick Test Commands - -### Run Unit Tests (No Database Required) -```bash -./vendor/bin/pest tests/Unit/ScheduledJobsRetryConfigTest.php -``` - -### Run Feature Tests (Requires Database - Run in Docker) -```bash -docker exec coolify php artisan test --filter=CoolifyTaskRetryTest -``` - ---- - -## Manual Testing - -### 1. Test ScheduledTaskJob ✅ (You tested this) - -**How to test:** -1. Create a scheduled task in the UI -2. Set a short frequency (every minute) -3. Monitor execution in the UI -4. Check logs: `storage/logs/scheduled-errors-2025-11-09.log` - -**What to verify:** -- Task executes successfully -- Duration is recorded (in seconds with 2 decimal places) -- Retry count is tracked -- Timeout configuration is respected - ---- - -### 2. Test DatabaseBackupJob ✅ (You tested this) - -**How to test:** -1. Create a scheduled database backup -2. Set frequency to manual or very short interval -3. Trigger backup manually or wait for schedule -4. Check logs for any errors - -**What to verify:** -- Backup completes successfully -- Retry logic works if there's a transient failure -- Error logging is consistent -- Backoff timing is correct (60s, 300s) - ---- - -### 3. Test CoolifyTask ⚠️ (IMPORTANT - Not tested yet) - -CoolifyTask is used throughout the application for ALL SSH operations. Here are multiple ways to test it: - -#### **Option A: Server Validation** (Easiest) -1. Go to **Servers** in Coolify UI -2. Select any server -3. Click **"Validate Server"** or **"Check Connection"** -4. This triggers CoolifyTask jobs -5. Check Horizon dashboard for job processing -6. Check logs: `storage/logs/scheduled-errors-2025-11-09.log` - -#### **Option B: Container Operations** -1. Go to any **Application** or **Service** -2. Try these actions (each triggers CoolifyTask): - - Restart container - - View logs - - Execute command in container -3. Monitor Horizon for job processing -4. Check logs for errors - -#### **Option C: Application Deployment** -1. Deploy or redeploy any application -2. This triggers MANY CoolifyTask jobs -3. Watch Horizon dashboard - you should see: - - Jobs being dispatched - - Jobs completing successfully - - If any fail, they should retry (check "Failed Jobs") -4. Check logs for retry attempts - -#### **Option D: Docker Cleanup** -1. Wait for or trigger Docker cleanup (runs on schedule) -2. This uses CoolifyTask for cleanup commands -3. Check logs: `storage/logs/scheduled-errors-2025-11-09.log` - ---- - -## Monitoring & Verification - -### Horizon Dashboard -1. Open Horizon: `/horizon` -2. Watch these sections: - - **Recent Jobs** - See jobs being processed - - **Failed Jobs** - Jobs that failed permanently after retries - - **Monitoring** - Job throughput and wait times - -### Log Monitoring -```bash -# Watch scheduled errors in real-time -tail -f storage/logs/scheduled-errors-2025-11-09.log - -# Check for specific job errors -grep "CoolifyTask" storage/logs/scheduled-errors-2025-11-09.log -grep "ScheduledTaskJob" storage/logs/scheduled-errors-2025-11-09.log -grep "DatabaseBackupJob" storage/logs/scheduled-errors-2025-11-09.log -``` - -### Database Verification -```sql --- Check execution tracking -SELECT * FROM scheduled_task_executions -ORDER BY created_at DESC -LIMIT 10; - --- Verify duration is decimal (not throwing errors) -SELECT id, duration, retry_count, started_at, finished_at -FROM scheduled_task_executions -WHERE duration IS NOT NULL; - --- Check for tasks with retries -SELECT * FROM scheduled_task_executions -WHERE retry_count > 0; -``` - ---- - -## Expected Behavior - -### ✅ Success Indicators - -1. **Jobs Complete Successfully** - - Horizon shows completed jobs - - No errors in scheduled-errors log - - Execution records in database - -2. **Retry Logic Works** - - Failed jobs retry automatically - - Backoff timing is respected (30s, 60s, etc.) - - Jobs marked failed only after all retries exhausted - -3. **Timeout Enforcement** - - Long-running jobs terminate at timeout - - Timeout is configurable per task - - No hanging jobs - -4. **Error Logging** - - All errors logged to `storage/logs/scheduled-errors-2025-11-09.log` - - Consistent format with job name, attempt count, error details - - Trace included for debugging - -5. **Execution Tracking** - - Duration recorded correctly (decimal with 2 places) - - Retry count incremented on failures - - Started/finished timestamps accurate - ---- - -## Troubleshooting - -### Issue: Jobs fail immediately without retrying -**Check:** -- Verify `$tries` property is set on the job -- Check if exception is being caught and re-thrown correctly -- Look for `maxExceptions` being reached - -### Issue: "Invalid text representation" errors -**Fix Applied:** -- Duration field changed from integer to decimal(10,2) -- If you see this, run migrations again - -### Issue: Jobs not appearing in Horizon -**Check:** -- Horizon is running (`php artisan horizon`) -- Queue workers are active -- Job is dispatched to correct queue ('high' for these jobs) - -### Issue: Timeout not working -**Check:** -- Timeout is set on job (CoolifyTask: 600s, ScheduledTask: configurable) -- PHP `max_execution_time` allows job timeout -- Queue worker timeout is higher than job timeout - ---- - -## Test Checklist - -- [ ] Unit tests pass: `./vendor/bin/pest tests/Unit/ScheduledJobsRetryConfigTest.php` -- [ ] ScheduledTaskJob tested manually ✅ -- [ ] DatabaseBackupJob tested manually ✅ -- [ ] CoolifyTask tested manually (server validation, container ops, or deployment) -- [ ] Retry logic verified (force a failure, watch retry attempts) -- [ ] Timeout enforcement tested (create long-running task with short timeout) -- [ ] Error logs checked: `storage/logs/scheduled-errors-2025-11-09.log` -- [ ] Horizon dashboard shows jobs processing correctly -- [ ] Database execution records show duration as decimal -- [ ] UI shows timeout configuration field for scheduled tasks - ---- - -## Next Steps After Testing - -1. If all tests pass, run migrations on production/staging: - ```bash - php artisan migrate - ``` - -2. Monitor logs for the first 24 hours: - ```bash - tail -f storage/logs/scheduled-errors-2025-11-09.log - ``` - -3. Check Horizon for any failed jobs needing attention - -4. Verify existing scheduled tasks now have retry capability - ---- - -## Questions? - -If you encounter issues: -1. Check `storage/logs/scheduled-errors-2025-11-09.log` first -2. Check `storage/logs/laravel.log` for general errors -3. Look at Horizon "Failed Jobs" for detailed error info -4. Review database execution records for patterns