Commit graph

15 commits

Author SHA1 Message Date
Andras Bacsai
c8a332a3bc fix(servers): isolate cloud status checks from SSH checks
Track provider state independently, skip SSH work for placeholder IPs, and clean up failed cloud server provisioning.
2026-07-11 22:52:10 +02:00
Andras Bacsai
c303c34cfd feat(digitalocean): add droplet provisioning support 2026-07-08 10:42:58 +02:00
Andras Bacsai
c58b4fd69c Merge remote-tracking branch 'origin/next' into feature/vultr-cloud-provider 2026-07-07 13:44:37 +02:00
Andras Bacsai
9b060958aa
fix(ray): remove Ray debug hooks from runtime (#10847) 2026-07-03 11:40:20 +02:00
Cornelis Terblanche
94abbe1590 Add Vultr cloud provider integration 2026-06-03 21:32:18 +02:00
Andras Bacsai
b8226be774 refactor(server): dispatch event for reachability notifications, drop retry loop
Move reachability notification triggering out of isReachableChanged into
a dedicated ServerReachabilityChanged event dispatched by
ServerConnectionCheckJob. Remove the blocking 3-attempt sleep loop from
isReachableChanged — unreachable_count threshold alone now gates the
Unreachable notification. Add feature and unit tests covering all
notification dispatch paths.
2026-04-28 15:28:22 +02:00
Andras Bacsai
dd2c9c291a feat(jobs): implement exponential backoff for unreachable servers
Reduce load on unreachable servers by implementing exponential backoff
during connectivity failures. Check frequency decreases based on
consecutive failure count:
  0-2: every cycle
  3-5: ~15 min intervals
  6-11: ~30 min intervals
  12+: ~60 min intervals

Uses server ID hash to distribute checks across cycles and prevent
thundering herd.

ServerCheckJob and ServerConnectionCheckJob increment unreachable_count
on failures. ServerManagerJob applies backoff logic before dispatching
checks. Includes comprehensive test coverage.
2026-03-26 10:51:36 +01:00
Andras Bacsai
fef8e0b622 refactor: remove verbose logging and use explicit exception types
- Remove verbose warning/debug logs from ServerConnectionCheckJob and ContainerStatusAggregator
- Silently ignore expected errors (e.g., deleted Hetzner servers)
- Replace generic RuntimeException with DeploymentException for deployment command failures
- Catch both RuntimeException and DeploymentException in command retry logic
2026-03-20 15:57:26 +01:00
Andras Bacsai
6d9dbb4ba1
fix(jobs): handle queue timeouts gracefully in Horizon (#8360) 2026-02-15 13:43:23 +01:00
Andras Bacsai
a34d1656f4 chore: prepare for PR 2026-02-15 13:42:58 +01:00
Andras Bacsai
1b2c03fc2d chore: prepare for PR 2026-02-15 13:28:52 +01:00
Andras Bacsai
00cb06150e fix: improve error logging and handling in ServerConnectionCheckJob for Hetzner server status 2025-10-10 10:12:59 +02:00
Andras Bacsai
bf5c08d071 work work on hetzner integration 2025-10-09 16:54:13 +02:00
Andras Bacsai
2ef139bc42 fix(server): update server usability check to reflect actual Docker availability status 2025-09-16 10:18:35 +02:00
Andras Bacsai
7d10711a65 fix(servercheck): Properly check server statuses with and without Sentinel 2025-08-22 11:50:56 +02:00