feat: add Docker build cache preservation toggles and development logging

Add two new application settings to control Docker build cache invalidation:
- inject_build_args_to_dockerfile (default: true) - Skip Dockerfile ARG injection
- include_source_commit_in_build (default: false) - Exclude SOURCE_COMMIT from build context

These toggles let users preserve Docker cache when SOURCE_COMMIT or custom ARGs change frequently. Development-only logging shows which ARGs are being injected for debugging.

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Andras Bacsai 2025-11-26 13:42:02 +01:00
parent 4e896cca05
commit 837391c31b
6 changed files with 286 additions and 103 deletions

View file

@ -270,6 +270,84 @@ ### Build Optimization
- **Build artifact** reuse
- **Parallel build** processing
### Docker Build Cache Preservation
Coolify provides settings to preserve Docker build cache across deployments, addressing cache invalidation issues.
#### The Problem
By default, Coolify injects `ARG` statements into user Dockerfiles for build-time variables. This breaks Docker's cache mechanism because:
1. **ARG declarations invalidate cache** - Any change in ARG values after the `ARG` instruction invalidates all subsequent layers
2. **SOURCE_COMMIT changes every commit** - Causes full rebuilds even when code changes are minimal
#### Application Settings
Two toggles in **Advanced Settings** control this behavior:
| Setting | Default | Description |
|---------|---------|-------------|
| `inject_build_args_to_dockerfile` | `true` | Controls whether Coolify adds `ARG` statements to Dockerfile |
| `include_source_commit_in_build` | `false` | Controls whether `SOURCE_COMMIT` is included in build context |
**Database columns:** `application_settings.inject_build_args_to_dockerfile`, `application_settings.include_source_commit_in_build`
#### Buildpack Coverage
| Build Pack | ARG Injection | Method |
|------------|---------------|--------|
| **Dockerfile** | ✅ Yes | `add_build_env_variables_to_dockerfile()` |
| **Docker Compose** (with `build:`) | ✅ Yes | `modify_dockerfiles_for_compose()` |
| **PR Deployments** (Dockerfile only) | ✅ Yes | `add_build_env_variables_to_dockerfile()` |
| **Nixpacks** | ❌ No | Generates its own Dockerfile internally |
| **Static** | ❌ No | Uses internal Dockerfile |
| **Docker Image** | ❌ No | No build phase |
#### How It Works
**When `inject_build_args_to_dockerfile` is enabled (default):**
```dockerfile
# Coolify modifies your Dockerfile to add:
FROM node:20
ARG MY_VAR=value
ARG COOLIFY_URL=...
ARG SOURCE_COMMIT=abc123 # (if include_source_commit_in_build is true)
# ... rest of your Dockerfile
```
**When `inject_build_args_to_dockerfile` is disabled:**
- Coolify does NOT modify the Dockerfile
- `--build-arg` flags are still passed (harmless without matching `ARG` in Dockerfile)
- User must manually add `ARG` statements for any build-time variables they need
**When `include_source_commit_in_build` is disabled (default):**
- `SOURCE_COMMIT` is NOT included in build-time variables
- `SOURCE_COMMIT` is still available at **runtime** (in container environment)
- Docker cache preserved across different commits
#### Recommended Configuration
| Use Case | inject_build_args | include_source_commit | Cache Behavior |
|----------|-------------------|----------------------|----------------|
| Maximum cache preservation | `false` | `false` | Best cache retention |
| Need build-time vars, no commit | `true` | `false` | Cache breaks on var changes |
| Need commit at build-time | `true` | `true` | Cache breaks every commit |
| Manual ARG management | `false` | `true` | Cache preserved (no ARG in Dockerfile) |
#### Implementation Details
**Files:**
- `app/Jobs/ApplicationDeploymentJob.php`:
- `set_coolify_variables()` - SOURCE_COMMIT conditional (~line 1960)
- `generate_coolify_env_variables()` - SOURCE_COMMIT conditional for forBuildTime
- `generate_env_variables()` - SOURCE_COMMIT conditional (~line 2340)
- `add_build_env_variables_to_dockerfile()` - ARG injection toggle (~line 3358)
- `modify_dockerfiles_for_compose()` - Docker Compose ARG injection (~line 3530)
- `app/Models/ApplicationSetting.php` - Boolean casts
- `app/Livewire/Project/Application/Advanced.php` - UI binding
- `resources/views/livewire/project/application/advanced.blade.php` - Checkboxes
**Note:** Docker Compose services without a `build:` section (image-only) are automatically skipped.
### Runtime Optimization
- **Container resource** limits
- **Auto-scaling** based on metrics

View file

@ -1957,7 +1957,12 @@ private function deploy_to_additional_destinations()
private function set_coolify_variables()
{
$this->coolify_variables = "SOURCE_COMMIT={$this->commit} ";
$this->coolify_variables = '';
// Only include SOURCE_COMMIT in build context if enabled in settings
if ($this->application->settings->include_source_commit_in_build) {
$this->coolify_variables .= "SOURCE_COMMIT={$this->commit} ";
}
if ($this->pull_request_id === 0) {
$fqdn = $this->application->fqdn;
} else {
@ -2242,12 +2247,15 @@ private function generate_coolify_env_variables(bool $forBuildTime = false): Col
$coolify_envs = collect([]);
$local_branch = $this->branch;
if ($this->pull_request_id !== 0) {
// Add SOURCE_COMMIT if not exists
if ($this->application->environment_variables_preview->where('key', 'SOURCE_COMMIT')->isEmpty()) {
if (! is_null($this->commit)) {
$coolify_envs->put('SOURCE_COMMIT', $this->commit);
} else {
$coolify_envs->put('SOURCE_COMMIT', 'unknown');
// Only add SOURCE_COMMIT for runtime OR when explicitly enabled for build-time
// SOURCE_COMMIT changes with each commit and breaks Docker cache if included in build
if (! $forBuildTime || $this->application->settings->include_source_commit_in_build) {
if ($this->application->environment_variables_preview->where('key', 'SOURCE_COMMIT')->isEmpty()) {
if (! is_null($this->commit)) {
$coolify_envs->put('SOURCE_COMMIT', $this->commit);
} else {
$coolify_envs->put('SOURCE_COMMIT', 'unknown');
}
}
}
if ($this->application->environment_variables_preview->where('key', 'COOLIFY_FQDN')->isEmpty()) {
@ -2283,12 +2291,15 @@ private function generate_coolify_env_variables(bool $forBuildTime = false): Col
add_coolify_default_environment_variables($this->application, $coolify_envs, $this->application->environment_variables_preview);
} else {
// Add SOURCE_COMMIT if not exists
if ($this->application->environment_variables->where('key', 'SOURCE_COMMIT')->isEmpty()) {
if (! is_null($this->commit)) {
$coolify_envs->put('SOURCE_COMMIT', $this->commit);
} else {
$coolify_envs->put('SOURCE_COMMIT', 'unknown');
// Only add SOURCE_COMMIT for runtime OR when explicitly enabled for build-time
// SOURCE_COMMIT changes with each commit and breaks Docker cache if included in build
if (! $forBuildTime || $this->application->settings->include_source_commit_in_build) {
if ($this->application->environment_variables->where('key', 'SOURCE_COMMIT')->isEmpty()) {
if (! is_null($this->commit)) {
$coolify_envs->put('SOURCE_COMMIT', $this->commit);
} else {
$coolify_envs->put('SOURCE_COMMIT', 'unknown');
}
}
}
if ($this->application->environment_variables->where('key', 'COOLIFY_FQDN')->isEmpty()) {
@ -2331,7 +2342,11 @@ private function generate_coolify_env_variables(bool $forBuildTime = false): Col
private function generate_env_variables()
{
$this->env_args = collect([]);
$this->env_args->put('SOURCE_COMMIT', $this->commit);
// Only include SOURCE_COMMIT in build args if enabled in settings
if ($this->application->settings->include_source_commit_in_build) {
$this->env_args->put('SOURCE_COMMIT', $this->commit);
}
$coolify_envs = $this->generate_coolify_env_variables(forBuildTime: true);
$coolify_envs->each(function ($value, $key) {
@ -3344,100 +3359,121 @@ private function add_build_env_variables_to_dockerfile()
{
if ($this->dockerBuildkitSupported) {
// We dont need to add build secrets to dockerfile for buildkit, as we already added them with --secret flag in function generate_docker_env_flags_for_secrets
return;
}
// Skip ARG injection if disabled by user - preserves Docker build cache
if ($this->application->settings->inject_build_args_to_dockerfile === false) {
$this->application_deployment_queue->addLogEntry('Skipping Dockerfile ARG injection (disabled in settings).', hidden: true);
return;
}
$this->execute_remote_command([
executeInDocker($this->deployment_uuid, "cat {$this->workdir}{$this->dockerfile_location}"),
'hidden' => true,
'save' => 'dockerfile',
'ignore_errors' => true,
]);
$dockerfile = collect(str($this->saved_outputs->get('dockerfile'))->trim()->explode("\n"));
// Find all FROM instruction positions
$fromLines = $this->findFromInstructionLines($dockerfile);
// If no FROM instructions found, skip ARG insertion
if (empty($fromLines)) {
return;
}
// Collect all ARG statements to insert
$argsToInsert = collect();
if ($this->pull_request_id === 0) {
// Only add environment variables that are available during build
$envs = $this->application->environment_variables()
->where('key', 'not like', 'NIXPACKS_%')
->where('is_buildtime', true)
->get();
foreach ($envs as $env) {
if (data_get($env, 'is_multiline') === true) {
$argsToInsert->push("ARG {$env->key}");
} else {
$argsToInsert->push("ARG {$env->key}={$env->real_value}");
}
}
// Add Coolify variables as ARGs
if ($this->coolify_variables) {
$coolify_vars = collect(explode(' ', trim($this->coolify_variables)))
->filter()
->map(function ($var) {
return "ARG {$var}";
});
$argsToInsert = $argsToInsert->merge($coolify_vars);
}
} else {
$this->execute_remote_command([
// Only add preview environment variables that are available during build
$envs = $this->application->environment_variables_preview()
->where('key', 'not like', 'NIXPACKS_%')
->where('is_buildtime', true)
->get();
foreach ($envs as $env) {
if (data_get($env, 'is_multiline') === true) {
$argsToInsert->push("ARG {$env->key}");
} else {
$argsToInsert->push("ARG {$env->key}={$env->real_value}");
}
}
// Add Coolify variables as ARGs
if ($this->coolify_variables) {
$coolify_vars = collect(explode(' ', trim($this->coolify_variables)))
->filter()
->map(function ($var) {
return "ARG {$var}";
});
$argsToInsert = $argsToInsert->merge($coolify_vars);
}
}
// Development logging to show what ARGs are being injected
if (isDev()) {
$this->application_deployment_queue->addLogEntry('[DEBUG] ========================================');
$this->application_deployment_queue->addLogEntry('[DEBUG] Dockerfile ARG Injection');
$this->application_deployment_queue->addLogEntry('[DEBUG] ========================================');
$this->application_deployment_queue->addLogEntry('[DEBUG] ARGs to inject: '.$argsToInsert->count());
foreach ($argsToInsert as $arg) {
// Only show ARG key, not the value (for security)
$argKey = str($arg)->after('ARG ')->before('=')->toString();
$this->application_deployment_queue->addLogEntry("[DEBUG] - {$argKey}");
}
}
// Insert ARGs after each FROM instruction (in reverse order to maintain correct line numbers)
if ($argsToInsert->isNotEmpty()) {
foreach (array_reverse($fromLines) as $fromLineIndex) {
// Insert all ARGs after this FROM instruction
foreach ($argsToInsert->reverse() as $arg) {
$dockerfile->splice($fromLineIndex + 1, 0, [$arg]);
}
}
$envs_mapped = $envs->mapWithKeys(function ($env) {
return [$env->key => $env->real_value];
});
$secrets_hash = $this->generate_secrets_hash($envs_mapped);
$argsToInsert->push("ARG COOLIFY_BUILD_SECRETS_HASH={$secrets_hash}");
}
$dockerfile_base64 = base64_encode($dockerfile->implode("\n"));
$this->application_deployment_queue->addLogEntry('Final Dockerfile:', type: 'info', hidden: true);
$this->execute_remote_command(
[
executeInDocker($this->deployment_uuid, "echo '{$dockerfile_base64}' | base64 -d | tee {$this->workdir}{$this->dockerfile_location} > /dev/null"),
'hidden' => true,
],
[
executeInDocker($this->deployment_uuid, "cat {$this->workdir}{$this->dockerfile_location}"),
'hidden' => true,
'save' => 'dockerfile',
'ignore_errors' => true,
]);
$dockerfile = collect(str($this->saved_outputs->get('dockerfile'))->trim()->explode("\n"));
// Find all FROM instruction positions
$fromLines = $this->findFromInstructionLines($dockerfile);
// If no FROM instructions found, skip ARG insertion
if (empty($fromLines)) {
return;
}
// Collect all ARG statements to insert
$argsToInsert = collect();
if ($this->pull_request_id === 0) {
// Only add environment variables that are available during build
$envs = $this->application->environment_variables()
->where('key', 'not like', 'NIXPACKS_%')
->where('is_buildtime', true)
->get();
foreach ($envs as $env) {
if (data_get($env, 'is_multiline') === true) {
$argsToInsert->push("ARG {$env->key}");
} else {
$argsToInsert->push("ARG {$env->key}={$env->real_value}");
}
}
// Add Coolify variables as ARGs
if ($this->coolify_variables) {
$coolify_vars = collect(explode(' ', trim($this->coolify_variables)))
->filter()
->map(function ($var) {
return "ARG {$var}";
});
$argsToInsert = $argsToInsert->merge($coolify_vars);
}
} else {
// Only add preview environment variables that are available during build
$envs = $this->application->environment_variables_preview()
->where('key', 'not like', 'NIXPACKS_%')
->where('is_buildtime', true)
->get();
foreach ($envs as $env) {
if (data_get($env, 'is_multiline') === true) {
$argsToInsert->push("ARG {$env->key}");
} else {
$argsToInsert->push("ARG {$env->key}={$env->real_value}");
}
}
// Add Coolify variables as ARGs
if ($this->coolify_variables) {
$coolify_vars = collect(explode(' ', trim($this->coolify_variables)))
->filter()
->map(function ($var) {
return "ARG {$var}";
});
$argsToInsert = $argsToInsert->merge($coolify_vars);
}
}
// Insert ARGs after each FROM instruction (in reverse order to maintain correct line numbers)
if ($argsToInsert->isNotEmpty()) {
foreach (array_reverse($fromLines) as $fromLineIndex) {
// Insert all ARGs after this FROM instruction
foreach ($argsToInsert->reverse() as $arg) {
$dockerfile->splice($fromLineIndex + 1, 0, [$arg]);
}
}
$envs_mapped = $envs->mapWithKeys(function ($env) {
return [$env->key => $env->real_value];
});
$secrets_hash = $this->generate_secrets_hash($envs_mapped);
$argsToInsert->push("ARG COOLIFY_BUILD_SECRETS_HASH={$secrets_hash}");
}
$dockerfile_base64 = base64_encode($dockerfile->implode("\n"));
$this->application_deployment_queue->addLogEntry('Final Dockerfile:', type: 'info', hidden: true);
$this->execute_remote_command(
[
executeInDocker($this->deployment_uuid, "echo '{$dockerfile_base64}' | base64 -d | tee {$this->workdir}{$this->dockerfile_location} > /dev/null"),
'hidden' => true,
],
[
executeInDocker($this->deployment_uuid, "cat {$this->workdir}{$this->dockerfile_location}"),
'hidden' => true,
'ignore_errors' => true,
]);
}
}
private function modify_dockerfile_for_secrets($dockerfile_path)
@ -3510,6 +3546,13 @@ private function modify_dockerfiles_for_compose($composeFile)
return;
}
// Skip ARG injection if disabled by user - preserves Docker build cache
if ($this->application->settings->inject_build_args_to_dockerfile === false) {
$this->application_deployment_queue->addLogEntry('Skipping Docker Compose Dockerfile ARG injection (disabled in settings).', hidden: true);
return;
}
// Generate env variables if not already done
// This populates $this->env_args with both user-defined and COOLIFY_* variables
if (! $this->env_args || $this->env_args->isEmpty()) {
@ -3600,6 +3643,18 @@ private function modify_dockerfiles_for_compose($composeFile)
continue;
}
// Development logging to show what ARGs are being injected for Docker Compose
if (isDev()) {
$this->application_deployment_queue->addLogEntry('[DEBUG] ========================================');
$this->application_deployment_queue->addLogEntry("[DEBUG] Docker Compose ARG Injection - Service: {$serviceName}");
$this->application_deployment_queue->addLogEntry('[DEBUG] ========================================');
$this->application_deployment_queue->addLogEntry('[DEBUG] ARGs to inject: '.$argsToAdd->count());
foreach ($argsToAdd as $arg) {
$argKey = str($arg)->after('ARG ')->toString();
$this->application_deployment_queue->addLogEntry("[DEBUG] - {$argKey}");
}
}
$totalAdded = 0;
$offset = 0;

View file

@ -37,6 +37,12 @@ class Advanced extends Component
#[Validate(['boolean'])]
public bool $disableBuildCache = false;
#[Validate(['boolean'])]
public bool $injectBuildArgsToDockerfile = true;
#[Validate(['boolean'])]
public bool $includeSourceCommitInBuild = false;
#[Validate(['boolean'])]
public bool $isLogDrainEnabled = false;
@ -110,6 +116,8 @@ public function syncData(bool $toModel = false)
$this->application->settings->is_raw_compose_deployment_enabled = $this->isRawComposeDeploymentEnabled;
$this->application->settings->connect_to_docker_network = $this->isConnectToDockerNetworkEnabled;
$this->application->settings->disable_build_cache = $this->disableBuildCache;
$this->application->settings->inject_build_args_to_dockerfile = $this->injectBuildArgsToDockerfile;
$this->application->settings->include_source_commit_in_build = $this->includeSourceCommitInBuild;
$this->application->settings->save();
} else {
$this->isForceHttpsEnabled = $this->application->isForceHttpsEnabled();
@ -134,6 +142,8 @@ public function syncData(bool $toModel = false)
$this->isRawComposeDeploymentEnabled = $this->application->settings->is_raw_compose_deployment_enabled;
$this->isConnectToDockerNetworkEnabled = $this->application->settings->connect_to_docker_network;
$this->disableBuildCache = $this->application->settings->disable_build_cache;
$this->injectBuildArgsToDockerfile = $this->application->settings->inject_build_args_to_dockerfile ?? true;
$this->includeSourceCommitInBuild = $this->application->settings->include_source_commit_in_build ?? false;
}
}

View file

@ -15,6 +15,8 @@ class ApplicationSetting extends Model
'is_container_label_escape_enabled' => 'boolean',
'is_container_label_readonly_enabled' => 'boolean',
'use_build_secrets' => 'boolean',
'inject_build_args_to_dockerfile' => 'boolean',
'include_source_commit_in_build' => 'boolean',
'is_auto_deploy_enabled' => 'boolean',
'is_force_https_enabled' => 'boolean',
'is_debug_enabled' => 'boolean',

View file

@ -0,0 +1,30 @@
<?php
use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
/**
* Run the migrations.
*/
public function up(): void
{
Schema::table('application_settings', function (Blueprint $table) {
$table->boolean('inject_build_args_to_dockerfile')->default(true)->after('use_build_secrets');
$table->boolean('include_source_commit_in_build')->default(false)->after('inject_build_args_to_dockerfile');
});
}
/**
* Reverse the migrations.
*/
public function down(): void
{
Schema::table('application_settings', function (Blueprint $table) {
$table->dropColumn('inject_build_args_to_dockerfile');
$table->dropColumn('include_source_commit_in_build');
});
}
};

View file

@ -22,6 +22,14 @@
@endif
<x-forms.checkbox helper="Disable Docker build cache on every deployment." instantSave
id="disableBuildCache" label="Disable Build Cache" canGate="update" :canResource="$application" />
<x-forms.checkbox
helper="When enabled, Coolify automatically adds ARG statements to your Dockerfile for build-time variables. Disable this if you manage ARGs manually in your Dockerfile to preserve Docker build cache."
instantSave id="injectBuildArgsToDockerfile" label="Inject Build Args to Dockerfile" canGate="update"
:canResource="$application" />
<x-forms.checkbox
helper="When enabled, SOURCE_COMMIT (git commit hash) is available during Docker build. Disable to preserve cache across different commits - SOURCE_COMMIT will still be available at runtime."
instantSave id="includeSourceCommitInBuild" label="Include Source Commit in Build" canGate="update"
:canResource="$application" />
@if ($application->settings->is_container_label_readonly_enabled)
<x-forms.checkbox