Terraform State Management
Video reference: here
Check out the previous blog if not: S3 Bucket creation using Terraform.
Understanding Terraform State Updates
We will look into how Terraform updates its infrastructure.
Here, the DevOps engineer updates the desired state through terraform apply. The state file will keep track of the resources in the local system state to the state in actual state, and then it leads to the changes in the infrastructure.
Terraform maintains infrastructure by keeping the actual state synchronised with the desired state defined in configuration files.
Here, the DevOps engineer will be
Run
terraform applyand it updates any changes toterraform.tfstatefile.Terraform compares the current state with the desired configuration (actual state)
Only resources requiring modification are updated
We always make sure that Desired state == Actual state.
Primarily, we can’t just call the infrastructure directly to make changes. The terraform.tfstate file is easier to manage our infrastructure.
Also this file shouldn’t be open to publishing or exposing. This is kind of secret and contains a lot of confidential information.
The Terraform state file is a JSON document containing:
Resource metadata and current configuration
Resource dependencies and relationships
Provider information
Resource attribute values
The state file serves as Terraform's source of truth for tracking managed infrastructure.
State File and Remote Backend
Every time we perform a tf command (tf == terraform). Instead of checking the local state file, we can look at the changes in a remote backend. This can be any storage, no need to be S3, blob, or GCP cloud storage.
By default, Terraform will use the local state file as a single source of truth. Remote backend is a feature in Terraform. We can use it to trigger more resources and manage the infrastructure well.
Import points to remember:
Store the state file in a remote backend
Do not update/delete the file
State locking (If a process executes the infrastructure, never allow any other process to use the state file at the same time on the same infrastructure. Like a package manager in Linux. Mutual exclusion principle)
Isolation of statefile
Regular backup
State File Best Practices
Never edit state files manually: Manual edits can corrupt state tracking and cause infrastructure drift.
Store state files remotely: Local storage prevents team collaboration and creates single points of failure.
Enable state locking: Prevents concurrent modifications that could corrupt state or cause conflicts.
Backup state files regularly: State loss can make infrastructure unmanageable.
Use separate state files per environment: Isolate development, staging, and production states.
Restrict access to state files: State files contain sensitive data, including passwords, API keys, and connection strings.
Encrypt state files: Enable encryption both at rest and in transit.
Remote Backend Bene
Collaboration: Multiple team members can access the shared state safely.
State Locking: Prevents simultaneous modifications from multiple users or processes.
Security: Centralised access control and encryption capabilities.
Backup and Versioning: Automatic state versioning enables rollback if needed.
High Availability: Remote storage services provide durability and availability guarantees.
S3 - State Locking
Go to the docs. We will use the given example in the official documentation and modify the code.
But before we do this, we need to make sure that the bucket exists. Or else, we will get this error:
Error: Failed to get existing workspaces: S3 bucket "bucket-name" does not exist.When you runterraform init
Go to the AWS console, choose the location as us-east-1 and create a bucket you want. Create a main.tf file in your desired local system. Make sure the bucket you are storing the state file should is a different bucket from what you created.
# backend configuration
terraform {
backend "s3" {
bucket = "amal-s3-bucket-terraform30days-1"
key = "dev/terraform.tfstate"
region = "us-east-1"
encrypt = true
use_lockfile = true
}
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
resource "aws_s3_bucket" "first_s3_bucket" {
bucket = "amal-s3-bucket-terraform30days"
tags = {
Name = "My bucket 2.0"
Environment = "Dev"
}
}
Key Parameters:
bucket: S3 bucket name for state storagekey: Path within the bucket where the state file will be storedregion: AWS region for the S3 bucketuse_lockfile: Enable S3 native state locking (set totrue)encrypt: Enable server-side encryption for the state file
After that, you should run:
terraform init
This will output to:
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
....................
If it’s all good, run:
terraform plan
terraform apply
This will create a new bucket with the name we intended to create using Terraform. Now, there are two S3 buckets in our hands.
Work as a remote backend
Work as a storage bucket for operations
Also, now the state file will be displaced in the local system, and from now onwards all the operations will we perform be performed on the remote backend. No state file changes will be reflected on the local system.
Go to the backend bucket, and we can see a folder called dev and inside, we can see the adapted state file.
Backend Migration
# Initialize with new backend configuration
terraform init
# Terraform will prompt to migrate existing state
# Answer 'yes' to copy existing state to new backend
# Verify state is now remote
terraform state list
State Commands
# List resources in state
terraform state list
# Show detailed state information
terraform state show <resource_name>
# Remove resource from state (without destroying)
terraform state rm <resource_name>
# Move resource to different state address
terraform state mv <source> <destination>
# Pull current state and display
terraform state pull
Make sure to run the destroy command once you create the backend.
terraform destroy
Arigato!