Handling Secrets in Software Development

Security is not a product, but a process.

• Bruce Schneier, Secrets and Lies (2000)

Throughout my development career, I have tried to follow best practices and work in a secure, professional way. Handling secrets is only a small slice of the overall software development process, yet getting it wrong can cause dramatically outsized damage. It is a topic that deserves deliberate thought rather than ad hoc decisions made under deadline pressure. So I spent some time reflecting on everything I have learned, and here is what every developer should keep in mind.

Secrets leak. Not because developers are careless, but because the surface area is wide and the feedback loop is slow. A committed credential disappears into git history; a CI log line gets archived; a container image sits in a registry for years. Good secret management is not about being perfect, it is about reducing exposure time, detecting breaches early, and recovering quickly when they happen.

Local Development

Development environments are where discipline tends to slip first. Every developer needs credentials to run the application locally, and the easiest solution is a shared .env file passed around in a team chat or stored in a shared drive. Resist that habit.

Environment variables are the standard delivery mechanism for injecting secrets into an application at runtime. They are not a security boundary, they do not encrypt values, they are visible to any code running in the same process, and they can be inherited by child processes and subshells. Their value is isolation from source code, not protection of the value itself. Keep them in the right place and know their limits.

Use per-developer .env files that never leave the machine, backed by service accounts specific to the development environment with reduced permissions. Commit a .env.example file to the repository that contains all variable names alongside safe, non-functional placeholder values. That file is documentation. The real .env is listed in .gitignore and stays local.

Using production credentials in development should be treated as a policy violation, not a shortcut. Separate dev accounts limit the blast radius if a laptop is lost or a developer tool is compromised.

Repositories and Version Control

Never commit secrets to version control, even in private repositories. Private does not mean safe, repositories get forked, cloned, backed up, and eventually audited. A token that was committed and then deleted still lives in git history until the repository is deliberately cleaned or rebuilt from scratch.

Tools such as git-secrets, gitleaks, or pre-commit hooks can scan for credential patterns before a commit lands. If a secret is already in history, rotate the credential immediately and treat the old value as fully compromised regardless of whether anyone appears to have accessed it.

CI/CD Pipelines

Pipelines are high-value targets because they typically hold broad deployment permissions. Store secrets in the CI/CD platform’s native secret facility (GitHub Actions Secrets, GitLab CI variables, or equivalent) and restrict which pipelines, branches, and environments can access each value.

Mask all secrets in logs and never print them for debugging. Prefer short-lived tokens issued just-in-time over long-lived static credentials that accumulate over months. Split permissions by stage: build jobs, test runners, and deployment jobs should receive only what they specifically need. A compromised test runner should not be capable of pushing to production.

Keyless Authentication with OIDC

The best static credential is one that does not exist. Modern CI/CD platforms support OIDC federation, which lets a pipeline authenticate to cloud providers using a short-lived identity token issued by the CI platform itself, no long-lived access keys required. With GitHub Actions and AWS, for example, the workflow exchanges a GitHub-issued OIDC token for temporary IAM credentials scoped to the specific repository, branch, and job. The credentials expire automatically and are never stored anywhere. The same model works with GCP Workload Identity Federation and Azure Federated Identity Credentials. If your pipeline currently holds a static cloud access key, replacing it with OIDC is one of the highest-value security improvements you can make.

SSH Keys

SSH keys are credentials too, and they deserve the same discipline as any other secret. Every private key should be protected with a strong passphrase. Without one, a key file sitting on a developer’s machine or inside a container is a plaintext credential, anyone who can read the file can use it immediately.

Set an expiry on keys where the target system supports it, and prefer shorter validity windows over long-lived keys that outlive the project they were created for. Use ed25519 keys over the older RSA defaults; they are shorter, faster, and based on more modern cryptographic foundations.

On developer machines, use ssh-agent to avoid typing the passphrase repeatedly without sacrificing protection. Never copy private keys between machines or share them across team members. Each person and each automated system should have its own key pair, so individual keys can be revoked without disrupting anyone else.

SSH Keys in Pipelines

Pipelines that need SSH access, for deployment, repository cloning over SSH, or remote command execution, require special care. Store the private key in the CI/CD platform’s secret facility, not as a file committed to the repository and not as a plain environment variable printed in logs.

Generate a dedicated key pair for each pipeline or deployment target. Restrict that key on the server side using ~/.ssh/authorized_keys options such as command= (to limit what commands the key can run) and no-port-forwarding. Rotate pipeline keys on a schedule and immediately when a pipeline configuration changes hands.

Avoid passphrase-protected keys in pipelines unless the toolchain supports ssh-agent forwarding securely, since an unattended pipeline cannot prompt for a passphrase. Instead, rely on the platform’s secret injection and keep the key’s server-side permissions as narrow as possible.

Secret Managers and Vaults

Environment variables and platform secret facilities are useful transit mechanisms, but in production the source of truth for secrets should be a dedicated secret manager: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager.

Secret managers provide what flat files and environment variables cannot: fine-grained access control, a full audit log of every read and write, automatic rotation, dynamic short-lived credentials, and versioned secret history. An application retrieves the secret at runtime rather than having it baked into the environment at boot. If the application process is compromised, the attacker does not automatically have every credential, only those the process retrieved during its lifetime.

The pattern to aim for: the application authenticates to the vault using a platform identity (IAM role, Kubernetes service account, or cloud workload identity), retrieves only the secrets it needs, and never persists them beyond the current request or connection pool. The vault is the only place a secret lives at rest.

Docker and Containers

Do not bake secrets into Docker images. Build arguments passed as --build-arg and environment variables set during docker build can persist in image layers and appear in plain text in image history. Inject secrets at container runtime through the orchestrator’s secret mechanism, Kubernetes Secrets, Docker Swarm secrets, or a cloud-provider secret sidecar.

When running Docker Compose locally, source secrets from environment variables or a .env file that is explicitly excluded from source control, never from files committed to the repository.

Infrastructure as Code

Terraform state files are a common and underestimated leak vector. State files routinely contain database passwords, API keys, and private keys in plaintext, because Terraform must store resource attribute values to detect drift. Store state remotely in a backend that supports encryption at rest and access control, an S3 bucket with server-side encryption and a strict bucket policy, or Terraform Cloud, and never commit state files to version control.

For Ansible, use Ansible Vault to encrypt sensitive variable files at rest. Never pass secrets as inline --extra-vars on the command line; they appear in shell history and process listings.

Application Layer and the Database

Secrets that must be persisted in a database deserve their own protection. Never store raw API keys, tokens, or third-party credentials as plain text columns. For values that must remain retrievable, apply application-level encryption with the encryption key stored outside the database in a dedicated secret manager. For passwords, use a strong adaptive one-way hash such as bcrypt or Argon2.

Restrict access at the application level too. A read-only reporting service should not share the same database role or secret scope as the authentication service.

Frontend and the Browser

Any secret loaded into a browser is no longer a secret. This sounds obvious, but it is violated routinely: API keys embedded in bundled JavaScript, private tokens included in build-time environment variables that get inlined into HTML, or credentials fetched from a public endpoint and cached in localStorage.

The rule is absolute: secrets must never reach the client. Backend-for-frontend patterns, server-side API proxies, and token exchange endpoints exist precisely to keep sensitive credentials server-side while giving the frontend limited, scoped access tokens. Public-facing API keys, analytics tags, map service keys, are not secrets in the traditional sense, but they should still be restricted by HTTP referrer, domain allowlist, and rate limit so they cannot be abused if extracted.

Logging and Observability

Error monitoring tools, structured loggers, and observability platforms are another surface where secrets escape unnoticed. A database connection error that includes the connection string in its message, a stack trace that serializes a full request object containing an authorization header, or a JSON log event that dumps an internal config struct, all of these can push raw credentials into Datadog, Sentry, Elasticsearch, or similar systems that have very different access controls than the application itself.

Sanitize secrets before they touch a logger. Many frameworks support log scrubbing or field masking, configure them explicitly. In structured logging, prefer allowlisting the fields you log over logging entire objects. Treat third-party observability services as untrusted external systems: they should receive context and signals, never raw credentials.

Lifecycle and Rotation

Secrets have a lifecycle, and every long-lived credential is a liability. Prefer short-lived tokens wherever possible and establish automated rotation schedules for credentials that must persist. Verify that the application handles rotation gracefully without downtime before relying on it in production.

Build detection and response into the plan from the start: configure alerts for unexpected usage patterns, maintain audit logs, and document the revocation procedure. When a secret leaks, and eventually one will, the team should know exactly what to do within minutes, not hours.

If you think technology can solve your security problems, then you don’t understand the problems and you don’t understand the technology.

• Bruce Schneier

Good secret hygiene is not a one-time checklist. It is a practice maintained across every environment, every pipeline, and every member of the team.