End-to-End Secrets Security: Making a Plan to Secure Your Machine Identities

At the heart of every application are secrets. Credentials that allow human-to-machine and machine-to-machine communication. Machine identities outnumber human identities by a factor of 45-to-1 and represent the majority of secrets we need to worry about. According to CyberArk's recent research, 93% of organizations had two or more identity-related breaches in the past year. It is clear that we need to address this growing issue. Additionally, it is clear that many organizations are OK with using plaintext credentials for these identities in private repos, thinking they will stay private. However, poor hygiene in private code leads to public leaks, as we see in the news too often. Given the scope of the problem, what can we do?

What we really need is a change in our processes, especially around the creation, storage, and working with machine identities. Fortunately, there is a clear path forward, combining existing secrets management solutions and secret detection and remediation tools, all while meeting the developers where they are.

Making an end-to-end secrets security game plan

When we think of remediating the machine identity problem, also known as secrets sprawl, we can lay out the problem in a couple sentences.

"We have an unknown number of valid long-lived plaintext secrets spread throughout our code, configurations, CI pipelines, project management systems, and other sources, which we can not account for, and without a coherent rotation strategy. Meanwhile, developers continue to work with secrets in plaintext since it is a reliable, although problematic, way to get the application to work."

Thinking through this working definition, we can make a multi-step plan to address each concern.

1. Secrets Detection - Search through code and systems involved in the software development lifecycle to identify existing plaintext credentials, gathering as much information as possible about each.
2. Secrets Management - Accounting for all known secrets through a centralized vault platform.
3. Developer Workflows - Adjust processes and tools to make it easier to properly create, store, and call secrets securely.
4. Secrets Scanning - Continuously monitoring for any new secrets that get added in plain text.
5. Automatic Rotation - Regular replacement of valid secrets shortens their potential exploitation by malicious actors.

You can take this journey one step at a time, treating this as a phased rollout. Before you know it, you will be much closer to eliminating secrets sprawl and securing all your machine identities.

Finding your secrets

The first problem every team encounters when trying to get a handle on secret sprawl is determining what secrets they even have. A manual search effort to track down unknown secrets would quickly overwhelm any team, but fortunately, there are secrets scanning tools, such as GitGuardian's, that can automate this process and give insight into critical details. From a stable platform, you should provide a communication path to work with the developers for remediation.

Implementing a centralized secrets vault

Central to any good secrets management strategy is managing how secrets are stored and utilized. Enterprise vaults transparently allow you to account for all known secrets, encrypting them at rest and in transit. A good vault solution, including Conjure from Cyberark and Hashicorp Vault Enterprise. If all of your infrastructure is from the same provider, such as AWS or GCP, those are very good options as well.

Securing the developer workflow

Secrets management has historically been left in the hands of developers to figure out, leading to a wide variety of solutions like `.env` files and, unfortunately, hardcoding secrets into the codebase. Leveraging a centralized vault solution gives developers a consistent way to safely invoke the credentials from their applications throughout all environments. If you can offer a standardized approach that is just as easy to implement as what they are currently doing, you will find many developers will jump at the chance to guarantee their deployments are not blocked due to this security concern.

You will also want to consider shifting left. Command-line tools, such as ggshield, allow developers to add automatic Git hooks to scan for plaintext credentials before any commit is made. Stopping a secret from ever reaching a commit means no incident to deal with later and fixing the problem at the least expensive point in the software development life cycle.

Secret scanning at every shared interaction

You also need a way to account for the reality that sometimes accidents happen. Ongoing monitoring is needed to watch for any new issues that come from existing devs making a mistake or when new teams or subcontractors are hired who simply don't know your processes yet. Just as when first doing secrets detection, using a platform that gathers the information into a coherent incident will help you respond rapidly to these new issues. GitGuardian, for example, integrates at the code repository level to catch new plaintext credentials in seconds, automatically at every push or comment.

Short-lived credentials should be the goal of automatic rotation

If an attacker finds a valid secret, that makes their job a lot simpler, as they can just unlock any doors they encounter. If that same attacker finds an invalid secret, they can't do much with it. With a centralized vault in place, you can put auto-rotation plans in place. Most modern platforms and services have a way to generate new credentials through an API call and a way to invalidate existing secrets. With a little scripting, following one of the many guides put out by platforms such as AWS or CyberArk, it is possible to automate the safe replacement of any credential on a regular, even daily, schedule.

End-to-end secrets security requires a plan

The best time to address the issues surrounding end-to-end secrets security is right now. If you do not already have a game plan in place, today is the best time to start having those conversations. Start by asking questions like" "what secrets do we even have?" or "Do you have a vault in place?" Ultimately, we must empower developers with workflows and guardrails that let them focus on their development flow.

Keeping vigilant that new secrets are discovered and dealt with immediately is an ongoing process. it will take effort, including raising awareness and adopting the right processes and technologies, but any company can get a better handle on machine identities and secrets, end to end, across the organization.