Developer errors lead to long-term exposure of sensitive data in Git repos

Credentials, API tokens, and passkeys – collectively referred to as secrets – from organizations around the globe were exposed for years, according to Aqua Security’s latest research.

By scanning the most popular 100 organizations on GitHub, which collectively includes more than 50,000 publicly accessible repositories, researchers found active secrets from open source organizations and enterprises such as Cisco and Mozilla providing access to sensitive data and software. The exposed secrets could lead to significant financial losses, reputational damage, and legal consequences.

Phantom secrets in Git-based systems

Aqua Security’s research team, Aqua Nautilus, revealed that “phantom secrets” can persist in Git-based infrastructure used by most Source Code Management systems (SCMs), including GitHub, Gitlab, Bitbucket and others. This is due to the way in which even deleted or updated code commits are saved in those systems, such that even a one-time developer mistake can expose secrets to savvy threat actors over extended periods.

“Our findings are truly alarming, and it is crucial that everyone involved in software development grasps the seriousness of this issue,” says Yakir Kadkoda, Aqua Nautilus Lead Security Researcher.

“For years, we’ve been educating developers not to hard-code secrets into their code. Now it turns out that even doing this just once permanently exposes that secret – even when they thought it was deleted or overwritten. The impact of a sensitive data leak can lead to unauthorized access, compromised security controls and significant financial or reputational damage. This would be devastating,” Kadkoda continued.

Among the exposed secrets found by scanning open GitHub repositories were API tokens of Cisco Meraki and the Mozilla project. The Cisco security team confirmed the findings: “We discovered privileged Meraki API tokens used by some Fortune 500 companies. These tokens could allow attackers to access network devices, Simple Network Management Protocol secrets, camera footage, and more, serving as an initial foothold for the exposed parties.”

The Mozilla project acknowledged that “An API token for the Mozilla FuzzManager with read-write privileges” and that “an employee’s API token for sql.telemetry.mozilla.org was leaked”; both were assigned a “Critical” score. Not only does the FuzzManager allow access to many potential security vulnerabilities in Firefox and Tor, but the telemetry gave access to confidential information related to Mozilla products and business.

Additionally, Nautilus found an Azure service principal token belonging to a large healthcare company exposed in a Git commit. This token had high privilege and high access to obtain credentials to the internal Azure Container Registry, which could have led an attacker to perform a supply chain attack impacting the organization, and customers.

In all cases, the exposed secrets were immediately revoked.

Commit once, expose forever

While secure coding best practices already require that secrets should not be hard coded, many developers continue this practice. They rely on secrets scanning tools to ensure that such secrets are not pushed into production, and often re-commit the updated code without those secrets.

Phantom secrets exist because of underlying processes within Git-based SCMs, which cause code that was overwritten or deleted in repositories to remain accessible within the underlying system. Most secrets scanners only look at repos accessible via the Git clone command, which overlooks almost 18% of secrets.

“The findings once again reinforce the best practice that secrets should never be put into code, not even for testing purposes, and security teams must be able to monitor this,” says Amir Jerbi, CTO of Aqua Security. “The software supply chain is optimized for speed and convenience, but this cannot come at the expense of secure engineering practices.”