CircleCI is currently investigating a security incident. We reached out to our customers using CircleCI as their development platform, but thought it important to share this information with the wider community.
Their official announcement can be read here, but the key takeaways are:
- The incident is currently being investigated, but took place as early as December 21st, 2022.
- Users should immediately rotate all secrets stored in CircleCI (this could be in project environment variables or in contexts).
- If your project uses Project API tokens, they have all been invalidated and will need to be replaced.
- CircleCI published an article to help their users walk through rotating their secrets.
While CircleCI is sharing information on the key compromise, it’s important to understand that a breach is a bit like an iceberg, with only a small portion of the total breach being visible. It’s completely reasonable to assume the attackers have compromised other systems in the network and CircleCI customer account information (secrets, source code, billing information, etc..) may also have been exposed.
Somewhat ironically, the timing of the breach seems to coincide with the “reliability update” pushed out on December 21st.
One security researcher has posted some of his investigation into related activity in his organization’s environment, including an IP address seen to be making “GetAuthorizationToken” API calls to the “ecr.amazonaws.com” service endpoint in a manner that seemed to indicate an interactive session.
Secrets are something of a necessary evil (a study in 2019 showed thousands of secrets leaked via GitHub projects every day) in development, but you can take steps to mitigate the security issues.
- Avoid using the same API keys in both staging and production environments.
- Ensure that temporary passwords used during testing do not get pushed to production.
- Scan your repositories for secrets (ShiftLeft can help with this) and determine if any you find are following security best practices. If you do need to use a hard coded secret, is it reachable by attackers?
Our users are of course welcome to reach out to us if they need any help modifying ShiftLeft jobs within their CircleCI environment.
