4 Lessons from the CircleCI Breach

Published on 2nd March 2023 Author: Owen Wight

‘Twas the night before Christmas, and all through the business, no security professionals were sleeping. Not even the CISO…

The holidays are a happy season for many. For cybersecurity professionals, they have become times of anxiety. Not because of the holidays themselves, but because cyber criminals have learned that launching their attacks while security teams are away on vacation is profitable.

Whether you are a small business or a large company with robust cybersecurity controls, you are in danger because the large, critical vendors that handle your data are likely to come under attack at this time. The United States Cyber Infrastructure and Security Agency (CISA) even recommends increased threat monitoring over holiday weekends!

Once we get back after the new year, it’s only a matter of time before vendors start releasing blog posts and updates disclosing breaches they discovered over the break. CI/CD vendor CircleCI was one of the most serious breaches of 2022’s holiday season.

Here’s what happened in the CircleCI Breach:

According to CircleCI, an attacker compromised an employee’s machine in mid December – though the specific method used to compromise the machine was not shared. The attacker deployed malware to steal a session cookie from the employee’s laptop, bypassing the need to steal their password and defeating multi-factor authentication.

The compromised employee had a high level of privilege within CircleCI’s environment. They are allowed to generate production access tokens. Using this ability, the attacker could gain access to and exfiltrate customer data from several different places.

To CircleCI’s credit, they took immediate action by alerting affected customers, conducting an internal investigation, and implementing remediation measures to prevent a recurrence of such an incident. If you are using CircleCI you should have rotated any and all secrets stored in CircleCI, and reviewed internal logs for your systems for any unauthorized access starting from December 21, 2022 through January 4, 2023. If you haven’t, you need to do so ASAP!

This was an urgent action item because of the importance CircleCI has in many development pipelines.

What does CircleCI do?

But first, let’s quickly review the service CircleCI provides for those who are not involved in software development. Its unique role is tied to some of the more unpleasant surprises of the breach.

CircleCI is a cloud-based Continuous Integration/Continuous Delivery (CI/CD) platform that automates the software build, test, and delivery process. It is used to build software by a lot of companies.

The CI/CD process aims to streamline the software development lifecycle by making it faster and more efficient, while reducing the risk of bugs being introduced into the production environment. Vendors that offer a CI/CD solution will often be handling huge amounts of unsanitized code and data through its pipeline.

Despite the large amount of data they process, they are often overlooked as critical vendors because they don’t frequently have external facing components. As this breach demonstrates, this doesn’t mean they shouldn’t be considered when evaluating the risk of your vendors!

Every cybersecurity incident presents a learning opportunity. There are several lessons we can and should take away from CircleCI’s breach, that will better protect you from suffering in the event you or one of your critical vendors is hacked.

4 Lessons to Learn from the CircleCI Breach

There are four key lessons to be learned from the 2022 CircleCI Breach:

Beware of hidden vendors.
MFA and SSO aren’t infallible.
The holidays are a bad time for cybersecurity.
How a vendor responds to a breach matters.

1. Beware of hidden vendors

At Fractional CISO we have a daily process that involves monitoring and analysis of global security events so that we can quickly inform our clients should any be affected or at-risk.

I was the one performing the monitoring and analysis when news about the CirlceCI breach broke. I found that, when notifying clients about this breach, that a few who were using Circle for CI/CD didn’t have them as a critical vendor or in their vendor list at all!

Thus, the most critical lesson I took from this incident is the importance of visibility when maintaining and managing your 3rd party vendors. CI/CD tools feel more like infrastructure than a specific software vendor, so it blends into the background. If we take for granted the ease and convenience of using CI/CD tools to manage our software development, we may also overlook them and their potential risks when performing vendor management.

Regardless of how protected the data you are hosting is, it’s pivotal to note any and all vendors that have any visibility to sensitive or confidential information. It’s equally important to regularly review the security practices and compliance of any of these tools or services.

2. MFA and SSO aren’t infallible

Multi-Factor Authentication (MFA) and Single Sign-on (SSO) are some of the strongest and most widely used authentication security controls available. While these are often seen as foolproof security measures, all of the data that was exfiltrated during the CircleCI breach was supposed to be protected by both MFA and SSO.

One of the more common circumvention methods right now is to steal a session cookie, and that’s what happened here. The session cookie is the data saved on your browser to grant you access to an account once you authenticate. That cookie is a piece of data that can be stolen and placed elsewhere, allowing an attacker to bypass the need to steal a password or interact with MFA at all. This is exactly what happened in this breach, as a piece of malware was used to steal the cookie. On top of that, the compromised laptop was protected by antivirus software that also failed to spot the malware!

While CircleCI didn’t say exactly how the malware got onto the employee’s compromised machine, it’s possible that it happened through some sort of social engineering attack. A phishing email with a fake file that’s actually malware might seem old-school, but phishing remains the most common cyber attack today!

This is why simply implementing security controls is not sufficient for protecting yourself from the sophisticated threats that we are seeing today. It’s also vital to ensure that the people using or interacting with the controls are aware of them and trained appropriately.

3. The holidays are a bad time for cybersecurity.

It seems like we can’t have a holiday without a half dozen major breaches occurring.

The timing of the CircleCI breach serves as a reminder that cyber criminals love to strike during the holiday season. Most people will want to kick their feet up and toss security to the side when enjoying the holidays. However, companies need to remain vigilant and continue to monitor their environment more closely during this time.

4. How vendors react to cybersecurity incidents matters.

CircleCI deserves credit for taking immediate action to investigate and remediate the breach, along with their prompt and thorough notifications – which were updated continuously as more information became available.

Not all vendors are as proactive with this process. Compare CircleCI’s breach notification page with LastPass’s security breach notification. CircleCI was more forthright with the severity of the breach, gave frequent updates, and simple, easy-to-follow action steps.

LastPass meanwhile buried the severity of their breach (in which all of their customer’s password vaults were stolen), and didn’t bother notifying anybody of this until Thursday, December 22. This was the last business day before most people left for their Christmas holiday weekend!

LastPass, you were supposed to protect from attackers, not join them!

A vendor’s response to an incident is an indicator to how seriously they take cybersecurity, and should be a crucial consideration when selecting a 3rd party vendor.

Despite the fact that CircleCI faced this breach, we are not advising any clients to switch away from them. We are advising our clients who use LastPass to drop it and select an alternative password management vendor.

Conclusion on the CircleCI Breach

The CircleCI breach serves as a strong reminder that businesses need to be aware of the risks their 3rd party vendors expose them to. A strong vendor management program is essential to managing this risk, and is required for SOC 2 and ISO 27001 compliance to boot.

Every security breach has its own lessons to teach. Many of them are applicable to everybody. Failure to learn from them will leave you more likely to suffer the same sort of attack in the future. It is far better to learn from what happens to others, than to learn from experience yourself.

Want to get great cybersecurity content delivered to your inbox? Click here to sign up for our monthly newsletter, Tales from the Click.

Owen Wight

As a cybersecurity analyst, Owen helps companies secure themselves by collecting and evaluating security program data. In college, he researched and developed improvements for cyber ranges used in capture-the-flag competitions and hackathons. Owen has a Bachelor’s degree in Computer Science Engineering, with a focus in cybersecurity, from University of Connecticut.