_scurity

SupplyChainSecurityCon @ Open Source Summit Europe 2022

Last week I attended the Linux Foundation’s Open Source Summit Europe here in my hometown of Dublin where I mainly camped out at the SupplyChainSecurityCon event. The talks focused on SBOMs, SLSA, VEX, provenance, attestation and signing. He’s what I picked up at the event over the 4 days.

image

I think everyone had a slide with this XKCD to drill home the problem the industry is facing.

The Keynotes

There were lots of great announcements at the keynotes like Europe setting up its own chapter of the Linux Foundation, Meta handing over PyTorch to the Linux Foundation, lots of great work around sustainability and Linus pushing to merge Rust into the Linux Kernel. Among the keynotes were two security focused talks from Christopher Robinson aka CRob and Paul Vixie.

image

Christopher’s talk “The Future of Open Source is Trust” focused on as open source software has evolved so has the threat landscape and highlighted the great work being done by the OpenSSF which has the following initiatives

Paul Vixie’s talk was a great history lesson entitled “ Revisiting Total Cost of Ownership for External Dependencies” where he spoke about how a patch was distributed which contained DNS back in the 80s by posting it to the Usenet and retrieved by an FTP server. People would just download the patch which eventually became abandonware “Then everything got big!” Anyone with a network device needed DNS. People just copied the code and made their own changes divorced from the upstream and all embedded systems are running a fork of a fork of a fork. Paul admits to writing bugs in DNS but also addressing them in the 1990s. “It’d be nice to have an internet when we were building one”. His key takeaways for today’s problems is that dependencies don’t just go one deep, orphaned dependencies are a huge cost and people should be monitoring their supply chains. We’re all in this together so let’s organize ourselves better where he’s working with the OpenSSF to build tools to make traceability simpler. He reckons once people start sharing SBOMs that’ll take a “decent whack” against the problem and shirk it by half. But we currently we don’t have a plan…

SupplyChainSecurityCon

  1. Terminology
  2. What’s in a Name? Vulnerabilities, SBOMs, and the Challenge of Software Identity - Justin Murphy, Department of Homeland Security (DHS), Cybersecurity & Infrastructure Security Agency (CISA)
  3. Trusted-SBOM: On the Critical Importance of Verifying SBOMs - Haoxiang Zhang, Huawei & Ahmed E. Hassan, Queen’s University
  4. Composing the Ultimate SBOM - Ivana Atanasova & Velichka Atanasova, VMware
  5. VEXing Open Source Security: Vulnerability Data for Everything - Andrew Martin, ControlPlane & Andres Vega
  6. Attesting Practically: Exploring the Glue Behind Secure Runtime Environments - Jim Bugwadia, Nirmata
  7. How to Identify and Avoid Cracks and Bumps in your Digital Infrastructure? By Considering the Health of Open Source in your Intake Process - Johan Linåker, RISE Research Institutes of Sweden
  8. Improving Package Repository Security – From White Papers to Practice - Jussi Kukkonen, Google
  9. Tracking Attackers in Open Source Supply Chain Attacks: The New Frontier - Tzachi ( Zack) Zorenshtain, Checkmarx
  10. BoF: State of SBOM Tools
  11. Other Tools

The problem with supply chains was almost always communicated through the SLSA diagram below, with the incentive to do something about the problem coming from the EO14028, and the solution being implementing SLSA and producing SBOMs. The SBOM analogy everyone reached for was it’s basically the ingredients label used to communicate what is contained in the food you’re buying - with VEX being the allergy advice.

image

Terminology

The following terms were spoken about a lot so its handy to have some definitions

SLSA - Supply chain Levels for Software Artifacts (SLSA) “salsa”. It’s a security framework, a check-list of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure in your projects, businesses or enterprises. It’s how you get from safe enough to being as resilient as possible, at any link in the chain.

SBOM - Software Bill of Materials is a nested inventory, a list of ingredients that make up software components.

VEX - Vulnerability Exploitability eXchange is a form of a security advisory with the goal allowing a software supplier or other parties to assert the status of specific vulnerabilities in a particular product. Allow both suppliers and users to focus on vulnerabilities that pose the most immediate risk, while not investing time in searching for or patching vulnerabilities that are not exploitable and therefore have no impact.

image

Provenance

Attestation - is an authenticated statement (metadata) about a software artifact or collection of software artifacts. A key feature of SLSA Level 2, which requires organizations to protect against software tampering and add minimal build integrity guarantees. It is also found in the NIST Secure Software Development Framework and ISACA’s Certified Information Security Auditor training.

What’s in a Name? Vulnerabilities, SBOMs, and the Challenge of Software Identity - Justin Murphy, Department of Homeland Security (DHS), Cybersecurity & Infrastructure Security Agency (CISA)

As this was a talk from DHS the room was packed. Justin spoke about the challenge of software identity and about the problem of discovering if you’re at risk when using a 3rd party where he highlighted four ways of achieving this and the problem with each one

1) Security Advisories - scale, vendor hasn’t issued one, information needed is not mentioned 2) Ask the maintainer - will you get a response 3) Own investigation - time-consuming, expensive, where to start? 4) SBOM - Do you have it, does it have enough information, are there tools available to process it

Justin said that at a minimum vendors should be declaring their top level dependencies. He spoke about how it took Executive Order 14028 to get organizations to take this seriously. One way they’re tackling this is using a machine-readable way of issuing security advisories with CSAF. He highlighted the problem where there are many ways of identifying a package - CPE, PURL, SWID tags, SWIHD and gitBOM and there needs to be a working and agreed upon way. He said SBOMs are coming, so vendors should be making plans. The overall problem is the number of vulnerabilities is increasing and there’s a lack of comprehensive global identification solutions that impedes the full benefits of SBOM and VEX. Help out here https://www.cisa.gov/sbom

Trusted-SBOM: On the Critical Importance of Verifying SBOMs - Haoxiang Zhang, Huawei & Ahmed E. Hassan, Queen’s University

This talk was about the quality of SBOMs as both Haoxiang and Ahmed felt that if the industry produces SBOMs that are of low quality then it won’t be adopted. Low quality includes both false-postives and false-negatives something that plagues the application security tool ecosystem. There were two measures they devized: correctness of SBOM and completeness of SBOM.

Correctness = # of packages found in software / # of packages declared in the SBOM

Completeness = # of files in software for which provenance can be determined based on SBOM / # of files in binary

They took a set of open source software and measured how correct the SBOMs were and the results weren’t great. This was a small set and they outlined the need to have a comprehensive study. They also highlighted the need for a community effort to achieve trusted SBOMs which included an agreed upon definition of SBOM quality metrics under correctness and completeness. There’s also a need for tools to measure the quality where organizations can quality stamp a vendor’s SBOMs “like how farms are verified organic”.

Composing the Ultimate SBOM - Ivana Atanasova & Velichka Atanasova, VMware

Ivana and Velichka made the case that the ultimate SBOM is one that’s composed of many “micro” SBOMs. That is each of the packages within your dependency graph has an SBOM and you combine these “micro” SBOMs into a single SBOM that describes your package and its dependencies. They spoke about the dependency graph nightmare and the current tooling that is out there to work with SBOMs

They highlighted that 90% of organizations have started their SBOM journey and the problem with not having a standard for those SBOMs

They listed the current standards

They highlighted the 3 things you need to generate an ultimate SBOM

They said the most comprehensive SBOM is from source or at build time, but you can’t always rely on getting them. Could an SBOM exist for each of the components like having micro SBOMs that describe each component but that might end up with 100s or 1000s+ of SBOMS. The good thing is that you can stitch them together into a single SBOM. SBOM composer takes micro SBOMs and combines them.

They closed by saying there’s both a technical challenge and a getting together challenge, every actor in the supply chain needs to generate their own SBOM.

Links https://automatecompliance.org/ Tools https://github.com/act-project/TAC SBOM and SLSA https://slsa.dev/blog/2022/05/slsa-sbom

VEXing Open Source Security: Vulnerability Data for Everything - Andrew Martin, ControlPlane & Andres Vega

Andrew and Andres explain that VEX is a standard for communicating the vulnerability risk of exploitation for software and is machine readable. It grew out of the NTIA multistakeholder process and the framing working group. Started in 2020 and continued into 2022 facilitated by CISA

They outlined 3 solutions for addressing vulnerable code?

They argue that an SBOM should be static and should be definitive. But a VEX document should be dynamic and exists fleetingly at the moment of creation. Should they both co-exists? They say that shipping Vex with SBOM makes no sense as its static vs temporal data where a VEX needs to be updated with new information as it emerges around certain vulnerabilities. Both formats should be signed but VEX must be API-retrievable. SBOM is trusted only when done at build time where at package and reverse engineering there’s less accuracy.

The outline the current issues

Attesting Practically: Exploring the Glue Behind Secure Runtime Environments - Jim Bugwadia, Nirmata

Jim presented on how you can improve your supply chain security using attestation. He spoke on how metadata are artifacts that provide information on artifacts or build environments such as provenance, SBOMs, VEX etc.

He defined attestation as signed statements from a trusted entity about the build environment and artifacts. It conveys that the build was executed in a trust environment and the artifacts comply with organizational and regulatory requirements. He highlighted tools from Sigstore that could be used to handle signing, verification and provenance - cosign, fulcio and rekor.

He closed with the following points

How to Identify and Avoid Cracks and Bumps in your Digital Infrastructure? By Considering the Health of Open Source in your Intake Process - Johan Linåker, RISE Research Institutes of Sweden

Johan raised something I didn’t really think about until attending his talk and that is how do you measure the health of open source projects. His motivation was to help organizations to adopt open source as some view it as sub par or insecure especially those in the public sector. His framework which is based on CHAOSS and consists of 107 metrics over 15 themes and includes things like how active is the communication on the project, how much conflict is there, how much diversity, what is the license, how capable are the contributors, the security, governance, processes etc. He plans on releasing this framework and future work includes

Improving Package Repository Security – From White Papers to Practice - Jussi Kukkonen, Google

Jussi spoke about how difficult it is to get everyone on the same page which makes securing open source supply chains that more difficult. The concept he spoke about was The Update Framework (TUF) used by Google, Amazon, DataDog and Sigatore and how it addresses the security of software update systems. Within the open source world he was involved in PEP458 and PEP480 which features concepts from TUF. However, these were documented in 2013 and 2014 where it took almost 10 years to get PEP458 into PyPI and he reckons PEP480 simply won’t happen. In reflection he said that the TUF specification should have considered repository operations and not only focus on the client and also should have involved those who are developing in the process - don’t just focus on attacks but more on workflows and processes.

With the assistance of his colleague he also gave a live demo of where they signed releases with Sigstore and showed how to achieve tamper proof builds with SLSA.

Tracking Attackers in Open Source Supply Chain Attacks: The New Frontier - Tzachi ( Zack) Zorenshtain, Checkmarx

Tzachi outlined a problem with package repositories but also the mindset of the security community when it comes to vulnerable code vs malicious code. He started by outlining a couple of high profile cases in npm and PyPI such as UAParser.js, coa, rc (all contained the same malicious code and occurred because of account takeovers being sold on an underground Russian marketplace), he also highlighted a big phishing campaign in August 2022 against PyPI contributors. Unlike the malicious attacks against contributes there also exists protestware like Brandons’s node-ipc, where he decided to add code to wipe people’s computers if they reside in Russia or Belarus. Another problem he raised is the speed at which attackers punch holes in defences such as MFA where EvilProxy added “Phishing as a Service” that can bypass MFA in Github, NPM and PyPI. He argues that no one is taking responsibility to stop these attackers. His team at Checkmarx along with Sentinel Labs have collaborated and tried bring attention, even going so far as to create websites around them

He also spoke about typosquatting and how successful it really is and with a tool called “Package Lab” you can simply steal the reputation of any other package on any registry which he calls “star jacking” where a malicious package called pampyio did just this where it replicated the code of pampy and made it look like it had the exact same reputation.

He closed by saying that we’ve been working with vulnerable code for years now but with malicious code we’re still deleting it as soon as its discovered. We need to recognize these are different problems, we should be analyzing and not wiping. He draws distinctions between vulnerable vs malicious

His summary

BoF: State of SBOM Tools

I joined this discussion thinking it was a talk but in fact it was a room full of SBOM pros. There were lots of ideas thrown around that I’ve tried to capture here. The idea of the discussion was to come up with a good bottom up approach to creating SBOMs - something that you can integrate into your build pipeline. For when the CISO or legal is like I need an SBOM now!

Why do we need SBOMs

Existing Tools

The room listed off some SBOM tools which are captured in the table below. It was funny that we’d spent a long time going over the list when someone shouted up there’s a Github with this info https://github.com/awesomeSBOM/awesome-sbom

Type: Produce, Consume, Transform, Share

Name Type Description
tern Produce, consume Generate SBOM for contiaers and filesystem
spdx-sbom-generator produce Generate SBOM for projects
syft Produce, transform Genreates SBOM for containers and filesystems
oss-review-toolkit Produce, consume Generates SBOM for source code, projects, and files
bom produce K8s SBOM generator
tejolote produce Consumes SBOMs and creates attestations
bom-tool produces C & C++
sbom-composer transform Combines SBOMs
sbom-tool produce
github/trustsource produce
cyclonedx produce Cyclonedx format
fossology

Gaps in functionality

Other Tools

Gitsign – Keyless Git Commit Signing - Billy Lynch, Chainguard Inc

Gitsign - Keyless git commit that tackles the A in SLSA where submit unauthorized changes to the code. Developed by Sigstore and uses an IODC provider to sign each git commit. Question was does this break the git ethos that you don’t need an internet connection to use git - yes it kind of does.

From Kubernetes With Open Tools For Open, Secure Supply Chains - Adolfo García Veytia, Chainguard

Tejote - A highly configurable build executor and observer designed to generate signed SLSA provenance attestations about build runs.

Do You Know What’s in the Software You Run? Introducing GitBOM - Nell Shamrell-Harrington, Microsof

gitbom - inspired how git works to produce a SBOM produced at build time

Conclusion

My main takeaway is that I was impressed by the work that the community has done to solve very complex problems. It’s obvious that it’s still early days, and we’ve a long way to go but things are progressing and moving in the right direction. There’s so many open questions around SBOMs such where do vendors store the SBOM, can consumers trust an SBOM generated by a vendor or a VEX document? What is the best tool to generate an SBOM and where in the build process to generate it? How can you consume an SBOM and how do you measure the correctness and completeness of it?

There’s also problems around the package registries not just technical but also aligning the people. Enabling the developers who devote their time into a hobby they adore to be more secure. Measuring the health of a project itself is not something I’ve ever done before but its definitely something that I think would help with the adoption of open source tools - I’ll be looking more closely into CHAOSS Metrics.

In the meantime I’m going to continue to play with SBOM tools, and hopefully we see its adoption take off and its potential fully realised.