Using security metrics to increase confidence in the third-party ecosystem

The need for security trust

We rely on suppliers and third parties to deliver secure services. As third-party ecosystems become increasingly complex, it is imperative to ensure that we have the required level of security trust in relation to these parties. 

Third-party ecosystems are the backbone of any business. These living ecosystems ensure that products and services are efficiently and securely delivered from suppliers to customers. An inefficient and fragmented supply chain is a major hindrance to business operations. This makes it critical to monitor and optimise relevant supply chain metrics.

With some ecosystems being identical across organisations, there is likely to be considerable overlap and therefore opportunity to work together in the area of metrics. However, it is difficult to monitor the level of trust in supplier security, not only because trust is hard to define and objectively measure, but also because the metrics available in the marketplace are in silos and cannot easily be combined. 

The PCSI security metrics project

Metrics generated by supplier risk profiling tools and techniques can potentially be used to measure security trust, especially when combined with internal metrics. However, the lack of uniformity of internal and external metrics not only hinders the added value of risk profiling tools and techniques, but also reduces the possibility of these metrics being reused within organisations. This prompted an exploratory project by PCSI, which started by identifying the various approaches to tackling this challenge. This resulted in a model that makes it possible to combine different metrics with the aim of quantifying the overall level of security trust. A guiding principle was to increase security through collaboration by developing a unified model that will improve communication and thereby reduce risk levels associated with the supplier chain. This model will be trialled in a practical use case in 2022.

The need to reuse and combine existing metrics

Early on it became clear that the availability of metrics is not the limiting factor. Even when metrics are readily available, creating added value for supplier (trust) management that can be seamlessly integrated with internal risk frameworks is not standard practice. The Information Security Forum (ISF)⁴ published a research paper that describes the various types of metrics and suggests ways in which they can be used. However, it became apparent that, without clear guidelines and methods, these suggestions were not sufficient to enable the combination and application of these metrics with the aim of achieving continuous supply chain assurance.

A holistic view of supply chain security

The PCSI model was developed to provide a holistic view of supply chain security in a generic and scalable way. The project examined different concepts of trust, the processes already in place for supplier risk management in the partner organisations involved and the different metrics available and then developed a model to map metrics of different kinds from different sources in relation to specific security controls in a control framework. This approach allows organisations to form a more nuanced, unified and comprehensive view of their supplier risk profile.

The model incorporates the non-binary aspect of trust and focuses on specific control areas with the overall objective of providing input that supports dialogue between the cyber security ecosystem, risk profile suppliers, suppliers in general and their customers. This will increase the overall maturity in continuous monitoring of the entire ecosystem, improve confidence and strengthen cyber resilience.

The concepts of trust and confidence

When engaging in dialogue with other stakeholders, it is important to be aware that the terms ‘trust’ and ‘confidence’ are often used interchangeably. However, as the Trust, Confidence and Cooperation (TCC) model shows⁵ (see figure 1), trust is social and relational while confidence is instrumental and calculative. In defining security metrics we focused on instrumental and calculative variables and with this as a basis, continued to iterate on metrics in relation to confidence. The TCC model indicates that the social aspect of trust may influence confidence results. In our project we assume that this is organisation-specific. Although very relevant, it is not our primary focus.

Figure 1. The TCC Model of Trust, Confidence and Cooperation

Different aspects of confidence

While developing the model, it became clear that it was necessary to distinguish component elements of metrics. Rather than basing an affirmation on a metric, we use the model shown in figure 2 to distinguish the level of confidence for the various elements that result in the metrics. One of the aims of the project is to determine fit-for-purpose levels for each aspect that influences the overall confidence in metrics.

Figure 2. Model of confidence and uncertainty

We distinguish the following factors that contribute to overall confidence:

• Confidence in the result of the measurement 
• Confidence in the measuring method/procedure
• Confidence in the measuring instrument
• Confidence in the measuring organisation
• Confidence in the relevance of the scope of the measurement

A concrete example of this would be using a calibrated thermometer (instrument) to measure (method) the temperature (metric) in Groningen (scope of measurement). This results in a certain level of affirmation that the measured temperature is the actual temperature in Groningen at the time of measurement. The affirmation that this is the actual temperature in The Hague (relevance of the scope of measurement) is substantially lower, although it will give an indication. The first part of the PCSI model contains elements that express the relationship between aspects relevant for a metric and supports variations in confidence for each of these elements. The resulting metric indicates the level of trust. The second part of the model consists of modules that express the relationship between the various metrics. This allows for integration and flexible reuse of a number of different types of metrics available within an organisation.

The anticipated added value of the model

For metrics suppliers, the objective is to enable better positioning of existing metrics, thereby increasing their value. This depends on the possibility of unifying measurements and metrics. Maybe not for all metrics but at least for measurement methods used to produce common metrics. It is thought that this will enhance the added value of individual services while raising the maturity level of the domain of metrics suppliers as a whole.

For suppliers, the objective is to increase the transparency and usability of existing product and service metrics for reporting purposes. We believe that the use of this model-based approach helps identify metrics and measurements for reusability across organisations. This increases transparency for (internal) customers by making their considerations explicit and creates an opportunity to improve services while reducing waste in the third-party risk process.

For customers and/or users, the objective is to contribute to an informed risk statement that can be used as input for decision making by increasing the potential for reuse of metrics collected for a specific purpose, thereby making them more generically applicable. This will result in better metrics underpinned by a better understanding of levels of trust, while at the same time reducing waste in the third-party risk process.

Next steps

The model will be further developed into an interactive dashboard and trialled in a specific use case that focuses on one or more security controls, such as patch management. The PCSI members currently involved are large companies that are both service providers and customers of many suppliers in the Netherlands. To enable them to take the next step, they need additional partners who specialise in supplier risk management and trusted third parties who are willing to engage in open dialogue with all participants and scientists with the aim of finding ways to contribute to a safer society. 

If you would like to contribute to or discuss this PCSI project, please contact the project lead, Puck van den Brink: puck.vandenbrink@tno.nl. 

1 Such as Trust Centres and white papers on security, privacy and compliance. 
2 Such as ISO27001, SOC1 Type II, PCI DSS, HIPAA and BAA.
3 Such as data processing agreements, online service terms, EU Standard Contractual Clauses and service level agreements.
4 ISF. 2020. Continuous Supply Chain Assurance. Monitoring supplier security. Continuous Supply Chain Assurance: Monitoring supplier security - Information Security Forum
5 Earle, Timothy & Siegrist, Michael & Gutscher, Heinz. (2012). Trust, Risk Perception and the TCC Model of Cooperation. Trust in Cooperative Risk Management: Uncertainty and Scepticism in the Public Mind. 10.4324, 9781849773461.