Lessons Learned from OT:ICEFALL – New Vulnerabilities and Insights on OT Security Design and Patching
In our final OT:ICEFALL report, Forescout Vedere Labs presents three new vulnerabilities and concludes the project after one year of research following the original disclosure.
The OT:ICEFALL research, including 61 vulnerabilities affecting 13 vendors, has yielded three key insights into the current state of OT product security:
- Vendors still lack a fundamental understanding of secure-by-design. Our research shows the continuing prevalence of insecure-by-design practices in OT products and highlights that existing security controls are often broken. We found recurring design issues that demonstrate a lack of understanding of basic security control design, such as plaintext and/or hardcoded credentials, client-side authentication, stateful control on stateless protocols, missing critical steps in authentication, broken algorithms and faulty implementations. In older product lines, some issues persist because of the need for backward compatibility, but some of these problems are also found on newer designs.
- Vendors often release low-quality patches. Incomplete patches can lead to the discovery of new vulnerabilities, exemplifying how a bad patch increases risk rather than decreasing it. This situation has previously been acknowledged in IT but is even more critical in OT, where security patches are harder to apply. Patches are often incomplete due to a lack of variant analysis and piecemeal fixes for vulnerabilities, instead of addressing their root causes.
- Vendors must improve their security testing procedures. The shallow nature of many vulnerabilities we found in the project casts doubt on the quality of the security testing these products currently undergo. Again, a possible explanation is that in some cases products and protocols must remain backward compatible with legacy designs. Notwithstanding, some vendors have a certified software development lifecycle, which leads us to wonder how the bugs were missed by those vendors in the first place.
Each of the points above reflects the posture of some vendors, but not necessarily every vendor affected by OT:ICEFALL.
Below, we summarize the new vulnerabilities and discuss the consequences of this research for OT security management. Full details are available in the technical report.
New OT product vulnerabilities
The table below summarizes the new vulnerabilities we are disclosing. CVE-2022-46680 is the last issue found in the original OT:ICEFALL research and was not initially made public at the request of the affected vendor. CVE-2023-1619 and CVE-2023-1620 are new findings on WAGO controllers using the popular CODESYS V2 runtime.
| CVE ID | Affected devices | Description | CVSS v3.1 | Potential Impact |
|---|---|---|---|---|
| CVE-2022-46680 | Schneider Electric ION and PowerLogic power meters | The ION/TCP protocol transmits a user ID and password in plaintext with every message. This allows an attacker with passive interception capabilities to obtain these credentials and authenticate to the ION/TCP engineering interface as well as SSH and HTTP interfaces to change configuration settings and potentially modify firmware. | 8.8 | Compromise of credentials |
| CVE-2023-1619 | WAGO 750 controllers | An authenticated attacker could send a malformed packet to trigger a device crash. After triggering the vulnerability, the affected device must be manually rebooted to return to its operating state. | 4.9 | DoS |
| CVE-2023-1620 | WAGO 750 controllers | Due to an insufficient session expiration, an authenticated attacker can crash an affected device by sending specific requests after being logged out. After triggering the vulnerability, the affected device must be manually rebooted to return to its operating state. | 4.9 | DoS |
Remediation and mitigation for CVE-2022-46680 are available through the vendor’s advisory. There was close collaboration between Forescout and Schneider Electric on CVE-2022-46680. The fix developed to secure this legacy protocol designed 30 years ago is a significant achievement and shows Schneider Electric’s commitment to adopt secure-by-design to protect existing customers. Remediation and mitigation for CVE-2023-1619 and CVE-2023-1620 are also available through the vendor’s advisory.
ION and PowerLogic power meters provide power and energy monitoring in sectors such as manufacturing, energy, water and wastewater systems. WAGO 750 is a line of automation controllers with variants supporting several different protocols, such as Modbus, KNX, Ethernet/IP, PROFIBUS, CANopen, BACnet/IP, DeviceNet and LonWorks, that are used in sectors such as commercial facilities, manufacturing, energy and transportation.
Although these devices are not supposed to be exposed online, we see between 2,000 and 4,000 potentially unique devices directly accessible when querying Shodan. The most popular exposed protocols are HTTP for WAGO controllers and Telnet for ION meters. WAGO controllers are most popular in Europe, while ION meters are most popular in North America.
On the Forescout Device Cloud – a repository of data from 19 million devices monitored by Forescout appliances – we see around 500 WAGO controllers and 500 ION power meters. Both types of devices are most commonly seen in manufacturing, but they are also popular in utilities and healthcare, in the latter case mainly for building automation.
“Shift left” to achieve OT security by design
OT:ICEFALL demonstrates the need for tighter scrutiny of, and improvements to, processes related to secure design, patching and testing in OT device vendors.
There are increasing discussions about the need for more vendor liability and better security by design and by default. One of the strategic objectives in the U.S. National Cybersecurity Strategy is to “shift liability for insecure software products and services,” which would entail legislation to establish liability of device vendors for insecure or vulnerable products.
Regardless of how these regulatory discussions evolve, one way to improve the state of OT security is to ensure that vendors address obvious design flaws such as the ones outlined in our research. Shifting security efforts to the left will also break the current culture of inefficient and disruptive “piecemeal patching” in OT.
After all, patches can be risky in OT. Fixing the patching process by ensuring that patches undergo strict security testing, with variant analysis, and are given priority over new product features would automatically decrease the number of new vulnerabilities.
For asset owners using insecure by design and vulnerable OT devices, deciding when to patch is a challenge. Currently, there is a push to focus on the likelihood of exploitation to drive this decision. Although likelihood is important to consider, it can also change fast, influenced by factors such as attacker motive and publicly available capabilities. For instance, CVE-2015-5374 was first reported without details in July 2015 before being used in December 2016 as part of Industroyer – but its details could be found in a presentation in May 2016, half a year before the attack. In March 2018, the exploit was integrated into Metasploit, rendering it available to the wide public. Similar Metasploit modules for other protocols and devices have recently been used by opportunistic attackers. Defense-in-depth is designed to deal with this kind of likelihood volatility, but if a defender banks on low likelihood alone, they might not be able to patch rapidly enough if that likelihood suddenly changes.
For all of these reasons, we recommend that asset owners carry out a careful, consequence-driven analysis of which vulnerabilities to patch, in which assets, rather than either blindly following vendor guidance or relying exclusively on compensating controls.