Moving Process Data Across Segmented Networks

Summary

It is vital to keep our industrial processes secure. Governments at the highest level are weighing in on this as they recognize the link between industrial data security and national security. A recent White House memo to corporate executives and business leaders across the USA urges them to protect their companies against unwelcome intruders. One of the key action items is to segment networks, to isolate OT (operations technology) from IT.

The memo says, “It’s critically important that your corporate business functions and manufacturing/production operations are separated, and that you carefully filter and limit internet access to operational networks, identify links between these networks, and develop workarounds or manual controls to ensure that ICS networks can be isolated and continue operating if your corporate network is compromised.”

In Europe, the EU has gone further in its NIS 2 Directive, which will become mandatory in October 2024. This document specifies a number of basic security practices, including standards for networking data between the production and corporate levels of a company. Again, its recommended approach is to segment networks. This is spelled out in NIST document SP-800-82, which says: “The most secure, manageable, and scalable control network and corporate network segregation architectures are typically based on a system with at least three zones, incorporating one or more DMZs.”

However, digital transformation, Industrial IoT, and Industry 4.0 all underscore the need to use OT data outside the plant. Clearly there must be a secure way to get to it.

What happened to VPNs?

Until recently, many companies have used virtual private networks (VPNs) to access data on their OT networks. But this is what the recommendations to segment networks advise against doing. Rather than segmenting networks, VPNs join them. A VPN extends the IT security perimeter into the plant network, effectively connecting the two networks. Anyone hacking the IT network and accessing the VPN can use it to reach every other connected node, including those on a linked OT network.

Rather than using VPNs, and to eliminate the risks they pose, the OT and IT networks must be segmented with a demilitarized zone (DMZ) between them to allow the secure transmission of data.

Connecting through a DMZ

To segment networks, NIST SP-800-82 recommends a system of three zones: the control zone (OT), the corporate zone (IT), and the DMZ. Using a DMZ ensures that there is no direct link between corporate networks and control networks, and that only known and authenticated actors can enter the system at all. The NIST document describes the value and use of firewalls to separate these zones, and to ensure that only the correct data passes from one to the other.

However, implementing a DMZ in an Industrial IoT environment is problematic for the two most commonly used IoT protocols: OPC UA and MQTT. Getting data out of a plant through a DMZ typically requires two or more servers, chained together one after the other. The OPC UA protocol is simply too complex to reproduce well in a daisy chain like this. Information will be lost in the first hop. The synchronous multihop interactions needed to pass data across a DMZ would be fragile on all but the most reliable networks and would result in high latencies. And there would be no access to the data at each node in the chain. MQTT, on the other hand, can be chained, but it requires each node in the chain to be aware that it is part of the chain and to be individually configured. The Quality of Service (QoS) guarantees in MQTT cannot propagate through the chain, making data at the ends of the chain unreliable.

Since neither OPC UA nor MQTT is well suited to passing data through a DMZ, another approach is needed—one that integrates well with both of these protocols. Secure tunnel/mirroring middleware can do this and pass the data along daisy-chained connections, securely crossing a DMZ. The middleware connects to either MQTT or OPC UA at each end of the tunnel and mirrors the full data set through each server in the chain. It provides access to the data to registered, qualified clients at each node, as well as at the final destination. The mirroring capability of the tunnel/mirroring middleware guarantees consistency so that any client or intermediate point in the chain remains consistent with the original data source.

Securing open firewall ports

To provide the highest level of cybersecurity, data connections to the DMZ must not open any inbound firewall ports on either OT or IT. This is something that most industrial protocols were not designed to do. For example, OPC DA and OPC UA both use a client/server architecture, in which the client initiates a connection and the server accepts it. The server must listen for the connection on a TCP port, and that port must be open for incoming connections on the server’s firewall, and on any other upstream firewalls between the client and the server. To provide access to the data via OPC means opening at least one port on each of those firewalls, a significant risk.

Any open incoming firewall port constitutes a security exposure. Network attacks are not made on a port, nor are they made on a protocol—they are made on an application. The risk is that the listening application has an exploitable flaw that may or may not be due to the protocol implementation.

For example, an application may perfectly implement the OPC UA protocol yet be vulnerable to flaws in OpenSSL. The application may have no exploitable flaws in OPC UA or SSL but still have a buffer overrun in a string processing function executed on incoming data. No application is free of bugs. Every open incoming port is an opportunity for an attacker to probe an application for exploitable flaws and can give instant access to a network if one is discovered.

The best security practice is to not open any incoming ports at all in the secure network’s firewall. Properly designed tunnel/mirroring middleware can meet this requirement. As long as it can make an outbound TCP connection from the server side to the client side, then there is no need to open any inbound firewall ports. This eliminates the attack surface altogether.

Securing the tunnel/mirror connection

In addition to using a DMZ and keeping all inbound firewall ports closed, security best practices should be implemented in the tunnel/ mirror connection itself. For example:

• SSL encryption is essential, preferably with support for the most recent versions, such as TLS 1.2 and TLS 1.3. The system should use and enforce server certificates.
• User authentication is best applied on a per-connection basis. Each client program should be required to transmit a username and password, an access token, or a client certificate to authenticate.
• Any potential connection over plain-text transports should be avoided because unencrypted data, possibly including passwords, could be retrieved with a network packet capture program.

MQTT: advantages and drawbacks

MQTT is a lightweight messaging protocol originally developed for the oil and gas industry. Devices and programs called “clients” connect to a server called a “broker” and publish their data to it and/or receive data from it. The MQTT broker does not examine the data payload itself, but simply passes it along as a message from each publishing client to all subscribing clients.

One of the benefits of using MQTT is its ability to connect outbound through a firewall. This “push” architecture avoids the client-server architecture problem of OPC, allowing devices to make outbound connections without opening any inbound firewall ports. And, by using a central broker, it is possible to establish many-to-many connections, allowing multiple publishers to connect to multiple subscribers. MQTT thus solves some communication and security problems.

Despite these architectural advantages, MQTT has drawbacks that must be addressed if it is to be truly useful for OT/IT and Industrial IoT communication scenarios.

• MQTT is a transport protocol. It does not specify a payload format, which poses interoperability problems among applications from different vendors. Sparkplug has been created to address this issue and is starting to gain traction among vendors and users.
• An MQTT broker does not preserve time order among data values on different topics. This means that events in the physical system that occur in the order A then B then C could be delivered to an application as C then B then A, or any other ordering, which is an error in many industrial-control use cases and could lead to serious consequences.
• MQTT brokers do not have a way to indicate that a data source is disconnected. The consuming application cannot tell the difference between an old value from a sensor that has failed or a current value that has simply not changed recently. The “last will” mechanism in MQTT designed to deal with this requires unreasonable levels of coupling between the producers and consumers of data, resulting in duplicate configuration and increased integration and maintenance costs.
• MQTT quality of service (QoS) levels are not adequate for use across a DMZ, as they only apply to single connections. Since connecting through a DMZ requires multiple hops, the data producer will never know whether an MQTT message has reached a user, regardless of the QoS it selects. What is needed is guaranteed eventual consistency, so that each value transmitted by a producer will either reach the user or be superseded by a more recent value.
• MQTT brokers do not handle overload situations well. If data is arriving faster than a consumer can process it, it is likely that data consistency and time order between data producer and consumer will be lost.

To put it briefly, MQTT keeps inbound firewall ports closed, making it useful for some Industrial IoT applications, particularly “first mile” data collection from embedded devices. But its inability to guarantee consistency of data between producer and consumer, particularly through a DMZ, does not make it an ideal candidate for OT/ IT connections. Connecting MQTT to tunnel/mirroring middleware can significantly expand its flexibility and value.

Meeting the needs of the times

Connecting OT and IT offers enormous benefits for those able to make it happen. But above all, these connections must be secure. Business leaders and policy makers worldwide see that their process, manufacturing, and critical infrastructure is under attack as never before. Even as they acknowledge the value of accessing industrial data, they are issuing calls for urgent action to raise the levels of industrial system security. Without it, millions of dollars will continue to be lost or wasted in ransomware attacks and other exploits.

Most industrial cyberattacks are directed towards corporate IT systems that present a broad attack surface. OT systems need not share that risk. OT networks can have zero attack surface. There is no need to expose an OT system to the internet, or to join OT and IT networks, either directly or through a VPN. The NIST and NIS 2 guidelines are clear—segmenting networks and sharing data through a DMZ is far more secure. The technology is here, available now. Tunnel/mirroring middleware can be deployed to link to existing protocols and pass data securely and consistently from OT to IT across a DMZ.

_{This feature originally appeared in AUTOMATION 2023 Volume 2: Connectivity & Cybersecurity.}