Case Study: Risk Mitigation of Critical Data Loss

By Josh Stewart and Kevin Stout

Introduction

A commercial drug manufacturer in collaboration with a controls systems integrator recently implemented a stability monitoring enhancement project that required critical environmental data recorded by data-loggers to be stored in a 21 CFR Part 11 compliant manner.

The end-user chose instrumentation with the capability to provide data buffering to protect against critical data loss.  However, the buffering capability did not integrate “off the shelf” with the existing SCADA.  The buffered data from the loggers didn’t automatically populate data to the historian if a gap existed in the historian data.

The solution included software that polls the SCADA for gaps in data.  If a gap is found, the solution pulls data from the buffers and populates it into the SCADA system.  This provides redundancy and verification that critical data resides in a single repository, ensuring data integrity.  The customer relied solely on the existing SCADA, and used the most robust instrumentation to protect against critical data loss.

Architecture

The architecture consisted of 150+ temperature and humidity sensors mounted throughout the facility.  Each sensor was connected to a data logger, which read data from the instruments, buffered the data, and transmitted real-time data to an OPC server.  The OPC server transmitted real-time data to the existing HMI server for monitoring and to the historian server for data retention.  The OPC server was also the home of the custom application described within.

Figure 1 – System Architecture

Problem

The end-user chose to use the data loggers to buffer data locally at the environment sensors.  This selection allowed the customer to divert from the more typical control system implementation which would include programmable logic controllers and remote I/O nodes.  The data loggers chosen did not send buffered data automatically to the end-user’s existing SCADA system when data loss occurred.  If the end-user would be required to retrieve data from the instrument data loggers, there was no solution in place to insert the missing data from the data loggers into the SCADA system.

Unique Solution

The software solution developed was designed to actively poll the historian to find gaps in critical data.  This polling occurred on a configurable, timed interval.  A gap in historical data could occur for a variety of reasons, including power loss, communication loss, or servers not running.  Once a gap in data was identified, the software solution retrieved data from the buffers in the instrument data loggers and populated the gap with relevant data at the historian. If a critical data gap is detected and the solution is unable to retrieve data or unable to populate the data gap, a SCADA alarm is generated and relevant personnel are notified.  The software solution decision tree that highlights the functionality is detailed below.

Figure 2 – Solution Decision Tree

Value Add

The solution has been qualified, is now running, and has successfully inserted otherwise lost critical data.  Currently, power/communication could be lost for 2.8 months (worst-case scenario) and the provided solution would insert critical data for all 150+ sensors upon communication restoration.  This solution has allowed the end-user to implement a critical monitoring system that consists of instrumentation with local data buffering which connects directly to the existing SCADA system. This eliminated the need for multiple sources of critical data, as well as a programmable logic controller with remote input/output nodes spread across the facility.

For more information, click here to visit E-Volve Systems, a controls systems integrator specializing in industrial automation, engineering, and computer systems validation.

http://e-volvesystems.com/