St. Clair Township Crest
1155 Emily Street
Mooretown ON
N0N 1M0

Phone: (519) 867-2021
Email: webmaster@twp.stclair.on.ca

Office Hours
Monday to Friday 8:30 am - 4:30 pm
Menu

study_scada_communication_revised

study_scada_communication_revised

NorfolkControls Norfolk Controls Inc.
Innovation through automation 1063 Fertilizer Road, Simcoe ON, N3Y 4K6

July 20, 2021
Attention: Mr. Brian Black, P.Eng.
Township of St. Clair 1155 Emily St.
Mooretown, ON. N0N 1M0

Re: RFP 2020-653-1
Efficiency & Cost Saving Study Based on Modernization
of the Sanitary Pump Station SCADA & Communication System

Dear Mr. Brian Black,

On behalf of Norfolk Controls Inc. please accept our revised final report on the Efficiency & Cost Savings Study Based on Modernization of the Sanitary Pump station SCADA & Communication system.

Rev B: Revisions include the addition of items 8 through 10 in Table 3 – Annualized Cost Savings. Additionally, the cost saving calculations were updated to show greater detail in their formulation.

We look forward to your review and feedback on this final report and should you wish to discuss
any of our idea’s further, please do not hesitate to contact us.

Regards,

Norfolk Controls Inc.

Michael Blizman, P.Eng. Controls Engineer | Owner C: 226-931-0443
E: mblizman@norfolkcontrols.ca

NorfolkControls
Innovation through automation

Efficiency & Cost Saving Study Based on Modernization of the Sanitary Pump Station SCADA & Communication System

RFP 2020-653-1 – Final Report Submittal (Rev B)

Submitted by: Michael Blizman, P.Eng.
Date: July 20/2021

Prepared for: St. Clair Township
1155 Emily Street Mooretown Ont.
N0N 1M0

Table of Contents
List of Figures 4
List of Tables 5
1. Introduction 6
2. Current System 7
Asset Summary 7
Remote Terminal Unit 10
Telemetry 10
SCADA Server 11
Current SCADA Screen Examples 12
3. Pain Points & Efficiencies 14
4. Annualized Cost Savings 28
5. Risk 33
6. Final Thoughts 34

List of Figures
Figure 1 – USRobotics 56K Fax Modem 10

Figure 2 – Mactec MTC-COM 6 Serial Communication Aggregator 11

Figure 3 – SCADA Server 12

Figure 4 – App 721 SCADA Screen Example 12

Figure 5 – FMC 200 SCADA Screen Example 13

Figure 6 – MultiSmart SCADA Screen Example 13

List of Tables
Table 1 – Asset Summary 8
Table 2 – Pain Points & Efficiencies 14
Table 3 – Annualized Direct Cost Savings 28

1. Introduction

Norfolk Controls has been retained to generate a report that details the efficiency and cost savings that can be achieved by modernizing the Township of St. Clair’s current Supervisory Control and Data Acquisition (SCADA) & Communication Systems.
This report will focus on a number of areas that address not only the reliability of St. Clair’s current SCADA system, but also the functionality, security, and cost savings that can be realized by migrating towards a modern SCADA system.
A number of sub-topics will be analyzed from an efficiency, cost savings and functionality standpoint including:
• Reporting
• Scheduling
• Alarming
• Real-time Monitoring/Troubleshooting
• Controls Integration Support
• Software & Hardware Obsolescence
• Security

The benefits of migrating towards a modern SCADA system will ultimately ensure operators have the ability to reliably monitor and control remote assets in real-time and in a secure fashion. The significance of SCADA modernization can be summarized by some of the most important attributes of a modern system,

Modern SCADA is vendor neutral, scalable, secure, and fully upgradable

Through these attributes, cost savings and efficiencies can ultimately be realized. The importance of a universal platform that can be used to collect process data within an enterprise is an extremely powerful concept that can have far reaching business benefits. Interfacing with typical business applications such as Enterprise Resource Planning (ERP) or Preventative Maintenance software allows streamlining of manual processes, ultimately saving time and money.

There are many pain points associated with the current SCADA system and these will be addressed throughout this report, highlighting the cause, consequences and how modern SCADA systems can solve these issues. It’s important to understand that it’s very difficult to place an exact ‘cost savings’ value on many of these pain points as detailed troubleshooting hours for each pain point is not available. There is a general understanding that as a process becomes more efficient, cost savings are a natural by-product of the efficiency. This can manifest through a reduction in troubleshooting, hardware, support, and manual process costs.

2. Current System

The Township of St. Clair owns and operates a sanitary collection system that includes twenty-nine
(29) Collector Pump Stations & three (3) Main Pump Stations. These stations are monitored and controlled by an Aquaview SCADA system (by Flygt), that was last updated in March of 2012. This SCADA software is used by operations staff to continually monitor and control each pump station remotely from St. Clair’s Civic Centre, 1155 Emily Street, Mooretown Ontario.
The current Aquaview SCADA software has been discontinued by Flygt, and is no longer eligible for support from the manufacturer. Discontinuation of Aquaview software support has resulted in the inability to make any updates to the existing SCADA software, including:
• Operator screens and controls
• Alarms
• Trends
• Reports
• Configurations
• Security
• Making updates or modifications of any kind

In addition to discontinued support, the current Aquaview SCADA system lacks functionality typically found in modern SCADA systems that allow operators to utilize real-time control and monitoring functions of remote assets. Lift station pumps and chlorine analyzers are examples of equipment that could be monitored in real-time to ensure vital sanitary and drinking water infrastructure is operating correctly.

There are a number of manual processes that include reporting, scheduling and alarm management that require a form of manual intervention from operations personnel. Modern SCADA systems have the ability to automate many of these tasks resulting in efficiency and cost saving gains.

Legacy hardware issues including a Dell Server used to operate Aquaview SCADA software is no longer supported by Dell and there is difficulty in finding replacement parts. Coupled with an operating system that has surpassed the manufacturers ‘end of life’ date, the SCADA server presents a security risk to the municipality.

The current Remote Terminal Unit (RTU) platform used in each remote station is manufactured by Flygt. A number of stations currently utilize RTU’s that are no longer supported by Flygt and difficult to source replacement parts. Additionally, Controls Integrator support for these controllers is very limited as these are very specialized in nature and not widely used throughout other industries.

Asset Summary

An asset breakdown of each site is listed in Table 1 – Asset Summary.

Table 1 – Asset Summary

ID
Facility Name
Location
RTU Model Network Communications
Type
SWF-001 Clay Creek PS 2851 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-002 White Line PS 2998 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-003 Cundick Park PS 3229 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-004 Sombra Main PS 3538 St. Clair Parkway MULTISMART Cellular Modem (Flygt Cloud Trial) Main PS
SWF-005 Reagan Park PS 3617 St. Clair Parkway MULTISMART USRobotics 56K Fax Modem Collection PS
SWF-006 Sombra Lagoon Chem Feed 161 Smith Line MULTISMART USRobotics 56K Fax Modem Alum Station
SWF-007 Holt Line PS 3775 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-008 Lambton Baptist Church PS 3896 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-009 Leeland Gardens PS 3946 Seaway Road MULTISMART USRobotics 56K Fax Modem Collection PS
SWF-010 Concession 8 North PS 4077 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-011 Concession 8 South PS 4159 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-012 Queen Street PS 502 Queen St. APP 721 USRobotics 56K Fax Modem Collection PS
SWF-013 Port Lambton Main PS 4406
McDonald MULTISMART Cellular Modem (Flygt Cloud Trial) Main PS
SWF-014 Port Lambton Storm PS 4420 Hill St. FMC 400 USRobotics 56K Fax Modem Storm PS
SWF-015 Port Lambton Lagoon Chem Feed 371 Broadway MULTISMART Cellular Modem (Flygt Cloud Trial) Alum Station
SWF-016 Brander Park PS 4628 Old River Road APP 721 USRobotics 56K Fax Modem Collection PS

ID
Facility Name
Location
RTU Model Network Communications
Type
SWF-017 Riverside Drive PS 4648 Riverside Dr. MULTISMART USRobotics 56K Fax Modem Collection PS
SWF-018 Island View Moorings PS 4723 Riverside Dr. MULTISMART USRobotics 56K Fax Modem Collection PS
SWF-019 Maplewood Subdivision PS 4761 Old River Road MULTISMART Cellular Modem (Flygt Cloud Trial) Collection PS
SWF-020 Indian Road PS 4697 St. Clair Parkway APP 721 USRobotics 56K Fax Modem Collection PS
SWF-021 Beresford Street PS 379 Beresford St. MULTISMART USRobotics 56K Fax Modem Collection PS
SWF-022 Tom Street PS 481 Tom St. MULTISMART USRobotics 56K Fax Modem Collection PS
SWF-024 Industrial Park PS 426 Polymoore Dr. MULTISMART Cellular Modem (Flygt Cloud Trial) Collection PS
SWF-025 Brigden Main PS 2393
Courtright Line FMC 200 USRobotics 56K Fax Modem Main PS
SWF-026 Frog Point South PS 1801 St. Clair Parkway FMC 200 USRobotics 56K Fax Modem Collection PS
SWF-027 Frog Point North PS 1735 St. Clair Parkway FMC 200 USRobotics 56K Fax Modem Collection PS
SWF-028 Clairwood PS 230 Clairwood Crescent APP 721 USRobotics 56K Fax Modem Collection PS
SWF-029 Wilkesport PS 3241 Mill Street APP 721 USRobotics 56K Fax Modem Collection PS
SWF-030 Curran Avenue PS 260 Curran Avenue APP 721 USRobotics 56K Fax Modem Collection PS
SWF-031 Church Street PS 113 Church Street APP 721 USRobotics 56K Fax Modem Collection PS
SWF-032 Brigden Lagoon Chem Feed 2393
Courtright Line N/A (uses Brigden Main PLC) N/A (uses Brigden Main Modem) Alum Station
N/A Brigden Elevated Water Tower 1581 Mill St. N/A (comms to Brigden Main Via
Antenna) USRobotics 56K Fax Modem for bulk water only Water Tower

Remote Terminal Unit

Each station uses a Remote Terminal Unit (RTU), with the exception of Brigden Water Tower as it sends information to SWF-025 (Brigden Main PS) via radio communication antenna. The RTU’s located at each station are used to interface pump and sensor data to the SCADA system via telemetry signals sent over dedicated phone line modems.
RTU hardware for each station is manufactured by Flygt and there are four (4) different RTU models currently in production:
• APP 721 (x14)
• MultiSmart (x12)
• FMC 200 (x3)
• FMC 400 (x1)

Telemetry

Telemetry signals are sent between stations and SCADA via phone line and in some cases cellular modem. The cellular modems are currently being used in a trial/evaluation period with Flygt. There are two (2) telemetry methods currently in production:
• USRobotics 56K Fax Modem (x25)
• Cellular Modem (Flygt Cloud Trial) (x5)

Figure 1 – USRobotics 56K Fax Modem

There are four (4) dedicated USRobotics 56K Fax Modems located in IT’s server room that are used
for SCADA communications to each remote station:
• Modem 1 – Communicates with Non-MultiSmart RTU’s

• Modem 2 – Communicates with Non-MultiSmart RTU’s
• Modem 3 – Receives all RTU Alarms from each remote station
• Modem 4 – Communicates with all MultiSmart RTU’s

Modems within each remote station and those used in the IT server room, each use a dedicated phone line in order to establish communications. A Mactec MTC-COM 6 communication unit is used as a communication aggregator in order to interface serial communications with the SCADA server.

Figure 2 – Mactec MTC-COM 6 Serial Communication Aggregator

SCADA Server

Flygt’s Aquaview SCADA software runs on a Dell PowerEdge T320 desktop server located in IT’s
main server room, installed in 2012. SCADA applicable software packages include:
• Microsoft Windows Server 2008 R2
• Microsoft SQL Server 2008 R2
• Aquaview SCADA (version 1.60.03)

Figure 3 – SCADA Server

Current SCADA Screen Examples

Example SCADA screen associated with a remote station using Flygt’s APP 721 RTU.

Figure 4 – App 721 SCADA Screen Example

Example SCADA screen associated with a remote station using Flygt’s FMC 200 RTU

Figure 5 – FMC 200 SCADA Screen Example

Example SCADA screen associated with a remote station using Flygt’s MultiSmart RTU

Figure 6 – MultiSmart SCADA Screen Example

3. Pain Points & Efficiencies
The existing SCADA system has a number of pain points that can be solved by the majority of modern SCADA platforms. This section will analyze some of the critical short comings of the existing system and provide insight into how modern systems are able to solve these pain points.
Table 2 – Pain Points & Efficiencies

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
1 Alarming – Remote station communication failure alarms

– FREQUENCY: from Feb.2019
to Feb.2020, roughly 200 communication alarms have occurred

– DURATION: the duration of communication alarms varies depending on the root cause of the problem. For example, if the cause is a malfunctioning modem, typical configuration and installation requires minimum of 2hr. Modem replacements occur on average six (6) times per year. If the root cause is found to be a defective phone line, typical utility provider repair lead time is 24hrs. Many occurrences simply require a manual communication request from SCADA to re-establish communications – Phone line needs repair by utility provider

– Modem has malfunctioned

– RTU has malfunctioned

– Phone line is busy (this is a security risk, if a person was to dial the phone number of the modem, it will lock the phone line and render SCADA communications faulted) -Operator has no visibility into the affected stations status from SCADA screens. Operator no longer has control functionality of station

– Operator must stop current task (i.e. dig job, sewer flushing etc.) and check the affected station. Mis- allocation of personnel hours

– Operator must troubleshoot/diagnose station by completing a number of tasks to determine if problem is with the modem, phone line, RTU, or any other relevant piece of equipment

– If problem cannot be remedied, operators must make daily visits to the affected pump station to verify everything is working correctly. Visits would be spaced three (3) times per day; morning, mid- afternoon, and late night. Late night visits (after 11pm) typically require overtime at
1.5 times hourly rate. Weekends and holidays are 2 times hourly rate

– Often after a communication alarm has been remedied a number of queued alarms flood the alarm notification system.
This creates confusion for the operator as they are not sure which alarms are currently active and which ones have been resolved.
Requires investigation by the operator to verify and clear each alarm – Modern SCADA uses cellular, fiber, or cable networks for communications to remote assets. These communication infrastructures typically utilize the robustness of the internet for SCADA monitoring and control

– Updated communication hardware enhances bandwidth, security and reliability of the SCADA network

– Easily integrates with IT infrastructure where modern IT tools used for security, administration and performance monitoring can be deployed

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
2 – Lack of adequate alarms to notify operator of specific issues within a remote station.

– Current Wastewater alarms include: power failure (i.e. generator running), high level, high level float, pump overloads (P1 & P2), pump leakage, and communication failure

– Current Water Tower alarms include: high level, high pressure, low pressure, Radio Frequency (RF) link failure, sump failure, and communication failure

-Current Chlorine Analyzer alarms include: low chlorine, high chlorine, and communication failure – Due to lack of software support from Flygt, additional alarms that could provide both functionality and safety are not available to add to the current SCADA system.

– There are a number of alarms that are missing from the current SCADA system that should be present as a matter of safety (this is not an all inclusive list, these are some of the most obvious):

1) There are five (5) stations that have generator backups. Currently, there is no alarm to indicate the generator has failed to start when requested to do so

2) No alarm on station building louvers to indicate a failure to open status while generators are running

3) Many of the remote station electrical panels are located outside. These panels should be monitored for temperature, water ingress and panel door open/closed status for both security and safety purposes

4) No alarms present to indicate a pump contactor failed open or closed – Not having adequate alarming can pose operational and safety challenges to operators. If for example, building louvers are assumed to be opened, when they are in fact not, CO2 may accumulate in station buildings equipped with diesel or gas generators

– Operators may think pumps are on or off when they are not

– Electrical panel high temperatures may cause overheating or shorten lifespan of critical electrical components including RTUs.

– Water ingress has occurred in some outdoor electrical panels creating electrical hazards and potentially shortening life of affected equipment

– Outdoor electrical panel doors may be accidently left open or experience a security breach where they have been forced open by a bad actor – Modern SCADA systems have no limits for alarming and are not bound by a ‘standard’ alarm set. As systems scale and equipment added, modern SCADA can add any custom alarm easily and efficiently with virtually no limits

– Alarms can be customized to trigger custom functions or scripts based on any parameter the programmer can imagine. For example, if an alarm occurs that has not been acknowledged by an operator for any reason, secondary logic can be setup to perform another task, such as open a valve, start another pump etc. An unlimited amount of customization can be added to every alarm and applied globally across the system or in a fine grain fashion amongst select alarms

– Each alarm can be tracked and logged into any standard database. Metadata such as duration, average time to acknowledge, average time to clear, most frequent, longest duration, etc. can be tracked and logged to gain insight into the alarming system. This is very useful to understand alarms that frequency cause trouble and assist in prioritizing maintenance
tasks

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
3 – Rely on third party alarm notification provider, Pager Duty, to send text message alarms (SMS) to operators.

– Pager Duty subscription fee is $1500 annually

– There is a multi-stage process to send out SMS notifications to operators. When an alarm is received by the SCADA software a specially formatted email is generated (this special format was setup by IT department) and sent via the municipalities email server to Pager Duty. Pager Duty will then send an SMS to the on-call operator (they have an on-call schedule maintained by Chris
Westbrook) – This is a complicated method to send SMS messages to operators for alarm notification. Not only does it rely on a third-party alarm notification provider, it also relies on specialized knowledge from IT staff in order to maintain custom email formats specific to Pager Duty. – Recurring annual subscription fee

– Reliance on IT specialized knowledge and maintenance of custom email configurations and routing

– Maintenance of on-call schedule by single point of contact (i.e. Chris Westbrook) – Modern SCADA has the ability to utilize third party alarm notification services OR can manage alarm notifications independently. By allowing SCADA to manage alarm notifications sent via SMS, voice notification or email, St. Clair could save annual subscription fees

– St. Clair would save roughly
$1500 annually on alarm notification fees

– No custom email formats required to be setup and managed by IT
4 – SMS notifications are ‘one- way’ notifications, meaning operators can receive an alarm SMS notification, but cannot acknowledge the alarm within the SCADA system via SMS. – Aquaview SCADA does not have the ability to receive SMS alarm acknowledgement – When an operator receives an SMS alarm notification, they are required to go to the SCADA system and physically acknowledge the alarm. Remote alarm acknowledgement is not configurable with the current SMS notification setup

– Operator must pause their current task to physically go to the SCADA server and acknowledge the alarm – Modern SCADA can be configured to allow remote alarm acknowledgement in addition to many other features. Alarm management and control is a powerful feature and can be used to setup alarm notification hierarchy levels. For example, if an operator does not acknowledge an alarm within a pre-set amount of time, the alarm management system will notify the next person on the
hierarchy chain

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
5 Real-time Monitoring – The current Aquaview SCADA platform is not capable of real-time monitoring

– Aquaview SCADA can ‘call’ or request data from each remote station a maximum of five (5) times per day – Built in limitation of current Aquaview SCADA platform- Each station can be called based on fixed settings within Aquaview SCADA. Setting options are: 1, 3, or 5 times per day. – Historical data is stored in each stations RTU and only polled (collected) five (5) times per day for storage in an MSSQL database (located on SCADA server). For example, chlorine residual is not monitored in real-time, data is only brought into the SCADA system during one of the five polling times. The result of monitoring chlorine residual in this fashion is the inability for an operator to monitor live residuals

– There is a risk of losing chlorine residual history for periods of time when an RTU fails. As SCADA is not collecting residual data in real-time, it relies on the RTU to store the data until the next scheduled poll. If an RTU fails, the collected chlorine residual block is lost and there will be gaps in residual reporting to the regulatory body – Real-time monitoring and control are vital for operators to have actionable insight into their system. Modern SCADA can poll or request large amounts of data at very fast time intervals. Depending on the granularity required of the collected data, polling rates can be set in the millisecond range

– Often, when cellular data plans are in use, communication protocols such as MQTT, that are based on a ‘change of state’ model are deployed. This model relies on a system that sends data to SCADA only when there is a change in state/value thus reducing redundant data being sent over the cellular network.
This network setup is very efficient and has been shown to reduce data usage in real-time monitoring systems
significantly
6 – Lack of real-time monitoring puts operators in a situation of “Operating by Alarm” as real-time situational awareness cannot be realized – Built in limitation of current Aquaview SCADA platform – “Operating by Alarm” restricts operators’ ability to be pro-active when pieces of equipment begin to trend out of operational specification. Operators are only alerted to a problem after it has occurred, due to a lack of situational and contextual awareness of each remote station operating status – Modern SCADA can effectively monitor the live status of all remote assets and give the operator context to it’s operating state. For example, it would be beneficial to an operator if the graphic of a pressure sensor showed not only the live value of the sensor, but also the operating parameters of the sensor, such as low alarm setpoint, high alarm setpoint, and the normal operating range.
Adding these setpoints, in combination with ‘high performance’ SCADA graphic design techniques provide an operator with situational awareness and context as to the
performance of an asset

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
7 – Real-time trending is currently not possible with the current SCADA software

– Trending is limited to historical graphs based on the last data set received from the most recent SCADA polling event as described in ‘Current Pain Point’, ‘ID 5’ – Built in limitation of current Aquaview SCADA platform – Without the ability to monitor live trending of key system operating parameters such as flow rates, pressures, tank levels etc. operators are forced to infer system operating performance based on the last historical data set received

– The ability to make pro- active decisions based on real-time trending is not possible. If for example, chlorine residual levels are trending down, the operator will be un-aware of it until there is an active alarm to
indicate its below acceptable levels – Real-time trending allows an operator to understand both the past and current performance of an asset and make pro-active decisions before an alarm occurs. Modern SCADA excels at trending, database logging, and ad- hoc trend graphics used to analyze system data
8 – Dial-up faxmodems are inherently slow with maximum data transfers of up to 56kbps over phone lines. This slow speed limits the amount of data that can be transferred at a
single time – Built in limitations of modem hardware and dial- up internet access via Internet Service Provider (ISP) – Modem and dial-up internet service limit the capacity of the overall SCADA network in both retrieval and control functionality – Modern cellular, fiber, and cable gateways provide reliable and efficient network connectivity for todays modern SCADA.
9 Security – The Aquaview SCADA platform currently operates on a Microsoft Windows Server 2008 R2 operating system. This operating system is no longer officially supported by Microsoft as of January 14, 2020 – Operating system is at end of life – Regular security updates have officially ended from Microsoft for this operating system. Any security holes within this operating system will remain in place and can present an opportunity for a bad actor to exploit. – Version of Microsoft Windows Server needs to be upgraded to the latest version in order to maintain operating system security

– Modern SCADA is not locked into a specific operating system. Many can run on Windows, Linux or macOS. Depending on the SCADA platform, some can run clients on anything that runs Java, including
Raspberry Pi’s
10 – Database software is Microsoft SQL Server 2008 R2. This database version is no longer supported by Microsoft as of July 9, 2019 – Database software is at end of life – Regular security updates have officially ended from Microsoft for this database version. Any security holes within this database version will remain in place and can present an opportunity for a bad actor to exploit – Version of Microsoft SQL Server needs to be upgraded to the latest version in order to maintain operating system security

– Modern SCADA is not locked into a specific database. Many can run on IT’s preferred database including, MySQL, MSSQL, MariaDB, Oracle Express,
PostgreSQL, SQLite etc.

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
11 – The current Aquaview SCADA platform is no longer supported by the manufacturer Flygt – SCADA software is at end of life – Security updates or modifications of any kind are no longer supported by Flygt. Any security holes within this software version will remain in place and can present an opportunity for a bad actor to exploit – Many SCADA systems use support contracts as a method of upgrade protection for the end users. Typically, for a yearly subscription amount, the end user is entitled to free software upgrades on all purchased modules within the SCADA platform. The value of the subscription contract typically is based on the number and value of the platform’s modules installed

– As software bugs are found and new versions of SCADA platforms are released end users can continually maintain their SCADA platform at the newest release. There is no obligation to do so if a support subscription is not desired, although we highly
recommend subscribing

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
12 – The current Aquaview SCADA server is not being backed up, with the exception of the database that holds chlorine residual and alarming data – Automatic SCADA backups do not appear possible with the current version of Aquaview SCADA – A catastrophic failure of the Dell Server running Aquaview SCADA has the potential to lead to extended periods of downtime.

– St. Clair’s IT staff with the help of Chris Westbrook would need to commission new server hardware and install and re-configure a copy of the SCADA application – Modern IT practices have drifted towards virtualization where operating systems are ran within a virtual machine or hypervisor. These virtual machines contain everything from the operating system to the SCADA application. The advantage to a virtual machine is that it’s very portable. Once a machine is created and everything is configured, it can be easily copied or backed up. The idea is, if there is a catastrophic hardware failure, a backed-up copy of the SCADA virtual machine, can be loaded onto another computer or server and once it’s installed, it will behave 100% the same as it did on the failed server and retain all of the same configurations. This is very common within the IT world and many IT departments are utilizing virtual environments in one form or another

– From an operations standpoint, the advantage to a virtualized environment is the speed in disaster recovery. For critical water infrastructure it’s imperative to recover as quickly as possible from a catastrophic event such as a hardware failure. Virtual machines deployed within a SCADA system play an important role in maintaining a robust
recovery solution
13 – Chris Westbrook is currently St. Clair’s only staff member that has experience with configuration of Aquaviews SCADA platform – Chris has assumed this role in order to fill in Flygts support gap – If Chris goes on vacation, becomes ill, or leaves St. Clair’s employment, there will be no staff on hand that have direct experience in the maintenance or troubleshooting of the Aquaview SCADA system – It’s important to develop a relationship with a number of Controls Integrator firms that can provide support in addition to St. Clair’s staff to mitigate over-reliance on any single person

– Modern SCADA solutions are supported by large networks of Controls
Integrators

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
14 – Software audit logging does not exist or is not setup – Built in limitation of current Aquaview SCADA platform – If a change to SCADA’s setting parameters is made, there is no log of who made the change, what the change was, and when the change was made

– If any modification to St. Clair’s SCADA code base is made, there is no log of the specific changes – Modern SCADA platforms utilize auditing systems used to track and log changes made within the SCADA environment. Logging of code base changes, parameter changes, logins, etc. can all be logged in SCADA’s database

– Audit event logging can be used to monitor what changes are being made, by who, when and from what location the change
originated
15 Reporting / Historian – Report generation of chlorine residuals required for the Ministry of Environment, Conservation and Parks (MECP) is not automatic and requires manual generation by St. Clair’s IT staff

– Reports are parsed into excel format and require on average 1/2hr to generate by IT – Built in limitation of current Aquaview SCADA platform – Rely on specialized IT knowledge to query and export chlorine residuals into excel format

– Reports are manually generated requiring IT resources – Modern SCADA does not rely on specialized IT knowledge to extract data from a database and parse into excel. Reports can easily be generated in pdf or excel formats along with any required data filtering. This can all be done with the click of a button if manual generation is required, or can be done automatically on a schedule

– Reports can be scheduled for automatic generation at specific times and then emailed to the appropriate
people
16 – The current Aquaview SCADA platform can store a maximum of 400 days of alarm logs

– MECP requires minimum of five (5) years worth of alarm record keeping for wastewater systems (as per conversation with Chris Westbrook)

– MECP requires minimum of seven (7) years worth of alarm record keeping for drinking water systems (as per conversation with Chris
Westbrook) – Built in limitation of current Aquaview SCADA platform – Alarm record keeping may not be in compliance with MECP regulations*

*Norfolk Controls is not an expert in MECP regulations. Municipality should verify record keeping requirements – Modern SCADA systems place no limit on historical logging of any nature. From trending to alarm logs and everything in between, modern SCADA will not restrict data collection and can meet all regulatory requirements

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
17 – Reporting system being improperly used as part of operator’s daily routine to verify correct operation of pumping stations

-For example, at 7am each morning, the on-call operator will log into SCADA and generate a report to view the daily hours each pump has ran. If the duty versus standby pump hours are not roughly equal, it could mean there is a potential issue (i.e. check valve, clog etc.).

– Reporting systems are not designed as diagnosis or alerting systems – Updates to Aquaview SCADA’s alerting and diagnosis logic are not possible as Flygt has discontinued support – Manually having to verify each station’s pump runtimes can be prone to error and possibly be missed by operational staff

– The daily task of generating the pump hour report and analyzing it requires on average, 1/2hr of operation staff time each morning – Modern SCADA platforms utilize scripting languages to code specific logic required of the end user’s application. Scripting is a powerful tool that can be used to analyze real-time and historical data in any required context. For example, scripts could be coded to compare pump runtimes and send out an alarm or email to an operator if the runtimes are not roughly equal

– The main advantage to scripting is that it can eliminate a number of manual processes and automatically complete tasks with repeatability and
reliability
18 Scheduling – Chris Westbrook is St. Clair’s sole staff member that has the credentials to generate and update the on-call schedule used by Pager Duty for direction of SMS alarms

– Initial schedule generation for the year requires roughly 8hrs of Chris Westbrooks time- Scheduling updates require roughly 15min. For example, when an operator is absent and the schedule needs to be updated so the correct operator on-call receives SMS alarm messages, Chris Westbrook must spend 15min. to
update the schedule for Pager Duty – Additional personnel have not been trained to perform this task

– Scheduling process is in- efficient – If Chris goes on vacation, becomes ill, or leaves St. Clair’s employment, there will be no staff on hand that have credentials or experience to create/modify the on-call schedule used by Pager Duty

– Time required to create and update the on-call schedule is tedious and overly reliant on Chris Westbrook – Modern SCADA can manage any type of ‘on-call schedule’ for system operators. Each operator could be given the ability to modify their on-call schedule as required, thereby reducing reliance on a single administrator

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
19 Hardware – Dell T320 desktop server is obsolete and is difficult for IT to source replacement parts

– Server is currently showing a High Temperature Mother Board Fault on it’s front panel diagnostics display – Server was installed in 2012 and has reached end of life

– Server motherboard appears to be at or has reached a high temperature state – If server hardware failure occurs, may cause excessive SCADA downtime until a solution can be sourced by IT

– Typically, obsolescence hardware is sourced at a premium price point as supplies may be limited in secondary markets (depending on the failed hardware component)

– Server motherboard may be close to reaching a catastrophic event whereby the component fails. IT may have difficulty sourcing a
replacement – Large scale modern SCADA solutions often run on IT server infrastructure in a virtualized environment. This eliminates the need for and costs associated with a dedicated SCADA server and creates an efficient environment for IT to manage and secure the SCADA system
20 – Eighteen (18) out of thirty
(30) Remote Terminal Units (RTU) are obsolete and no longer available from the manufacturer – Obsolete hardware from manufacturer – Manufacturer no longer supports legacy units

– Replacement parts must be sourced in secondary markets – RTU’s should be replaced with updated and supported models

– If greater functionality and efficiency is desired, upgrading to Allen Bradley Programmable Logic Controllers (PLC’s) would provide a more modern system with vast Controls Integrator support networks. PLC’s have greater programming flexibility than RTU’s and would ensure future system additions
could be easily added
21 Controls Integration Support – Support for both Aquaview SCADA and RTU’s manufactured by Flygt is very limited. Very few Controls Integrators are familiar with Flygt software/hardware as it’s very specialized for the water industry and cannot be used in other industries – Flygt software/hardware is specialized and not commonly used by Controls Integrators – Obsolete software/hardware is currently not supported by Flygt or third-party Controls Integrators

– Specialized software/hardware that relies on the manufacturer only for support may not be cost effective. With a limited knowledge base within the Controls Integration community, competitive bids for support cannot reliably be obtained

– Chris Westbrook is the municipalities sole knowledge base for replacement and configuration of both
SCADA and station RTU’s. – Modern SCADA systems are not industry specific, rather universal platforms used across many different industries. Universal acceptance ensures a wide range of Controls Integrator expertise is available to support these install bases

– Competitive bids can be tendered for upgrades or maintenance tasks as Controls Integrator support networks for universal platforms is large

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
22 General Functionality – Reliability of the SCADA system as a whole appears to be lacking. Alarms frequently are lost and not received by the on-call operator

– Remote station serial communications are slow with limited capacity

– Custom logic to monitor or control remote stations cannot be added

– Trending options are very limited and do not provide operator with ability to create ‘ad-hoc’ trends to assist in troubleshooting

– Reporting options are limited and must be manually ran without the ability to substantially generate custom reports among all assets

– Graphics, screen navigation, and overall look and feel of Aquaview SCADA is dated and does not adhere to modern day high performance design standards – Built in limitation of current Aquaview SCADA platform

– Built in limitation of remote station RTU and HMI’s

– Built in limitation of communication network

– Built in limitation of SCADA server and network setup – Missed alarms do not alert operator to actual conditions at remote stations. As an example, pumps may be faulted and need manual intervention and the operator is not aware.

– Missed alarms increase public health risk under low chlorine conditions (outside of regulatory requirements) that may lead to in- sufficiently treated water consumed by residents. May also lead to regulator fines

– Public health risks associated with basement sewage backups caused by pump malfunctions not being relayed to operations staff reliably. These backups occur at least once per year and result in insurance claims. There have been claims upwards of $150k to date.

– Not adhering to modern day, high performance SCADA design standards results in operator screens that can be difficult to interpret and navigate. Situational awareness is
greatly affected by lack of available graphical objects – Modern SCADA is robust, efficient, secure, scalable and unlimited. There are multiple components to a modern SCADA system including network infrastructure, software applications, hardware infrastructure and edge device infrastructure (RTU’s or PLC’s). Each component plays an important role in the modern SCADA system

‘- Modern SCADA is reliable which in turn mitigates risk associated with critical system functions such as chlorine dosing and pump monitoring

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
23 – Aquaview SCADA lacks functionality to interface with third party software. For example, St. Clair currently uses Cartegraph to create work orders, track and bill hours against projects etc. Aquaview SCADA cannot be interfaced with this software to automatically create a work order based on a custom set of parameters (i.e. a specific alarm could be used to trigger a work order) – Built in limitation of current Aquaview SCADA platform – Efficiencies that could be gained via automatic work order creation is an opportunity lost – Modern SCADA systems have the ability to interface with many different third- party applications via different communication interfaces. Some SCADA platforms use ‘Web Services’ as a method to communicate with compliant third party applications, such as ERP systems.

– As an example, St. Clair’s Cartegraph software has the ability to receive emails or SMS commands that could be used to generate work orders based on any number of conditions present within SCADA. If a pump fails, a work order could automatically be generated and the
appropriate staff resources allocated
24 – Aquaview SCADA is not modular in nature where features can be added as the system grows or more functionality is desired – Built in limitation of current Aquaview SCADA platform – Not being able to ‘add-on’ additional features to the Aquaview SCADA severely limits its capability and functionality. For example, the inability to add additional communication protocols, mobile functionality, advanced database logging, reporting, trending etc. – Modern SCADA platforms are typically modular in nature allowing great flexibility when designing a SCADA system. End users have the ability to customize a system that utilizes features they need, and not the ones they don’t. This creates cost savings as you only purchase your required modules. In the future as the system grows, new modules can be added at any time that simply ‘plug in’ to the SCADA platform creating a fully scalable architecture. There are virtually no limits to
what modern SCADA systems can do

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
25 – Aquaview SCADA does not use modern IT communication protocols that ensure high speed and security. – Built in limitation of current Aquaview SCADA platform – Limited bandwidth available for real-time monitoring (in addition to Aquaview SCADA not able to perform real-time monitoring)

– Communication protocols are not encrypted which may result in security vulnerabilities – The standard communication protocol to remote assets has emerged as MQTT over the last number of years. MQTT is often used in the ‘Internet of Things’ (IOT) devices. This modern protocol has become an industry standard in both the industrial and IT spaces.
Enterprises such as Amazon Web Services, Microsoft Azure and Google Cloud services use MQTT and so do modern SCADA systems. It’s a very robust, reliable, and secure protocol based on modern IT practices

– There are modern SCADA systems available that use bank level security encryption in their communication networks to ensure all data remains
secure
26 – Aquaview SCADA is not optimized for use on mobile phones or tablets – Built in limitation of current Aquaview SCADA platform – Lost efficiency opportunity to monitor and control remote stations via mobile phone or tablets – Modern SCADA is designed with mobility in mind. Many different SCADA vendors offer software solutions that make programming industrial mobile applications simple and standard. Mobile and tablet applications are typically built in HTML5 and run natively in any major browser

– Allowing an operator to monitor and control assets from a mobile platform can create many efficiencies. Alarms can easily be received and acknowledged remotely, thereby not taking an operator away from his/her current task unnecessarily. Asset statuses such as pump status, level indicators, pressure indicators etc. can all be monitored remotely giving operators flexibility throughout the day as they no longer need to visit a central SCADA workstation
to check up on things

ID Category Current Pain Point Cause of Pain Point Consequence of Pain Point Modern SCADA Solution
27 – Remote station Human Machine Interface (HMI) screens are standalone systems and cannot be remotely updated or
customized – Built in limitation of current RTU’s – Lost efficiency opportunity to administer and make updates to remote station HMI’s – Modern HMI screens are customizable and can be administered remotely over the SCADA network

4. Annualized Cost Savings

Migrating to a modern SCADA system will ultimately lead to cost savings. These savings are the result of efficiencies, both direct and indirect and can be realized throughout all aspects of a modern SCADA system.
There are numerous manual tasks performed by operators with the SCADA system that incur hourly costs. Reporting, scheduling, troubleshooting, and general maintenance all require time. Many of these tasks are difficult to quantify a direct cost as the number of hours spent is not tracked or known. In addition to direct cost obscurity, there are opportunity costs that can be difficult to quantify. For example, if an operator is pulled away from a task such as sewer flushing or a dig job, there is an opportunity cost associated with having to stop mid-way through a task.
Does interruption of an unrelated task to troubleshoot the SCADA system result in overtime expense? There are times it does, and times it does not. These are indirect/opportunity costs associated with an unreliable SCADA system and should be considered along side quantifiable direct costs.
Table 3 – Annualized Direct Cost Savings list some direct costs associated with the existing SCADA system and compares them with a modern system to estimate annual cost savings.

Table 3 – Annualized Direct Cost Savings

ID
Current Annual Cost
Modern SCADA Annual Cost Annual Net Savings (Annual Net Cost Increase)
1 – Third party SMS alarm notification company annual subscription:

– $1500 annually – Modern SCADA does not require the use of third party for SMS notifications:

– $0 annually – Approx. $1500
2 – Dial-up modem replacements occur on avg. six (6) per year

– Hardware Cost (each): Approx. $100

– 2hr of installation @ $70.56/hr +
$18.50/hr vehicle costs = $178.12

– Avg. annual modem replacement cost (each) = Hardware Cost + Installation Cost
$100 + $178.12 = $278.12

– Total avg. annual modem replacement cost (qty 6) = Avg. annual modem replacement cost (each) x Number of Avg. yearly failures (six)
$278.12 x 6 = $1668.72 – Modern industrial gateways typically are designed with mean-time before failures anywhere from 100k to 1mil hours. The likelihood of having to replace industrial grade units over a 10 year period is quite small.

– Estimated current cost of cellular industrial gateways:
$1500

– If we take an aggressive failure rate of 10 years, the annualized cost of each cellular gateway is approx.
$1500 / 10 = $150 per year

– 2hr of installation @ $70.56/hr + $18.50/hr vehicle costs
= $178.12 (annualized over 10 years, $17.81)

– Avg. annualized cellular gateway replacement cost= (Gateway Annualized Cost + Annualized Installation Time) x six (6) units
($150 + $17.81) x 6 = $1006.86 for replacement of six ($167.81ea) – Approx. $662

ID
Current Annual Cost
Modern SCADA Annual Cost Annual Net Savings (Annual Net Cost Increase)
3 – There are four (4) dedicated USRobotics 56K Fax Modems located in IT’s server room that are used for SCADA communications to each remote station. Each of these modems requires a dedicated phone line/number that occurs annual costs

– Total monthly phone line charges for the SCADA system add up to $1485.85. There are a total of 29 phone lines currently in use among remote stations and the four dedicated modems in IT’s server room.

– Avg. monthly cost of each phone line
= Total Monthly Phone Charges / Total Number of Phone Line:
$1485.85 / 29 = $51.24ea

– Total annual cost of four (4) dedicated modems in IT server room = Monthly Cost(ea) x 4(units) x 12(months):
$51.24ea x 4units x 12months = $2459.52

– It should be noted that we are calculating the cost of removing four (4) modems only from the system. The remaining modems would be replaced with cellular modems in a modern system where annual cellular costs would be comparable, resulting in no net increase/decrease in cost savings – Modern SCADA systems use the power of the internet for remote communications. As the municipality already subscribes to business internet services, there would be no additional costs associated with granting the SCADA server with internet access

– Avg. annual cost = $0 – Approx. $2460
4 – Communication failures between Aquaview SCADA and remote stations is roughly 200 times per year. We will estimate conservatively that 5% of those communication failures results in an operator driving to a remote station for diagnosis/troubleshooting: 200×5% = 10 trips

– Estimate 2hr total of driving/troubleshooting on avg. @
$70.56/hr + $18.50/hr vehicle costs =
$178.12 per trip

– Avg. annual communication diagnostics costs = Num trips x Cost per trip:
10trips x $178.12per trip = $1781.20 – Modern SCADA networks are reliable and very rarely would require a site visit to diagnose a communication failure. Estimate one trip per year to troubleshoot communication failures

– Estimate 2hr total of driving/troubleshooting on avg. @
$70.56/hr + $18.50/hr vehicle costs = $178.12 per trip

– Avg. annual communication diagnosis cost estimate:
$178.12 for one (1) trip – Approx. $1603

ID
Current Annual Cost
Modern SCADA Annual Cost Annual Net Savings (Annual Net Cost Increase)
5 – Flygt’s latest Remote Terminal Unit (RTU), MultiSmart RTU has approx. replacement/upgrade cost of
$10,000ea. We will estimate mean-time before failure at 10 years

– Annualized cost over 10 years = Unit Cost / 10years:
$10,000(per unit) / 10 years = $1000ea remote station

– Total annualized cost for all thirty (30) remote stations = Annualized Remote Station Cost x Num Remote Stations:
$1000ea x 30 remote stations = $30,000
total annualized cost – There are numerous RTU and PLC models that can be used in a modern SCADA system. We’ll use Allen Bradley PLC’s as an example as it’s the most common in North America. Mean-time before failure estimate of 10 years

– Depending on PLC model chosen, replacement cost can be anywhere from $1000 to greater than $10,000 for PLC hardware only. For St. Clair’s needs, lower end PLC’s will work well. We’ll estimate PLC hardware costs at $5000ea

– PLC solutions will require an external human machine interface (HMI). Flygt units have a form of HMI built into their 10k price tag. HMI screens vary greatly in price, from $1000 to greater than $6000 for HMI hardware only. For St. Clair’s needs, lower end HMI’s will work well. We’ll estimate HMI hardware costs at $3500ea

– Total annualized cost over 10 years (each): PLC Hardware + HMI Hardware / 10 years:
$5000 + $3500 / 10 = $850ea remote station

– Total annualized cost for all thirty (30) remote stations:
$850ea x 30 = $25,500 – Approx. $4500
6 – Aquaview SCADA currently uses a dedicated desktop server box that is due for replacement. Replacement costs vary depending on many factors, and could range from $3000 to greater than $10,000. We will estimate replacement cost at $6000. Mean-time before failure estimated at 6 years

– Annualized cost over 6 years = Replacement Cost / 6years:
$6000 / 6 = $1000 – Modern SCADA platforms often reside on IT infrastructure already in use for corporate operations. Typically, virtualized environments are used to segregate SCADA from business applications. The advantage to this is there is no dedicated SCADA server required, and SCADA can make use of IT’s existing virtual machine environment and licenses

– Annualized cost over 6 years: $0 – Approx. $1000

ID
Current Annual Cost
Modern SCADA Annual Cost Annual Net Savings (Annual Net Cost Increase)
7 – Aquaview SCADA currently does not require annual subscription costs as this product is no longer supported by Flygt

– Annualized cost = $0 – Modern SCADA platforms generally utilize annual subscription models for support and software upgrades. Pricing models for these contracts vary among manufacturers but we’ll use Inductive Automation’s Ignition product as an example. Ignitions support contracts are calculated on the retail price of the SCADA software at time of purchase multiplied by 16% for a ‘basic care’ contract

– For example, if Ignition SCADA retails for $30,000 and has a support multiple of 16%, annual support costs will be $30,000*0.16 = $4800

– Although there is an annual subscription cost associated with modern SCADA, the benefits are immense. The end user is entitled to all software updates as they are released from the manufacturer. As new versions are released, end users can take advantage of new features and security updates to ensure the SCADA system is always up to date. This cost should be viewed as a preventative maintenance cost and a requirement in building a secure, scalable, and
sustainable SCADA system – Approx. ($4800)
8 – Operators generate daily reports with respect to each pump stations runtime in order to ensure the system is running as expected. On average it currently takes 30min of time each morning to manually generate these reports.

– The average charge out rate for an operator is $70.56/hr

-Daily cost to generate reports = Avg. Operator Charge Out Rate x 0.5hr:
$70.56 x 0.5hr = $35.28

– Annualized cost to generate reports = Daily Cost x 365days:
$35.28 x 365 = $12,877.20 – Modern SCADA systems have the ability to automatically generate reports and email them to the appropriate personnel without user intervention, eliminating the need for an operator to manually generate any reports.

– Annualized cost: $0 – Approx. $12,877
9 – Operators can access the Aquaview SCADA system remotely in order to perform monitoring/troubleshooting tasks. There are times throughout a given year where the SCADA system will freeze and require an operator to physically go into the office to reset the system. Log books indicate this occurs on average 11 times per year after hours (4hours charge out per event)

– Total Annualized Cost to reset SCADA system = Number of events x Charge Out Hours x Hourly Rate:
11events x 4hrs x $70.56/hr = $3104.64 – Modern SCADA systems are inherently robust in nature and do not require “resetting” very often. To be conservative we’ll estimate a modern SCADA system needs to be rebooted by an operator one (1) time per year (4hours charge out per event)

– Total Annualized Cost to reset SCADA system = Number of events x Charge out hours x Hourly Rate: 1event x 4hrs x $70.56/hr = $282.24 – Approx. $2822

ID
Current Annual Cost
Modern SCADA Annual Cost Annual Net Savings (Annual Net Cost Increase)
10 – A review of St. Clair Townships Insurance claims over the past 10 years has resulted in eighteen (18) water related losses, totaling $89,831 of gross incurred losses. The insurer was not able to identify the portion of losses related to “water damage” versus claims specifically related to “wastewater”.
Conservatively we will estimate wastewater claims to be 50% of all water related claims resulting in
$44,915.50 of gross incurred loss. It should be noted that St. Clair’s deductible is approx. $20,000 and all individual claims fell within the deductible resulting in direct net loss to St. Clair Township.

– Wastewater insurance claims for the most part can be attributed to the current ineffective AquaView SCADA system. Operators do not reliably receive equipment failure notifications resulting in sewage backups in resident homes. This poses a significant environmental and health risk to affected parties. It should also be noted the townships insurer has indicated the loses incurred by St. Clair Township are higher than the provincial average for a municipality of St. Clair’s size.

– Annualized net cost of wastewater insurance claims = Gross incurred losses over 10 years / 10 years:
$44915.50 / 10yr = $4491.55 – Modern SCADA systems are very reliable and will ensure operators receive all equipment notifications as required. Risk of sewage backups as a result of a malfunctioning Modern SCADA system is very low and should never be the direct cause of a wastewater claim.

– $0 Annually – Approx. $4492
Total estimated direct cost annual savings Approx. $27,116

5. Risk

The ultimate goal of modern SCADA is not only to increase efficiency and reduce costs, but also manage risk. There are many operational risks associated with the current Aquaview SCADA system that increase the municipalities risk exposure from a financial, regulatory compliance and public health standpoint. Many of these risks have been laid out in Section 3 – Pain Points & Efficiencies but some of the most significant risks will be elaborated in this section. The following are practical examples of operational risks associated with the current SCADA system:
• Chlorine Limits – Chlorine levels below regulatory standards caused by insufficient monitoring and notification systems via the current Aquaview SCADA could result in fines and orders by the regulatory body in addition to ill residents. There is a heightened risk of the current unreliable SCADA system failing to correctly notify the operator of a chlorine malfunction. Although adverse conditions have not occurred to date as a result of SCADA malfunction, there are on average six (6) times per year where operators are notified of either a chlorine monitor or communication malfunction with the chlorine monitor. Under these conditions’ operators are required to implement manual testing protocols including manual sampling from their homes periodically and checking the affected station daily. Manual sampling increases the risk of data loss in the chlorine residual records required by regulatory bodies in addition to in-efficient resource utilization.
• Sewage Backups – On average once per year, a number of resident’s experience sewage backups in their basements as a result of the current Aquaview SCADA’s malfunctioning alarm system. These sewage backups are not only a public health risk, but have also incurred significant insurance claims and litigation, some in the order of $150k. In general, it can reasonably be inferred that as insurance claims are processed, the municipalities insurance premiums are likely to increase as a result of rulings against the municipality.
• Security – Outdated operating systems (OS) and SCADA software present bad actors with opportunity to exploit critical water infrastructure. As security patches are no longer released for the current version of OS, there is risk of exposure to cyber attacks on the SCADA system that could jeopardize the operator’s ability to effectively monitor and manage remote assets, putting public health at risk.

These risks are quite significant as they are both directly and indirectly concerned with public health. The importance of reliable, consistent alarming and real-time monitoring for critical public infrastructure is of significant importance and must be carefully considered.

6. Final Thoughts

As shown throughout this report, there are a number of areas within the existing SCADA system that present efficiency opportunity and security risks. Modern SCADA systems have come a long way over the last number of years and continually become more efficient and secure while reducing capital and maintenance costs. Modern SCADA systems can benefit St. Clair in the following ways: *
• Reduce public health risk associated with in-effective alarming and lack of real-time monitoring
• Modern SCADA is reliable, scalable, modular, and does not place restrictions on the end user
• Server based licensing models keep install costs low, combined with unlimited clients that can be launched from anywhere and on any device, creating a system rooted in efficiency
• Built on standard IT best practices that allow seamless merging of Operational Technologies (OT) and Information Technologies (IT)
• Subscription based support models eliminates the need for large capital expenditures that
typically occur at software ‘end of life’
• Virtualized IT environments reduces server and network hardware capital costs and maintenance
• Security vulnerabilities are minimized as software is kept up to date through annual support contracts
• Controls Integrator support networks are large for modern SCADA systems, ensuring competitive bids can be procured for programming updates and reduce the reliance on any single vendor for support
• Third-party application integration with platforms such as Enterprise Resource Planning or Preventative Maintenance Software has become common place
• Modern SCADA platforms have put a lot of emphasis on mobile monitoring and control in order to utilize the full power of mobile device hardware. GPS, accelerometer, camera, barcode scanner, touch gestures and other sensors can all be utilized to increase efficiencies within an operator’s daily routine
• Reduction in opportunity costs related to unnecessary troubleshooting and manual tasks that can be automated.

Although the direct cost savings when updating to a modern SCADA system appear low, the fact is, modern SCADA systems solve all of the major pain points of St. Clair’s current Aquaview platform. Modern SCADA gives end users control of their process, and the ability to track, monitor and control from any device, anywhere, and without limits ultimately saving time and money, while considerably reducing public health risks.
*Not all modern SCADA platforms contain the same feature sets and some may not be capable of addressing each of the beneficial points laid out for the Municipality of St. Clair. Care should be taken when selecting a modern SCADA platform.