Redundant virtualization of a supervisory control system

V diplomskem delu je opisana nadgradnja obstoječih strežnikov pri stranki, katere primarna dejavnost je proizvodnja cementa, na dva virtualna strežnika VMWARE ESXi 6.0 s pripadajočo programsko opremo. Odločitev za nadgradnjo je bila sprejeta zaradi več razlogov. Prvotno smo želeli odpraviti drago in nepotrebno vzdrževanje dvaindvajsetih različnih fizičnih strežnikov in izpade proizvodnega sistema, ki so povezani z izpadi katerega koli od teh strežnikov. Prav tako pa smo želeli odpraviti izpad poročilnega sistema, vezanega na strežnik, kjer je za zbiranje podatkov bil prej uporabljen programski paket za arhiviranje Proficy Historian 3.5. Ob izpadu strežnika, kjer je potekal zapis zgodovine, smo ostali brez podatkov tudi za načrtovano in sprotno porabo, ker nismo mogli priti do stroškovne bilance za določeno preteklo obdobje. Hkrati smo se rešili nepotrebnih stroškov, povezanih s porabo električne energije, ki jo je povzročilo delovanje prejšnjih strežnikov, prepotrebnih UPS sistemov in hlajenje strežniških omar. Pridobili smo tudi na skrajšani mrežni povezavi, ki jo je zahtevalo večje število strežnikov. Ker sta virtualna strežnika povezana preko skupinske nastavitve, smo ob izpadu enega od strežnikov dosegli neprekinjeno delovanje sistema z visoko razpoložljivostjo. Strežnika sta tudi lokacijsko ločena in povezana z optičnim kablom, diskovna polja in 10G stikalno omrežje pa so podvojeni in redundantni. Ob izpadu enih, druga takoj prevzamejo vso operativnost, s čimer smo želeli poskrbeti za nemoten delovni proces proizvodnje, nizke stroške vzdrževanja, lažje oddaljeno upravljanje strežnikov, lažje bodoče nadgradnje, varnost in zanesljivost sistema ter nižje stroške investicij v novo opremo. Z zgoraj naštetim smo ob izpadu strojne opreme zagotovili nemoteno obratovanje delovnega procesa. Na programskem nivoju je bilo potrebno rešiti preklop ob izpadu aktivnih SCADA strežnikov na tiste, ki so v pripravljenosti. Za rešitev naštetih težav smo izbrali ponudnika, ki se nam je zdel najbolj primeren. Na sami proizvodni lokaciji so bili nameščeni strežniki s starejšo verzijo tega ponudnika, in sicer Proficy HMI/SCADA iFix 3.5, zato smo se odločili za nadgradnjo na verzijo 5.8. Za lažjo preglednost operaterjem je bila sprejeta odločitev za povečanje ločljivosti s 1280x1040 na polno HD resolucijo 1920x1200. Namesto desetih fizičnih računalnikov, kjer so bili postavljeni odjemalci, smo jih postavili 6, ki preko ethernet omrežja dostopajo do virtualnih strežnikov, preklop SCADA sistema pa je rešen na nivoju virtualnega strežnika, tako da operater niti ne zazna, kdaj se je preklop ob morebitnem izpadu zgodil. Ob prej omenjenem problemu z arhiviranjem podatkov smo na virtualni strežnik namestili tri virtualne strežniške računalnike. Dva potrebujemo za najnovejšo opremo za arhiviranje podatkov Proficy Historian 6.0, ki po novem vsebuje redundantno rešitev v primeru izpada enega od Historian strežnikov. Na enem je postavljen Historian strežnik, na drugem pa njegova zrcalna kopija. Vse skupaj pa je povezano s SQL strežnikom, ki je postavljen na tretjem virtualnem računalniku. S tem smo poskrbeli, da z izgubo katerega koli od teh treh strežnikov ne izgubimo potrebnih podatkov za obdelavo. Tako sem na vseh področjih poskrbel za rešitev, pri kateri sistem in proizvodni procesi nemoteno delujejo po izpadu katerega koli zgoraj naštetega sklopa. Seveda pa smo v določenih primerih, kot je izpad elektrike (določen čas ob izpadu je sistem pokrit z UPS sistemi) ali krmilne enote, nemočni. Prostor za nadgradnjo vidim predvsem na krmilnem nivoju, saj bi lahko naročili krmilnike, ki bi bili redundantni obstoječim in bi imeli pokrit izpad krmilne enote, vendar je ta rešitev ekonomsko neupravičena in se zanjo nismo odločili.This thesis describes how to upgrade existing servers within the customer, whose primary business is the manufacture of cement, into two virtual VMware ESXi 6.0 with associated software. The decision to upgrade was accepted for several reasons. Originally, we wanted to solve costly and unnecessary maintenance with 22 different physical servers, downtime of the production system associated with a failure of any of these servers in the failure reporting systems tied to the server, where previously for the data collection Proficy Historian 3.5 was used. With the failure of the server, where the record of history is held, we are left without data for planned and ongoing consumption, because we were unable to get the balance of the cost for a particular prior period. At once we saved unnecessary costs, associated with the power consumption, caused by earlier servers, a lot of UPS systems and server cabinets cooling. We have also acquired new network connection by decreasing the number of servers required. Since the virtual server is connected via a cluster configuration, in the event of failure of one of the servers, we achieved continuous operation of the system with high availability. Servers are also separated on 2 locations and connected by fiber optic cable. Disk arrays and 10G switching network are duplicated and redundant so that with the failure of ones, the others immediately assume full operability. In this way we ensure a smooth production process, low maintenance costs, facilitate remote management of servers, easier future upgrades, security and reliability of the system, and lower costs of investment in new equipment. With all that is stated above, we cover workflow hardware shortfalls. However, at the software level, it was necessary to configure the switch upon failure of the active SCADA back to the backup. To resolving these problems, we have chosen a provider that has a history of experience with ensuring redundant SCADA system. On the same production side, we have installed servers with the old version of this service provider Proficy HMI/SCADA iFix 3.5, which we decided to upgrade to version 5.8. To facilitate transparency for operators, we decided to increase the resolution from 1280x1040 to 1920x1200 full HD. Instead of 10 physical computer clients we set 6, which access virtual servers via an Ethernet network. Switching between both SCADA systems is solved at the level of the virtual server, so that the operator does not even detect when the turnover between active and backup SCADA on the possible failure occurred. To resolve problems with archiving data on a virtual server, we installed three servers. Two are needed for the latest equipment for data archiving Proficy Historian version 6.0, which now contains a redundancy solution in the package of a Historian server. On one side Historian server and on the other his mirror node. All this is linked to the SQL server, which is installed on a third server. In this way we can make sure that the loss of any of these three servers does not lose the data required for processing. So I took care for all possible distractions to find a solution where the system and manufacturing processes operate smoothly after the failure of any of the above mentioned elements. Nevertheless, nothing can be done in certain situations, such as power loss (during the failure the system it is covered with UPS systems for a certain period of time). I see possibility for improvement mainly on a control unit, as we could order PLCs, which would be redundant to the existing ones and would cover the failure of the control unit. Unfortunately, this solution is economically unjustified and it was not our final decision