Control system reliability at Jefferson Lab

At Thomas Jefferson National Accelerator Facility (Jefferson Lab), the availability of the control system is crucial to the operation of the accelerator for experimental programs. Jefferson Lab's control system, uses 68040 based microprocessors running VxWorks, Unix workstations, and a variety of VME, CAMAC. GPIB, and serial devices. The software consists of control system toolkit software, commercial packages, and over 200 custom and generic applications, some of which are highly complex. The challenge is to keep this highly diverse and still growing system, with over 162,000 control points, operating reliably, while managing changes and upgrades to both the hardware and software. Downtime attributable to the control system includes the time to troubleshoot and repair problems and the time to restore the machine to operation of the scheduled program. This paper describes the availability of the control system during the last year, the heaviest contributors to downtime and the response to problems. Strategies for improving the robustness of the control system am detailed and include changes in hardware, software, procedures and processes. The improvements range from the routine preventive hardware maintenance, to improving their ability to detect, predict and prevent problems. This paper also describes the software tools used to assist in control system troubleshooting, maintenance and failure recovery processes