Optimisation and automation of workflows from data capture to data processing in electron microscopy

Research instruments rarely come “out of the box” with an optimized or even any solutions for automated and high-speed data transfers. Often these solutions are up to the facility or end user (researcher) to develop, and these end up being done on a case-by-case basis. This work package explores how the automation work at UOW to manage the data from its Cryo Electron Microscope (CryoEM) facility can be extended or reused at other EM, microscopy and bigdata generating instruments. Here we layout details of the workflows developed to move big data and apply insights for other institutions wanting to develop a big data infrastructure for seamless data management. It is critical to build efficient workflows. UOW has noted that a non-uniform approach to solving this problem hurts productivity, wastes researcher time and can lead to the possibility of data being lost. This work has demonstrated that: A high level (architectural) and uniform (implementation) approach to ALL instruments with high data throughput requirements is essential. Some level of scripting on a per site/facility basis will be essential (UOW specific examples are outlined in the report). “Default” settings on computers are often not suitable for high-speed transfers – these need to be changed to get better performance. Building efficient workflows is as important when dealing with modern high throughput data requirements as fielding high end research instruments. Researchers and facility operators need tools to see end to end visibility of how networks are behaving to make decisions on workflow timing. There is an increasing trend for data sources i.e., instruments and data sinks i.e., storage and compute may not to be side by side. Often the computation and long-term storage services may be some distance (interstate or in the cloud) from where data is collected or generated. There can be parallel requirements for near real time data processing (e.g., quality control) and longer-term data processing (analysis, generating final maps, structions, decisions). Automation needs to address both these requirements. Data image formats and throughputs can change during the life of an instrument – any automation needs to have sufficient flexibility to deal with this. In particular, as instruments may be upgraded, data volumes have the potential to increase significantly. These changes need to be managed and the design must be modular in order to quickly pivot to new hardware, software, and workflow modifications. Automating data transfers is likely to be a multi-phase implementation as unexpected behaviours may be identified in early phases, i.e., your automation may actually put significant load on data capture or throughput implementation – leading to worse performance outcomes than expected. Automation is key to ensuring data can be managed across multiple parts of a complex workflow – without being lost or corrupted. Where data *is* lost or corrupted it’s extremely important to find out ASAP to ensure appropriate remedies can be applied