As the COVID-19 outbreak took hold in Italy, researchers working on a nuclear physics experiment called CUORE at an underground laboratory in central Italy scrambled to keep the ultrasensitive experiment running and launch new tools and rules for remote operations.
This Cryogenic Underground Observatory for Rare Events experiment – designed to find a never-before-seen process involving ghostly particles known as neutrinos, to explain why matter won out over antimatter in our universe, and to also hunt for signs of mysterious dark matter – is carrying on with its data-taking uninterrupted while some other projects and experiments around the globe have been put on hold.
Finding evidence for these rare processes requires long periods of data collection – and a lot of patience. CUORE has been collecting data since May 2017, and after upgrade efforts in 2018 and 2019 the experiment has been running continuously.
Before the pandemic hit there were already tools in place that stabilized the extreme cooling required for CUORE’s detectors and provided some remote controls and monitoring of CUORE systems, noted Yury Kolomensky, senior faculty scientist at the U.S. Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley Lab) and the U.S. spokesperson for CUORE.
The rapid global spread of the disease, and related restrictions on access to the CUORE experiment at Gran Sasso National Laboratory (Laboratori Nazionali del Gran Sasso, or LNGS, operated by the Italian Nuclear Physics Institute, INFN) in central Italy, prompted CUORE leadership and researchers – working in three continents – to act quickly to ramp up the remote controls to prepare for an extended period with only limited access to the experiment.
Just days before the new restrictions went into effect at Gran Sasso, CUORE leadership on March 4 made the decision to rapidly deploy a new remote system and to work out the details of how to best maintain the experiment with limited staffing and with researchers monitoring in different time zones. The new system was fully operational about a week later, and researchers at Berkeley Lab played a role in rolling it out.
“We were already planning to transition to remote shift operations, whereby a scientist at a home institution would monitor the systems in real time, respond to alarms, and call on-site and on-call personnel in case an emergency intervention is needed,” Kolomensky said, adding, “We were commissioning the system at the time of the outbreak.”
Brad Welliver, a postdoctoral researcher, served as Berkeley Lab’s lead developer for the new remote monitoring system, and Berkeley Lab staff scientist Brian Fujikawa was the overall project lead for the enhanced remote controls, collectively known as CORC, for CUORE Online/Offline Run Check.
Fujikawa tested controls for starting and stopping the data collection process, and also performed other electronics testing for the experiment from his home in the San Francisco Bay Area.
He noted that the system is programmed to send email and voice alarms to the designated on-shift CUORE researcher if something is awry with any CUORE system. “This alarm system is particularly important when operating CUORE remotely,” he said, as in some cases on-site workers may need to visit the experiment promptly to perform repairs or other needed work.
Development of so-called “slow controls,” which allow researchers to monitor and control CUORE equipment such as pumps and sensors, was led by Joe Johnston at the Massachusetts Institute of Technology.
“Now we can perform most of the operations from 6,000 miles away,” Kolomensky said.
And many participants across the collaboration continue to play meaningful roles in the experiment from their homes, from analyzing data and writing papers to participating in long-term planning and remote meetings.
Despite access restrictions at Gran Sasso, experiments are still accessible for necessary work and checkups. The laboratory remains open in a limited way, and its staff still maintains all of its needed services and equipment, from shuttles to computing services.
Laura Marini, a postdoctoral researcher at UC Berkeley who serves as a run coordinator for CUORE and is now living near Gran Sasso, is among a handful of CUORE researchers who still routinely visits the lab site.
“As a run coordinator, I need to make sure that the experiment works fine and the data quality is good,” she said. “Before the pandemic spread, I was going underground maybe not every day, but at least a few times a week.” Now, it can be about once every two weeks.
Sometimes she is there to carry out simple fixes, like a stuck computer that needs to be restarted, she said. Now, in addition to the requisite hard hat and heavy shoes, Marini – like so many others around the globe who are continuing to work – must wear a mask and gloves to guard against the spread of COVID-19.
The simple act of driving into the lab site can be complicated, too, she said. “The other day, I had to go underground and the police stopped me. So I had to fill in a paper to declare why I was going underground, the fact that it was needed, and that I was not just wandering around by car,” she said. Restrictions in Italy prevent most types of travel.
CUORE researchers note that they are fortunate the experiment was already in a state of steady data-taking when the pandemic hit. “There is no need for continuous intervention,” Marini said. “We can do most of our checks by remote.”
She said she is grateful to be part of an international team that has “worked together on a common goal and continues to do so” despite the present-day challenges.
Kolomensky noted some of the regular maintenance and upgrades planned for CUORE will be put off as a result of the shelter-in-place restrictions, though there also appears to be an odd benefit of the reduced activity at the Gran Sasso site. “We see an overall reduction in the detector noise, which we attribute to a significantly lower level of activity at the underground lab and less traffic in the highway tunnel,” he said. Researchers are working to verify this.
CUORE already had systems in place to individually and remotely monitor data-taking by each of the experiment’s 988 detectors. Benjamin Schmidt, a Berkeley Lab postdoctoral researcher, had even developed software that automatically flags periods of “noisy” or poor data-taking captured by CUORE’s array of detectors.
Kolomensky noted that work on the CORC remote tools is continuing. “As we have gained more experience and discovered issues, improvements and bug fixes have been implemented, and these efforts are still ongoing,” he said.
###
CUORE is supported by the U.S. Department of Energy Office of Science, Italy’s National Institute of Nuclear Physics (Instituto Nazionale di Fisica Nucleare, or INFN), and the National Science Foundation (NSF). CUORE collaboration members include: INFN, University of Bologna, University of Genoa, University of Milano-Bicocca, and Sapienza University in Italy; California Polytechnic State University, San Luis Obispo; Berkeley Lab; Lawrence Livermore National Laboratory; Massachusetts Institute of Technology; University of California, Berkeley; University of California, Los Angeles; University of South Carolina; Virginia Polytechnic Institute and State University; and Yale University in the US; Saclay Nuclear Research Center (CEA) and the Irène Joliot-Curie Laboratory (CNRS/IN2P3, Paris Saclay University) in France; and Fudan University and Shanghai Jiao Tong University in China.
(Note: This is the first part in a recurring series highlighting Berkeley Lab’s ongoing work in international physics collaborations during the pandemic.)
