19th May 2020

API Loss of data

7 days ago, we had a database go over quota. At that time, we updated the quota of the database. Today we were making preparations to support larger experiments and updated our database configuration. When we updated the configuration of the database and rolled the container hosting it in Kubernetes, an old replica of the database container came up with the old quota, truncating data beyond the original quota point from 7 days ago. We are currently working on trying to recover this data. The effected data does not include your home directories or project directories, but does include experiment definitions, topology sources and realization/materialization state.