Honesty in Pandemic Modeling. OpenAI Microscope. Analytic Indices? Lossless Image Compression via DL. [DSR #224]

Hiding infections in the future is not the same as avoiding them.

A keen figure-reader will notice something peculiar in Kristof’s figure. At the tail end of his “Social distancing for 2 months” scenario, there is an intriguing rise in the number of infections (could it be exponential?), right before the figure ends. That’s because of an inevitable feature of realistic models of epidemics; once transmission rates return to normal, the epidemic will proceed largely as it would have without mitigations, unless a significant fraction of the population is immune (either because they have recovered from the infection or because an effective vaccine has been developed), or the infectious agent has been completely eliminated, without risk of reintroduction. In the case of the model presented in Kristof’s article, assumptions about seasonality of the virus combined with the longer mitigation period simply push the epidemic outside the window they consider.

Technologies such as columnar block-based data organization and compression have become standard practice in most commercial database systems. However, the problem of best assigning records to data blocks on storage is still open. For example, today’s systems usually partition data by arrival time into row groups, or range/hash partition the data based on selected fields. For a given workload, however, such techniques are unable to optimize for the important metric of the number of blocks accessed by a query. This metric directly relates to the I/O cost, and therefore performance, of most analytical queries.

In this paper, we propose a new framework called a querydata routing tree, or qd-tree, to address this problem, and propose two algorithms for their construction based on greedy and deep reinforcement learning techniques. Experiments over benchmark and real workloads show that a qd-tree can provide physical speedups of more than an order of magnitude compared to current blocking schemes, and can reach within 2× of the lower bound for data skipping based on selectivity, while providing complete semantic descriptions of created blocks.

This is the official implementation of SReC in PyTorch. SReC frames lossless compression as a super-resolution problem and applies neural networks to compress images. SReC can achieve state-of-the-art compression rates on large datasets with practical runtimes. Training, compression, and decompression are fully supported and open-sourced.