How to work with large number of small files

Singularity Overlays

• https://docs.sylabs.io/guides/3.5/user-guide/persistent_overlays.html

Persistent overlay directories allow you to overlay a writable file system on an immutable read-only container for the illusion of read-write access. You can run a container and make changes, and these changes are kept separately from the base container image.

From Python

Working with files inside a TAR archive

• tarfile module:

https://www.askpython.com/python-modules/tarfile-module

https://stackoverflow.com/questions/27220376/python-read-file-within-tar-archive

Using HDF5 Hierarchical Data file Format

• Official site: https://www.hdfgroup.org/solutions/hdf5/
• Wikipedia article: https://en.wikipedia.org/wiki/Hierarchical_Data_Format