Besides old card catalogs probably nothing activates bibliophiles' neurons more than paper marble. Who can resist a 1000 year old technique that became popular in Europe and the US during the mid-19th century turning books into unique psychedelic marblelike monotypes. I was browsing the great collection of them at the University of Washington Libraries Decorated and Decorative Paper Collection and I started thinking of how they would be arranged if you organized them somehow. Not by a vocabulary of characteristics and curation but visually in a continuum. And that made me think of this idea of multi-dimensional latent space used in machine learning and what if you had a space full of paper marble patterns? If you could traverse through that space there would be real patterns but what would the space between those look like? That’s what this post is about, training a model on paper marble and exploring that latent space as a tool of organization.

Before you throw your vintage LCSH red book at me since this relates to machine learning and an artistic practice: The goal here is not to build a paper marble machine, I’m using a relatively out of date technique not employed by modern generative AI that I run on my own computer. It produces low resolution cropped images that are not really useful outside of this little exploration.

After some research I ended up using this well known older (6 years is a long time anymore) Generative Adversarial Network approach developed by NVIDIA called StyleGAN2-ADA from 2020. Unlike modern diffusion models that today’s generative AI uses it allows you to “walk” the latent space of the model which is exactly what I wanted to explore. I picked this version of it because I read that it does well with a smaller amount of training images due to the Adaptive Discriminator Augmentation (ADA) part which maximizes the small dataset I’m using. Here are the technical steps I took:

• Generated about 1,100 1024x1024 pixel crops of paper marble from the 500 University of Washington Libraries collection images.
• On my 4090 24GB card I used stylegan2-ada-pytorch library and started with the FFHQ 70,000 human faces base model and fine tuned that. 1,100 images is way too small of a corpus to train a GAN from scratch so this starts off with a large defined model and then “untrains” the parts we don't want (human faces) out of it.

  

• Training took about nine hours to get the base after about 300,000 images reviewed by the discriminator. I found that the results really oversampled a few common marble patterns in the corpus so I did another fine tune of the first model to over sample the more rare marble patterns in the corpus. These are the cool unique one off sheets that were basically ignored in the original training. Otherwise a few of the dominant patterns took over the whole output.
• This GAN generates a 512 number vector. There is one vector value called the W space that behaves in a predictable way, nearby W space values are similar images. So by manipulating the W space on generation you can walk the latent space of the model.
• But real images, like the ones we started with don’t have a W space value to begin with, so if we do want to ultimately map these into a space we need to find that number via a process called projection or GAN inversion, it takes a long time but eventually you can generate where the real image would likely sit in this new model.
• With all that information we can MAP! I wanted a grid layout, no overlap, but still community so still do the expected UMAP reduction but then take the scipy.optimize.linear_sum_assignment solver to place them into a grid arrangement.
• The 128x128 grid has 16,384 marble sheets, 9,600 of those were just randomly selected, 381 are real sheets, I’m calling them anchors, and 6096 were generated by mixing the w latent space of the its neighbors around the anchors, and 307 are generated where there was no sheet, gaps in the mapping.

Explore the Marble Latent Map

Using this interface you can browse the 128x128 grid and explore the patterns generated between the real anchored paper marble samples which are outlined in blue. I wasn’t sure what to expect from the results, as I mentioned earlier it is more of an organizational tool than a generation tool. I was interested in what kind of marble patterns would emerge between the anchors but there is not a real surprise, they mostly look like the original samples with slight iterations. I think this could be viewed as a spectrum of the Libraries Decorated and Decorative Paper Collection as opposed to a overall paper marble explorer. But I do like this idea of a using latent space as a tool for organization and wayfinding.

I’ll end with an hour long tour of the latent space I made a video of the transitions between all the different anchors and the marbles you find inbetween: