Mining the Vatican Secret Archives with TensorFlow w/ Elena Nieddu - TWiML Talk #243

Model Development

The development of machine learning models for the Vatican archives project involved a meticulous process of experimentation and iteration. Elena Nieddu and her team began with simple models like logistic regression, gradually advancing to convolutional neural networks (CNNs) to better fit their data's unique characteristics. They faced challenges with TensorFlow, balancing between Keras for prototyping and pure TensorFlow for advanced features. Elena explains, "We were very torn apart between keras and pure tensorflow because we loved keras for prototyping, because it kept the code small and very readable."

We could have just downloaded one big model, but we knew our dataset wasn't that big and it wasn't as hard as imagenet.

--- Elena Nieddu

The team also leveraged TensorFlow's high-level API layers to streamline the transition between Keras and TensorFlow, enhancing their model's capabilities 1 2.

   

Character Recognition

Character recognition in historical texts presents unique challenges, particularly with cursive scripts. Elena Nieddu describes using a combination of old-school computer vision heuristics and CNNs to segment and classify characters. This approach involves breaking down words into sub-shapes, akin to puzzle pieces, which are then classified both in isolation and combination. Elena notes, "We have a few like old school computer vision heuristics that help us cut these whole words into sub shapes."

The network by itself has 94% average accuracy over 33 classes.

--- Elena Nieddu

Despite achieving high accuracy in character recognition, the pipeline's complexity introduces potential errors at each step, affecting overall transcription accuracy 3 4.

   

Contextual Ranking

To improve transcription accuracy, Elena Nieddu and her team employed contextual ranking using language models. This process involves ranking transcriptions based on their likelihood of being correct within the context of Latin text. Elena explains, "We ranked them according to a language model we built out of latent text."

The language model picks the one that sounds more latin ish.

--- Elena Nieddu

This method helps resolve ambiguities in character recognition, such as distinguishing between similar-looking letters, by leveraging the broader linguistic context 5 6.