Paul von Hindenburg in 3D – History Meets Technology (again!)

What do the walls of the world's largest brick castle, a 13th-century Teutonic fortress, and cutting-edge AI 3D modelling have in common? On a trip to Malbork Castle in Poland, I captured a short video of the bust of Paul von Hindenburg (bullet holes and all!) and recently used it for a fascinating tech experiment. It also happens to tie in with a little improvement I've made to the website.

I like to experiment a lot with all various types of technology and I had started to see examples of Neural Radiance field (NeRF) models, which were rendered very quickly from short segments of video. NeRF models use deep learning to render a 3D scene from 2D images or video; they've become a game-changer in photogrammetry. One day back home after the holiday, I remembered a quick video I had taken of a bust at the castle, of Paul von Hindenburg (and associated bullet holes!).

Malbork Castle is the largest castle in the world when measured in land area and is recognised as a UNESCO World Heritage Site. It's believed to have built by the Teutonic Knights in the 13th century, and served as the capital of the Monastic state. Officially known as the Castle of the Teutonic Order in Malbork, it is a huge complex brick fortress, and I'd recommend anyone who happens to be visiting Poland to strongly consider visiting. Hindenburg, a key figure in German military and political history, has a complex connection to this region, and the visible bullet holes add a visceral layer of history to the digital model.

I figured this would be ideal to test generating a model using a NeRF technique and I had intended on adding the functionality to change models on the fly on the website with a view to creating a gallery in the near future. Unlike traditional photogrammetry, NeRF models are excellent at handling complex light and slight movement, making it ideal for the low-quality, hand-held video I had captured. I turned to nerfstudio, which is an open-source effort to provide as many implementations of NeRf models as possible. This took a little bit of effort to get it up and running, requiring COLMAP and therefore a CUDA enabled graphics card.

The video above shows the viewer within the browser for nerfstudio, and rotating the model takes a moment to update the resolution displayed. You can also see each frame of the video which nerfstudio has used to produce the model. All-in-all the results are really impressive, and it's mesmerising to watch the model improve through the iterations. As you can see in the interactive point cloud currently loading behind this text, the results are really impressive for the quality of the video used as input. Whilst the result traditionally wouldn't be considered great given the lighting and the strong shadows produced as a result, I still like how it turned out. In a sense I feel like the shadows might capture the militaristic atmosphere of being inside a giant fortification.

Watch this space for a longer tutorial on how to start generating your own models from pictures and/or video using Nerfstudio. I've also been having a look at Reality Capture which is free for personal use from Epic Games, so check back in for a quick post on my experiences with it very soon!

Head back out to the homepage and use the gallery button on the bottom-right to check out the model in more detail!