How AI is expanding art history

Artificial intelligence (AI), machine learning, and computer vision are revolutionizing research from medicine and biology to earth and space sciences. Now it’s art history’s turn.

For decades, traditionally trained art scholars have been slow to embrace computational analysis, dismissing it as too limited and simplistic. But as I explain in my book, pixels and paintingsAnnounced this month, algorithms are advancing rapidly, with numerous studies proving the power of AI to shed new light on fine art paintings and drawings.

For example, AI-driven tools analyze brush strokes, color, and style to reveal how an artist’s understanding of optical science helps convey light and perspective. The program recovers the appearance of lost or hidden works of art and even calculates the “meaning” of some paintings, for example by identifying symbols.

It’s challenging. Works of art are compositionally and materially complex, full of human meaning, nuances that are difficult for algorithms to understand.

AI reads text from ancient Herculaneum scroll for the first time

Most art historians still rely on individual expertise, backed by laboratory, library, and homework to determine dates, materials, and provenance, when visually judging an artist’s skill. Computer scientists, on the other hand, find it easier to analyze 2D photographs and digital images than layers of oil paint styled with a brush or a palette or his knife. But collaborations are emerging between computer scientists and art scholars.

These early successes of “computer-assisted appraisal” fall into three categories: It processes subtle details in images beyond what is possible with normal human perception. and introduces new approaches and classes of problems to the study of art. Such methods, especially when powered by the digital processing of large volumes of images and texts about art, are beginning to empower art scholars just as microscopes and telescopes have done for biologists and astronomers.

Analysis of huge datasets

Think about poses. This is an important characteristic that portrait painters exploit for formal, expressive, and even figurative purposes. Some artists and art movements prefer certain poses. For example, during the Renaissance period from the 15th century to his 16th century, royalty, political leaders, and betrothed were often depicted in profile to convey solemnity and clarity.

Primitivist artists, such as the 19th century French painter Henri Rousseau, have no formal art training, and early 20th century French painters, such as Henri Matisse, have no formal training. Artists who intentionally imitate simplicity often depict everyday people head-on to help. A direct, unaffected style.Rotating and tilting poses are powerful: Japanese masters Ukiyo-e Floating world paintings, a genre that flourished from the 17th to the 19th century, often depict kabuki actors and geishas in twisted or distorted poses, evoking drama, dynamism, anxiety, and sensuality. Ta.

Using AI techniques, a computer can analyze tens of thousands of such poses for portraits in just an hour, much faster than an art scholar could do. Deep neural networks (machine learning systems that mimic biological neural networks in the brain) can detect the location of important points in a painting, such as the tip of the nose or the corners of the eyes. It then accurately estimates the angle of the subject’s pose around three orthogonal axes to create realistic, highly stylized portraits.

Consciousness: What is it, where does it come from, and whether machines can have it?

For example, earlier this year, researchers used deep neural networks to analyze the pose and gender of more than 20,000 portraits across a wide range of eras and styles, allowing art scholars to group works by era or art movement. I did it like this. There were some surprises. The tilt of the self-portrait’s face and body varied depending on the artist’s posture, and the algorithm was able to determine whether the self-portrait’s author was right-handed or left-handed.JP-P. Butterfly and DG Stork electronic. image. 35, 211-1–211-13. 2023).

Similarly, AI tools can reveal trends in landscape composition, color schemes, brushstrokes, perspective, and more across major art movements. Models are most accurate when they incorporate art historians’ knowledge of factors such as social norms, costume, and artistic style.

expand your awareness

Visual art analysis varies depending on how different scholars perceive a work of art. For example, from the exaggerated light and dark contrasts (chiaroscuro) and dark style (tenebrism) of the 16th century Italian painter Caravaggio, to the flat, graphic lighting of his 20th century works by American artist Alex Katz. Lighting is an expressive feature. Many experiments have shown that even careful observers are poor at estimating the overall direction and mismatch of illumination across a scene. For example, this is why the human eye is often fooled by photos that have been manipulated by cutting and pasting one shape onto another.

Computer-based methods may be able to do this better. For example, one source of information about illumination is the brightness pattern along the outer boundaries (or occluding contours) of objects such as faces. Leonardo da Vinci understood in his 15th century that this contour is brighter where the light hits it vertically, and darker where the light hits it at an acute angle. He used optical analysis to improve his paintings, but his “shape from shading” and “contour occlusion” algorithms use this rule in reverse, changing the direction of illumination from patterns of brightness along contours. guess.

Take a look at Johannes Vermeer’s 1665 painting. girl with a pearl earring, for example. The lighting analysis takes into account the highlights of the girl’s eyes, the reflections from the pearls, and the shadows cast across her nose and face. The occlusion contour algorithm allows a more complete understanding of the illumination of this tableau, revealing the extraordinary consistency in Vermeer’s illumination and proving that this figure study was carried out in the presence of a model. Ta (MK Johnson other.steps spy 6810, 68100I. 2008).

Similarly, advanced computational methods can be used to spot intentional lighting inconsistencies in works such as those of 20th century Belgian surrealist René Magritte. Additionally, some early painters, such as Jan van Eyck (ca. 1390-1441), were secretly using optical projection in their work a quarter of a millennium earlier. It has also proven its value in debunking theories such as British artist David Hockney’s bold hypothesis in 2000.Most scholars believe that optics were used in this way (see Nature 412860; 2001). Occlusion contour analysis, homography analysis (quantification of 3D shape differences at different sizes and pose angles), ray tracing, and other computational techniques advocated by other scholars using conventional techniques It systematically overturned Hockney’s theory much more decisively than any previous argument. Historical method.

Restoring lost cultural heritage

Computer methods can also identify missing attributes or parts of an incomplete work of art, such as perhaps the style or color of a ghost painting (a work that is painted over and later revealed by X-ray or infrared imaging). Restored. two sumo wrestlers Written by Vincent van Gogh. The painting was painted before 1886 and was mentioned by the artist in a letter, but was considered lost until it was discovered under another painting in 2012.

Neural networks trained on image and text data have also been used to recover the possible colors of some of Gustav Klimt’s lost ceiling paintings. medicine (look go.nature.com/47rx8c2). The original, depicting the intersection of life and death, was donated to the University of Vienna in 1901, but during World War II, the castle where it was kept for safety was burned down by the Nazis to prevent it from falling. , lost. into the hands of the Allies. Only preparatory sketches and photos remain.

Even more complex was the digital restoration of the missing parts of Rembrandt. night watchman (1642) — trimmed to fit into the space of Amsterdam City Hall — based on a modern copy by Gerrit Landens in oil on oak paneling. The algorithm learned how Lundens’ copy slightly deviated from Rembrandt’s original and “fixed” it to recreate the missing parts of the original. go.nature.com/46wvzmj).

Unleashing the full power of AI in arts research requires the same foundations as in other fields: access to vast datasets and computing power. Museums are posting more art images and supporting information online than ever before, and smart funding could accelerate ongoing efforts to collect and organize such data for research. there is.

Scholars predict that much of the information recorded about works of art will one day be available for calculations. Ultra-high resolution images of every major work of art (and countless smaller works of art), images taken using the extended electromagnetic spectrum (X-rays, ultraviolet, and infrared), chemical and physical measurements of pigments, Lecture videos recorded for every word written in any language, and art. After all, AI advances like chatbot ChatGPT and image generator Dall-E have been trained using nearly a terabyte of text and nearly a billion images from the web, and ongoing enhancements Data sets many times larger will be used.

But how will art scholars use existing and future computational tools? Here’s a suggestion. The known artefacts of the Western canon lost to fires, floods, earthquakes, and wars alone would fill the walls of every public museum in the world. Some like Diego Velasquez. Expulsion of the Morisco family (1627) was considered the pinnacle of artistic achievement before its destruction. Tens of thousands of paintings were lost in World War II, but an equal number of Chinese masterpieces were lost in Mao Zedong’s Cultural Revolution, to name just two. As a result, the world’s cultural heritage remains impoverished and incomplete.

Computation allows art historians to view the task of recovering the appearance of a lost work of art as a problem of information retrieval and synthesis. Data about lost works can be found in surviving sketches, copies by the artist and his followers, and written descriptions. The first tentative steps toward recovering lost works of art are showing promise, but much work lies ahead.

The study of art has expanded over the centuries through the introduction of new tools. Computing and AI appear to be the next steps in the never-ending intellectual adventure of understanding and interpreting our vast cultural heritage.