[Unstagram] AI That Said It Would 'Read Photos'

AI Vulnerable to Textualizing Visual Information
The Vast Ocean of Human Senses Beyond Machines' Reach

About a decade before the name artificial intelligence (AI) and its capabilities seemed impressive, there was a website where a computer would ‘narrate’ photos. I was curious how a machine would verbally describe what it saw. I uploaded a landscape photo taken during a trip to Jeju Island, featuring canola flowers and pine trees. After waiting about 30 minutes, the English sentence I received was complex and chaotic.

The photo of Jeju Island that I uploaded because AI said it would read it aloud ⓒ Heo Younghan

“Blue sky and fields, and tranquility. The blue sky remains an unknown realm. The fields evoke farming, and tranquility is a key virtue in Buddhism. Brightness is also called radiance, and the sunset has a rosy hue. Sunsets are colorful and generally beautiful. It is how God says goodbye....” It was a lengthy sentence resembling a poem that sometimes made sense and sometimes did not. Rather than describing the content, it conveyed associations and feelings, which was astonishing.

The website is gone now, but I still cherish this sentence?the first machine evaluation of my photo. The operator continues to work on interesting projects that mediate the relationship between text and images, such as creating a ‘camera’ that outputs sentences like receipts instead of photos when taking pictures. What he seems to want to show is not technology itself but an artistic way of looking at technology.

Today, AI is more ‘talented’ at imitating art and creating works that depart from reality. With just a few words, it can generate vivid and delicate images and write texts that mimic famous authors’ masterpieces. With some effort, it can produce images more realistic than reality and texts more factual than facts.

Ironically, what is more vulnerable is converting visual information that captures factual scenes, like photos or videos, into text. One current application of technology that textifies realistic visual information is in safety and alert industries. It involves extracting basic facts such as ‘people are fighting,’ ‘there is a fire,’ or ‘water is overflowing’ from CCTV footage and applying them. Although still in its infancy, it is a useful technology. The bridge connecting humans and machines to protect and advance the world is language.

Demonstration screen of PIA, a company specializing in AI screen text recognition technology. It was demonstrated at the Seoul Investor Forum held on September 30 to attract investment for Korean companies.

What AI finds difficult to express are not nouns?the names of things?but the living, moving aspects and relationships like verbs and adjectives. The world is full of all kinds of relationships, brushes, interactions, and conflicts, and machines find it hard to distinguish and verbalize them. Even the simplest feelings and emotions for humans?beauty, sadness, longing?are vast oceans for machines. Yet art sometimes crosses these oceans using a different language.

And the photo narrated by the computer reminded me of recurring sentences in a novel that felt like images. Sentences that are read more by sensation than grammar, vaguely understood.

‘He could feel time gathering in swarms around him,

those large, heavy masses

packed densely?from Bermuda to Buenos Aires?

so densely packed it was almost too much to see.’

(Anne Carson, Autobiography of Red, HanKyoreh Publishing)

This novel was written as poetry. Poetry is imagery. Could there be poetry written as a novel? Probably.

If one does not insist on strict distinctions...

Editor's NoteThis piece writes about photos, what is seen, human senses, and past time. The artist’s current works are available on the website rossgoodwin.com.