If a deep learning algorithm (more spezifically: A Convolutional Neural Network) recognizes a cat on a photo, then it's even for the developer of the AI algorithm hardly possible to understand the decision-making process of the algorithm. What characteristics of the picture have led the algorithm to classify the animal a cat? The process of abstraction (in dozens of neural layers) of image elements and image features is simply too confusing and too complex. It works, but we (rarely) know why.
One might come to the conclusion it's great that it works after all; what do we need understanding for, if it allows to make reliable predictions? Some AI experts even forecast a science without theories and explanations. But in fact this black box problem comes with a multitude of ethical and practical challenges. The author and futurist Amy Webb delivers in her book "The Big Nine. How the Tech Titans & Their Thinking Machines Could Warp Humanity" (year of publication 2019) a good example for illustration:
Researchers at the New York Ichan School of Medicine (located at Mount Sinai Hospital) developed an AI algorithm for cancer prediction, which they were able to train with extensive data from about 700,000 patients. Each data set contained hundreds of different variables. The AI algorithm (called "Deep Patient") identified many new patterns in the data, which the researchers could not always comprehend, but which formed viable indicators to identify early stages of many diseases (e.g. liver cancer) in patients.
Mysteriously, "Deep Patient" could also provide indicators of psychiatric disorders such as schizophrenia. But even the researchers who built the system did not know how the AI algorithm makes decisions ("black box problem"). So, "Deep Patient" makes clever predictions, but without any explanations. What does this mean in practice? How can such a system be used by medical teams that have to make decisions or recommendations with far-reaching consequences? Decisions about discontinuing or changing medication? Decisions on the administration of radiotherapy or chemotherapy or an operation?
