The Future of Cameras

ROMA AGRAWAL

In my lifetime so far, I've seen the camera go through several revolutions. Film, digital, smartphones. With all this rapid change, I can't help but wonder, what's next for the camera? And how might the next stage of my life be shaped by this technology. I'm Roma Agrawal and you're listening to Create The Future from the Queen Elizabeth Prize For Engineering. In this episode, I'm joined by Professor Abhijit Ghosh, professor of graphics and imaging at Imperial College London, and historian and curator of photography, science and visual culture Dr. Geoff Belknap. Together, we unpack the myriad ways in which camera technology transforms the world around us.

PROFESSOR ABHIJIT GHOSH

I think there will be a generation coming that is more used to seeing content in 3D superimposed with their reality, so they are much more tuned to living in mixed reality.

ROMA AGRAWAL

We also grapple with the ethics of this technology. The balance between power and responsibility in an age of facial recognition and surveillance culture.

DR. GEOFF BELKNAP

None of these technologies are neutral. We're all making choices.

ROMA AGRAWAL

And how a recognition of the camera's history can help us to understand its future.

DR. GEOFF BELKNAP

The real importance of photography and its legacy now is because it can break that threshold and it can give us views into worlds which we will never be able to see, imagine, look at, verify with our human eye.

ROMA AGRAWAL

Right. So everyone knows that I am very fond of an engineering pun, so this is what we've come up with. Can we get a snapshot of each of your careers, please, before we zoom into the future of advanced camera technology? So, Geoff, can we start with you?

DR. GEOFF BELKNAP

So I started my career working in academia, so I first did a PhD in the history and philosophy of science, focusing on 19th century photographic reproductions, particularly in kind of UK newspapers and looking at how science was demonstrated through photography in scientific and public press, in particular in the 19th century. And then I, as a kind of jobbing academic worked around to a number of different post docs. And then from there I was super lucky and museums accepted me into their heart about five years after the PhD, and I became a curator of photography first and then kind of a manager within the Science Museum group at the biggest and best photographic and Photographic Technology Museum in the world, the natural science and media Museum in Bradford, I was the photo curator and then the head curator there. And then about a year and a half ago, I've had the privilege of becoming the keeper of Science and Technology at the National Museum of Scotland.

ROMA AGRAWAL

Well, I love the kind of, the idea of, of the science, the engineering, the history, like everything kind of all comes together in your career. Abhijit, could you tell us a bit about your career?

PROFESSOR ABHIJIT GHOSH

Yeah. So I did my PhD from the University of British Columbia and Canada, focusing on computer graphics. That's where I basically got into working with cameras. We use it for measurements of the appearance properties of various things we want to model in computer graphics. And after my PhD, I went to work at the University of Southern California and Los Angeles with a very famous researcher in computer graphics named Paul Debelak, and we invented the light stage facial capture technology that's impacted majorly the entire film production industry in Hollywood. Academy Award technology for capturing realistic reproductions of digital doubles for visual effects. I moved to Imperial as a lecturer first in December 2012 and for the last 10 years I've been an academic at Imperial College and I'm a full professor. And I run my group focusing on essentially realistic computer graphics and computational imaging.

ROMA AGRAWAL

I want to start you off with a slightly philosophical question, if I may. So Abhijit, what does taking a photograph today mean to you?

PROFESSOR ABHIJIT GHOSH

Oh, it's a very difficult question to answer. I mean, the funny thing is my wife is a much better photographer but from an artistic perspective. She's an architect, by the way. She has more aesthetic sense. Whenever I'm taking a picture, it's usually for some sort of a scientific measurement, and it's hence much more technical in that sense rather than artistic. I don't know if that's necessarily the philosophical answer you're looking for.

ROMA AGRAWAL

No, I love that. I love, because again, you know, photography can be about art and visuals, but it could also be about science and I think we're gonna explore that more with you. Geoff, what about you? What does taking a photograph mean to you?

DR. GEOFF BELKNAP

Oh, that's, yeah, big question. I think for me, the obvious answer is what taking a photograph is, is that the camera on my iPhone or the camera on my phone that I might have and the WhatsApp pictures or pictures I send back and forth between my family or the random snaps I take out in the sunshine that you have in Edinburgh today. That's the obvious answer, answered. But for me the less obvious answer is that, most of my life is photographic in some way, so a lot of my interaction with the visual world is photographic, so, you know, living a life on X or Twitter or social media you're seeing photographic images all of the time, both obvious ones and less obvious ones. So I think that, you know, we live our lives so visually now watching TV, watching media formats, which are all photographic in most senses. So, the reason why it's a hard question to ask - answer is because I think photography invests, inveigles itself into so many other parts of our lives that it's easy to forget about where it exists and just how much we're doing photographic every single day.

ROMA AGRAWAL

So what are some of the latest innovations that you're seeing in this space? Could you give us a couple of examples?

PROFESSOR ABHIJIT GHOSH

I mean, if you just see self-driving cars, right? I mean, they're assisted with a whole bunch of sensors, including visible regular cameras, but increasingly rely on other sensors like time of flight imaging and radar to give additional signal. And then, those images are not being interpreted by a human. They're being interpreted by a computer, actually an AI algorithm. Which is fusing multiple channels of these information with a deep neural network that's been trained to interpret certain scenarios as some things, whether the car needs to slow down or speed up. A more consumer facing application is in modern devices like Apple just came out with the Vision Pro. Right? And that has various bunch of cameras on the outside looking at the world where the person is located to essentially, create the pass through image that their screens are showing them as if creating the, the presence of a mixed reality. But they're not just cameras. They also have time of flight sensors to do 3D detection and mapping and understanding of where you are within the room and where should the computer graphics be plotted in front of your eye within this 3D space?

DR. GEOFF BELKNAP

Photography is surprisingly key to so many different forms of scientific practice and experimentation. As one example, during the COVID pandemic, the rush to find a vaccine was predicated on the need to understand what the virus looks like and what its actual realistic idea of the image itself and the positioning of the corona on the top of the virus and and exact positionality of the parts of the virus. And in order to do that you need visual technologies. So that is, to a large extent, photographic right? So using some very modern technology, such as cryo-EM, which is electron microscope which we have been using for a long period of time but the amazing innovation that's happened over the last decade or so is being able to do electromagnetic imaging of biological live specimen. On many of the Covid teams working across the globe, were using tools like cryo-EM to be able to capture a series of what we might call photographic images. So that they could, as Abhijeet has explained, use computational methods to essentially create a video, again using a term that's not necessarily true, but to create a series of images, stacking those sets of singular microscopic - I mean past the kind of scope of human imagination of how small these things are - but using very small pseudo photographic technologies to capture series of what in history or history of photography we call verisimilitude, which is exactness. So what is in front of the camera is what you are seeing on the film. There are problems with that, but that's the the role of photography ultimately to represent something true in the world. So this cryo-EM is capturing the true positions of the corona on the virus in a series of images, and then being able to stack that together has allowed these group of scientists to be able to create a realistic model of what the virus actually looks like.

ROMA AGRAWAL

What strikes me from this conversation is that the current applications, you know, modern technology with photography and capturing images. Is trying to reduce the size of what we can see because obviously, visible light has a particular wavelength, you can't see really stuff smaller than that. So we're looking at how can we capture images smaller than that? But then there's also this speed thing, because some of these processes that happen in a biological cell are so quick that you need very, very short pulses in order to capture them. But if we go back in history, Geoff, what we were trying to do was simply replicate, like, my face onto a film and create a portrait. So could you maybe start us off from where the development of photography and the camera came? What were you trying to achieve?

DR. GEOFF BELKNAP

Photographs as portraits come slightly later than very early photography. So the things that people were taking photographs of, so the famous inventors of photography are the British inventor William Henry Fox Talbot and the French inventor Daguerre. And they're creating two very different processes. Daguerre is making one on metal, Talbot’s making one on paper, but the fundamental aspect of those is it takes minutes to expose an image. So whatever is in front of you can't be moving, ultimately. So, human subjects come later. Once you get greater photosensitivity and the ability to capture light quicker essentially and fix it onto the paper, so the very first forms of images are things that are easier to capture. So empty street scenes, architecture, landscape. I think portraiture comes later. It comes about a decade later for some very early in the 1840s, but really comes a bit later. There's many things I love about photography. But what, from a philosophical level, is amazing about photography is it is the tool, the primary tool that allows us to break the limits of human observation, right? So we are as humans only able to internalise and understand light as it moves. That's something that is faster than the 10th of a second, the 10th of a second is a real threshold for us, right? If you see something moving faster than a 10th of a second, then it's a blur to us or it's constant motion. Photography past the 1870s - by the time they get really fast emulsions and ability to capture photography at a movement that's quicker than that - can start to see things that we can't see with the human eye. So it can start to explore worlds that are not verifiable by us. We can't say that is true because I've seen it, which mostly before that period someone could say “I've seen that thing that looks like what I saw that thing looked like”. And that's the real amazing and that’s why we have photography, the real importance of photography and its legacy now is because it can break that threshold and it can give us views into worlds which we will never be able to see, imagine, look at, verify with our human eye.

ROMA AGRAWAL

And Geoff, could you also then tell us what the impact of digital photography, you know this sort of big change that happened in the 1970s, is on the trajectory of the camera?

DR. GEOFF BELKNAP

So we still call a analogue camera and a digital camera the same thing. We think of them in the same way, but they are fundamentally different technologies that allow us to do fundamentally different things. So you know an analogue camera is based on the concept of light and some kind of physical chemical emulsion capturing that image. When you get to the digital era and we get these, these amazing innovations that invent the pixel, the ability to capture parts of light onto pieces of silicon. Essentially, right, so the substrate is rather than paper or something else is silicon and it's capturing light into those small little pixels and then using mathematics and algorithms to combine that together to make and to show us an image. Once we have these innovations and the ability to capture that and to save that and record that and then the scope of photography, I think, fundamentally changes. You know if you look at the archives of Photographic history, there are billions of photographic images from the 19th century through the 20th century. Once you have the advent of the digital age. You have billions of photographs taken in a day, right? So the scale of images and content being created in a moment. It changes ultimately, what we can do with it.

PROFESSOR ABHIJIT GHOSH

I would take this further and say it's that innovation of digital recording which is powering, essentially, the AI Algorithms of today, because there is so much content to source from essentially.

ROMA AGRAWAL

Abhijit, could you expand a little bit on that? Maybe could you talk to us about how research and the technology of cameras has evolved to allow us to create these images of the, either the natural worlds or astronomical worlds, these worlds that Geoff was saying that, you know, we just simply can't comprehend with our own eyes?

PROFESSOR ABHIJIT GHOSH

The digital age ushered in essentially a commoditization of this technology and shrinkage of its size, right, so you first went from very big, heavy equipment with big lenses, the digital SLRs replacing the analogue SLRs, so to speak. Then you got the point and shoot cameras that were the first essentially, commercially used commodity item that everybody had. And then since the advent of the mobile phone and the high quality cameras on them, it's just gone up another order of magnitude in terms of how many images people are taking all the time. And once a lot of this data, not all of this data, but a lot of this data becomes available publicly, that becomes essentially what modern computer vision and computer graphics algorithms and increasingly now AI algorithms are essentially learning representations of the world. A lot of these images people would also tag them with, describing the scene or what it was, whether it was a landscape or a person and a person wearing a certain dress. So these become additional, essentially, labels that AI algorithms then learn to relate a certain set of text to certain images that are getting created.

DR. GEOFF BELKNAP

One of the really fascinating aspects of photography is in the modern age, it's just so much, how much it sits behind our understanding of data of where data informs the world's we are living in and are going to be living in, and whether that's creating the AI models. You know, you can't have an AI model without data. That data needs to be fed from somewhere and we haven't really, I don't think we've thought yet of the consequences of that being visual data, purely visual data and the complicated mess that is visual data, because it comes with things that we can't see in the metadata within it, has biases and complicated bits in it that might not have meant to have informed in that, in that content, and it has direct implications for how we're living in the world. Whether it's about privacy or the way in which we are using these tools for things like surveillance or understanding populations and, or ultimately, the consequences and or benefits of things like AI and AI modelling. Which we are so early on and that I don't think we yet understand the full ethics of it.

ROMA AGRAWAL

I want to pick up on that idea of bias that you just mentioned, again with some of the research I was doing I, I learnt that these early daguerreotypes, or these films that were used, were calibrated to white skin. And so if black people or people with darker skin were being photographed, they would look very kind of flat and washed out. So this is almost like there's a bias that's baked into the engineering itself. So maybe Geoff, could you tell us maybe a few of the kind of historical impacts of this and then Abhijit, I'd love to know of, you know how that's kind of carrying into the modern day.

DR. GEOFF BELKNAP

I think we need to remember that even though these tools are scientific and technological, so we have created them and we've made choices. But if we're looking at analogue photography, there isn't one type of photography, it is a number of different chemical emulsions and ways in which you use your, usually grains of silver salts, to create your photosensitivity, and then you get the industrialisation of photographic processes. There's many different, kind of, photographic emulsion makers and printmakers. And all of those were competing for markets, but they were also trying to create IP around their specific type of photographic emulsion, and many of those were built by white people, for white people, they would calibrate the sensitivity of their black and white and their colour film to the palette of the white skin tone, right when they were testing it, they were testing it on white people for white people, that wasn't just isolated to the 19th and 20th century. You know, you had early digital cameras and some of the digital makers in the last 20 or 30 years, which we're having the same problem. There is the famous problem of gaming technology, so I believe it was Xbox when they were coming out with their VR model. So essentially they’re cameras that were on top of the Xbox and you would be able to interact with the game through this camera. The early models of that didn't work on non white faces. So I think it's an important but also a stark reminder that none of these technologies are neutral, we're all making choices when we are creating these technologies. And if we don't have enough thinking into who the people are gonna use it and what kind of technologies we need to build then we are going to ultimately, well, we will always inform it with bias.

PROFESSOR ABHIJIT GHOSH

Absolutely. Yeah. So I mean, from a technical perspective, like, yeah. If most databases only contain majority light skinned people as the subjects in the database, then that database is used to train an AI algorithm, then it learns to only recognise, or with higher accuracy, essentially Caucasian skin and not work so well for either recognition or reconstructions. So in computer graphics we are mostly doing generation and reconstruction rather than recognition, but the same problem exists. So for instance, we study the biophysical tissue optics of human skin because we need to simulate that in computer graphics, but a lot of this tissue optics literature itself has had primarily only studied white skin. Until much more recently. When I was a PhD student I was reading skin papers in computer graphics, and they were depicting mostly only white skin.

ROMA AGRAWAL

Could you talk to me a little bit about the idea of the power that people hold when they can take images, or you know, what is the responsibility or the ethics for somebody that holds the camera? Geoff, maybe we can start with you.

DR. GEOFF BELKNAP

The power of taking a photograph is the power of making a record right? A record for posterity, for yourself, for data, for whatever it might be. It has influence. So you know, we can't think of photography without thinking of just how important it is for our own personal familial records. You know, we, most people listening to this will have a family archive of photographs somewhere, so that power is phenomenally important. It is phenomenally important for now, but it also means that it is our record of ourselves. I can't think of, my life now without thinking of just how many photographs of myself and my own past of my family members, my partner, my dog, my friends that I have in my phone or my archive. I don't know how to live in the world without understanding that, to a certain degree, like I don't know what the world would be like if it wasn't photographic, because I've never lived in a world where it wasn't. It is informed and changed who I am. It makes me see the world in a very specific way.

ROMA AGRAWAL

Abhijit, what's the power of photography and camera in science and research?

PROFESSOR ABHIJIT GHOSH

My focus is highest quality appearance and shape capture of real world objects that we want to digitise and subjects, human subjects or faces essentially to create very realistic digital representations of them to drive photorealistic avatars. So in this scenario we look at combination of computational photography, so some sort of special photography, not just regular photography and computational illumination, where we will design the lighting conditions around the subject in such a way that we will not take one photograph in one lighting condition, but several photographs in certain set of complementary lighting conditions. Which then give us information about the highest quality facial shape at the scale of skin pores and fine wrinkles, so like a 10th of a millimetre scale and then a highest quality decomposition of the texture into constituent surface reflection and subsurface reflection. So colour in skin is the result of light penetrating skin, getting absorbed in the various skin chromophores and coming back out, which is why my skin colour is different from your skin colour. And then there's the shine on the top of our skin which is the result of direct reflection of the oily layer and on top of the skin. So we usually want to separate these two components to drive a computer graphics simulation. So these are all technical terms that I'm mentioning, but I'm trying to explain them hopefully, so this is this decomposition that we want to do and we want to do this in as easy as possible setup. So, we started with light stages, which is like a full sphere of control lights in which a Hollywood actor would go in and be captured with hundreds of controlled LED lighting and high quality. The cameras down to now at Lumirithmic we are doing the highest quality scanning with just essentially a phone camera combining the capture with downstream computational algorithms and AI to essentially predict all of those appearance maps that cannot be directly captured with the mobile phone.

ROMA AGRAWAL

And the idea of this being that you drastically improve the quality of the output?

PROFESSOR ABHIJIT GHOSH

You improve the quality while reducing the size and cost of the apparatus, so this has applications not just in surveillance but also in astronomy. So telescopes required big fat lenses. Now you can make a much cheaper, lighter thing that you can easily send to space, for instance if you want to set it up on an International Space Station or even transport it from locations and it still captures very high quality images.

DR. GEOFF BELKNAP

The example of astronomy in the use of photography. It does go back to the 19th century, you know they've been using photography to capture the sky for more than a century. But I think the importance of photography as a tool for exploring the universe is, not little understood, but maybe publicly little understood. You know, when we think of Hubble, for example, that's just a really big camera. They're just gigantic cameras pointing at the sky. You know, we probably think they're just big tubes of telescopes, but ultimately they're capturing light and data from millions of years and miles away, billions of years and miles away. And that technology of the camera, of the photograph, even though it's hard to think of them as photographs. Have led to the big discoveries in astronomy, so that you know, the black hole discoveries recently, those are photographic discoveries and all of these, these big changes, the ability to map the skies and know where light is coming from, where planets are, those are all photographic discoveries.

ROMA AGRAWAL

What does the future of the camera look like to you, Abhijeet?

PROFESSOR ABHIJIT GHOSH

So to me it's increasingly more generalised sensing rather than just visible light sensing. So, as I mentioned, we're already looking at combining a camera with maybe things that are imaging either in the ultraviolet or infrared spectrum or even maybe not a visible camera at all, but like a time of flight, depth sensor or even millimetre wave of radiation which penetrate at different depths through certain things, if you want to do subsurface imaging. It's also happening in commodity apparatus like your phones, they all come with time-of-flight sensors, if it's an iPhone or even Apple's vision Pro headsets. So I think generalised imaging, I mean, Hubble was a great example, right? It was an actually optical telescope, but the next generation of it, the James Webb telescope, is a radio telescope. And now it's spanning the entire electromagnetic spectrum, in its search of the universe, right? That's how I see things expanding.

ROMA AGRAWAL

What about you, Geoff?

DR. GEOFF BELKNAP

I take what Abhijit has said, which is I think, that is absolutely the case that you know it's going to get further and deeper and more complicated. I think for me as a historian, where camera technology is going is a question of the social impact of these technologies on our lives. It has always had a social impact. But I think we're at a crux, dealing with some really complicated, sometimes largely problematic, but big issues that are going to fundamentally change how we experience and understand the world. So an example, I mean AI we've already talked about AI modelling, that is to a large degree driven by photographic data. Where AI takes us next I think is a big question that we haven't resolved. And then associated with that are other more obvious visual technologies such as deep fakes, or the ability to represent or to manipulate and change something that is happening in front of you that's not actually happened. That we can use these technologies to trick, to evade, to convince us that something is happening and we are getting to a point of photorealism, for lack of better word, in the work of this so that our sense of distrust in what's in front of us is becoming alighted, so we don't actually know what to trust anymore. That's not a new phenomenon and it's something that's always happened with photography in particular, but I think we're going to get to a point, especially with the confluence of social media and our online digital lives, where our ability to parse truths from untruth is no longer there. And this can have big implications to our political choices, to our social choices, to the way in which the world interacts with each other and I think that's, that's where we're going is we're, we haven't yet got to that crossroads where the consequences of us not being able to do that will have direct large scale impact on our lives, but I think there will be something like that right? We will reach a crossroads.

ROMA AGRAWAL

How do we deal with that?

DR. GEOFF BELKNAP

My simple answer is media literacy. You know, become more competent and cogent and fluent in how you think about these things. Be critical, be conscientious. Try to think about what you're looking at, at all times and understand the science as much as you can, understand where things are going. The amazing things that they're going, but also the problematic ways in which they're going. And just scrutinise things, especially when they're online.

PROFESSOR ABHIJIT GHOSH

Yeah. For me, another thing that's happening is in another inflexion point in technology, which is we went from maybe black and white photography to colour photography. And then colour display and that changed the world. And now we're getting to the point where we're moving from that to, essentially 3D displays and 3D capture and 3D imaging, which means with devices like head mounted displays, which are essentially potentially passed through mixed reality ones. I think there will be a generation coming that is more used to even seeing content in 3D superimposed with their reality, so they are much more tuned to living in mixed reality. And so cameras will start being tools to source and embed additional information or additional imagery that's not there, present to the naked eye, and I think that also has social implications. It will have a huge impact. Variables like smart glasses, essentially all of our glasses will have augmented reality, additional information that we could directly just see there.

ROMA AGRAWAL

One of the things that I'm picking out from what you're saying is about regulation and trying to, I guess, stay ahead of the curve in terms of regulation to make sure that our rights are protected.

PROFESSOR ABHIJIT GHOSH

Yeah, it's difficult, so for computer graphics applications and training AI neural networks in my lab whenever we've scanned any subject, we've had them actually fill out permission forms and we've been very explicit about what kind of permissions are they giving. And so, in our case it's mostly for, in the academic side, just academic research and publication, even on the company side, just for training a neural network. Never to reproduce that exact same face the way they appear to be and strictly like that, their data belongs to them and stuff like that. Now, not necessarily all companies do that and so that definitely has implications. With GDPR rules, it definitely is more restrictive in Europe, but in other countries rules could be much more relaxed and data could be used. But beyond that, right, I mean anytime you're on the street, everybody has a camera. Everybody's constantly recording, so we are constantly getting recorded by other people without necessarily our explicit permission. So we also have to be, probably in this mindset that, if I'm in a public place I am, I guess, kind of giving my implicit permission to be in someone's photograph in the background, even if that was not their intention.

ROMA AGRAWAL

Let's move on to the future then. How would you like to see the future of cameras develop or what do you see on the horizon?

DR. GEOFF BELKNAP

For me, I guess the answer is a hopeful one that I would like to see, rather than I can see happening immediately. And that is some kind of set of tools for all of us, global communities, to be able to engage with the current digital archive of photography that, you know, we have millions to billions of photographic images that include ourselves. But the problem with the digital age is we have more content than we're ever able to actually consume or go through or manage or look at. And I don't yet think that we actually have the tools other than you know, some of the tools that you have on your own digital archive to remind you of this thing happened 10 years ago. But I think we can get a lot more nuanced than that. And then combined with that from the, kind of scarcity aspect of that, is our digital record of our analogue collections is really, really scarce. So you know, the fact that we have these millions to billions of analogue photographic images that sit in our homes or our institutional archives. Only a very small percentage of those have actually been digitised and made available online through publics, and there's just so much history there. There's so much to see, to know and you know, as a historian I can go in and work with those archives in person. But there is, there's so much digital affordances available once we digitise things and connect them. I would love to see us go in that direction. You know some kind of way of combining the evidence of our present and our past of photographic images would be amazing.

ROMA AGRAWAL

I feel this way after a lot of my conversations but particularly today that it's not just about the engineering or the science, it's about how we create that engineering or science. What are the biases or the inputs that we as humans are putting into it? And how that then affects the impact of that engineering or technology on our society, these things are all intertwined. The ethics, the engineering, the power, the structures, the technology. One can't exist without the other. You've been listening to Create The Future, a podcast from The Queen Elizabeth Prize for Engineering and Peanut and Crumb. This episode was presented by me, Roma Agrawal and produced by Tess Davidson. If you enjoyed this, make sure to listen to future episodes with conversations from pioneering engineers, designers, technologists and thinkers. To find out more, follow QEPrize on Twitter, Instagram and Facebook. Thanks for listening. See you next time.