The Future of Swarm Robotics

GEORGE IMAFIDON

If you've ever looked up into the sky and seen a flock of birds dancing in the air, or a shoal of fish in the sea darting through the water, or perhaps a colony of ants marching and carrying food … you might have wondered how animals actually manage to coordinate their behaviour in such mesmerising ways. I was in Scotland recently. And I was doing a jog as I usually would and somehow I found a flock of birds that were actually chasing me. Very weird but in the middle of that terrifying experience there were some questions that were running through my mind. How do they know where to go? How do they make decisions? And how do they behave like one giant organism? Biologists have been trying to answer these questions and understand this animal collective behaviour for quite a long time. But in recent years, engineers have become very interested in this area too. What if we could harness the rules behind swarms to create systems of self-organising, synchronised robots? Could swarms help us with warehouse logistics, detecting pollution, or even search and rescue missions? I'm George Imafidon and you're listening to the Create The Future podcast from the Queen Elizabeth Prize for Engineering. And today we're exploring swarm engineering. This is an exciting new field that's dealing with some fascinating issues in computing and robotics.

RAZANNE ABU-AISHEH

Robots are just programmed with simple rules and then through that you get the emergent behaviour where an individual robot wouldn't be able to do that by itself.

GEORGE IMAFIDON

My guest today is Dr. Razanne Abu-Aisheh from Bristol University, an engineer who is particularly interested in the way we as humans interact with robots and how we perceive them when they're actually in the swarm.

RAZANNE ABU-AISHEH

Hi, I'm Razanne. I'm a Senior Research Associate at the University of Bristol in Swarm Engineering. So I started with a Bachelor's in Electronic Engineering, and then I just continued and did a Master's in Embedded Systems. So still more on the hardware side. And then I worked in industry for two years, but more on the software side. So I wanted to get a bit more experience in software engineering. And I worked on clouds and decentralised systems. And then that kind of like led me into feeling like I want to combine both together. I want like the best of hardware and software. And that kind of led me into getting a PhD in robotics because you get to work on like the hardware and the software and like the body and the brain kind of thing. And for my PhD, I worked on multi-robot systems. So I worked on exploration mapping with multi-robot systems, which we'll talk a bit about later on. And naturally that kind of like led me into an interest in swarms. And then I got a postdoc at Bristol in swarms and I've been there since.

GEORGE IMAFIDON

Many people would have heard of swarm engineering for the first time. And I'm very curious to learn more about the field. I know it's one that is very recent.

RAZANNE ABU-AISHEH

Funny enough, I didn't even hear about swarms until I started my PhD, even with a background in hardware and software and all of that. Swarm engineering is inspired by nature. So it's inspired by swarms that we see in nature. And the biggest example that I like to give is flocks of birds. So a lot of us, we see these kind of complex stances that flocks of birds do, and it looks like they're all told what to do. And they all have this big brain that kind of like instructs them and orchestrates them. But what's really happening is that each bird is just reacting to the local environment around it. So that could be other birds around it or just like the wind or the external environment. And then through those kind of like local interactions, when you zoom out, you get this complex behaviour where the group behaviour is bigger than the sum of its parts. And it's the same with swarms of robots. We took inspiration from that. So swarms of robots are basically groups of robots - typically at least three robots, the more the better so you can get that emergent behaviour - and these robots are just programmed with simple rules and behaviours so that they can react to the world around them. And then through that, you get the emergent behaviour where they end up completing complex tasks and missions as a group together, where an individual robot wouldn't be able to do that by itself.

GEORGE IMAFIDON

Are there any other examples, because that's the first thing that comes into my mind is flocks of birds? Are there any other examples that you think people might relate with or may have seen before?

RAZANNE ABU-AISHEH

In nature as well, there's bees and ants. I think ants is one of the very common ones because a lot of the behaviour that we programmed to swarm robots is inspired by ants. So there's foraging, for example, the way you see like a group of ants getting together and picking up these like really big items of food compared to their size. So one ant definitely couldn't get that done by itself. But then you see these groups kind of working together and collaborating, and then they can kind of like carry this food and take it back to the nest. And then they also like drop these pheromones. So that's something called stigmergy where they interact with each other through the environment, not directly. So for example, some ants go exploring, they find some sorts of food, they drop some pheromones and then the other ants pick up on that and they go to help it out and they just collaborate on that.

GEORGE IMAFIDON

Watching ants as a kid is probably one of the most fascinating things because everyone, everyone has seen it. Right? So from your perspective, why exactly are engineers interested in swarms? How can we use it? What properties do they have that we actually want to harness?

RAZANNE ABU-AISHEH

So if we go back to the example of birds, they're very flexible because they're just reacting to the environment around them. So for example, if you suddenly get a windy environment or something, they can just adapt. So they're quite adaptable because they're not pre-programmed with exact movements to make or exact things to do. It's kind of like conceptual behaviour, which makes them reactive. The other thing is, for example, they're scalable. So you can keep adding more birds. They'll just pick up on kind of like what's happening and they'll just become part of like the complex dance. Also, it's robust because if some of the birds just fall to the ground or decide that they don't want to be part of the flock anymore, the kind of collective behaviour will still continue. Again, there's this thing where the group behaviour is bigger than the sum of its parts. So the group of birds are better, for example, fighting off predators than an individual bird would be. So with robots, that's also very attractive because you want systems where you can scale them up.

GEORGE IMAFIDON

It's pretty amazing. We hear of the words ‘decentralisation’ in the context of blockchain, but I've never thought about it from a case of birds or nature, where each bird kind of relies on that local information. So it doesn't need any centralised control, which I think is very, very impressive. And the self-organisation, as you said, you can just add something in. So from the swarm engineering point of view, if you were to pre-program something, what happens? Can't they be pre-programmed and then just adapt accordingly?

RAZANNE ABU-AISHEH

They are pre-programmed with some stuff, right? But the difference is in how you pre-program them. So if you have a fully centralised system, you tend to need to know a lot of information about the environment. So they're usually customised to very particular environments, which means they need a lot of infrastructure. And then they're programmed to, again, do very specific things. Whereas with swarms or decentralised behaviour, we use something called behaviour trees a lot of the time. Which is kind of like, if this, you encounter this, then we act in this way. If you're in this kind of like scenario, then do this. So it's more of like, it's less specific. That's what makes it more adaptable.

GEORGE IMAFIDON

What are some of the applications of swarm engineering from your perspective?

RAZANNE ABU-AISHEH

So for me, I think the most interesting applications are the ones that reduce risks to humans. So if we think about very dangerous missions, like search and rescue, there's a lot of risk on the rescuers themselves. So imagine if you just have like a box with a bunch of robots and then you just put them out, take them out of the box and then they start exploring like say there was a fire or like a natural disaster. You just put the robots in the building. They start walking around building a map and then they discover that, oh there's victims here. And then they just send like some information back to the rescuer. So the rescuers know exactly where the victims are and they can go and they can then do their part, right? So that would, first of all, it would speed up the mission in terms of fighting the victims and it would also reduce the risk on the rescuers. So this is something where you can see where humans and robots can collaborate with each other to actually get, again, kind of like an output that is bigger than each one could do individually.

GEORGE IMAFIDON

Agreed. And this is an engineering podcast so we care a lot about the ‘how’. So my next question for you is, how do you actually go about engineering a swarm?

RAZANNE ABU-AISHEH

The first thing we think about is the application, because you want to customise the group behaviour to the application that you have in mind. One big challenge actually in swarm engineering is because you know what you want the group to do, but how do you take that and kind of like break it down into the small simple behaviours that are going to lead to that emergent behaviour? Cause it's very hard to kind of like reverse engineer that. So sometimes we can see, for example, while there's certain behaviour in nature that's already helpful, such as foraging, for example. So if I want to design a swarm for logistics, I can just take inspiration from that. You can be like, OK, well, I have a warehouse. I can have a certain area that can be the nest where I want them to take the food and store it. And then you just kind of break it down into the simple behaviour that each robot needs to be able to do. So for example, a robot needs to be able to detect that there is something it needs to pick up. And then after picking it up, it's like, okay, well, I need to take it and store it. So it needs to start navigating, making sure it doesn't bump into any other robot and making sure it's able to locate the place where it has to store the item. And then once it's there, it's like, okay, now I need to put it down and then I need to continue. So you see, it's like just a few instructions, but it's looping over and over and over again. And then if you have each robot that's just trying to do these simple behaviours, you end up with like all the boxes being sorted out within a certain amount of time.

GEORGE IMAFIDON

So that's where we started to speak about the robots. So can you tell us a little bit more about this hardware and the different robot platforms that people are currently using to do that?

RAZANNE ABU-AISHEH

So at the beginning, people were focusing a lot on you have to have a lot of robots, they have to be simple, low computational power. And they were looking into just applying that kind of swarming behaviour from nature onto robots. So for that, there were a lot of robots, like for example, the Kilobots, which are very small robots, and they just have these little kind of like legs that can vibrate so they can move and they can communicate through infrared. But you can have like a thousand of them. And then you can play with different things like how do they self-organise, make different shapes, how do they reach consensus, things like that. There's other robots, for example, that are slightly bigger with wheels and better communication abilities, like the E-Pucks, for example. And there you can kind of like explore more behaviour that requires maybe more movement. And then now if you look at what we're doing, we're breaking a few of the stereotypes because in our lab, for example, we have robots called the Dots. So those are a lot bigger. Ours has GPUs, it has cameras where it can use those cameras to localise itself with respect to that. It has a lifter that it can use to lift objects up and move them around. So what's happening now is some people are looking into creating more customised robots so that the hardware meets the needs of the mission that they're trying to design for.

GEORGE IMAFIDON

And we spoke about birds and fish. Are people using robots that can fly, swim at the moment? Is that a thing?

RAZANNE ABU-AISHEH

So there aren't really any examples of swarms deployed for the real world. They're still kind of confined to the lab environments mostly. But for example, there are robots that fly. So there are like UAVs, people use UAVs. So in our lab, for example, we have some people that are using robots to study sharks' mouths and how they pick up on (inaudible). To see how we can see if robots could clean the oceans and things like that. So in terms of technology, yes, there are robots that can fly and can swim, but it's more about, again, how they interact with each other, how they’re operated by people, what level of autonomy do they have, are they regulated, and things like that.

GEORGE IMAFIDON

So it would be great to actually speak a little bit more about your research. So you're interested in the way that humans interact with swarms and in particular, the way that we feel about swarms. So maybe you can tell me a little bit more about that.

RAZANNE ABU-AISHEH

So now the big thing in swarm engineering is like, how are we going to move these systems to the real world? And one of the biggest challenges in that is actually not the technology, more people's perception of these systems and how people accept these systems and trust these systems. And one thing that's been interesting is a lot of studies have looked into how to use group motion as a way to communicate information. So there are some studies where, for example, like they had robots move in a more synchronous way and people thought that they looked happier or they looked like they were friends or collaborating and stuff like that. But we realised that there wasn't many studies on like how that impacts trust because if you want robots and humans to collaborate and work together, you need to establish that level of trust. So we wanted to study the impact of swarm motion on people's perception in a system and trust in the system. And to do that, we used a little Dots robot that I talked about to kind of like act as a cloakroom. So we had our participants, they dropped their item off to the cloakroom, the robots would go deposit it and then they leave for a bit. And then when they come back, the robot just brings them the item back. But what they didn't know at the time is that there's two different types of cloakrooms. There's one cloakroom where the robots move in a very swarmy way. They move at angles that appear to be random, designed to look very reactive as if they're adapting on the go. The other cloakroom looks very structured. The robots move at 90 degree angles. They go straight to the box, but both scenarios take the same amount of time. And what we've seen … or the biggest comment that we've got from people is they're like, yeah, but I trust you, the human, right? Like, it's like, if you weren't in the room, I don't know if I would have trusted it. So one big thing that came up is like trust in developers. And then the other thing, which was very interesting, because we asked about reliability, because reliability is quite well related to trust. And for both motions, people were like, well, it deposited my item safely and it brought it back, which means it did its job. So it's reliable. The interesting thing is they seem to be looking at it holistically as a system rather than what each individual robot was doing, because we had multiple robots depositing different items at different times. There were like three people in the room.

GEORGE IMAFIDON

I've got a question for you. When it comes to people and humans in general, everything is about trust. Right? But how do you actually measure trust in that context?

RAZANNE ABU-AISHEH

Absolutely. So like you said, it's very complex because like it's not something where you can just measure it and then here's how much you trust, the trust on this percentage. So we used a validated questionnaire from another study on trust and autonomous systems and they broke it down into different subcategories. So you had familiarity with the system, reliability and predictability of the system, trust in developers. And then we had open-ended questions so people could elaborate on some of their answers. And then we also used eye-tracking glasses. So again, like we're still kind of putting all the pieces of the data together and analysing them. But what we're hoping on achieving is have the different pieces of data validate each other or tell a cohesive story to hopefully give us a better idea on whether they trust the systems or don't trust the systems.

GEORGE IMAFIDON

At the moment with the current level of research, what are some of the reasons for people not necessarily trusting swarms?

RAZANNE ABU-AISHEH

I think one big challenge is people's only understanding of swarms, have they only ever heard of swarms from like sci-fi movies and things like that. There's just this fear of like, what is this technology? Or even the fact that, oh, it's like a swarm of bees or it's like a swarm … like that sometimes scares people. And I think that's one of the biggest things as well, like people's preconception of the systems and what they're supposed to do. And again, like in our experiment we didn't have a huge number of robots. So imagine if you have like a hundred of these robots just roaming around, like we don't know how people as well will perceive that, people might generally not feel like they have enough control over something like that, but that's still a very new field that's recently being studied.

GEORGE IMAFIDON

Sure. And when it comes to perceptions and stereotypes about robots, how do we actually break those stereotypes? So when people come across them, you know, yeah, you can have those better human robot interactions.

RAZANNE ABU-AISHEH

I think the biggest thing there is first of all we have to listen to people. We have to truly understand what it is that they're afraid of, because there might be certain aspects where they do have a point and we need to consider that when we're designing the robots as well, because we, again, as engineers, we have a responsibility. We have to work with sociologists and ethicists and we have to be very considerate and responsible in terms of like how we design these systems. So we have to co-create these technologies with people and with communities. We need to sit with people, listen to them, see what their actual needs are, because not all robots need to exist. You know, we need to think about what robots will actually be impactful, what robots will be helpful, what people need, what they want, and how we can customise the robots we create to that.

GEORGE IMAFIDON

And that leads me on to the last question in terms of your research. What are you looking at next? What is exciting you at the moment? And yeah, what do you really want to explore?

RAZANNE ABU-AISHEH

So actually I've just recently started working with a group of sociologists. So now I'm kind of looking into robot futures. And again, it's a continuation of all the things that I've talked about. So how do we assure that we're designing robots for people? And that we're not just saying that as like a slogan or a term that we're actually listening to people. talking to people. Like for example, with swarms, like where should we really be deploying swarms? Where shouldn't we be deploying swarms? In terms of interaction, what do people prefer? What don't they prefer? What scares them? What excites them? And I think it's just about having a more holistic approach towards designing these systems.

GEORGE IMAFIDON

I asked you about applications and we spoke about some exciting use cases. We spoke heavily on logistics for example and swarms for good … but every coin has two sides. So what about the other side of the coin? There could be military users of swarms or potentially security concerns. So, what are some of your thoughts in terms of the darker sides of swarms and how do engineers actually deal with that? Because it's not something that we can just push to the side and say that has nothing to do with us … and this has a big effect on trust as well. So how do we mitigate the negative consequences of the swarm research?

RAZANNE ABU-AISHEH

So that's a very important thing to consider as well, because I think as engineers sometimes there's a tendency to get carried away with the exciting technology we're working on, right? And to just get stuck in the bubble of, this is so cool, I just want to explore and see where it goes, without actually considering the possible outcomes that can come out of it. So I think it comes down a lot to like the engineers having this understanding of the responsibility that lies on their shoulders, and having to think about the worst case scenario, and having to think about maybe even dystopian outcomes because we do have the tendency to think about utopian outcomes, right? Because, oh my God, this can change the world. This can improve things. This can make it better. But we really forget to think about sometimes the dark side, and thinking bigger, and thinking a bit further into time, maybe, when you're designing these things.

GEORGE IMAFIDON

It shows the importance of the non-technical skills that engineers need to have in terms of the ethics and emotional intelligence, essentially, in order to actually make this a success, which could be a force for good, but also do the reverse and have unintended consequences.

RAZANNE ABU-AISHEH

Absolutely. And I like that you said emotional intelligence because it really is that as well. It needs a lot of reflection. You need to sit there and be like, okay, why am I doing this? Why am I working on this? Like, what do I think the impact is going to be? And it's just that sense of reflection. So I really liked that you said that, yeah.

GEORGE IMAFIDON

Last question from me. What would you say if we zoomed out and think about what is the state of the field right now when it comes to swarm engineering, and where does it actually need to move in the future? We spoke a little bit about it, but what would you say if there was one main barrier, you know, to actually get swarms into the real world? All of these applications we speak of, what is that barrier and when might it happen? Is this something that I can see? I always like asking my guests, is this something that I can see in my local area of Peckham anytime soon?

RAZANNE ABU-AISHEH

I'm not going to give an exact timeline, but more or less, I think the main thing is moving these robots to the real world. And the challenge with that has been, I think, sticking to a lot of requirements that are probably no longer needed. These robots need to be safe, right, if they're going to be in the real world, right? So they need to be a bit smarter. So if we're going to just hold on to the thing of for them to be a swarm, they have to have low computational power and they have to be very simple, then that's going to contradict the need for them to be safe in the real world, for example, to be smart enough to interact with humans in the real world. So we need to probably find somewhere in the middle. Or, for example, like fully decentralised systems, again, like there are some benefits to having partial centralisation, or to be able to alternate between both. So maybe like systems that we call hierarchical systems where it's decentralised to an extent, but then there's also some ability to connect to a centralised entity, for example, if you need at some point, or to have these subgroups so that you can get the benefits of both. And of course, the human, I've said it all throughout, but the human element, so like considering the interaction. So I think we are closer to seeing early deployments of swarms in the real world. So hopefully within our lifetime. Yeah, but I'm not going to give an exact number because then like, if that year comes in and they’re not out there, I'm going to be like, damn it, my prediction was wrong.

GEORGE IMAFIDON

I'm going to hold you accountable, but this is why we do it, right? This is why we do it. We do it so we can accelerate the timelines and it's just good to know, you know, when is it likely to come? And I asked that question because this is a growing field with so many applications, whether it be robotics, AI, environmental science, and there's so many career opportunities. We spoke about the interdisciplinary learning across so many different disciplines, all the problem solving that needs to be had. So if people know, you know, this is around the corner, this is something that is an industry that is likely to come very soon, I think it gets people even more excited.

RAZANNE ABU-AISHEH

And even research, research is fun as well. Like, industry is nice, but it's also really fun and impactful to just be involved in the research phase of these technologies as well, because you can ask more questions and you can reflect more at that phase as well. So I think, I know like sometimes industry can come across as more exciting and stuff, but I really do think there's something very impactful and empowering sometimes about feeling like you're in the research stages of these technologies. So I would also encourage people to also consider those options.

GEORGE IMAFIDON

Agreed. 100%. I think it's a mixture of both, right? Yeah, I've learned a lot more about swarm engineering. I was chased by a flock of birds up in Scotland not too long ago. But yeah, it all kind of brings it all to life in terms of what we can be doing with technology to really mimic nature and exploit the potential out there. So thank you so much.

RAZANNE ABU-AISHEH

Thank you for having me.

GEORGE IMAFIDON

You've been listening to Create The Future, a podcast from the Queen Elizabeth Prize for Engineering and Peanut & Crumb. This episode was presented by me, George Imafidon, and was produced by Anand Jagatia. To find out more about the podcast and the work of the Queen Elizabeth Prize for Engineering, follow QEPrize on Twitter, Instagram, and Facebook.