Combining Event Cameras with Spiking Neural Networks for Bio-Inspired Event-Based Visual Inertial Odometry

The domain of Neuromorphic Computing applies models and concepts from
neuroscience, emulating the computational behaviours observed in biological
neurons through the medium of spiking neural networks, and applying them to
engineering tasks. In recent years, Neuromorphic Computing has seen rapid
growth, with applications that have transitioned from lab-based testing and
simulation, to real-world scenarios. Navigation tasks are the subject of extensive
and varied research due to the variety of implementations afforded by sensor
fusion and the range of possible solutions, e.g. Simultaneous Localisation and
Mapping (SLAM), Visual Inertial Odometry (VIO), visual place recognition.
An entirely event-based system capable of performing VIO on real-world data
has yet to be proposed. Here, we combine event cameras with spiking neural
networks to perform localisation tasks on complex real-world driving data,
creating a spike-based representation of position and orientation. We show how
navigation-oriented neural networks that have been observed in the mammalian
brain, such as head direction, grid, and place cells, can be adapted to track
both orientation and position in real-world driving scenarios through inertial
signals. It is also shown that stereo event-based visual place recognition can
be combined with inertial integration for loop closure, providing a robust
solution that corrects accumulated errors. These results show the viability of
a system that can take advantage of the asynchronous, low-power benefits of
neuromorphic processing in an end-to-end fashion by maintaining sparse event
encoding from the event camera input to the spiking neural network outputs.
It is anticipated that neuromorphic navigation systems can provide an optimal
solution for applications where low latency and low power are constraints,
such as autonomous robotics. This work should be expanded upon with the
development of more complex visual place recognition spiking neural networks,
refinement of the spatial network behaviours, and with increased dimensionality
to allow flying agents to also be tracked accurately through 3D space.