Motion energy filtering in Python
Published:
As a fun exercise I translated George Mather’s motion energy filtering demo from MATLAB into Python.
The GitHub repo with Python code and stimulus files can be found here.
What is motion energy filtering?
Motion energy filtering revolves around a surprisingly powerful observation: that motion manifests as oriented structure in space-time. If you’re reading about this for the first time and find that idea completely incomprehensible (it certainly was for me!) I’d recommend checking out this blog by Jacob Yates, which does a far better job of explaining things than I ever could. Seriously, go check it out, it’s entertaining AND informative. If, however, you want to just soldier on, really all you need to know is that motion = orientation in space-time.
Motion energy filtering involves building detectors (‘filters’) which are sensitive to orientation in space-time, providing an elegantly simple means of detecting motion. To make things extra interesting, its generally thought that this is (part of) what the visual system does to sense motion!
Below I’ve plotted the filters generated in the demo. Notice how they contain diagonally oriented streaks… that’s essentially what enables them to look for oriented structure in space-time (i.e., motion)! In fact, if you cycle through each time-slice of the filters you will see what looks like ‘pulses’ of leftwards and rightwards motion, giving a hint at what they are trying to detect. Jacob’s blog has a really nice demo of this.
Additional resources
The original paper on motion energy filtering by Adelson and Bergen can be found here. It’s somewhat dense, but is a beautiful paper once you wrap your head around things. I’d recommend reading it after going through Jacob’s blog, before diving into the code.
George Mather’s website contains detailed descriptions of the steps involved in the demo code. I’d recommend having this open as you go through the Python translation.
The current demo only considers 1D motion (e.g., motion along a single axis). For more complex (2D motion) implementations see this nice MATLAB toolbox by Anne Urai and Klaus Wimmer, which contains links to additional resources.