# Motion microscopy for visualizing and quantifying small motions

1. aComputer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139;
2. bDepartment of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139;
3. cHarvard-MIT Program in Health Sciences and Technology, Cambridge, MA 02139;
4. dResearch Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139;
5. eGoogle Research, Google Inc. Cambridge, MA 02139;
6. fDepartment of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA 02139;
7. gSchool of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138;
8. hDepartment of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218;
9. iHopkins Extreme Materials Institute, Johns Hopkins University, Baltimore, MD 21218
1. Edited by William H. Press, University of Texas at Austin, Austin, TX, and approved August 22, 2017 (received for review March 5, 2017)

1. View larger version:
Fig. S1.

Increasing the phase of complex steerable pyramid coefficients results in approximate local motion of the basis functions. (A) A 1D slice of a complex steerable pyramid basis function. (B) The basis function is multiplied by several complex coefficients of constant amplitude and increasing phase to produce the real part of a new basis function that is approximately translating. Copyright (2016) Association for Computing Machinery, Inc. Reprinted with permission from ref. 37.

2. View larger version:
Fig. S2.

A 1D example illustrating how the local phase of complex steerable pyramid coefficients is used to amplify the motion of a subtly translating step edge. (A) Frames (two shown) from the video. (B) Sample basis functions of the complex steerable pyramid. (C) Coefficients (one shown per frame) of the frames in the complex steerable pyramid representation. The phases of the resulting complex coefficients are computed. (D) The phase differences between corresponding coefficients are amplified. Only a coefficient corresponding to a single location and scale is shown; this processing is done to all coefficients. (E) The new coefficients are used to shift the basis functions. (F) A reconstructed video is produced by inverse transforming the complex steerable pyramid representation. The motion of the step edge is magnified. Copyright (2016) Association for Computing Machinery, Inc. Reprinted with permission from ref. 37.

3. View larger version:
Fig. S3.

Noise model of local phase. (A) A frame from a synthetic video with noise. (B) The real part of a single level of the complex steerable pyramid representation of A. (C) The imaginary part of the same level of the complex steerable pyramid representation of A. (D) A point cloud over noisy values of the real and imaginary values of the complex steerable pyramid representation at the red, green, and blue points in B. (E) The corresponding histogram of phases.

4. View larger version:
Fig. S4.

A comparison of our quantitative motion estimation vs. a laser vibrometer. Several videos of a cantilevered beam excited by a shaker were taken, with varying focal length, exposure times, and excitation magnitude. (A) A frame from one video. (B) Motions from the motion microscope at the red point are compared with the integrated velocities from a laser vibrometer. (C) B from 0.5 s to 1.5 s. (D) B from 11 s to 12 s. (E) The correlation between the two signals across the videos vs. the RMS motion size in pixels, measured by the laser vibrometer. (F) The correlation between the two signals across the videos vs. the RMS motion size in pixels measured by the laser vibrometer. (G) The correlation between the signals vs. focal length (exposure time, <mml:math><mml:mn>490</mml:mn></mml:math>490 <mml:math><mml:mi>μ</mml:mi></mml:math>μs; excitation magnitude, 15). (H) Correlation vs. exposure time (focal length, <mml:math><mml:mn>85</mml:mn></mml:math>85 mm; excitation magnitude, 15). Cropped frames from the corresponding videos are shown above. (I) Correlation vs. relative excitation magnitude (focal length, <mml:math><mml:mn>85</mml:mn></mml:math>85 mm; exposure time, <mml:math><mml:mn>490</mml:mn></mml:math>490 <mml:math><mml:mi>μ</mml:mi></mml:math>μs). Only motions at the red point in A were used in our analysis.

5. View larger version:
Fig. S5.

An evaluation of our motion estimation method and Ncorr (30) on a synthetic dataset of images. (A) Sample frames from real videos used to create the dataset. (B) Sample of synthetic motion fields of various motion size and spatial scale used to create the dataset. (C) The motion microscope and Ncorr are used to estimate the motion field, and the average relative error is displayed for both methods as a function of motion size and spatial scale. Both methods are only accurate for spatially smooth motion fields. Our method is twice as accurate for spatially smooth, subpixel motion fields. Ncorr is more accurate for larger motions.

6. View larger version:
Fig. S6.

Magnification of a spatial smooth and temporally filtered noise can look like a real signal. (A) Frames and time slices from a synthetic 300-frame video created by replicating a single frame 300 times and adding a different realistic noise pattern to each frame. (B) Corresponding frame and time slices from the synthetic video motion magnified 600<mml:math><mml:mo>×</mml:mo></mml:math>× in a temporal band of 40 Hz to 60 Hz. Time slices from the same parts of each video are shown on the right for comparison.

7. View larger version:
Fig. S7.

Using a probabilistic simulation to compute the noise covariance of the motion estimate. (A) A single frame from an input video, in this case, of an elastic metamaterial. (B) Simulated, but realistic noise (contrast enhanced 80<mml:math><mml:mo>×</mml:mo></mml:math>×). (C) Synthetic video with no motions, consisting of the input frame replicated plus simulated noise (noise contrast-enhanced 80<mml:math><mml:mo>×</mml:mo></mml:math>×). (D) Estimated motions of this video. (E) Sample variances and sample covariance of the vertical and horizontal components of the motion are computed to give an estimate of how much noise is in the motion estimate.

8. View larger version:
Fig. S8.

Synthetic experiments showing that our noise covariance estimation, which assumes that the motions are zero, is also accurate for small nonzero motions. (A) The motion between synthetic frames with noise and slightly translated versions (not shown) are computed over 4,000 runs at the marked point in red for several different translation amounts. Each time, different but independent noise is added to the frames. (B) The sample covariance vs. motion size. (C) Relative error of horizontal variance vs. motion size. (D) Relative error of vertical variance vs. motion size. C and D are on the same color scale.

9. View larger version:
Fig. S9.

Validation of our noise estimation on real data. (A) A frame from a real video of an accelerometer attached to a beam. (B) The accelerometer shows there are no motions in the frequency band 600 Hz to 700 Hz. (C) The variance of our motion estimate in the 600- to 700-Hz band serves as a ground-truth measure of noise, as there are no motions. (D) The estimated noise level vs. intensity for a signal-dependent noise model and a constant noise model. (E) The noise estimate produced by our Monte Carlo simulation with a signal-dependent model. All variances are of the motions projected onto the direction of least variance. Textureless regions, where the motion estimation is not meaningful, have been masked out in black. (F) Difference in decibels between ground truth and E. (G) Noise estimate produced by the Monte Carlo simulation with a constant noise model. (H) Difference in decibels between ground truth and G.

10. View larger version:
Fig. 2.

Exploring the mechanical properties of a mammalian tectorial membrane with the motion microscope. (A) The experimental setup used to stroboscopically film a stimulated mammalian tectorial membrane (TectaY1870C/+). Subfigure Copyright (2007) National Academy of Sciences of the United States of America. Reproduced from ref. 12. (B) Two of the eight captured frames . (Movie S1, data previously published in ref. 13). (C) Corresponding frames from the motion-magnified video in which displacement from the mean was magnified 20<mml:math><mml:mo>×</mml:mo></mml:math>×. The orange and purple lines on top of the tectorial membrane in B are warped according to magnified motion vectors to produce the orange and purple lines in C. (D) The vertical displacement along the orange and purple lines in B is shown for three frames. (E) The power spectrum of the motion signal and noise power is shown in the direction of least variance at the magenta and green points in B.

11. View larger version:
Fig. 3.

The motion microscope reveals modal shapes of a lift bridge. (A) The outer spans of the bridge are fixed while the central span moves vertically. (B) The left span was filmed while the central span was lowered. A frame from the resulting video and a time slice at the red line are shown. (C) Displacement and noise SD from the motion microscope are shown for motions in a 1.6- to 1.8-Hz band at the cyan, green, and orange points in B. Doubly integrated data from accelerometers at the cyan and green points are also shown. A time slice from the motion-magnified video is shown (Movie S2). The time at which the central span is fully lowered is marked as “impact.” (D) Same as C, but for motions in a 2.4- to 2.7-Hz band.

12. View larger version:
Fig. S10.

The motion microscope is applied to a pipe being struck by a hammer. (A) A frame from the input video, recorded at 24,096 FPS. (BF) (Top) A frame is shown from five motion-magnified videos showing motions amplified in the specified frequency bands (Movie S3). (Middle) Modal shapes recovered from a quantitative analysis of the motions are shown in blue. The theoretically derived modal shapes, shown in dashed orange, are overlaid for comparison over a perfect circle in dotted black. (Bottom) Displacement vs. time and the estimated noise SD is shown at the green point marked in A.

13. View larger version:
Fig. 4.

The motion microscope is used to investigate properties of a designed metamaterial. (A) The metamaterial is forced at 50 Hz and 100 Hz in two experiments, and a frame from the 50-Hz video is shown. (B) One-dimensional slices of the displacement amplitude along the red line in A are shown for both a finite element analysis simulation and the motion microscope. (C) A finite element analysis simulation of the displacement of the metamaterial. Color corresponds to displacement amplitude, and the material is warped according to magnified simulated displacement vectors. (D) Results from the motion microscope are shown. Displacement magnitudes are shown in color at every point on the metamaterial, overlayed on frames from the motion-magnified videos (Movies S4 and S5).

#### Online Impact

• 99132880 2018-01-23
• 802899879 2018-01-23
• 295573878 2018-01-23
• 352668877 2018-01-23
• 984633876 2018-01-23
• 545928875 2018-01-23
• 976569874 2018-01-23
• 871324873 2018-01-23
• 263462872 2018-01-23
• 577161871 2018-01-23
• 255603870 2018-01-23
• 117346869 2018-01-23
• 90982868 2018-01-23
• 663415867 2018-01-23
• 793874866 2018-01-23
• 843582865 2018-01-23
• 864971864 2018-01-22
• 258841863 2018-01-22
• 957295862 2018-01-22
• 553518861 2018-01-22