Video Analysis

Figure 10. Tracker: A Video Analysis Application

Tracker video analysis window

Video analysis refers to the process of tracking specific features of interest in a video image and extracting data related to those features. Typically, the data of interest fall into two categories: (a) spatial data (position, shape, orientation, etc.) and (b) pixel data (brightness, color, gradient, etc.). Most applications for introductory physics education fall into the first category, so that is the focus of this section.

Getting meaningful spatial data requires both a mechanism for locating a feature of interest and a coordinate system for converting the video image position of that feature into a scaled world position. In the media framework, the first requirement is met by the Trackable interface, and the second by the ImageCoordSystem class.

Imagespace and Worldspace

The position of a point may be expressed in imagespace or worldspace coordinates.

The imagespace coordinates (or image position) refer to the pixel position of the point on the video image, measured from the top left corner. The positive x-axis points to the right and the positive y-axis points down. Image units are like pixels except that they are doubles, not integers. This means that the center of the top left pixel has coordinates (0.5, 0.5); it is the top left corner of the top left pixel that is at (0, 0)! The lower right corner of a 320x240 pixel image is at (320.0, 240.0).

The worldspace coordinates (or world position) refer to the scaled position of the point relative to a specified world reference frame. The reference frame origin may be anywhere on or off the image and the positive x-axis may point in any direction. The positive y-axis is 90 degrees counterclockwise from the positive x-axis.

A video panel can draw trackable objects in both worldspace (where the world reference frame is fixed) and imagespace (where the video image is fixed). A standard drawing panel can draw only in worldspace.

When analyzing videos, features of interest are typically tracked in imagespace and compared with model overlays in worldspace.

Trackable Interface

The Trackable interface defines no new methods. It is simply an agreement by an implementing class to draw itself correctly in both imagespace and worldspace. Trackable object do this by defining their positions in image coordinates and using a video panel's ImageCoordSystem to convert them to world coordinates when drawing in worldspace.

ImageCoordSystem class

An ImageCoordSystem ("coords") encapsulates the relationship between imagespace and worldspace coordinates by defining for each frame of the video:

  1. the origin: the image coordinates of the world origin
  2. the angle: the counterclockwise angle in radians from the image x-axis to the world x-axis
  3. the scale: the world scale in image units (pixels) per world unit

When the origin, angle and scale are fixed, as they are by default, their values are the same at all times (or, equivalently, in all frames). When fixed, any change to one frame applies to all. If the fixed property of the origin, angle or scale is turned off with the ImageCoordSystem.setFixedXXX(boolean) method, then each frame is set independently.

With these definitions, the coords constructs a set of AffineTransforms, powerful Java objects for manipulating points and shapes in 2D space. Locking the coords prevents any changes to the transforms.

Methods for setting ImageCoordSystem properties are shown in Listing 13.

Listing 13. ImageCoordSystem Methods

public void setLocked(boolean locked);
public void setFixedOrigin(boolean fixed);
public void setFixedAngle(boolean fixed);
public void setFixedScale(boolean fixed);
public double getOriginX(int n); // n is frame number
public double getOriginY(int n);
public void setOriginXY(int n, double valueX, double valueY);
public void setAllOriginsXY(double valueX, double valueY);
public double getAngle(int n);
public void setAngle(int n, double theta);
public void setAllAngles(double theta);
public double getScaleX(int n);
public double getScaleY(int n);
public void setScaleXY(int n, double valueX, double valueY);
public void setAllScalesXY(double valueX, double valueY);

Drawing Trackable Objects

The Trackable interface extends Drawable, so a trackable object is added to a video panel using the usual DrawingPanel.addDrawable() method. Drawable objects that do not implement Trackable (animations, for example) can also be added to the video panel, and they will draw themselves correctly in world space. The table below summarizes the drawing behavior of (1) a trackable object and (2) a drawable (non-trackable) object on a standard drawing panel and a video panel drawing in worldspace or imagespace.

DrawingPanel VideoPanel Worldspace VideoPanel Imagespace
Trackable not drawn drawn drawn
Drawable drawn drawn not drawn

A trackable object saves its image position internally, so when a video panel is drawing in imagespace, the trackable simply draws itself at that position on the panel. When the video panel is drawing in worldspace the trackable first converts its image position to a world position, then draws itself at that position on the panel. The trackable does the conversions with the video panel's coords using the methods listed below.

To determine which drawing space to use, a trackable object gets the video panel's DrawingInImageSpace property using the VideoPanel.isDrawingInImageSpace() method.

ImageCoordSystem methods for converting coordinates all depend on frame number, so a trackable object saves its frame number internally and passes it to the coords when needed.

Note: A video can draw itself on both standard drawing panels and video panels. When drawn on a drawing panel, it uses its own internal coords. When drawn on a video panel, it uses the panel's coords. When a video is added to a video panel, it sets the panel's coords equal to its own.

Methods used to control drawing and convert between drawing spaces are shown in Listing 14.

Listing 14. Drawing and Converting Methods

VideoPanel methods:
public ImageCoordSystem getCoords();
public void setDrawingInImageSpace(boolean imagespace);
public boolean isDrawingInImageSpace();

ImageCoordSystem methods:
public double imageToWorldX(int n, double imageX, double imageY);
public double imageToWorldY(int n, double imageX, double imageY);
public double worldToImageX(int n, double worldX, double worldY);
public double worldToImageY(int n, double worldX, double worldY);
public double imageToWorldXComponent(int n, double imageX, double imageY);
public double imageToWorldYComponent(int n, double imageX, double imageY);
public double worldToImageXComponent(int n, double worldX, double worldY);
public double worldToImageYComponent(int n, double worldX, double worldY);
public AffineTransform getToImageTransform(int n);
public AffineTransform getToWorldTransform(int n);

Listing 15 shows the Drawable.draw() method for a simple trackable circle associated with frame number n. The methods getX() and getY() return its image coordinates.

Listing 15. Drawing a Trackable Circle

public void draw(DrawingPanel panel, Graphics g) {

if (!(panel instanceof VideoPanel)) return; // only draws on video panels
VideoPanel vidPanel = (VideoPanel)panel;

double x = getX(); // image position
double y = getY(); // image position

if (!vidPanel.isDrawingInImageSpace()) { // convert to worldspace
x = vidPanel.getCoords().imageToWorldX(n, getX(), getY());
y = vidPanel.getCoords().imageToWorldY(n, getX(), getY());
}

// standard drawing panel drawing from here on
int xpix = panel.xToPix(x) - pixRadius;
int ypix = panel.yToPix(y) - pixRadius;
g.setColor(color);
g.fillOval(xpix, ypix, 2*pixRadius, 2*pixRadius);

}

TPoint and TShape

The TPoint class, which implements Trackable and InteractiveDrawable, is a powerful and convenient base class for constructing trackable objects. TPoint extends Point2D.Double, a Java class designed to hold 2D position data with double precision that is used in many Shape and AffineTransform methods. TPoint adds several convenience methods of its own that are designed specifically for use with a VideoPanel.

However, a TPoint has an empty draw() method, and its findInteractiveDrawable() method returns null. Therefore, TPoints must be either extended or used as member objects in more complex Trackable objects. By convention, classes that extend TPoint have a class name that begins with T. For example, the framework also defines TShape, which extends TPoint and implements draw() and findInteractiveDrawable(), and TCircle, which extends TShape.

A few of the most useful TPoint methods are shown in Listing 16.

Listing 16. TPoint and TShape Methods

TPoint methods:
public int getFrameNumber(VideoPanel vidPanel);
public Point getScreenPosition(VideoPanel vidPanel); // useful for detecting mouse hits
public void setScreenPosition(Point p, VideoPanel vidPanel); // useful for setting with mouse
public Point2D getWorldPosition(VideoPanel vidPanel); // useful for graphing
public void translate(double dx, double dy); // dx and dy are in image units
public void center(Point2D pt1, Point2D pt2); // centers this between pt1 and pt2
public double angle(Point2D pt); // returns the angle of the line between this and pt

Tracker

Figure 12 at the beginning of this section shows Tracker, a video analysis package built on the osp media framework that uses TPoints extensively. Tracker features include object tracking with position, velocity and acceleration overlays and graphs, multiple reference frames, calibration points and line profiles for analysis of spectra and interference patterns. For more information see the Tracker Java Tutorial. Tracker downloads are available at http://www.cabrillo.edu/~dbrown/tracker/.

Note: OSP developers should pay particular attention to Tracker's WorldTView class. A world view is a video panel that always draws in worldspace, so animations and other drawable overlays can treat it like any other drawing panel. Some advantages of the world view are that it (1) has a player and video and (2) can access student-marked Track data for setting numerical parameters or other model-data interactions.

In the screenshot below, the lower drawing panel is a world view. The "video" in this instance is a simple white rectangle. The points were marked in the upper panel, which is drawing in imagespace. But the world view, graph and table all show the worldspace data.

Figure 11. Tracker: World, Plot and Table Views

Tracker with additional views