Using the Roto Brush Tool - Adobe After Effects CC Classroom in a Book (2017 release) (2017)

Adobe After Effects CC Classroom in a Book (2017 release) (2017)

9. Using the Roto Brush Tool

Lesson overview

In this lesson, you’ll learn how to do the following:

• Extract a foreground object from the background using the Roto Brush tool.

• Correct the segmentation boundary across a span of frames.

• Touch up a matte with the Refine Edge Tool.

• Freeze a matte across a clip.

• Animate properties for creative effects.

• Track a face in footage.


This lesson will take approximately an hour to complete. Download the project files for this lesson, following the instructions in the Getting Started section under “Accessing the lesson files and Web Edition,” if you haven’t already done so. As you work on this lesson, you’ll preserve the start files. If you need to restore the start files, download them from your Account page.



With the Roto Brush tool, you can quickly separate a foreground object from a background across many frames. You can achieve professional results in a fraction of the time you’d spend performing the same task with traditional rotoscoping.

About rotoscoping

When you draw or paint on the frames of a movie, you’re rotoscoping. For example, a common use of rotoscoping is to trace an object, using the path as a mask to separate it from the background so you can work with it separately. Traditionally in After Effects, you drew masks, animated the mask paths, and then used the masks to define a matte. (A matte is a mask used to hide part of an image so that another image can be superimposed.) While effective, this is a time-intensive, tedious process, especially if the object moves a great deal or the background is complex.

If a background or foreground object is a consistent, distinct color, you could use color keying to separate the object from the background. If the subject was shot against a green or blue background (green screen or blue screen), keying is usually much easier than rotoscoping.

The Roto Brush tool in After Effects is much faster than conventional rotoscoping, and for movies with complex backgrounds, it’s much easier than keying. You use the Roto Brush tool to define the foreground and background elements. Then, After Effects creates a matte, and tracks the movement of the matte over time. The Roto Brush tool does much of the work for you, leaving only a little cleanup work to be done.

Getting started

In this lesson, you’ll use the Roto Brush tool to isolate a young boy from a wet winter background so you can color-treat the background without affecting the boy. To finish up the project, you’ll add an animated title.

First, you’ll preview the final movie and set up your project.

1. Make sure the following files are in the Lessons/Lesson09 folder on your hard disk, or download them from your Account page at now:

• In the Assets folder:,

• In the Sample_Movies folder: Lesson09.avi,

2. Open and play the Lesson09.avi sample movie in Windows Media Player or the sample movie in QuickTime Player to see what you will create in this lesson. When you are done, close Windows Media Player or QuickTime Player. You may delete the sample movies from your hard disk if you have limited storage space.

When you begin this lesson, restore the default application settings for After Effects. See “Restoring default preferences” on page 2.

3. Start After Effects, and then immediately hold down Ctrl+Alt+Shift (Windows) or Command+Option+Shift (Mac OS). When prompted, click OK to delete your preferences. Close the Start window.

After Effects opens to display an empty, untitled project.

4. Choose File > Save As > Save As.

5. In the Save As dialog box, navigate to the Lessons/Lesson09/Finished_Project folder.

6. Name the project Lesson09_Finished.aep, and then click Save.

Creating the composition

You need to import one footage item for this lesson.

1. Choose File > Import > File.

2. Navigate to the Lessons/Lesson09/Assets folder, select the file, and then click Import or Open.

3. Drag the clip onto the Create A New Composition button (Image) at the bottom of the Project panel.

After Effects creates a composition named boy, based on the settings in the file. The composition is three seconds long, with a frame size of 1920 x 1080. The movie file was shot at a rate of 29.97 frames per second.


4. Choose File > Save to save the project so far.

Using Adobe Premiere Pro with After Effects

You can work with a clip in both Adobe Premiere Pro and After Effects, moving easily between the two applications as you edit your project.

To edit an Adobe Premiere Pro clip in After Effects, do the following:

1. Right-click or Control-click the clip in Adobe Premiere Pro, and choose Replace With After Effects Composition.

After Effects starts, and opens the Adobe Premiere Pro clip.

2. Save the project when you’re prompted by After Effects. Then, work in the composition just as you’d work in any other After Effects project.

3. When you’re finished, save the project, and return to Adobe Premiere Pro.

Your changes are automatically reflected in the timeline.

Creating a segmentation boundary

You use the Roto Brush tool to specify which parts of the clip are in the foreground and which are in the background. You add strokes to distinguish the two, and then After Effects creates a segmentation boundary between the foreground and background.

Creating a base frame

To use the Roto Brush tool to isolate a foreground object, you start by adding strokes to a base frame to identify foreground and background areas. You can start on any frame in the clip, but in this exercise, you’ll use the first frame as the base frame. Then you’ll add the strokes that identify the boy as the foreground object.

1. Move the current-time indicator across the time ruler to preview the footage.

2. Press the Home key to move the current-time indicator to the beginning of the time ruler.

3. Select the Roto Brush tool (Image) in the Tools panel.

You use the Roto Brush tool in the Layer panel, so you’ll open it now.

4. Double-click the layer in the Timeline panel to open the clip in the Layer panel.

5. Choose Fit from the Magnification Ratio pop-up menu at the bottom of the Layer panel if you don’t see the entire image.


By default, the Roto Brush tool creates green foreground strokes. You’ll start by adding strokes to the foreground—the boy. Generally, it’s most efficient to start with broad strokes and then use smaller brushes to refine the border.

6. Choose Window > Brushes to open the Brushes panel. Set up a hard round 100-pixel brush. (You may need to resize the Brushes panel to see the options.)

When you’re drawing strokes to define the foreground object, follow the skeletal structure of the subject. Unlike traditional rotoscoping, you don’t need to define a precise boundary around the object. Start with broad strokes, and work down to small regions as After Effects determines where the boundary is supposed to be.

Image Tip

You can quickly zoom in and out of the Layer panel using the scroll wheel on your mouse.

7. Draw a green stroke from the boy’s head down toward the bottom of the clip.


A magenta outline identifies the boundaries After Effects created for the foreground object. After Effects recognized only about half of the boy, because you initially sampled only a small area of the subject. You’ll help After Effects find the boundary by adding some more foreground strokes.

8. Still using a large brush, draw a green stroke from left to right across the boy’s coat, including the black strip on the right.

9. Use a smaller brush to add any neglected areas to the foreground.

Image Tip

To increase or decrease the brush size quickly, press Ctrl (Windows) or Command (Mac OS) as you drag. Drag right to increase the brush size, or left to decrease it.


It can be tricky to get some areas without accidentally adding background as well. It’s okay if you haven’t captured every detail in the foreground. You’ll use background strokes to remove any extraneous areas of the matte.

10. Press Alt (Windows) or Option (Mac OS) to switch to the red background stroke brush.

11. Add red strokes to background areas you want to exclude from the matte. Switch back and forth between the foreground and background brushes to fine-tune the matte. Don’t forget to deselect the area under the boy’s hat where the background shows through. In fact, one click may be all it takes to exclude that area from the matte.


Don’t worry about being exact about your brush strokes. Just make sure the matte is within 1 to 2 pixels of the edge of the foreground object. You’ll have an opportunity to refine the matte later. However, After Effects uses the information on the base frame to adjust the matte for the rest of the span, so you want the matte to be accurate.

12. Click the Toggle Alpha button (Image) at the bottom of the Layer panel. The selected area is white against a black background, so you can see the matte clearly.


13. Click the Toggle Alpha Overlay button (Image) at the bottom of the Layer panel. The foreground area appears in color, and the background has a red overlay.


14. Click the Toggle Alpha Boundary button (Image) at the bottom of the Layer panel to see the outline around the boy again.


As you use the Roto Brush tool, the Alpha Boundary is the best way to see how accurate your boundary is, because you can see everything in the frame. However, the Alpha and Alpha Overlay options let you see your matte without the distraction of the background.

Refining the boundary across the initial span

You used the Roto Brush tool to create a base frame, which includes a segmentation boundary that divides the foreground from the background. After Effects applied the segmentation boundary across a span of frames. The Roto Brush span appears below the time ruler at the bottom of the Layer panel. As you move forward and backward through the footage, the segmentation boundary moves with the foreground object (in this case, the boy).

1. Extend the span range by dragging the end of the span to 1:00 in the Layer panel.


You’ll step through the frames in the span and make adjustments to the segmentation boundary as necessary.

2. Press the 2 key on your main keyboard (not the numeric keypad) to move forward one frame.

Image Tip

To move forward one frame, press the 2 key on your keyboard; to move back one frame, press the 1 key.

Working from the base frame, After Effects tracks the edge of the object and attempts to follow its movement. Depending on how complex your foreground and background elements are, the boundary may or may not conform exactly to the area you hoped it would. In this case, you may notice changes to the segmentation boundary along the boy’s right sleeve (the left edge of the clip) as more of the coat is revealed in the frame. Likewise, the flaps of the cap and the edges of the hood need some adjustment, which means you need to refine the segmentation.

3. Using the Roto Brush tool, paint foreground and background strokes to refine the matte for this frame. If the matte is accurate, you don’t need to paint any strokes.


Image Note

As it propagates the segmentation boundary for a frame, After Effects caches that frame. Cached frames have a green bar in the time ruler. If you jump ahead to a frame further along the span, After Effects may take longer to calculate the boundary.

If you make a stroke you don’t like, you can always undo the stroke and try again. As you move through the span, each change you make affects the frames after it. The more you refine your selection, the better the overall results will be. You may find it useful to move forward a few frames to see how changes affect the boundary.

4. Press the 2 key again to move forward to the next frame.

5. Use the Roto Brush tool to add to the foreground or subtract from the background as necessary to refine the boundary.

6. Repeat steps 4 and 5 until you reach 1:00.


Adding new base frames

After Effects creates an initial Roto Brush span of 40 frames (20 in each direction). As you move through the frames, the span range may increase automatically, or you can drag to extend the span. However, the further you move from the base frame, the longer After Effects takes to propagate, or calculate, the segmentation for each frame, especially if the shot is complex. If the scene changes significantly, it may be better to create multiple base frames for your footage than to have one really long span. The scene in this project remains fairly consistent, so you could extend the span and make additional adjustments as needed. However, you’ll create additional base frames to gain experience using the tools, learn to connect spans, and see how the segmentation shifts as you move further from a base frame.

You’ve reached 1:00 in the refining process. Now you’ll add a new base frame to the project.

1. Go to 1:20 in the Layer panel. This frame isn’t included in the initial span, so the segmentation boundary includes the entire frame.

2. Use the Roto Brush tool to add foreground and background strokes, defining the segmentation boundary.


A new base frame (represented by a blue rectangle) is added to the time ruler, and the Roto Brush span extends several frames behind and ahead of the new base frame. Depending on how far the initial span propagated, there may be a gap between the two spans. If so, you’ll connect the two.

3. If necessary, drag the left edge of the new span to the edge of the previous span.

4. Press 1 to move backward one frame (from the new base frame), and then refine the segmentation boundary.

5. Continue to move backward through the span, refining the segmentation boundary until you reach the frame at 1:00.

6. Move back to the base frame at 1:20, and then press 2 to move forward, correcting the segmentation boundary for each frame if necessary.

7. When you reach the end of the span, drag the right edge to the end of the clip, and continue to refine each frame in the footage as needed. Pay special attention to the left earflap of the cap when it crosses in front of the tree. The dark areas overlap, making it more difficult to get a consistent edge. Remember that you are trying to get the segmentation boundary as close to the edge of the foreground object as possible.

8. When you have completed refining the segmentation boundary for the entire clip, choose File > Save to save your work so far.


Fine-tuning the matte

Roto Brush does a pretty good job, but there may be stray bits of background in the matte, or foreground areas that weren’t included. You’ll clean those up by refining the edge.

Adjusting the Roto Brush & Refine Edge effect

When you use the Roto Brush tool, After Effects applies the Roto Brush & Refine Edge effect to the layer. You can modify the effect using settings in the Effect Controls panel. You’ll use those settings to further refine the edge of the matte.

1. Press the spacebar to play the clip in the Layer panel; press the spacebar again to end the preview when you’ve seen the whole clip.

As you preview the clip, you may notice that the segmentation boundary area jumps around quite a bit. You’ll use the Reduce Chatter setting to make it smoother.

2. In the Effect Controls panel, increase the Feather amount to 10, and increase Reduce Chatter to 20%.


The Reduce Chatter value determines how much influence the current frame has when performing a weighted average across adjacent frames. Depending on how tight your matte was, you may need to increase Reduce Chatter to 50%.

3. Preview the clip again. Notice how much smoother the edge of the matte has become.

Refine Soft Matte and Refine Hard Matte effects

After Effects includes two related effects for refining mattes: Refine Soft Matte and Refine Hard Matte. The Refine Soft Matte effect does the same thing as Refine Edge Matte, except it applies the effect to the entire matte at a constant width. If you need to capture subtle variations across an entire matte, use this effect.

The Refine Hard Matte effect performs the same edge refinements as the Roto Brush when Fine-Tune Roto Brush Matte is turned on in the Roto Brush & Refine Edge effect in the Effect Controls panel.

Using the Refine Edge tool

The boy’s coat and face have hard edges, but his cap is fuzzy, and the Roto Brush tool didn’t pick up the nuanced edge. The Refine Edge tool lets you include fine details such as wisps of hair in designated areas of the segmentation boundary.

Though it might be tempting to use the Refine Edge tool immediately after creating the base frame, it’s best to wait until you’ve refined the segmentation boundary across the entire clip. Because of the way After Effects propagates the segmentation boundary, using the Refine Edge tool too early results in a matte that is difficult to use.

1. Zoom in so that you can see the edges of the cap clearly. Enlarge the Layer panel if necessary, and use the Hand tool to move the layer so you can see the entire cap.

2. Select the Refine Edge tool (Image), hidden beneath the Roto Brush tool in the Tools panel, and then move to the beginning of the clip in the Layer panel.

The cap is relatively soft, so a small brush size will work well. For a fuzzier object, you might have better results with a much larger brush. The brush needs to overlap the stray edges that emerge from the object.

3. Change the brush size to 10 pixels.

When you use the Refine Edge tool, draw strokes across or along the edges of the matte.

4. In the Layer panel, move the Refine Edge tool over the edge of the cap, straddling the segmentation boundary and including the fuzzy variations. You can use multiple strokes to move the tool around the entire cap.


When you release the mouse, After Effects switches to the Refine Edge X-ray view so that you can see how the Refine Edge tool changes the matte, capturing the detail in the edges.

5. Move to the second base frame in the Layer panel (1:20), and then repeat steps 1–4 to complete the rotoscoping process.

6. Zoom out to see the entire scene, resize the Layer panel if you enlarged it, and then choose File > Save to save your work.

Image Note

Use the Refine Edge tool only after you have cleaned up the matte across the entire clip.

Freezing your Roto Brush tool results

You’ve put a fair amount of time and effort into creating the segmentation boundary across the entire clip. After Effects has cached the segmentation boundary so it can recall it without having to make the calculations again. To keep that data easily accessible, you’ll freeze it. This reduces the processing demands on your system so you can work faster in After Effects.

Once the segmentation is frozen, you cannot edit it unless you unfreeze it. Refreezing the segmentation is time-consuming, so it’s best to refine the segmentation boundary as much as possible before freezing.

1. Click the Freeze button in the lower right area of the Layer panel.


The Freezing Roto Brush dialog box displays a progress bar as it freezes the Roto Brush and Refine Edge tool data. Freezing may take a few minutes, depending on your system. As After Effects freezes the information for each frame, the cache line turns blue. When it has finished freezing, a blue warning bar appears above the time ruler in the Layer panel, reminding you that the segmentation is frozen.

Image Depending on your system, this may take a while.

2. Click the Toggle Alpha Boundary button (Image) in the Layer panel to see the matte. Then click the Toggle Transparency Grid button (Image). Move the time marker across the time ruler to see the subject without the distractions of the background.


3. Click the Toggle Alpha Boundary button again to see the boundary.

4. Choose File > Save.

After Effects saves the frozen segmentation information with the project.

Changing the background

There are many reasons to isolate a foreground image from a background. Often, you want to replace the background entirely, moving the subject to a different setting. But rotoscoping is also useful if you simply want to change the foreground or background without modifying the other. In this lesson, you’ll make the background blue to enhance the theme of winter and help the subject stand out.

1. Close the Layer panel to return to the Composition panel, and then move the current-time indicator to the beginning of the timeline.


The Composition panel displays the composition, which includes only the layer, consisting only of the foreground you isolated from the clip.

2. Hide the properties for the layer, if they’re visible.

3. Click the Project tab to display the Project panel. Then drag another copy of the clip from the Project panel to the Timeline panel, and place it below the original layer.

4. Click the new layer, press Enter or Return, and rename the layer Background. Then press Enter or Return again.


5. With the Background layer selected, choose Effect > Color Correction > Hue/Saturation in the After Effects menu bar.

6. In the Effect Controls panel, do the following:

• Select Colorize.

• Change Colorize Hue to –122 degrees.

• Change Colorize Saturation to 29.

• Change Colorize Lightness to –13.


7. Choose File > Increment And Save.

If you save incrementally, you can return to earlier versions of your project to make adjustments later. This can be very useful if you’re experimenting or want to try alternative effects. The Increment And Save feature preserves the previously saved version of the project and creates a new project with the same name, with an increasing number added to the filename.

Adding animated text

You’re nearly done. All you need to do is to add the animated title between the boy and the background.

1. Deselect all layers, and then move the current-time indicator to the beginning of the time ruler.

2. Choose Layer > New > Text.

A new text layer appears in the Timeline panel at the top of the layer stack, and a cursor appears in the Composition panel.

3. Type WINTER BLUES in the Composition panel.

4. Select all the text in the Composition panel, and then do the following in the Character panel:

• Choose Myriad Pro for the font.

• Choose Black or Semibold for the font style.

• Type 300 px for the font size.

• Choose Optical from the Kerning menu.

• Select white for the fill color.

• Select black for the stroke color.

• Make sure the stroke width is 1 px and Stroke Over Fill is selected.


5. Select the text layer in the Timeline panel to deselect the text. Then press T to reveal the layer’s Opacity property. Change the Opacity to 40%.


6. Click the Effects & Presets tab to open the panel, and then type Glow in the search box. Double-click the Glow preset under Stylize.

The text gains some texture. The default settings are fine.

7. Drag the WINTER BLUES layer in the Timeline panel down to position it between the and Background layers. Then move the current-time indicator to the beginning of the time ruler, if it’s not already there.

You’ll animate the text so that it moves left behind the boy while he crosses to the right of the frame.

8. With the WINTER BLUES layer selected, press P to reveal its Position property. Change the Position to 1925, 540. Click the stopwatch icon (Image) for the Position property to set a keyframe.


The text moves offscreen, so it’s not visible when the movie begins.

9. Move the current-time indicator to 3:01 (the end of the clip), and change the Position to –1990, 540.


The text moves to the left. After Effects creates a keyframe.

10. Deselect all layers in the Timeline panel, and move the current-time indicator to the beginning of the time ruler. Press the spacebar to preview the clip.


11. Choose File > Save to save your work.

Outputting your project

Render your movie to complete the project.

1. Choose File > Export > Add To Render Queue.

2. In the Render Queue panel, click the blue words Best Settings.

3. In the Render Settings dialog box, choose Half from the Resolution pop-up menu, and make sure Use Comp’s Frame Rate is selected in the Frame Rate area. Then click OK.

4. Click the blue text next to Output Module. Then, at the bottom of the Output Module Settings dialog box, choose Audio Output Off, and click OK.

5. Click the blue text next to Output To. In the Output Movie To dialog box, navigate to the Lesson09/Finished_Project folder, and click Save.

6. Click Render in the top right corner of the Render Queue panel.

7. Save and close the project.

Congratulations! You’ve separated a foreground object from the background, including tricky details, modified the background, and animated some text to complete the movie. You’re ready to use the Roto Brush tool on your own projects.

Extra credit

Tracking faces

The face-tracking feature in After Effects makes it easier to track a face or specific facial features, such as lips or eyes. Previously, tracking a face required roto brushing or complex keying.

Image Note

If you open the Lesson09_extra_credit.aep file, you may need to relink the asset.

1. Choose File > New > New Project. Double-click in the Project panel to open the Import File dialog box, and then navigate to the Lessons/Lesson09/Assets folder. Select the file, and click Import or Open.

2. Drag the clip onto the Create A New Composition button at the bottom of the Project panel.

3. Select the layer in the Timeline panel. Then, select the Ellipse tool, hidden by the Rectangle tool in the Tools panel.

4. Drag an elliptical mask roughly covering the face.


5. Right-click the Mask 1 layer, and choose Track Mask.

6. In the Tracker panel, choose Face Tracking (Outline Only) from the Method menu.


The Face Tracking (Outline Only) option tracks the entire face. The Face Tracking (Detailed Features) option tracks the outline plus the lips, eyes, and other distinctive facial features. You can export detailed facial data for use in Character Animator, or you can use it in After Effects to apply effects or to match it with another layer, such as an eye patch or a hat.

7. Click the Track Forward button in the Tracker panel.


The Tracker tracks the face, changing the mask’s shape and position as the face moves.

8. Press the Home key to return to the beginning of the timeline, and then move the current-time indicator across the time ruler to see how the mask moves with the face.

9. In the Effects & Presets panel, search for Bright. Then drag the Brightness and Contrast effect onto the layer in the Timeline panel.

10. Expand Effects > Brightness & Contrast > Compositing Options in the Timeline panel.

11. Click the + icon next to Compositing Options. Choose Mask 1 from the Mask Reference 1 menu, and increase Brightness to 50.

The masked area of the face brightens, but nothing else does. This setting is too bright, and the edge of the mask is too abrupt. You’ll make it more subtle.

12. Reduce the Brightness to 20.

13. Expand the Mask 1 properties. Change the Mask Feather property to 70, 70 pixels.


14. Save the project to the Lesson09/Finished_Project folder. Then close the file.

You can use the face tracker to blur a face, brighten it, or add any other effect. You can also invert the mask to affect everything but the face.

Review questions

1. When should you use the Roto Brush tool?

2. What is the segmentation boundary?

3. When should you use the Refine Edge tool?

Review answers

1. Use the Roto Brush tool any time you would have used traditional rotoscoping. It’s particularly useful for removing a foreground element from the background.

2. The segmentation boundary is the boundary between the foreground and background. The Roto Brush tool adjusts the segmentation boundary as you progress through the frames in the Roto Brush span.

3. Use the Refine Edge tool when you need to rotoscope objects with fuzzy or wispy edges. The Refine Edge tool creates partial transparency for areas of fine detail, such as hair. Use the Refine Edge tool only after you’ve adjusted the segmentation boundary across the entire clip.