Lynda: Rotoscoping and Edges

After Effects Compositing 05 Rotoscoping and Edges by Mark Christiansen on Lynda.

My notes from this tutorial on are a bit lengthy but this is  an overview of what I learned.

Overview of what I learned:

Think ahead:

A simple shape mask can be used with After Effect’s tracker to track an area like a dark spot that doesn’t move behind any object and does not shift in perspective.

For more complex shots, use large simple masks as much as possible, then when you can’t do that anymore, start using articulated masks for moving parts (if necessary). Then after this, use the brush tools; paint, clone and roto for cleaning up.


01 Welcome Begin by planning ahead and avoid doing extra work. Fully articulated roto means that it has lots of moving parts such as limbs.

02 A simple shape mask can be used to track an area like a dark spot that doesn’t move behind any object and does not shift in perspective.

03 Rotoscoping means using manual means to select or clone footage over time.  Rotoscoping is necessary when green screening is not an option. Draw a mask onto the layer that you want to roto and use the tracking menu to track.

04 The changes in contrast within the tracking area can change the outcome of a track drastically.

05 Apply a mask to effects. For the grease spot removal: Look through the channels r.g.b. with alt+1/2/3. In the example, the spot is most apparent in the green channel. Do a levels adjust in the green channel and colour correct to match the image again?

06 Refine mask controls Use the mask controls to fix the edge of the mask.

07 Test the limits of the mask tracker. The mask tracker is limited by shifts in perspective. There are still ways to combine tracking with roto.

08 Evaluate before beginning roto. Rotoscoping takes a lot of work. You need to pay close attention to detail and be thorough.

09 Shortcuts – Draw and edit a mask: ‘g’ switches between the pen and feather tool. Use ‘ctrl’ to move points. ‘alt’ is used to adjust Bezier handles.

10 Start with a shape. You sometimes have the option of masking with a simple shape using the ‘q’ key. A shape can be more precise.

11 Lock-off for VFX Set playback to duration of selected layer ‘ctrl alt b’. You need to lock-off the footage with the after effects warp stabilizer.  Choose ‘no motion, method:position and framing:stabilize, crop autoscale. Having a locked off shot means not having any camera movement. A separate course looks at camera movement and roto. ***reminder

12 Break down the shot for a clean plate:  ‘j and k’ to set beginning and end of work space. Plan ahead: Duplicate your layer and call it ‘cp left’ (clean plate). ‘alt + /’ zoom to fit. You want to put the footage that has no people walking about over the time when people come into frame. Use a simple mask.

13 Create blank footage with a simple expression. To clean up the right footage, we cannot simply mask over a single frame as this would get rid of the grain and ambient light movement that occurs in footage. Create a ‘cp right’ and enable time remapping ‘ctrl alt t’. alt click on the time remap stopwatch, then from the ‘play-ish’ button menu>properties>loop out and this automatically loops the footage. Click enter and lengthen time bar. This expression looks like a cycle in the graph editor. ‘ee’ reveals the expression. What is he using to reveal the mask? Make sure the mask is inverted as needed. To loop a different set of time expressions, duplicate your plate again, find the area/time that is clean and create a new expression for this remapped time. This can be used to get rid of shadows.

14 Replace missing or mismatched background. We want to use our mismatched background plate to replace the extra as best as possible. Mask around the area you want to create and view the layers with difference. We want to line up and colour correct/curves adjust the new replacement background.

15 Decide what needs attention. This is important so that time isn’t wasted doing unnecessary tasks. For the first head, use an oval mask and set to subtract. The mask order is important as it is subtracting from the mask above. For a simple mask, use straight-ahead roto. E.g. if you want to remove the top of a head behind a wall, simply key the mask over the head as it moves.

16 Avoid unnecessary work such as full articulated roto. In the example, we can mask out the orange shirt guy with a simple 3-point Bezier. Name your masks to what is being removed.

17 Rotoscope straight ahead: Work in layer view if this is easier. ‘alt double click’ on layer. ‘tilde’ maximises and restores view. Alt shift m’ reveal and add mask path keyframe. You might want to remove a group of objects and then use another mask to add back in one of the objects.




With an object that is heavily motion blurred, you will need to straight-ahead roto. Place your path in the centre of the blur. You can move a layer in layer view but not in comp view.

018 Lock position only. Alt+click stopwatch on the position attribute to set an expression on this. Type in the position that’s displayed, into square brackets. This is better done by typing the expression [width/2, height/2]. This expression can be applied anywhere else. You can ‘copy expression only’ and paste to other layers for a quick way of locking the layer’s position. You can also >animation >save animation preset to save this expression. Go to animation>apply preset>choose one.

019 Build articulated roto with in-between frames. Start at a frame that does not have a lot of detail. Draw your mask with as few points as possible and place the points on areas that will be identifiable at the beginning and end of your roto. When animating the roto, keep the points on the same area of the roto and add extra points as needed for changing shapes. Otherwise the points will roll-around in the in-betweens.

20 Work with defocused mask. Your green screen might not be selectable alone and might cause artifacts when deleted this way. In this scenario, you could mask the edges of the hard surface car around the actor. Identify the lens behaviour in your footage e.g. rack focus, camera breathing, and then apply this same behaviour to your mask. Animate scale as needed for a zoom. Right click on your keyframe and add eases in/out. Then animate the feather to mask the focus shift on the foreground vs background masks. Use the mask to hide the car and this way the green will be easier to select by itself.

22 Get motion blur for free: For the motion blur on the board, we add motion blur instead of a feather. The motion blur will follow the motion of the mask. Turn on the motion blur icons in the composition area.

23 Brush tools come last. We want to paint back in the shadow that has been masked away. Choose the brush tool and give it soft edges. Double tap ‘u’ to reveal only modified properties. Keyframe the position of the brush stroke to follow where the shadow was.

24 Clone brush set up and shortcuts. Now we want to restore the areas of our skater which have been rotoscoped out. We paint in the layer view. Look for a detail that you can see both in the foreground and background. Alt+click with clone tool to set the source. Your source should be lined up with your top layer. The clone brush paints details from the clone source layer onto the layer you paint on dependant on position(?). It lifts details from the clone source layer? Paint back in the missing board, set to frame by frame. See the clone source overlay option box to see if the layers are overlayed correctly. Or check by holding alt to see the cross hairs of the clone brush.

25 Input devices for paint and roto. See the brush dynamics option.

26 Working with clone brush. Open your layers in layer view. Open a new viewer under view so that you have a split screen. Ctrl+0 closes/opens the project panel. Reveal ‘title action safe’ from menu below layer viewer. Make sure that your clone brush duration is set to single frame and that your eraser is set to only erase paint. You can use both views to figure out where you need to paint.




27 Make quick use of roto brush. Look up the basics of using the roto brush in AFx. Putting the motion blurred skater in front of the boards is a good example of where to use the roto brush. What’s an anchor frame? The roto brush deals well with heavily blurred edges.

28 Don’t get bogged down with roto brush. Roto brush relies on contrast of both motion and colour. If you’re spending too long trying to fix the brush’s edge selection, you might be better off just drawing a mask.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s