In this tutorial I will show you how to mask complex objects on single color backgrounds using alpha channels and image calculations and remove color cast. This technique can be applied to subtract objects from complex backgrounds too. You will not even need the brush tool.

This is a technique that I’ve seen on a DVD but it was applied on portrait retouching and the purpose was different.

On more complex backgrounds you will need to use this technique using additional tools like the brush tool and other selection techniques like Color Range.

Although it looks like it’s a long tutorial, once you practice this you can do all this actions in less than a minute.

This is the result that you will get using the Calculculations channels masking technique.

Advanced masking using calculations

Why using the Channels to create layer masks?

Since Photoshop already comes with so many selection tools like magnetic selection tools, magic wand, quick mask… why using channels? The reason for using this is because it will save you tons of work time and because it’s more accurate.

For most users that only use Photoshop to retouch their holidays photos or their friend’s portraits, using the channels to create masks might sound weird and difficult to understand and mostly because they don’t even know what channels are.

I’m not going to explain what channels are, you can learn about that yourself on the internet. Basically channels split an image in three levels of brightness corresponding to Red, Green and Blue (RGB). That being said, let’s get started with the tutorial.

Step 1

I used a photo of some branches against the blue sky. You can download the picture from here: http://www.sxc.hu/photo/1295777.

Load this photo in photoshop and duplicate it. Rename the duplicated layer to branches. We will apply the layer mask on this layer so this will be our branches without the background.

Step 2

As you can see, there are a lot of branches and using using the Pen Tool to create the mask it will take you forever.The Magic Wand Tool will also take you a lot of time and it always results in very jagged edges. Quick Mask will also take you a whole lot of time.

Since our background is a single solid color, we will use the layer channels to create an alpha layer. Don’t worry if you don’t understand it yet, just follow the steps. Select the branches layer and click the channels tab on the layers palette.

Step 3

Now you see three layers for each of the three basic colors, they are called alpha channels. The RGB channel is the combination of the other three. The objective here is to select one of the three layers that will show you the biggest contrast between the branches and the background.

Go ahead and click on the three layers and notice the difference in lightness. In this case you can see that the best option is the Blue channel because the contrast between the tree branches and the blue background is bigger than on the other two. So we will use the Blue channel, click to select it.

Step 4

With the Blue channel selected, go to Image>Calculations. This tool will allow us to blend alpha channels and it will show us a preview of the result. We will use this tool to create a fourth alpha channel along the three that we already have. Use the settings in the screenshot below.

Let me explain what this does to our channel. It takes the Source 1 which in this case is the blue channel of the locked background layer and it blends it with the blue channel of the branches layer using the Multiply mode.

Blue multiplied by Blue results in a much more contrasted image on a new channel called Alpha 1. It boosts the contrast between the black and white. Our final objective is to have the branches totally black and the background totally white, otherwise we cannot use this channel as a layer mask.

Step 5

On Step 4 we increased the contrast but we have to increase it even more. Select the Alpha 1 channel that has been created and go again to Image>Calculations. This time change the Blending mode to Overlay.

The effect of applying the second calculation using the Overlay blending mode is whiten the whites and darken the darks even more, creating another channel called Alpha 2.

Step 6

Even after applying the two calculations the top part of the background still has some gray so we cannot use the channel as a layer mask yet because part of the sky will still be visible.

Select the Alpha 2 channel and use the shortcut Ctrl+L to open the levels window. We want to get rid of the gray tone on the background and make it completely white.

Move the Highlight slider to the left like in the image below until the background becomes white. Also increase the darks a bit.

Now we have exactly what we wanted, the background is white and the tree branches are black. We are now ready to create our layer mask from this.

Step 7

All we have to do now is invert the Alpha 2 channel to make the branches white and background dark because we want the branches to be visible and the background invisible. You can invert the Alpha 2 channel with the shortcut Ctrl+I.

Press and hold Ctrl and click the Alpha 2 thumbnail when the cursor changes to a hand under a square selection icon. That will load a selection around the branches like you see on the screenshot below.

Step 8

With the selection still loaded on the inverted Alpha 2 channel, exit the Channels section by clicking the Layers tab and then select the branches layer.

Now you see the selection around the branches. This selection includes all the details, something you will never be able to do with the pen tool or with the quick mask or magic wand.

Step 9

All you need to do now is go to Layer>Layer Mask>Reveal Selection or click the layer mask icon located on the bottom of the layers palette. As soon as you do that you will see that the background is gone leaving just the tree branches visible. If you zoom in you will see that the edges are sharp and very accurate and all the fine details are there.

The masking job is done. To make sure everything is ok, create a background layer and fill it with a solid color and check for unwanted results.

Color cast problems

If you want to use this branches over a new background you will immediately find a problem which many Photoshop users find difficult if not very difficult to solve. The problem I’m talking about is the blue color cast on the thiner branches. I used another stock image as background so you can see exactly what I’m talking about.

Removing the color cast in Photoshop

There is an easy way of removing the color cast using the Hue/Saturation filter and using a layer mask  reduction. If the color cast is only on the edge, you can contract the layer mask 1pixel.

Since we already have a layer mask on the branches layer, we cannot create another one. The way to tackle this is to put the branches layer on a group and apply the layer mask to the group.

Select the branches layer and press Ctrl+G to place it on a group.

Select the branches layer mask using Ctrl+click. When the selection is loaded, go to Select>Modify>Contract and contract by 1 pixel. When you have it, create a layer mask on the group layer by going to Layer>Layer Mask>Reveal Selection.

If you are using a low resolution image, this might not be a good option because contracting the layer mask by one pixel can completely hide the smaller details.

Once you are done with that, there is still some blue color cast on some of the branches specially on the smaller ones on the left side. You can remove the color cast using the Hue/Saturation filter to desaturate the blues and cyans or to change the Hue and match the new background.

Create a Hue/Saturation adjustment layer, select blues from the drop-down list and desaturate it. Do the same with the cyans.

This is the result after applying the contracted layer mask and the Hue/Saturation filter.

I hope you found this tutorial useful. I made this masking in less than a minute and writing the tutorial and preparing all the images took me several hours.

Your job now is to spread the word, somebody out there might need help with this too 🙂