• Premium Lifetime access

  • Calculations Photoshop tutorial – Advanced Masking

    Share on pinterest
    Share on facebook
    Share on tumblr
    Share on twitter
    Share on whatsapp
  • In this tutorial I will show you how to mask complex objects on single color backgrounds using alpha channels and image calculations and remove color cast. This technique can be applied to subtract objects from complex backgrounds too. You will not even need the brush tool.

    This is a technique that I’ve seen on a DVD but it was applied on portrait retouching and the purpose was different.

    On more complex backgrounds you will need to use this technique using additional tools like the brush tool and other selection techniques like Color Range. Although it looks like it’s a long tutorial, once you practice this you can do all this actions in less than a minute.

    This is the result that you will get using the Calculations channels masking technique.

    Calculations Photoshop tutorial. Advanced masking in Photoshop

    Advanced masking using calculations


    Why using the Channels to create layer masks?

    Since Photoshop already comes with so many selection tools like magnetic selection tools, magic wand, quick mask… why using channels? The reason for using this is because it will save you tons of work time and because it’s more accurate.

    For most users that only use Photoshop to retouch their holidays photos or their friend’s portraits, using the channels to create masks might sound weird and difficult to understand and mostly because they don’t even know what channels are.

    I’m not going to explain what channels are, you can learn about that yourself on the internet. Basically channels split an image in three levels of brightness corresponding to Red, Green and Blue (RGB). That being said, let’s get started with the tutorial.

    Step 1

    I used a photo of some branches against the blue sky. You can download the picture from here: https://www.freeimages.com/photo/branches-1344103.

    Load this photo in photoshop and duplicate it. Rename the duplicated layer to branches. We will apply the layer mask on this layer so this will be our branches without the background.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Step 2

    As you can see, there are a lot of branches and using using the Pen Tool to create the mask it will take you forever.The Magic Wand Tool will also take you a lot of time and it always results in very jagged edges. Quick Mask will also take you a whole lot of time.

    Since our background is a single solid color, we will use the layer channels to create an alpha layer. Don’t worry if you don’t understand it yet, just follow the steps. Select the branches layer and click the channels tab on the layers palette.

    Photoshop channel masking tutorial

    Step 3

    Now you see three layers for each of the three basic colors, they are called alpha channels. The RGB channel is the combination of the other three. The objective here is to select one of the three layers that will show you the biggest contrast between the branches and the background.

    Go ahead and click on the three layers and notice the difference in lightness. In this case you can see that the best option is the Blue channel because the contrast between the tree branches and the blue background is bigger than on the other two. So we will use the Blue channel, click to select it.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Step 4

    With the Blue channel selected, go to Image>Calculations. This tool will allow us to blend alpha channels and it will show us a preview of the result. We will use this tool to create a fourth alpha channel along the three that we already have. Use the settings in the screenshot below.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Let me explain what this does to our channel. It takes the Source 1 which in this case is the blue channel of the locked background layer and it blends it with the blue channel of the branches layer using the Multiply mode.

    Blue multiplied by Blue results in a much more contrasted image on a new channel called Alpha 1. It boosts the contrast between the black and white. Our final objective is to have the branches totally black and the background totally white, otherwise we cannot use this channel as a layer mask.

    Step 5

    On Step 4 we increased the contrast but we have to increase it even more. Select the Alpha 1 channel that has been created and go again to Image>Calculations. This time change the Blending mode to Overlay.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    The effect of applying the second calculation using the Overlay blending mode is whiten the whites and darken the darks even more, creating another channel called Alpha 2.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Step 6

    Even after applying the two calculations the top part of the background still has some gray so we cannot use the channel as a layer mask yet because part of the sky will still be visible.

    Select the Alpha 2 channel and use the shortcut Ctrl+L to open the levels window. We want to get rid of the gray tone on the background and make it completely white.

    Move the Highlight slider to the left like in the image below until the background becomes white. Also increase the darks a bit.

    Difficult masking in Photoshop

    Now we have exactly what we wanted, the background is white and the tree branches are black. We are now ready to create our layer mask from this.

    Step 7

    All we have to do now is invert the Alpha 2 channel to make the branches white and background dark because we want the branches to be visible and the background invisible. You can invert the Alpha 2 channel with the shortcut Ctrl+I.

    Press and hold Ctrl and click the Alpha 2 thumbnail when the cursor changes to a hand under a square selection icon. That will load a selection around the branches like you see on the screenshot below.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Step 8

    With the selection still loaded on the inverted Alpha 2 channel, exit the Channels section by clicking the Layers tab and then select the branches layer.

    Advanced selections Photoshop tutorial Calculations

    Now you see the selection around the branches. This selection includes all the details, something you will never be able to do with the pen tool or with the quick mask or magic wand.

    Step 9

    All you need to do now is go to Layer>Layer Mask>Reveal Selection or click the layer mask icon located on the bottom of the layers palette. As soon as you do that you will see that the background is gone leaving just the tree branches visible. If you zoom in you will see that the edges are sharp and very accurate and all the fine details are there.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    The masking job is done. To make sure everything is ok, create a background layer and fill it with a solid color and check for unwanted results.

    Color cast problems

    If you want to use this branches over a new background you will immediately find a problem which many Photoshop users find difficult if not very difficult to solve. The problem I’m talking about is the blue color cast on the thiner branches. I used another stock image as background so you can see exactly what I’m talking about.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Removing the color cast in Photoshop

    There is an easy way of removing the color cast using the Hue/Saturation filter and using a layer mask  reduction. If the color cast is only on the edge, you can contract the layer mask 1pixel.

    Since we already have a layer mask on the branches layer, we cannot create another one. The way to tackle this is to put the branches layer on a group and apply the layer mask to the group.

    Select the branches layer and press Ctrl+G to place it on a group.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    Select the branches layer mask using Ctrl+click. When the selection is loaded, go to Select>Modify>Contract and contract by 1 pixel. When you have it, create a layer mask on the group layer by going to Layer>Layer Mask>Reveal Selection.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    If you are using a low resolution image, this might not be a good option because contracting the layer mask by one pixel can completely hide the smaller details.

    Once you are done with that, there is still some blue color cast on some of the branches specially on the smaller ones on the left side. You can remove the color cast using the Hue/Saturation filter to desaturate the blues and cyans or to change the Hue and match the new background.

    Create a Hue/Saturation adjustment layer, select blues from the drop-down list and desaturate it. Do the same with the cyans.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    This is the result after applying the contracted layer mask and the Hue/Saturation filter.

    Calculations Photoshop tutorial - Advanced Masking Photoshop Tutorials 2022

    I hope you found this tutorial useful. I made this masking in less than a minute and writing the tutorial and preparing all the images took me several hours.

    Your job now is to spread the word, somebody out there might need help with this too 🙂

    Other tutorials

    Leave a Reply

    Your email address will not be published.

  • Keep in touch

  • © PSD Box 2010-2022 | Andrei Oprinca

    What's new in 2022

    In this video I will highlight the most important changes and new releases in 2022.

    If you are a member and have any problems accessing the site or downloading the premium content, please click the “I need help” link on the top menu bar.

    Log in

    Access your account

    Don't have an account?

    Create a free account and unlock hundreds of free downloads: PSD files and tutorial resources

    Novedades en 2022​

    En este vídeo destacaré los cambios más importantes y las novedades de 2022.

    Si eres miembro y tienes algún problema para acceder al sitio o descargar los contenidos premium, haz clic en el enlace “Necesito ayuda” de la barra de menú superior.

    Iniciar sesión

    Accede a tu cuenta

    ¿No eres miembro?

    Cree una cuenta gratuita y desbloquee cientos de descargas gratuitas: Archivos PSD y recursos