Automatic Background Removal With Pixel Bender

How do you get rid of the product backgrounds?

This was a problem that has plagued me and the other user interface developers at Pixsta ever since I started working here. Our efforts to build nice homepages and cool Facebook / Twitter apps seems almost hampered every try by the limitation that our product images always had a white background.

I decided to spend a little time investigating how we could remove backgrounds on the client-side, and immediately thought of Pixel Bender. In this post, I share the results of my experimentation and show that Pixel Bender is a viable alternative to server-side image manipulation and calculation. It really highlights the power of Flash and Flex over other client technologies like JavaScript.

Pixel Bender is a technology by Adobe that enables image processing at the pixel level introduced with Flash 10. It is fast and uses the native graphic capabilities of the machine. It is much faster and optimized than plain ActionScript.

I started with a naive approach, copying a colour-based pixel removal algorithm from an old stackoverflow thread. This algorithm simply replaces the white pixels of the image within a certain threshold with a transparent pixel. The kernel code is as follows:

input image4 src;
output float4 dst;

// How close of a match you want
parameter float threshold
<
  minValue:     0.0;
  maxValue:     1.0;
  defaultValue: 0.4;
>;

// Color you are matching against.
parameter float3 color
<
  defaultValue: float3(1.0, 1.0, 1.0);
>;

void evaluatePixel()
{
  float4 current = sampleNearest(src, outCoord());
  if((distance(current.rgb, color) < threshold) ){
      dst = float4( 0.0 );
  } else {
      dst = current;
  }
}

This algorithm works well for a big chunk of the time and uses Pixel Bender’s capabilities fully. However, it fails when the item itself has a lot of white and doesn’t do a very good job of removing the edges. The other problem here is that a threshold that might work for some images might look horrible on other images.

After finally getting the hang of Pixel Bender, I walked over to my well respected collegue Tuncer, who is a mad image processing PhD genius and asked him if he could recommend something we could use for  client side background removal. He sent me a version of the background removal algorithm we currently use in the Empora site, and I went ahead trying to see if it could work on Pixel Bender.

After tweaking it for a day and converting it from Matlab and Java into ActionScript, I arrived at a nice solution that worked with Pixel Bender, Flex and Flash.

Unfortunately, I cannot post the code for this as it involves some algorithms that a) I don’t really understand, b) are the brainchild of somebody else and c) are probably proprietary for the company. However, I can share some results.

The following rows show the original image, the white background removed image with an arbitrary threshold, and the automatically calculated image.

Original / White Removal / Pixsta Special

As you can see, this works really well and shows the possibility of the Adobe Pixel Bender technology in processing images on the client side.

There were a few limitations and challenges that I ran into implementing these algorithms in Pixel Bender, which were the following:

  • Pixel Bender works at a per pixel level. Some of the pre-computations in these algorithms worked with the entire image. This means that a pure Pixel Bender implementation would be almost impossible to do. However, once these automatic thresholds were calculated, the rendering speed of Pixel Bender was simply amazing.
  • There are a few kinks in the Flash player implementation of Pixel Bender. Most annoyingly, it doesn’t allow For loops. I had to write a simple groovy script to unravel my scripts. Hopefully this technology matures more and the language becomes a little more than a crippled toy language.
  • The path to integrating Pixel Bender Shaders into Flex were not very clear. Neither the Pixel Bender page nor the official Adobe documentation goes into these technologies in depth. Fortunately, Mike Chambers has blogged about integrating Pixel Bender into Flex, and I would be completely lost without his blog posts. [ 1 ] [ 2 ] [ 3 ]
  • To make it better, I would probably add another shader that will blend the edges into the image more. But this would be easy with Pixel Bender. The background in the example is very dark, we would display our images in a white or semi-white background, reducing the contrast of the edges.
  • The algorithm is a little aggressive sometimes, removing more than it needs. But the Flex UI would allow for either a slider to be incorporated to manage individual thresholds or enable a more dynamic background removal approach such as the one presented here.

It was really neat to be able to explore these kind of image manipulations and see the limits of this technology. Makes me really appreciate working with people that actually know how to use Mathematics for good. It’s really exciting to see Pixel Bender and the possibility it brings to image processing.

If you are interested in reading more about the image processing work done at Pixsta, visit the homepage of our research team, Dr. Tuncer Aysal and Dr. Daniel Heesch. It contains all their publications to date.

To see some of this stuff at work in a real live product, visit Empora.com.

Please note: I cannot and will not share the code to the Pixel Bender Kernel for background removal.

3 thoughts on “Automatic Background Removal With Pixel Bender

  1. Noj

    A few days ago I wrote a similar algorithm for trimming out backgrounds. I used the concept of choosing the colours that fall in a certain hsv range and it worked like a charm.

    It allows u to remove dark colours, light colours, a bluish shade of white only e.t.c

    Will post my code to pb exchange.

    Reply
    1. zinabnm

      I am going to write background removal. And I also want to do that with help of Pixel Bender. Unfortunately author do not share the code, it will save my time. But I want to say thank you! You confirm my approach. [To Noj] – can you put your solution to pb exchange ASAP?

      Reply
  2. Ed Bratt

    would it be possible to write a similar program that would take out or reverse the black backgrouds on web pages as they open on my screen, that are designed to “look cool” but are slow to read and hamper retention?

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s