Remove the background from images using Swift

Removing backgrounds from images is a common task in modern apps, whether for photo editing, social media, or automation. In this article, you'll learn how to build a cross-platform background remover in Swift using Apple's Vision and Core Image frameworks. We'll walk through the concepts, code, and practical tips to help you integrate this feature into your iOS or macOS projects.

Technologies Used

• Swift

• Core Image

• Vision Framework

• UIKit / AppKit

Why Background Removal?

Background removal lets you isolate subjects, create transparent images, and enhance user experiences. Apple's Vision framework makes this possible with powerful machine learning tools, while Core Image handles image processing. We'll use both to create a reusable Swift class.

Note: This implementation is cross-platform (iOS & macOS) For a ready-to-use version use SwiftExtras.

Step 1: Cross-Platform Setup

We'll create a BackgroundRemover class that works on both iOS and macOS. To handle platform differences, we use a typealias for the image type: NSImage for macOS and UIImage for iOS.

@available(iOS 17.0, macOS 14.0, *)
class BackgroundRemover {
#if os(macOS)
    // Support macOS
    typealias PlatformNativeImage = NSImage
#else
    // Support iOS
    typealias PlatformNativeImage = UIImage
#endif
}

Step 2: Error Handling

To make our code robust, we define custom errors for common failure cases. This helps with debugging and makes the API safer to use.

enum Errors: Error {
    case failedToUnwrapImage
    case failedToCreateCIImage
    case failedToRenderCGImage
    case failedToApplyMask
    case failedToCreateMask
    case invalidImageData
}

Step 3: Generate the Foreground Mask

The Vision framework can detect the subject in an image and create a mask. This mask is used to separate the foreground from the background.

Key APIs:

• VNGenerateForegroundInstanceMaskRequest()

• VNImageRequestHandler(ciImage:)

• CIImage(cvPixelBuffer:)

⚠️ Note: This code does not run on the iOS simulator due to Vision framework limitations. Use a real device or macOS.

private func createMask(from inputImage: CIImage) -> CIImage? {
    // Create a request to generate the foreground instance mask
    let request = VNGenerateForegroundInstanceMaskRequest()
    // Create a handler to perform the request
    let handler = VNImageRequestHandler(ciImage: inputImage)
    do {
        // Perform the request
        try handler.perform([request])
        // Get the first result
        if let result = request.results?.first {
            // Generate the mask from the result
            let mask = try result.generateScaledMaskForImage(
                forInstances: result.allInstances,
                from: handler
            )

            // Return the mask as a CIImage
            return CIImage(cvPixelBuffer: mask)
        }
    } catch {
        // Something went wrong
        print(error)
    }

    // Something went wrong
    return nil
}

Step 4: Apply the Mask to Remove the Background

With the mask generated, we use Core Image to blend the mask with the original image, effectively removing the background.

Key APIs:

• CIFilter.blendWithMask() (requires import CoreImage.CIFilterBuiltins)

private func applyMask(mask: CIImage, to image: CIImage) -> CIImage? {
    // Create a blend filter
    let filter = CIFilter.blendWithMask()
    // Set the input image
    filter.inputImage = image
    // Set the mask image
    filter.maskImage = mask
    // Set the background image to be empty
    filter.backgroundImage = CIImage.empty()
    // Return the output image
    return filter.outputImage
}

Step 5: Render the Final Image

After applying the mask, we render the result to a platform-native image type (UIImage or NSImage).

Key APIs:

• CIContext.createCGImage(_:from:)

• UIImage(cgImage:) (iOS)

• NSImage(cgImage:size:) (macOS)

private func removeBackground(from inputImage: CIImage) throws -> PlatformNativeImage {
    // Create a CIContext to render the output image
    let context = CIContext(options: nil)

    // Create a mask image, see createMask(from:)
    guard let maskImage = createMask(from: inputImage) else {
        throw Errors.failedToCreateMask
    }
    // Apply the mask to the input image, see applyMask(mask:to:)
    guard let outputImage = applyMask(mask: maskImage, to: inputImage) else {
        throw Errors.failedToApplyMask
    }

    // Render the output image
    guard let cgImage = context.createCGImage(outputImage, from: outputImage.extent) else {
        throw Errors.failedToRenderCGImage
    }

    #if os(iOS)
    return PlatformNativeImage(cgImage: cgImage)
    #else
    return PlatformNativeImage(cgImage: cgImage, size: CGSize(width: cgImage.width, height: cgImage.height))
    #endif
}

Step 6: Parse and Remove the Background

The parse(image:) method ties everything together. It converts the input image to a CIImage, processes it, and returns the result with the background removed.

/// Parses the image and removes the background.
///
/// - Parameter image: The image to parse.
/// - Returns: The image with the background removed.
public func parse(image: PlatformNativeImage) throws -> PlatformNativeImage {
#if os(iOS)
    guard let ciImage = CIImage(image: image) else {
        throw Errors.failedToCreateCIImage
    }
#else
    guard let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) else {
        throw Errors.failedToCreateCIImage
    }
    let ciImage = CIImage(cgImage: cgImage)
#endif

    return try removeBackground(from: ciImage)
}

Complete Implementation

#if canImport(Vision) && canImport(CoreImage.CIFilterBuiltins)
import Vision
import CoreImage.CIFilterBuiltins
#if canImport(UIKit)
import UIKit
#endif
#if canImport(AppKit)
import AppKit
#endif

@available(iOS 17.0, macOS 14.0, *)
class BackgroundRemover {
    enum Errors: Error {
        /// Failed to unwrap the image.
        case failedToUnwrapImage

        /// Failed to create a CIImage from the input image.
        case failedToCreateCIImage

        /// Failed to render the CGImage.
        case failedToRenderCGImage

        /// Failed to apply the mask to the image.
        case failedToApplyMask

        /// Failed to create the mask.
        case failedToCreateMask

        /// Invalid image data.
        case invalidImageData
    }

#if os(macOS)
    /// Platform-specific image type. (macOS)
    typealias PlatformNativeImage = NSImage
#else
    /// Platform-specific image type. (iOS)
    typealias PlatformNativeImage = UIImage
#endif

    /// Removes the background from the image.
    ///
    /// - Parameter inputImage: The image from which to remove the background.
    /// - Returns: The image with the background removed.
    private func removeBackground(from inputImage: CIImage) throws -> PlatformNativeImage {
        // Create a CIContext to render the output image
        let context = CIContext(options: nil)

        // Create a mask image, see createMask(from:)
        guard let maskImage = createMask(from: inputImage) else {
            throw Errors.failedToCreateMask
        }
        // Apply the mask to the input image, see applyMask(mask:to:)
        guard let outputImage = applyMask(mask: maskImage, to: inputImage) else {
            throw Errors.failedToApplyMask
        }

        // Render the output image
        guard let cgImage = context.createCGImage(outputImage, from: outputImage.extent) else {
            // Failed to render CGImage
            throw Errors.failedToRenderCGImage
        }

        // Create the final output image
#if os(iOS)
        return PlatformNativeImage(cgImage: cgImage)
#else
        return PlatformNativeImage(cgImage: cgImage, size: CGSize(width: cgImage.width, height: cgImage.height))
#endif
    }

    /// Creates a mask from the input image.
    ///
    /// - Parameter inputImage: The image from which to create the mask.
    /// - Returns: The mask image, or nil if creation failed.
    private func createMask(from inputImage: CIImage) -> CIImage? {
        // Create a request to generate the foreground instance mask
        let request = VNGenerateForegroundInstanceMaskRequest()
        // Create a handler for the image request
        let handler = VNImageRequestHandler(ciImage: inputImage)

        do {
            // Perform the request
            try handler.perform([request])
            // Get the first result
            if let result = request.results?.first {
                // Generate the mask for the image
                let mask = try result.generateScaledMaskForImage(
                    forInstances: result.allInstances,
                    from: handler
                )

                // Create the final mask image
                return CIImage(cvPixelBuffer: mask)
            }
        } catch {
            // Failed to generate mask
            print(error)
        }

        // Failed to generate mask
        return nil
    }

    /// Applies the mask to the image.
    ///
    /// - Parameters:
    ///   - mask: The mask image to apply.
    ///   - image: The image to which the mask will be applied.
    /// - Returns: The image with the mask applied, or nil if the operation failed.
    private func applyMask(mask: CIImage, to image: CIImage) -> CIImage? {
        // Create a blend filter
        let filter = CIFilter.blendWithMask()
        // Set the input image
        filter.inputImage = image
        // Set the mask image
        filter.maskImage = mask
        // Set the background image to be empty
        filter.backgroundImage = CIImage.empty()
        // Return the output image
        return filter.outputImage
    }

    /// Removes the background from the image.
    ///
    /// - Parameter image: The image from which to remove the background.
    /// - Returns: The image with the background removed.
    public func parse(image: PlatformNativeImage) throws -> PlatformNativeImage {
#if os(iOS)
        // iOS: Create a CIImage from the UIImage
        guard let ciImage = CIImage(image: image) else {
            throw Errors.failedToCreateCIImage
        }
#else
        // macOS: Create a CIImage from the NSImage
        guard let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) else {
            throw Errors.failedToCreateCIImage
        }
        let ciImage = CIImage(cgImage: cgImage)
#endif

        // Remove the background from the image
        return try removeBackground(from: ciImage)
    }
}

For a production-ready version and more utilities, see SwiftExtras on GitHub.

Wrap Up

With just a few lines of Swift, you can leverage Apple's frameworks to perform advanced image background removal on both iOS and macOS. This approach is efficient, easy to integrate, and can be extended for more complex use cases.

Conclusion

You now have a solid foundation for implementing background removal in Swift. Experiment with the code, adapt it to your needs, and explore the resources below for further reading and inspiration.

Further Resources:

• Image Background Remover CLI

• Remove Background from Image in SwiftUI

Read more

• Building SwiftUI Debugging Utilities • 3 minutes reading time.
• Gradients in text using foregroundStyle • 3 minutes reading time.
• NumberFormatter • 3 minutes reading time.

Share