Lumia Imaging SDK v3 (No, it's not just for phones...)

Discussion in 'Live RSS Feeds' started by News, Oct 12, 2015.

  1. News

    News Extraordinary Robot
    News Feed

    Joined:
    Jun 27, 2006
    Messages:
    26,205
    Likes Received:
    20
    This week, surprisingly enough, we're going to be doing a "Holly Cow, that was an awesome Hardware Event last week!" theme, focusing on related SDK's, projects and such.

    I'm sure you've already heard all about it, but incase not, check out this post, A New Era of Windows 10 Devices from Microsoft and catch the entire event on-demand, Windows 10 Devices

    Today, we're going to highlight a new Imaging SDK version, one that given the name you might think is for phones only, but it's not!

    First Clint Rutkas, yes, that Clint, introduces us it in a recent post. Then we detail what's new, the cool Samples Repo and finally two posts Igor Ralic on upgrading to it.

    Lumia Imaging SDK 3.0 now available with UWP support


    Lumia Imaging SDK, a popular toolset that Microsoft also internally uses to implement some of its imaging apps, has been updated to version 3.0. Previous versions of the SDK have already powered applications with a total of over 100 million downloads and now has support for Windows 10 UWP (Universal Windows Platform) to target smartphones, tablets and computers running Windows 10. In addition to Windows 10, development for both Windows 8.1 and Windows Phone 8.1 platforms continues to be supported. The SDK is implemented as Windows Runtime Component, allowing its APIs to be called from applications written in C#, C++, JavaScript and Visual Basic. To get started with Lumia Imaging SDK 3.0 development for Windows 10, install the SDK using NuGet Package Manager straight in Visual Studio or download the package from nuget.org

    The key feature introduced in Lumia Imaging SDK 3.0 is support for GPU based image processing through Direct2D. For most applications, this leads to significant performance improvement compared to the same application implemented with an earlier CPU-only version of the SDK. Where possible, exact implementation details of GPU support have been abstracted away from the API so that majority of the developers don’t need to be aware of what’s happening under the hood.

    Another focus area in the 3.0 release is interoperability with other Windows 10 APIs: rendering to SwapChainPanel is supported, and Direct3DSurface, SoftwareBitmap and WriteableBitmap can be used both as image sources as well as rendering targets. With its high abstraction level and wide range of features, Lumia Imaging SDK 3.0 complements other Windows imaging APIs like Win2D and Windows Imaging Component (WIC).

    ...
    Lumia Imaging SDK 3.0


    [​IMG]

    Lumia Imaging SDK, now as version 3.0, makes technologies that Microsoft uses in its own imaging apps available to third-party developers. The features provide the ability to decode and encode images, apply effects, blend, crop, rotate, and resize. It gives developers easy access to world-class imaging algorithms and effortless GPU utlilization (utilizing Direct2D under the hood). It also allows the developers to extend the functionality for custom scenarios.

    Note: Despite the name, the Lumia Imaging SDK actually works on any Windows 10 or Windows 10 Mobile device, as well as on any Windows 8.1 or Windows Phone 8.1 device.

    The Lumia Imaging SDK is a rich set of image and video frame processing tools for developers to provide amazing visual experiences in their apps. It provides more than 60 pre-made effects, many of which can be applied without decoding the whole image. That means that you can apply effects to high-resolution images without worrying about your memory budget. You can add an effect to your existing project with just a few lines of C#. The library can also be called from C++ code. The effects can be chained to create the exact effect you're looking for.


    Tip: Check out our open source project Lumia Imaging SDK Extras in GitHub. It will make it even easier for the developer to build amazing imaging apps. The source code project can be found in Lumia Imaging SDK Extras Git repository .

    New in SDK 3.0

    • GPU image processing - The performance has been improved by utilizing the GPU.
    • Video Frame processing - Lumia Imaging SDK 3.0 provides a Microsoft Media Foundation Transform (MFT) making it possible to apply the effects to video frames.
    • Windows interoperability - Lumia Imaging SDK 3.0 supports rendering to SwapChainPanel as well as rendering to or from the new Windows 10 imaging APIs SoftwareBitmap and IDirect3DSurface.
    • Information about valid ranges for property values are now available at runtime through PropertyDescriptions. This allows the developer to easily match slider values to the supported range.
    • Windows Phone 8.0 and Windows Phone 8.1 Silverlight applications are no longer supported.

    For more details, check the Version history and Compatibility.

    ...

    [​IMG]

    ...
    Version History - New in SDK 3.0


    Lumia imaging SDK has been updated with GPU capabilities and Universal Windows Platform (UWP) support. The development was focused on enabling these two scenarios with other improvements on the side.

    The Lumia Imaging SDK version 3.0 is not directly backward compatible with earlier versions. The main difference is that we have changed the way the image processing graph is organized. We have moved away from the concept of filters; all filters are now instead effects. For information about how to upgrade from 2.0 to 3.0, see How to upgrade guide.

    Lumia Imaging SDK 3.0 is split into three NuGets. You always install only one, depending on the platform you are targeting. For example, if you want to develop for the new Universal Windows Platform (UWP), you should use Lumia Imaging SDK UWP. For Windows Phone 8.1 projects, use Lumia Imaging SDK Win81. For Universal Windows Platform (UWP) JavaScript projects use Lumia Imaging SDK Javascript

    With SDK version 3.0, the Silverlight application framework is no longer supported. For Windows Phone 8.0 development, use Lumia Imaging SDK 2.0 that is available in NuGet.org.

    New classes and Windows 10 interoperability


    In Windows 10, new classes SoftwareBitmap and Direct3DSurface have been added to make it easier to work with images. We have added an image source and a renderer class for each of them. This allows the application to interact with the new Windows 10 types in a direct and efficient way.

    New building block interfaces such as IImageProvider2 and IImageConsumer2 have also been added. These interfaces improve and extend the image provider model that the Imaging SDK exposes; however, the developers can continue using IImageProvider and IImageConsumer when not implementing custom types. For more information, see Custom sources, effects and effect groups.

    Hardware accelerated effects


    The effects in the list below are capable of being rendered by GPU, which vastly improves the rendering time.

    Lumia.Imaging.Adjustments

    • BlurEffect
    • BrightnessEffect
    • ColorAdjustGpuEffect
    • ColorBoostEffect
    • ColorizationEffect
    • ContrastEffect
    • CurvesEffect
    • ExposureEffect
    • GrayscaleEffect
    • HueSaturationEffect
    • HueSaturationLightnessEffect
    • LevelsEffect
    • RgbLevelsEffect
    • SaturationLightnessEffect
    • SharpnessEffect
    • TemperatureAndTintEffect
    • WhiteBalanceEffect
    • VibranceEffect

    Lumia.Imaging.Artistic

    • AntiqueEffect
    • ColorSwapEffect
    • EmbossEffect
    • FogEffect
    • GrayscaleNegativeEffect
    • MirrorEffect
    • MonoColorEffect
    • NegativeEffect
    • PosterizeEffect
    • SepiaEffect
    • SolarizeEffect
    • SplitToneEffect

    Lumia.Imaging.Compositing

    • AlphaToGrayscaleEffect
    • BlendEffect
    • CachingEffect
    • ChromaKeyEffect

    Lumia.Imaging.Transforms

    • CropEffect
    • FlipEffect
    • ReframingEffect
    • RotationEffect
    • ScaleEffect

    Lumia.Imaging

    • CachingEffect
    • PassThroughEffect
    Microsoft/Lumia-imaging-sdk

    Lumia Imaging SDK projects

    Contents

    1. Quick start


    Quick Start is a sample project accompanying the tutorial that helps to get your first app that utilizes the Lumia Imaging SDK up and running. This sample implements the following basic tasks: picking an image from the camera roll, applying an effect to it, and processing the filtered image to be rendered and saved as a full resolution JPEG.

    Quick Starts

    Quick Start (C#) for Windows 10 and Windows 10 Mobile

    Quick Start (Visual Basic) for Windows 10 and Windows 10 Mobile

    Quick Start for Windows Phone 10 HTML5/JScript

    Quick Start for Windows 8.1 and Windows Phone 8.1

    2. Sample projects


    Samples

    Edit Showcase
    Edit Showcase is an example app that demonstrates the use of effects on still images. The photo is processed with the predefined effects. In addition, custom controls are implemented for manipulating the effect properties in real time. The processed image can be saved in the JPEG format into the camera roll.

    Video Effect
    Video Effect sample is an example app that demonstrates the capabilities and performance of the Lumia Imaging SDK by allowing the user to preview and apply a number of real-time effects to camera preview. The effects are applied to the stream received from the camera and shown in the viewfinder. The effects can be changed using the buttons in the application bar. This example app supports recording and capturing of videos and photos.

    Lumia Imaging SDK and Win2D Demo
    Lumia Imaging SDK and Win2D Demo is an example app that demonstrates the use of the Lumia Imaging SDK together with the Win2D API.

    Image Sequencer
    Image Sequencer is an example app that demonstrates the use of the Image Aligner and GIF Renderer APIs for creating Cinemagraph-style animations in the animated GIF format. There are also some example image sequences that can be used as a basis for the alignment and animation.

    Custom Effect Sample
    Custom Effect sample demonstrates how to create Custom Effects to do image manipulation both on the CPU and the GPU.

    3. Lumia Imaging SDK Extras


    This repository contains extra functionality and sample code for the Lumia Imaging SDK.

    ... [Go to the Repo]

    Finally we have those two posts from Igor...

    Get valid effect range using property descriptions in Lumia Imaging SDK 3


    Lumia Imaging SDK 3 is finally out. The long awaited SDK that was blocking some devs from porting their Windows 8.1 apps to Windows 10 UWP is now available with great new performance improvements and new features, but also some breaking changes. Highlights of the release are definitely Windows 10 support and GPU image processing that brings the performance improvements, but many more changes were squeezed in - you can find more details in the documentation.

    Silverlight apps are no longer supported though so if you want the latest magic from this SDK, porting to WinRT is necessary. If you're just migrating from v2 to v3, there's a great document for that called (believe it or not :)) How to upgrade from Lumia Imaging SDK 2.0 to 3.0. One thing that you'll notice right away is that filters are no longer called filters, but effects, but a more important (and breaking) change is that there is no FilterEffect. This means that instead of chaining the filters by setting the IFilters array of FilterEffect

    ...

    Property descriptions


    Imaging SDK 3 also brings a new interface called IPropertyDescriptions implemented by effects that gives you information about the minimum, maximum and default values of an effect property at runtime. It's super useful when you want to use sliders to change property values, which is quite often with range effects, but don't want to hardcode the value. Let's create a quick demo to see how this works.

    ...

    Conclusion


    It's amazing to see Lumia Imaging SDK going forward. With Windows 10 support, performance improvements and even new features, there's no better way to manipulate images in Windows 10 apps. For range effects, PropertyDescriptions should be used to detect minimum, maximum and default effect values.

    I remember when I built my Photo Light app, a photo editor for Windows Phone 7. There were no libraries like this one at the time, I pretty much had to do everything myself, pixel by pixel, taking care of memory, performance... Nowadays, all we need is Lumia Imaging SDK! :)

    [Check out the entire post]
    Using SwapChainPanelRenderer to improve real-time rendering in Lumia Imaging SDK 3


    In my previous blog post about PropertyDescriptions in Lumia Imaging SDK 3 I wrote some code for rendering effect results in real-time that wasn't really that good and resulted in performance issues. To be fair, the blog post was about using PropertyDescriptions so rendering was used only for a more complete demo, but as I was reading the documentation I noticed a couple of ways to improve the code I wrote.

    In that code, whenever the slider was moved to change the strength of the blur effect, I would create a new WriteableBitmapRenderer to render the result.

    This is bad resource management - creating a new renderer has a performance cost so it should actually be kept alive and only disposed when it's no longer needed.

    Doing some performance analysis proved that this code was bad and what happened when I loaded an image (3072x1728 pixels) and moved the blur effect slider around.

    ...

    Conclusion


    Using WriteableBitmapRenderers and WriteableBitmaps should be avoided as much as possible and renderers should not really be recreated all the time for real-time rendering as that causes serious performance issues. Instead, renderers should be kept alive and reused by the app if needed again. When it comes to interactive/real-time rendering, using a SwapChainPanelRenderer is recommended, and it should live as long as SwapChainPanel lives.

    [Click through to read it all]

    A few of the past times we've highlighted the SDK


    Follow @CH9
    Follow @coding4fun
    Follow @gduncan411

    [​IMG]

    Continue reading...
     

Share This Page

Loading...