Transforming the image with an arbitrary matrix

[vpenades]

Could it be possible to transform an image by an arbitrary matrix?

Something like this:

Matrix3x2 xform = Matrix3x2.CreateRotation(2) * Matrix3x2.CreateTranslation(5,0);
pl.AddTransform(new TransformMatrix(xform));

[saucecontrol]

Yep, that's entirely possible if you create your own IPixelTransform implementation.

I'm not clear what such a transform would do, though. Typically a 3x2 matrix transform is something you'd apply to vectors, not to pixels.

[vpenades]

The usual transformations applied to an image are scaling, rotation and translation, which happen to be a generalized as a matrix3x2.

So what this pixel transform would do is:

• loop over the destination pixels
• for every destination pixel, transform its pixel coordinates by the inverse of the input matrix
• copy the pixel from source, using the transformed pixel coordiantes

That would allow to have translation, rotation and scaling in a single pixel transformer

[saucecontrol]

It's not quite that simple, because a pixel value cannot be arbitrarily moved without distorting the image. In most cases, the transform will place the destination sample somewhere other than the center of a pixel, and in those cases, the destination pixel value must be interpolated by sampling several values from the source. This can be done, of course, but the general implementation is quite expensive in terms of memory bandwidth and computation (though well suited to GPU acceleration).

In general, I don't believe that kind of expensive transform fits with the philosophy of MagicScaler, so I'd not likely include it in-box. I do want people to be able to build those things or to be able to bridge into other libraries that provide them (e.g. SkiaSharp or ImageSharp) however, so if you attempt that and run into any issues, please let me know.

[vpenades]

I took some time to write a naive IPixelTransform that does more or less what I was looking for:

class MatrixTransform : IPixelTransform
{
    public MatrixTransform(Matrix3x2 transform, int dstWidth, int dstHeight)
    {
        Width = dstWidth;
        Height = dstHeight;
        Matrix3x2.Invert(transform, out _Transform);
    }

    public void Init(IPixelSource source)
    {
        _Source = source;
        _SourceBpp = 0;
        if (source.Format == PixelFormats.Grey8bpp) _SourceBpp = 1;
        if (source.Format == PixelFormats.Bgr24bpp) _SourceBpp = 3;
        if (source.Format == PixelFormats.Bgra32bpp) _SourceBpp = 4;            
    }    

    private IPixelSource _Source;
    private int _SourceBpp;

    private Matrix3x2 _Transform;

    public Guid Format => _Source.Format;        

    public int Width { get; }
    public int Height { get; }    

    public void CopyPixels(Rectangle sourceArea, int cbStride, Span<byte> buffer)
    {
        // sampling one pixel at a time.
        var srcRect = new Rectangle(0, 0, 1, 1);

        // step vector to the next pixel in source.
        Vector2 srcV = Vector2.TransformNormal(Vector2.UnitX, _Transform);

        for (int y=0; y < sourceArea.Height; ++y)
        {
            var dstP = new Vector2(sourceArea.X, sourceArea.Y + y);
            var srcP = Vector2.Transform(dstP, _Transform);

            var dstRow = buffer.Slice(y * cbStride, cbStride);                

            for (int x = 0; x < sourceArea.Width; ++x)
            {
                var dstPixel = dstRow.Slice(x * _SourceBpp, _SourceBpp);

                // set source sampling pixel.
                srcRect.X = (int)srcP.X;
                srcRect.Y = (int)srcP.Y;

                // sample the pixel at Src and copy it to Dst

                if (srcRect.X < 0 || srcRect.X >= _Source.Width-1 || srcRect.Y < 0 || srcRect.Y >= _Source.Height-1)
                {
                    dstPixel.Fill(0);
                }
                else
                {
                    _Source.CopyPixels(srcRect, dstPixel.Length, dstPixel);
                }                    

                srcP += srcV; // advance to the next pixel in the source bitmap
            }
        }
    }
}

The result of applying this transform with a matrix that has translation, rotation and scaling looks like this:

The biggest problem is that it's using a very naive "point" sampling mechanism, and ideally it would have to use bilinear interpolation or maybe something more advanced.

Now, many things of MagicScaler are internal, and I don't know if doing this with a IPixelTransform is the right approach, but as I understand it, part of the process of scaling an image implies "sampling" the source image. What it would just need is to apply a matrix transform to the coordinates of these "sampled" pixels.

My plan for now is to use this to rotate the image keeping the scale, and then resize it to its final dimensions, but it's a two step operation.

Just leaving this here for reference and proof of concept.

[rickbrew]

If you're on Windows, I'd recommend something like Direct2D or Win2D (which is a Direct2D wrapper). It has a built-in 2D Affine Transform effect which does exactly what you're looking for. In fact, it's what powers Paint.NET's Move Selected Pixels tool. It has several sampling modes including Nearest Neighbor, Linear, and Cubic.

[vpenades]

Sadly I need a robust, cross platform solution