GSoC 2026: Generalized graphical data rendering representation #4162
Replies: 12 comments
Community Bonding Period ~ Week 1Made the renderer to use the render-fill-and-stroke-via-list-graphic.mp4
|
Week 2Made the Gradient tool work with gradient node chains feeding a Fill node. Screen.Recording.2026-06-23.at.09.39.02.movLet the Gradient tool to read from and write to a "Gradient Value" node chain (optionally paired with "Transform", "Gradient Type", and "Spread Method" nodes) when it feeds a "Fill" node, not just when it feeds a layer directly. |
Week 3Started working on using the Meanwhile, a bug in the gradient tool has been fixed, which could result in the gradient selection and writing target being different. |
Week 4Continued working on #4257. In addition to that, created two PRs.
Screen.Recording.2026-06-23.at.10.35.16.movScreen.Recording.2026-06-23.at.10.37.33.mov |
Week 5Continued working on and discussed #4257 in order to investigate about the performance regression. Although the PR introduces more power to the fill and stroke nodes, it adds more complexity than just assigning paint values to the corresponding vector. |
Week 6
|
Week 7Continued working on tiling model design. In addition to that, opened some PRs.
|
Week 8Started researching on mesh gradient. The current provisional mesh gradient model reference is from the version that was part of the SVG2 specs before (and then removed), which defines the mesh using Coons patch. mesh-gradient-demo-1.mp4 |
Week 9This week was to research the possibility and limitations of the subdivision approach. It turned out noticing that approximating the shape and color of a Coons patch mesh gradient could require 1,000 ~ 5,000 subpatches at 1080p size even adaptive subdivision is introduced. While this can be rendered by Vello, which is hardware accelerated, it is nearly impossible to edit as an SVG because the file size becomes massive. mesh-gradient-demo-2.mp4mesh-gradient-demo-3.mp4 |
Week 10Investigated the capability of Rendered by subdivision mesh-gradient-by-subdivision-svg.mp4Rendered by displacement map mesh-gradient-by-displacement-map.mp4 |
Week 11The main focus of the research on the mesh-gradient-by-displacement-map-2.mp4 |
Week 12Support for additional interpolation color spaces has been added, including OKLab, Lab, and linear sRGB. These spaces can also be approximated using the convex combination property of source-over blending. mesh-gradient-demo-4.mp4The mesh gradient implementation was wrapped up and the PR was opened. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
About
My name is Yohei Yamasaki (@YohYamasaki) and I will be working on the "Generalised graphical data rendering representation" project as a Graphite GSoC '26 contributor.
Synopsis
This project refactors Graphite to create a more generalized graphic representation of paints. It introduces dedicated Gradient and Pattern nodes, as well as updated Fill and Stroke nodes that can consume generalized graphic sources, not just the current fill and color types. The project also enables the standalone rendering of colors, gradients, patterns, and other compatible graphics as ordinary layers. The implementation will update Graphite's graphics, nodes, and Vello/SVG renderers to support gradient strokes and pattern fills.
Deliverables
Final Report
My project aimed to implement generalized painting, which allows the use of any renderable graphics as a paint source. This lays the groundwork for Graphite's flexible painting capabilities, including stacked paint, tiling, and complex gradients. Next, I researched and implemented a Coons patch-based mesh gradient. This led me to an interesting approach for rendering it without embedding the final gradient as a raster image.
Contributions
9 merged PRs
4 ongoing PRs
Code changes: 4.6k lines added, 2.5k lines removed
1. Generalized paints
Before the project started, a vector had its own fields to store the paints for fill and stroke, which only supported solid color and gradient (gradient only for fill). The naive idea was to add support for storing
Graphic<T>in the Vector's field, but that causes a contradiction in the module dependencies. Then we discussed storing them as attributes ofList/Item, which was introduced in #4050 to allow inserting dynamic attributes. This made it possible to store the fill and stroke paints as attributes, but it affects every single vector paint in the application, so the refactoring was divided into several chunks: 1. implement the rendering part, 2. update the Fill / Stroke nodes and related tools to use the attributes, 3. refactor to remove the legacy structures.1-1. Refactor the renderer to use attributes as paint source
#4111
This change only targeted the renderer, making it use attributes as a paint source, while the node graph was still using the fields of
Vector. Thus, a temporary thin layer to convert from the fields to the corresponding attributes was introduced. While the solid color and gradient painting implementation stays almost the same as before, a new clipping-based paint was introduced to enable using vectors, rasters and other complex structures as paints. For SVG, a bounding box-sized<pattern>is utilized instead of<clip-path>, as it is a paint server element, which means it can be fed into the paint source of elements. This also lays the foundation for tiling rendering. This change deprecated the usage of thePathStyleandFillstructs in the renderer.1-1.mp4
1-2. Update Fill / Stroke nodes to set attributes
#4257
#4177
#4241
After making sure the renderer internally uses attributes as its paint source, these PRs migrate the paint data flowing through a graph to use
Graphicin the corresponding attributes instead ofVector's fields. This required refactoring the related tools that read and write paint data.Along with that, the Gradient tool gained support for working with node chains of
Graphic::Gradientcarrying transform and other attributes. The renderer and the tool were also fixed to treat the transform for a gradient as an absolute value in the geometry's local space.1-2.mp4
1-3. Remove legacy paint-related structs
#4297
As a wrap-up of the first version of the paint generalization, the legacy structs for paints, such as
Fill,GradientandPathStyle, were removed from the code base.2. Mesh Gradient
The initial plan was to work on tiling next, but since another data format refactoring was ongoing at the time, experiments of mesh gradient was chosen instead. An early version of the proof of concept had already been implemented right before GSoC began, and the paint infrastructure was ready to render it in production.
Example of a mesh gradient
Coons patch
A mesh gradient is a two-dimensional, grid-based gradient. In a vector graphics context, a mesh gradient is often modelled by a set of Coons patches; the most notable spec is defined by PostScript's type 6 shading, and some of the modern vector graphics editors support editing it. Making it part of the SVG 2 specs was also discussed, which was deferred and moved to SVG Next (svgwg/master/pservers.html at svg-next · w3c/svgwg · GitHub).
Coons-patch — Ag2gaeh / Wikimedia Commons / CC BY-SA 4.0
A Coons patch is one of the methods to express a surface, using the four curves that make up the surface's edges. The mesh gradient is constructed from multiple connected Coons patches. The patch calculation is done by a parametric function that is derived from several interpolations of the edge curves. The color interpolation, though, is a separate concern from the shape, which means bilinear interpolation cannot ensure that the color connectivity between patches is C1, thus bicubic interpolation is often used to have a smooth gradient.
Polyfill for Vello and SVG
As of 2026, neither Vello nor SVG supports a native way to render a mesh gradient. A noteworthy point is that for SVG polyfill, most of the major vector graphics editors rasterize the gradient and use the image as a paint source.
We have investigated methods to approximate the mesh gradient without rasterizing the final gradient. In short, subdividing into tiny subpatches was the feasible way for Vello, and using the
feDisplacementMapfilter to map a rectangle to a patch shape was the one for SVG.Patch shape approximation
Assume that it is possible to interpolate 4 corner colors in a rectangle nicely. (The detailed approach is discussed in the following section.) Then, here we can focus on how to approximate the shape of a Coons patch using rectangles.
Patch subdivision
When a patch is subdivided into infinitesimally sized subpatches, the shape approximates a parallelogram. Every parallelogram is affinely equivalent to a rectangle, and a Coons patch can be represented to a desired accuracy by the subdivision. In a real-life scenario, we can approximate the shape by dividing a patch recursively until every subpatch's spatial error becomes less than a certain threshold. Heuristically, this approach requires ~10k subpatches to render a relatively complex mesh gradient, which can still be rendered by Vello.
adaptive_subdivision.mp4
Displacement map
In SVG, however, the subdivision approach is impractical, as the file size could be 5~10 MB and the rendering could also be < 1 fps with that depth of subdivision. Instead of that,
feDisplacementMapis utilized, as the filter enables non-affine transformations through displacement mapping.feDisplacementMapsamples the source image at an offset encoded in the map channels. We bake the inverse warp into those channels by storing the displacement from each destination position to its corresponding source position. For each destination sample, the inverse Coons patch mapping is solved numerically using Newton's method. Editing is constrained by sampling the Jacobian determinant to avoid local foldovers.Transforming from a rectangle to a Coons patch
Color interpolation
Color interpolation is a separate problem from the shape, so here we can focus on how to interpolate 4 colors at the corners of a rectangle. The simplest approach is bilinear interpolation, which can be replicated by blending two linear gradients for the X direction with a linear gradient alpha mask for the Y direction. However, this cannot ensure C1 continuity of color between patches. Interestingly, human eyes are sensitive to the derivative discontinuity of luminance and exaggerate the connecting point, which is called the Mach banding effect.
To make the interpolation smoother, it is necessary to consider the slope of the color at each corner. Hermite interpolation is a commonly used approach for the purpose, and we can utilize it by calculating the color slopes at each corner using the finite difference method.
Comparison of two interpolation methods, linear interpolation causes Mach banding
The rendering of bicubically interpolated color is not natively supported in SVG either, but we can approximate it by combining 4 linear gradients for the horizontal direction and 3 linear gradient alpha masks for the vertical direction, if the color space is gamma sRGB.
One key observation here is that we can consider the source-over composition as a calculator of a convex combination of colors. Also, a Hermite spline is interchangeable with a Bezier curve, which is a convex combination that takes the Bernstein basis as its weights. This means that it is possible to bake the control net and the Bernstein basis into linear gradients with multiple stops to approximate the polynomials.
Given the$4 \times 4$ Bezier control colors $P_{ij}$ , the bicubic color patch can be evaluated as four horizontal Bezier curves $Q_j(x)$ blended in the vertical direction:
These curves are represented as opaque RGB layers ordered from$Q_3$ at the back to $Q_0$ at the front. To reproduce the Bernstein weights through nested source-over compositing, the required vertical mask opacities for $Q_0$ , $Q_1$ , and $Q_2$ are:
For the color spaces not natively supported by SVG, we need to approximate the Y-direction too, since the conversion from gamma sRGB is not an affine transform. Yet, we can utilize the convex combination property to blend two gradients for the horizontal direction with one linear gradient alpha mask for the vertical direction, in this case it is actually a "linear" mask, and approximate the vertical curve by having multiple steps, just as we do for a 1D gradient. This requires more layers than for gamma sRGB, but in our experiments, it is up to around 16 layers at most for a black and white mesh, and heuristically it uses 5~10 layers for blending medium-luminance colors.
Current state & What's left to do
The mesh gradient itself has not been merged yet, so the remaining work is to land it. The following improvements for the mesh gradient are also considered in the near future.
Challenges
The hardest part of the paint generalization was that it touched every single paint in the application. Moving the paints from
Vector's fields to attributes could not be done in one step without breaking the renderer, the node graph and the tools at the same time, so the work was split into chunks that each kept the application working. Also, the architecture of Graphite was evolving in parallel, which required understanding not only the overview of the codebase but also the target shape the architecture is moving toward. This could not have been done without the kind help of @Keavon.For the mesh gradient, I underestimated the gap between a mathematically correct interpolation and a production-ready renderer. Evaluating the Coons patch and bicubic color field was relatively straightforward, but much of the work became an iterative investigation of rendering artifacts. Subpatch subdivision caused anti-aliasing gaps, while inflating the subpatches fixed opaque rendering but produced darker overlaps when transparency was present. For SVG, subdivision generated excessively large files and was too slow for interactive editing, which led to the displacement map approach. That introduced another set of challenges, including numerically solving the inverse Coons patch, handling image quantization and filter padding, and preventing fold-overs so the inverse mapping remains well-defined. Also balancing visual accuracy with interactive performance was challenging. Reaching this design required testing and discarding several approaches, but the process significantly improved my understanding of the field.
Acknowledgements
I would like to thank @Keavon for his guidance, thoughtful feedback, and extensive help throughout the project. I am also grateful to the Graphite community and Google Summer of Code for providing the opportunity and support that made this work possible.
All reactions