In the Miraviso Colore app you point the front camera at your face, tap a copper brown, and your hair turns copper brown while you move your head. No frame leaves the phone. Put like that it sounds like a filter app feature. Doing it properly, on an ordinary Android phone and with a privacy promise you can check by watching the network, took more rounds than I had planned for. This is the engine as it stands today, with the numbers I measured.
Why there is a browser inside a Flutter app
The first thing that surprises people who open the code: the try-on does not use tflite_flutter, there is no Kotlin plugin and no Flutter shader. The screen is a WebViewWidget, and inside it runs a page with MediaPipe Tasks Vision in WebAssembly and a WebGL2 fragment shader.
The engine started life as a browser prototype and found its shape there. Porting it to Dart would have meant rewriting the segmentation, the shader and every fix described below, to end up with the same result. Today it lives in a package, live_color, shared between the salon tablet app and the app for people who want to try a colour at home. Before that it was copied into both apps, and the copies had already drifted apart once: one engine, two presentations.
You pay for that choice straight away. ES modules do not load from file://, and getUserMedia requires a secure context. The fix is a small HTTP server written in Dart that listens on 127.0.0.1, on a port picked by the OS, and serves the engine assets from the app bundle:
final server = await HttpServer.bind(InternetAddress.loopbackIPv4, 0);
// the WebView opens http://127.0.0.1:<port>/index.htmllocalhost counts as a secure context, so the camera starts. In the Android manifest cleartext traffic is allowed only to 127.0.0.1 and localhost, and the camera permission requested by the WebView is granted on the Flutter side, after the user has given it to the app.
From camera to mask
The front camera opens at 640×480 with an ideal 30 fps, and the working canvas has its long side at 480 pixels. More resolution does not help: a hair mask is soft by nature, and every extra pixel is paid for in every later step.
The model is hair_segmenter.tflite, the one from MediaPipe's old hair demo. Two classes, background and hair, 780 KB. Until mid-August I used selfie_multiclass_256x256, which segments six classes (hair, skin, clothes...) of which I needed one. The switch had three measurable effects:
- the model went from 16 MB to 780 KB, and the APK from 40.7 to 25.8 MB;
- on Google's official Pixel 6 numbers, CPU segmentation drops from 217 to 58 ms (GPU from 71 to 52);
- the CPU path, which is what phones with an uncooperative GPU end up on, became usable.
The segmenter runs in VIDEO mode and returns only confidenceMasks: a value between 0 and 1 per pixel instead of a binary mask. I try the GPU delegate first and fall back to CPU if creation fails. The detail that makes the difference is that the segmenter gets the same WebGL canvas I draw on: the mask arrives as a texture through getAsWebGLTexture() and never goes back to JavaScript memory.
Colour: multiply to go darker, relight to go lighter
Recolouring is a GLSL ES 3.0 fragment shader. The first version, in JavaScript over the canvas pixels, cost 66 ms per frame on its own, and no mirror survives that.
The basic idea is simple: compute the hair's luminance (Rec.601 weights, 0.299, 0.587, 0.114) and multiply it by the tint. Shadows and highlights stay where they are, because they come from the real hair. It works beautifully when the tint is darker than the hair. When it is lighter it does not work at all: multiplying black by platinum gives a salt-and-pepper grey, because multiplication cannot lighten.
For lighter tints the shader switches strategy, which I call "relight": it brings the hair's mean tone up to the tint's luminance and keeps only the relative shading around that mean. The numbers, picked by eye in front of the mirror and then kept:
- 60% of the local texture is preserved;
- below a 0.12 gap between tint and hair the relight does not kick in, so browns and blacks stay on the multiplicative path;
- relative shading is clamped between 0.45 and 1.5, against black holes and burnt patches;
- if the result falls outside the gamut, the whole tint is scaled instead of clipping each channel, so a light blonde does not turn white.
The mask confidence acts as alpha: thin hair and edges are coloured in proportion, and below 0.10 nothing is touched. There is no blur or feathering on the mask, because confidence is already soft. The hair's mean luminance, which the relight uses as its reference, goes through an exponential moving average with weight 0.25 and resets if it jumps by more than 0.15, for example when you walk into a brighter room.
The catalogue has 54 entries: 53 colours in 7 families (blondes, browns, reds, copper, cool, blacks, fashion) plus "Natural", which removes the tint. On top of solid colour there are techniques: balayage, ombré and sombré live, the brush on photos only.
Three bugs that looked like something else
The most useful fixes came from symptoms that did not point at their cause.
The colour flickered. I was binding the mask texture returned by MediaPipe directly. But the library recycles that texture for the next frame, and now and then the shader read a mask that had already been overwritten. Now I copy it into a texture of my own with a shader pass and keep the last good one.
The hair was always one step behind. I segmented the video and then drew the video: by then the video had already moved on by a frame. Now the frame is frozen into a texture before segmentation, so mask and image are the same snapshot by construction.
The mirror "breathed". Once a second the image seemed to zoom slightly. It was a synchronous read of about 300 KB of pixels from the GPU to compute the hair's mean luminance: it stalled the pipeline just long enough to be visible. Now the GPU does the average, reducing the texture with mipmaps down to 1×1, and I read a single pixel asynchronously with a pixel buffer object and a fence, collecting the value on the next frame. Zero synchronous GPU→CPU reads in the loop.
Then there is the worst case: the Galaxy A56 (Exynos 1580, Xclipse 540 GPU) returns empty masks on the GPU delegate without any error. No exception, no log, just a mirror that does not colour. The engine now notices and rebuilds the segmenter on CPU by itself. On CPU one segmentation took between 437 and 483 ms on that phone, so there the cadence is adaptive: segment every N frames, up to one in 20, with the colour pinned to the last mask. On GPU I never dilute the cadence: when you are tracking a turning head, staying locked on matters more than smoothness.
Last lesson: an unhandled exception inside the draw loop killed requestVideoFrameCallback, and the mirror froze on the first frame without a word. The loop now sits in a try that writes the error to the status line.
How fast it really is
In mid-August, with the engine's own work already under 25 ms per frame and the camera delivering 30, I measured three phones: 15 fps on the Pixel 8 Pro, 14 on the A56, 5 on a Galaxy S21. The next day, with the new model and the loop moved entirely onto the GPU, the Pixel 8 Pro went up to 22 fps, with 10 ms segmentation and a mask present on 398 frames out of 398.
The ceiling, though, is no longer the engine: it is compositing the WebView inside Flutter. On the Galaxy S21 about 180 ms per frame remain that do not show up in any of my measurements. I tried and ruled out the rounded clipping of the mirror; the next suspect is the composition mode of webview_flutter on Android. It is the cost of the architecture I picked at the start, and I would rather write it down than hide it.
The telemetry I did not expect
"No frame leaves the phone" has always been true: segmentation and colour run locally. In the early versions, though, MediaPipe, the model and the fonts were fetched from jsDelivr and Google on every launch; in August I bundled all of them into the app, about 39 MB more, and since then none of those files comes from outside. Then, watching the network from a headless Chromium, I found that MediaPipe's bundle, 60 seconds after the segmenter is created, sends a POST to odml.pa.googleapis.com/v1/log with an API key assembled at runtime. About 90 bytes of protobuf with version, delegate and counters, no image. It fires even if you never start the camera, and Google answers 200. Grepping the sources for it is pointless: the key never appears in plain text.
I did not patch the minified bundle, because a surgical edit to third-party code disappears on the first update. I closed it with a Content-Security-Policy served by the same loopback server:
connect-src 'self';
script-src 'self' 'wasm-unsafe-eval';connect-src 'self' takes away the engine's permission to talk to anything that is not the local server. 'wasm-unsafe-eval' is required, without it MediaPipe cannot compile its WASM; 'unsafe-eval' is not. Measured over 200 seconds, three flush windows: the request fires at +60 seconds, the browser blocks it, outbound connections stay at zero. Since a missing header breaks nothing you can see, there is a test (csp_test.dart) that queries the real server and checks the response.
For completeness: the app does have the INTERNET permission, because the salon directory, the map and the demo request form need it. The promise is about the colour engine, and the engine has no way out.
What I take away
Two things. First, "on-device" is not a property of an architecture, it is a property you measure: I had a fully local engine and a library phoning home, and I only noticed by watching the traffic. Second, the worst bugs in a video engine are not the ones that break it but the ones that make it feel slightly off: a flicker, a lag, a breath. Each had a precise cause, and none was the one I would have guessed.
If you run a salon and want to see how it works there, it is all on miraviso.it. If you have a product with this kind of problem (on-device vision, Flutter, data that must not leave) I am happy to talk: this is how I work, and here is the Miraviso case study. I also wrote about the same engine in when you don't need an LLM.