README.md (8059B)
1 # go-libwebp 2 3 Status: Experimental. 4 5 This package provides a means to encode webp images. 6 It does _not_ wrap the entire libwebp API surface. 7 It _does_ provide a way to avoid CGO. 8 9 This package provides several options for interaction with libwebp: 10 1. dynamic libraries (purego) 11 2. WebAssembly (wazero) 12 3. transpiled C to Go (ccgo) 13 14 Each option is called a "backend". 15 16 Package dynamic provides a dynamic binding to libwebp. This requires the consumer to acquire the shared objects. 17 Package wasm runs libwebp as a sandboxed WebAssembly module under wazero, from a single 18 embedded artifact that serves every platform. 19 Package transpiled provides a Go translation of the libwebp source. 20 21 If the shared objects are available at runtime, dynamic bindings will be preferred. 22 23 ## Backends 24 25 Which backends are compiled in is a build-time choice, so a binary carries only what it uses. 26 27 | build | backends | binary | 28 | ---------------------------- | ------------------------ | ------- | 29 | *(default)* | dynamic, then wasm | 5.7 MiB | 30 | `-tags transpiled` | dynamic, then transpiled | 3.4 MiB | 31 | `-tags nowasm` | dynamic only | 1.8 MiB | 32 | `-tags nodynamic` | wasm only | 5.7 MiB | 33 | `-tags nodynamic,transpiled` | transpiled only | 3.3 MiB | 34 | `-tags nodynamic,nowasm` | none — build error | — | 35 36 Sizes are for a trivial program built with `-ldflags='-s -w'`; a bare one is 1.4 MiB. 37 Most of the wasm backend's footprint is wazero's compiler, not the 0.6 MiB module. 38 39 The `transpiled` tag displaces the wasm backend rather than adding to it, so no build 40 carries both pure-Go fallbacks. 41 42 `webp.Backend()` reports which backend is live. The dynamic backend binds whatever 43 libwebp the host provides, which may be a different version than this package was built 44 against, and may encode to different bytes. 45 46 Tags select what is linked, not what is downloaded: the module requires wazero, purego 47 and `modernc.org/libc` regardless. 48 49 ## Performance 50 51 Geomean over the 14-image corpus in `./bench`, relative to dynamic, on a Ryzen 9 7900X3D. 52 Lower is better. 53 54 | operation | dynamic | wasm | transpiled | 55 | --------------------- | ------- | ---- | ---------- | 56 | encode lossless | 1.0x | 2.6x | 2.8x | 57 | encode lossy (q=0.75) | 1.0x | 2.6x | 4.3x | 58 | decode | 1.0x | 2.2x | 4.5x | 59 60 Megapixels per second, 1280x720 photographic: 61 62 | operation | dynamic | wasm | transpiled | 63 | --------------------- | ------- | ---- | ---------- | 64 | encode lossless | 4.0 | 1.7 | 1.7 | 65 | encode lossy (q=0.75) | 25.7 | 10.0 | 5.1 | 66 | decode | 309 | 133 | 56 | 67 68 Same source, six builds. SIMD accounts for the whole wasm/transpiled difference, and 69 lossless barely uses it: 70 71 | operation | native SIMD | native scalar | wasm zig | wasm em | wasm em+SIMD | transpiled | 72 | --------------- | ----------- | ------------- | -------- | ------- | ------------ | ---------- | 73 | encode lossless | 1.0x | 1.2x | 3.2x | 3.2x | 2.6x | 2.8x | 74 | encode lossy | 1.0x | 2.3x | 4.8x | 4.7x | 2.6x | 4.3x | 75 | decode | 1.0x | 2.1x | 4.8x | 4.7x | 2.2x | 4.5x | 76 77 All backends encode byte-identically; the harness checks that before measuring. 78 79 Reproduce: 80 81 ``` 82 tools/build-native-bench-libs.sh # native shared objects to compare against 83 tools/build-wasm-emscripten.sh # SIMD and scalar modules 84 tools/build-wasm.sh # zig scalar module 85 86 go run ./bench -so bench/lib -wasm \ 87 "wasm-zig=bench/lib/libwebp-zig-scalar.wasm,\ 88 wasm-em=bench/lib/libwebp-em-scalar.wasm,\ 89 wasm-em-simd=bench/lib/libwebp-em-simd.wasm" 90 ``` 91 92 Package transpiled is a [`ccgo`-based](https://pkg.go.dev/modernc.org/ccgo/v3) translation from [libwebp](https://github.com/webmproject/libwebp/) 93 in the hope to bring webp encoder into Go space. 94 95 ## Example 96 97 `go run ./cmd encode ./cmd/megopher.png` 98 99 Can consume JPEG and PNG images. 100 101 ## Rationale 102 103 There exists a webp decoder for Go `golang.org/x/image/webp` that does not include an encoder. Using it is obtuse: if you need an encoder you need to hack together another dependency. 104 This package extends `golang.org/x/image/webp` with an encoder translated from the libwebp project, providing a single package webp codec. 105 106 ## WebAssembly 107 108 `lib/wasm/webp/libwebp.wasm` is checked in, built from the vendored `c-lib` source. Two 109 build scripts are kept, both reproducing a module wazero can run: 110 111 - `tools/build-wasm-emscripten.sh` — produces the checked-in module. Emscripten is 112 required for SIMD specifically: libwebp has no simd128 DSP backend, so its SSE2/SSE4.1 113 kernels are reached through Emscripten's SSE-to-simd128 compat headers, which wasi-sdk 114 and zig do not ship. Also emits a scalar module for comparison. 115 - `tools/build-wasm.sh` — a zig-only scalar build. No emsdk needed; useful as a 116 lighter-weight toolchain and as a control when comparing. 117 118 SIMD roughly halves lossy-encode and decode time but does nothing for lossless encode, 119 which is entropy-coding bound. 120 121 Run `tools/test-matrix.sh` to build and test every supported tag combination. 122 123 ## Toolchain 124 125 `tools/transpile.sh` regenerates `lib/transpiled` with 126 [`ccgo`](https://pkg.go.dev/modernc.org/ccgo/v4) v4, one file per GOOS/GOARCH, on an 127 amd64 Linux box. It needs: 128 129 | tool | for | 130 | ------------------------ | ------------------------------------------- | 131 | `ccgo` v4 | the transpiler | 132 | `zig` | darwin headers | 133 | `x86_64-w64-mingw32-gcc` | windows headers | 134 | `aarch64-linux-gnu-gcc` | linux/arm64 headers | 135 136 ccgo v4 supplies target predefines through `--goos`/`--goarch` but not target libc 137 headers, so each cross target needs a header source. zig covers darwin from its bundled 138 `any-macos-any` headers, which is why no macOS SDK or osxcross install is required. It 139 does not cover linux: ccgo cannot parse zig's `generic-glibc` `bits/types.h`, so 140 linux/arm64 uses a real cross compiler and linux/amd64 uses the host. 141 142 Two patches are applied during transpilation. libwebp declares `GetCoeffs` as a 143 `volatile` function pointer, which ccgo cannot emit a call through; threading is 144 disabled here so dropping `volatile` is safe. And `tools/sharpyuv_stub.c` replaces the 145 sharpyuv library, whose `sharpyuv_gamma.c` is libwebp's only caller of `expf`/`logf` — 146 `modernc.org/libc` provides those for linux only. Nothing reaches sharpyuv unless 147 `WebPConfig.use_sharp_yuv` is set, which this package never does. 148 149 ## Dynamic 150 151 To make use of the dynamic bindings, ensure you supply the appropriate shared objects at runtime. 152 It is the consumer's responsibility to acquire the shared object. This projects contains a script 153 that will build amd64 shared-objects and place them in `lib/dynamic/webp/blobs` (`tools/compile-dynamic-libraries.sh`). 154 155 ## Testing 156 157 Go side testing involves a lossless compression test; byte-wise comparison against a golden test image. 158 In addition, there is a fuzz harness that executes against the transpiled code - encoding and subsequently 159 decoding images of random size and color. 160 161 ## Features 162 163 - support for amd64 Windows, Linux, Darwin 164 - support for dynamic linking and transpiled backends 165 - if you want speed and must avoid CGO, use dynamic linking 166 - if you want to avoid CGO without further hassle, use transpilation 167 168 ## Contributors 169 170 This repo represents a joint effort between Chris Waldon (~whereswaldon) and myself. 171 Thanks Chris for figuring out the build process for compiling libwebp to Go! 172 173 We owe a great debt to [@cznic](https://gitlab.com/cznic) for both creating `ccgo` and fixing numerous small 174 problems we encounted while compiling `libwebp`. 175