README.md (7124B)
1 # jm — Odin for scripts 2 3 A collection of small packages and one runner that make Odin comfortable for 4 the scripts Python and bash usually get. Everything builds on `core:`. The 5 only system library is libcurl through `vendor:curl`; SQLite is vendored and 6 linked statically, so a script that uses it still installs nothing. 7 8 ```odin 9 #!/usr/bin/env odin-run 10 package main 11 12 import "core:fmt" 13 import "jm:prelude" 14 import "jm:sh" 15 import "jm:path" 16 17 must :: prelude.must 18 die :: prelude.die 19 20 main :: proc() { 21 context = prelude.init() 22 branch := must(sh.out("git rev-parse --abbrev-ref HEAD")) 23 for f in must(path.walk(".")) { 24 fmt.println(branch, f) 25 } 26 } 27 ``` 28 29 `chmod +x` and run it. The first run compiles; later runs start the cached 30 binary. 31 32 ## Packages 33 34 | Package | Job | 35 |-----------|-----| 36 | `prelude` | `must`, `die`, `env`, `args`; arena or debug allocator; logfmt log file plus `deaths.log` audit trail | 37 | `sh` | `out`, `lines`, `ok`, `run`, `capture` through the shell; `exec`, `exec_run` with argv; `which`, `quote`, `error` | 38 | `http` | `get`, `post`, `post_json`, `get_json`, `download`, `request` over libcurl | 39 | `path` | `expand`, `join`, `mkdirs`, `read`, `read_lines`, `write`, `append_file`, `list`, `walk`, `temp_dir`, `same`, per-user app dirs | 40 | `timefmt` | strftime `format`, `local`, `parse`; `iso`, `stamp`, `date`, `duration` | 41 | `debug` | guard-byte debug allocator (origin: sonar, which now imports this copy) | 42 | `flow` | `width`, `each`, `manage`: lock-free worker pools where each worker owns one state slot and the caller merges afterwards | 43 | `tar` | `read`, `extract`: `git archive` output without a tar program | 44 | `sqlite3` | `open`, `exec`, `exec_args`, `query`/`next`, `prepare`, `transact` over a statically linked SQLite | 45 | `sqlite3/fuzz` | property fuzzer for `jm:sqlite3`: generated values and damaged SQL, replayable by seed | 46 47 `tools/odin-run` is the runner. Every package reads on its own; the doc 48 comment at the top of each file is the reference. 49 50 ## Conventions 51 52 - Scripts never free. `prelude.init` puts a growing arena in 53 `context.allocator`; the exit reclaims it. `ODIN_SCRIPT_DEBUG=1` swaps in 54 the debug allocator, which reports overflow, double free and write after 55 free at exit. 56 - Anything that can fail returns `(value, ok)` or `(value, os.Error)` so 57 `must(...)` wraps it. `die` writes the message and the call site to the 58 script log, to stderr, and to one shared `deaths.log`. 59 - `prelude` never writes to stdout. Stdout is the script's data channel. 60 - Logs: `<log dir>/odin/<name>/<name>.log`, rotated to `.1` at 8 MiB, and 61 `<log dir>/odin/deaths.log`. The log dir is `~/.local/state` on Linux, 62 `~/Library/Logs` on macOS and `%LOCALAPPDATA%` on Windows. `ODIN_LOG=debug` 63 echoes everything to stderr; the default echoes warning and above. 64 - Platform splits live in `_unix.odin` / `_windows.odin` files. `just check` 65 type-checks every package for `linux_amd64`, `darwin_arm64` and 66 `windows_amd64` from one machine. 67 68 ## Runner 69 70 ``` 71 odin-run script.odin [args...] compile if stale, then run 72 odin-run -clean drop the cache 73 ``` 74 75 Cache key: script bytes and path, every `.odin` in the collection, the odin 76 binary, target and flags. `ODIN_RUN_FLAGS="-debug"` builds unoptimised; 77 `ODIN_RUN_VERBOSE=1` prints the build line. On Windows the shebang does 78 nothing; call `odin-run script.odin` or associate `.odin` with it. When 79 `vendor/curl/lib/libcurl.dll` exists under the Odin root it is copied beside 80 the cached binary. 81 82 ## Recipes 83 84 ``` 85 just build debug odin-run just release optimised odin-run 86 just test all package tests just check 3-target type-check 87 just install odin-run -> ~/.local/bin (BINDIR overrides) 88 just sqlite compile the vendored SQLite just clean 89 just example run examples/hello.odin just fuzz 30s of jm:sqlite3 fuzzing 90 ``` 91 92 `just install` bakes this checkout's path into the runner as the `jm` 93 collection root; `ODIN_RUN_COLLECTION` overrides it. 94 95 ## SQLite 96 97 `sqlite3/vendor/` holds the SQLite **3.53.4** amalgamation (`sqlite3.c` and 98 `sqlite3.h`, source id `bf7c7f30031888f4e796e429ab3978879485813aaca6f641c7b33e4e09459bcc`), 99 taken from sqlite.org and verified against the SHA3-256 that page publishes. 100 SQLite is public domain, so vendoring it carries no licence obligation. 101 102 `just sqlite` compiles it once into `sqlite3/lib/sqlite3.a`, which is 103 gitignored and rebuilt when the amalgamation changes. `foreign import` 104 resolves that archive relative to the package directory. `odin check` never 105 opens a foreign import, so `just check` still type-checks all three targets on 106 one machine with no archive built. 107 108 The compile options are sqlite.org's recommended set, with three deliberate 109 departures, all of them in the justfile: 110 111 - `SQLITE_THREADSAFE=1`, not the recommended `0`. `jm:flow` exists, and a 112 connection per worker has to be safe. 113 - `SQLITE_OMIT_AUTOINIT` is **not** set, though it is recommended. With it, any 114 call made before `sqlite3_initialize` is a segfault rather than an error. 115 - `SQLITE_ENABLE_FTS5` is added, for a full-text index, and 116 `SQLITE_OMIT_LOAD_EXTENSION` keeps the link from needing libdl. 117 118 One trap is worth knowing even though the package handles it: the bytes behind 119 a text or blob column are freed on the next `step`, and SQLite reuses its own 120 pool rather than returning them to libc, so reading a stale pointer yields the 121 *next* row's data instead of crashing. AddressSanitizer cannot see it. That is 122 why `text` and `blob` clone into the allocator the query was given. 123 124 ## Fuzzing jm:sqlite3 125 126 `jm:sqlite3/fuzz` generates values and damaged SQL and checks the properties 127 the package promises: a bound value reads back as itself, a value is never 128 parsed as SQL, broken SQL faults and leaves the connection usable, an 129 argument list that does not match the statement is refused, a failed 130 transaction leaves nothing behind, and a reused statement stays honest. 131 132 ``` 133 just fuzz 30 seconds, roughly a million cases 134 just fuzz "-for=5m" longer 135 just fuzz "-seed=12345" replay a reported seed exactly 136 just fuzz-asan the same under AddressSanitizer 137 ``` 138 139 A run is a pure function of its seed, and a report always names the seed it 140 used, so a failure found by a random run replays deterministically. `just 141 test` runs 600 cases on each of four fixed seeds. 142 143 Two things about it are worth knowing, because both were found by building 144 it: 145 146 - **The properties read every row before comparing any of them.** Comparing a 147 column while the cursor is still on its row passes even when the read handed 148 back SQLite's own memory instead of a copy, because the bytes have not been 149 reused yet. Deleting the clone from `text` leaves the whole example-based 150 test suite green and fails 1 case in 3 here. 151 - **Each case has a watchdog.** Nothing in SQLite bounds how long a statement 152 runs, and a recursive CTE whose recursion stops advancing returns rows 153 forever, so a case that overruns is interrupted and reported as a hang 154 rather than stopping the run. `sqlite3.interrupt` is what does it.