Introducing Microlighter
daverupert.com
I made a smol client-side syntax highlighter that uses the CSS Custom Highlights API called MicroLighter. I’ll talk about why I made it in a bit but first I wanted to… ahem… highlight… some of the features.
  • lobsters__andyc@crustaceanson.theshelf.topM
    0·
    2 months ago

    Leverages Textmate’s language grammars

    I’m not sure if the author is here, but doesn’t leveraging Textmate’s grammars require embedding the Oniguruma regex engine (used in TextMate, Ruby, etc.)?

    In the browser, that means WASM. That seems to conflict with “zero dependencies” and 2 KB minified

    As far as I understand, Shiki has exactly this issue. It interpretes TextMate grammars in JS, and JS regexes are not enough for TextMate grammars. So it embeds Oniguruma

    https://github.com/shikijs/shiki

    score 1 · canonical

      • lobsters__andyc@crustaceanson.theshelf.topM
        0·
        2 months ago

        Yup, so this should say Textmate-like grammars, not “TextMate grammars”

        MicroLighter uses the CSS Custom Highlight API and TextMate grammars

        The grammars look like they are forks of upstream, eliminating all the Oniguruma constructs and so forth:

        https://github.com/davatron5000/microlighter/tree/main/src/grammars

        To be fair the README does address this, which I missed the first time. But the tagline is not accurate

        MicroLighter reads TextMate grammars with the browser’s native RegExp … It does not use Oniguruma, WebAssembly, or generated token markup.

        But it also munges regexes with regexes, which is pretty gross:

        /**
 * Replace TextMate end-pattern backreferences with escaped begin captures.
  ...
 */
const expandEnd = (pattern, beginMatch) => pattern.replace(/\\(\d+)/g, (reference, index) => {
  return beginMatch[index] === undefined ? reference : escapeRegex(beginMatch[index]);
});

        So yeah it seems poorly-specified – not something I’d want to build on. TextMate itself is a bit poorly specified, which is why everyone has to drag along Oniguruma

        I would like something like this, but not confined to JavaScript regexes. I am thinking about a “lexing VM” for syntax highlighting and other tools like sloccount, that’s built on top of regular languages and constructs like “find constant string”, “capture delimiter”, etc. I’ve been thinking about that since using writing syntax highlighters in Vim and C++/re2c: https://github.com/oils-for-unix/oils.vim/blob/main/doc/algorithms.md

        score 1 · canonical

        • lobsters__andyc@crustaceanson.theshelf.topM
          1·
          18 days ago

          OK now I see what expandEnd is trying to do – based on the capture of the begin pattern, it’s dynamically compiling a regex for an end pattern. (This is trying to match TextMate semantics, and TextMate also apparently does this dynamic compilation …)

          Except that rewrite is unsound:

          • \1 versus \\1 – the latter is a backslash and then the digit 1, and gets messed up
          • [\1] – is in a character class, not a backreference
          • \10 is not a backreference – there are only 9 in JS, with a single digit

          This is gross … I wanted to call it slop, but it’s actually inherited from TextMate. So there is pre-AI slop too, or Unix sludge …

          score 1 · canonical