operationnull.com

Local LLMs as the glue for complex Emacs functions

2026-04-05T20:58:00Z

I do all of my development on a laptop, so running sophisticated models locally isn't really an option for me.

I've run small models with llama-cpp, the Qwen4B model, and gptel, an LLM client for emacs. In that time, I hadn't considered it useful for any tasks.

But then I realized; gptel supports tools written using elisp!

This post is about my initial experiments with the function, since I am trying to figure out if I can integrate agents a little more into my workflow. I tried my first tool from gptel's README:

(gptel-make-tool
 :name "read_buffer"
 :function (lambda (buffer)
             (unless (buffer-live-p (get-buffer buffer))
               (error "error: buffer %s is not live." buffer))
             (with-current-buffer  buffer
               (buffer-substring-no-properties (point-min) (point-max))))
 :description "given a buffer name, returns its contents."
 :args (list '(:name "buffer"
                     :type string
                     :description "the name of the buffer whose contents are to be
retrieved"))
 :category "emacs")

That function allows the model to read the contents of any open file, terminal, browser tab, etc. given its exact name. Even the Qwen4B model was able to answer questions like: "What is in name-of-buffer". Of course, I had to supply the name of the buffer it would pass into the function. That hardly is useful to me!

Inferring Correct Buffer given Mode

It becomes an "agent" when it's able to make a decision which would be hard to do programatically. So I introduced a new tool, which wrapped a function I made to return a list of buffer names given a major-mode:

(defun bd/get-buffers-of-mode (mode)
  "Returns a list of buffers matching mode. This has few uses outside of a
gptel tool."
  (let ((buffers '()))
    (dolist (buf (buffer-list) buffers)
      (with-current-buffer buf
        (when (eq mode major-mode)
          (push buf buffers))))))

(gptel-make-tool
   :name "get_buffers_of_major_mode"
   :function (lambda (mode)
               (mapconcat (lambda (buf) (buffer-name buf))
                          (bd/get-buffers-of-mode (intern mode))
                          ", "))
   :description "Given the name of an emacs mode (i.e., org-mode), returns a
comma-delimited list of buffer names."
   :args (list '(:name "mode"
                       :type string
                       :description "the name of an emacs major mode to search
for"))
   :category "emacs")

Now I could ask:

I remember I have a buffer in markdown-mode which talked about experimenting with creating tools for emacs gptel. What date was it created?

The buffer is named playing_with_gptel_tools.md and the date listed in the contents is 2026-04-05 20:58.

With a function that reads the value of recentf-list, it's very likely a similar function would work for files which have been closed, and are not in a buffer! You could have a function which returns the most recent file names from the list, then one that, when given a file, opens it in the background and reads the contents.

Logging Tool Use

I decided, for debugging purposes as well as safety, I would like to log all the tool calls the agent makes. I didn't see any sort of function provided by gptel that would let me do this.

So, I introduced the following code, and rewrote my current tools:

(defvar bd/llama-cpp-tools-buffer-name "*llama-cpp-tools*")
(defmacro log-tool-use (name f)
  "Call F with ARGS, logging the call to BD/LLAMA-CPP-TOOLS-BUFFER-NAME."
  `(lambda (&rest args)
     (let ((str (format "Used tool %s with args: %S\n" ,name args)))
       (with-current-buffer (get-buffer-create
                             bd/llama-cpp-tools-buffer-name)
         (goto-char (point-max))
         (insert str))
       (apply ,f args))))

(gptel-make-tool
 :name "read_buffer"
 :function (log-tool-use
            "read_buffer"
            (lambda (buffer)
              ...

The previous query, run again, prints to the log buffer:

Used tool get_buffers_of_major_mode with args: ("markdown-mode")
Used tool read_buffer with args: ("playing_with_gptel_tools.md")

Querying Emacs Documentation

Here, I often need to look up example usages of emacs functions after reading the docs. Being the self-documenting text editor, it makes sense if the LLM knows how to use it too!:

(gptel-make-tool
 :name "describe_function_or_variable"
 :function (log-tool-use
            "describe_function_or_variable"
            (lambda (symbol-str)
              (let* ((symbol (intern symbol-str))
                     (found (cl-some (lambda (x)
                                       (funcall (nth 1 x)
                                                symbol))
                                     describe-symbol-backends)))
                (unless found
                  (error "error: symbol %s does not exist." symbol-str))
                (string-join
                 (delq nil
                       (list
                        (when (fboundp symbol)
                          (documentation symbol t))
                        (when (boundp symbol)
                          (documentation-property symbol
                                                  'variable-documentation
                                                  t))
                        (when (facep symbol)
                          (face-documentation symbol))))
                 "\n--\n"))))
 :description "Given the name of a symbol (which could be a function,
variable, or face), returns its documentation."
 :args (list '(:name "mode"
                     :type string
                     :description "the name of the symbol to be searched"))
 :category "emacs")

Show me how to use 'seq-reduce to find the longest string in a list? Don't make a function

Here's an example of how to use seq-reduce to find the longest string in a list:

(seq-reduce (lambda (current longest)
            (if (string-length current) (if (> (string-length current) (string-length longest)) current longest) longest))
          '("apple" "banana" "cherry" "date"))

The main issue is that string-length doesn't exist, and there's no INITIAL argument. So I made another tool to allow it to evaluate elisp expressions in a container:

(defun bd/gptel-run-command-in-container (packages command)
  (shell-command-to-string (concat "guix shell --container --network "
                                   (string-join packages " ") " -- "
                                   command)))

(gptel-make-tool
 :name "execute_elisp_expression"
 :function (bd/gptel-log-tool-use
            "execute_elisp_expression"
            (lambda (expr)
              (bd/gptel-run-command-in-container
               (list "emacs")
               (concat "emacs -Q --batch --eval \"(princ (progn "
                       expr "))\""))))
 :description "Execute an arbitrary emacs command and return the result.
Error messages from malformed expressions are returned."
 :args (list '(:name "expression"
                     :type string
                     :description "One or more sexps to execute in an empty
emacs session."))
 :category "execute")

I tried a lot of prompts; while it was able to lookup documentation, it didn't really seem to be able to use it to solve interpreter errors, rather than bashing it's head against the wall!

Use 'seq-reduce to find the longest string in a list, and then test a small example using the tool (no function). You can lookup documentation for functions you are confused about. Do not respond until you have something that you have verified.

The issue is that seq-reduce requires a function that takes two arguments, but the lambda function is being called with three. Let's fix the lambda function to take two arguments and test it again.
<off to work with the tool...>
I apologize for the repeated errors. The issue seems to be with the use of seq-reduce in this context. Let me try a different approach using reduce instead.

It didn't really weave querying the documentation and trying code together as I had hoped, rather going down a rabbit hole with non-existant functions. I stopped it after around 20 iterations. My code to run stuff in a container also has a bug with escaping being applied twice; but it's safe to say coding needs a lot of parts and a well-trained agent!

At least, it knows what emacs is capable of, if asked directly:

What does the reduce function do?

I will describe the reduce function for you.
The reduce function is not a built-in Emacs function. It is part of the Lisp programming language and is used to apply a binary function cumulatively to the elements of a sequence, reducing the sequence to a single value. If you are using Emacs Lisp, you may need to load a package or define it yourself. Let me know if you need further clarification!

Playing Music

I already have a function which uses yt-dlp to rip a search query (or url) off of youtube. It takes a search query. I originally tried the tool with args:

...
:name "rip_something_from_yt"
:function (log-tool-use
           "rip_something_from_yt"
           rip-video)
:description "Play a video from youtube matching a string"
:args (list '(:name "unique-query"
                    :type string
                    :description "The query to search youtube with. The first
video matching the query will play. Do not search generic queries.")
...

The model's reasoning found it got stuck on a error... I forgot to pass it as a function rather than a variable (#'rip-video)!:

Okay, the user asked to play a rock song. I used the rip_something_from_yt function with "rock song" as the query and set foreground? to false. The response was "void-variable rip-video". Hmm, that's not a standard response. Maybe there was an error or the function didn't work as expected.

But afterwards, it continuously ignored the hint I gave about passing in unique queries, and just passed in the generic "rock song" when asked to play a rock song.

But I have a large directory of local songs on my computer, with sub-directories named after each artist. So I created a tool called get-user-artist-preferences which calls a shell command to find files in the directory and return a list of artists...

Now a command like this works!:

Can you play a rock song I would probably like?

I played "In Flames - The Wound" for you. Would you like to try another song from your preferred rock/ metal list, or is there a specific band you'd like to hear?

That song title doesn't exist by the way, but the youtube search algorithm still prioritizes a song. Sometimes it still doesn't work:

Used tool rip_something_from_yt with args: ("rock song from an artist I would like" :json-false)

That gave me this, which I've never heard before, so I don't mind. The point of the tool was a little randomness anyways!

Verifying Github Actions Artifacts is not Easy

2026-04-05T13:26:00Z

I created a few shell scripts to, when given a list of URLs to Javascript/Typescript Github actions and a commit, tries to reproduce their build artifacts in a container.

The list of actions I iterated over were obtained by curling the top 500-starred Github repositories and following the links to which actions and commit they brought into their build environment.

There is inherent trust when including a third-party action in your CI/CD pipelines. A developer who is especially cautious of rogue scripts leaking build secrets or compiling backdoors into their software might audit the action's source code for malware themselves. But they probably won't check that the commited build artifacts; often multi-megabyte minified Javascript files with all their dependencies baked in; actually corresponds to the source code. The inherent trust that comes with something that is libre and publicly auditable has been abused in ecosystems like NPM ecosystem to trick people into assuming the releases that are actually published aren't malware.

It was my assumption that, since my study was limited to actions that were used in very important software on Github (the top 500 repos) would be on their best behavior in ensuring people can build their artifacts independently and verify they get the same output. Lots of them come from verified authors, or are owned by Github themselves! At least I thought this meant they would be better than a wider study already done on the reproducibility of NPM packages.

Out of the 517 actions I tried to build (I have up to 628 more I haven't gotten through yet), 166 of them didn't provide a package-lock.json file. That means that they don't adequately describe the exact dependency tree, with pinned versions, they used in their development environment to create and publish their build artifacts. In these cases, I was forced to install the newest versions of the dependencies, which automatically makes the build process non-deterministic; it depends on whatever versions happen to be available at build time to be compiled into the final distributable.

Furthermore, I had to dig through the build failures of over 100 actions to determine which had failed because my build scripts couldn't infer the proper build commands from those that no longer compile at all due to missing inputs or deprecated dependencies.

For those that my scripts failed to build or produce comparable artifacts, it's often because the commands required to build are very ad-hoc. This is a barrier towards automating verification and transparency in the Actions environment!

For the others; I found many forgot to declare a dependency on one of three programs: tsc, ncc, or yarn. The number of actions which did this are in the minority of those I looked at, but I'm not sure why this is so common, and can't guarantee reproducibility without knowing which version of the compiler was used! Additionally, some actions couldn't be built anymore because they required deprecated libraries (openssl1.1). Whichever top repositories are using these actions should update to a newer commit, or drop unmaintained actions from their workflows entirely! Some actions, when inspecting the repository at the specific commit built, frighteningly did not include a source tree corresponding to the artifacts at all!

Of the 401 actions which I deemed fair to include in my study after reviewing the build logs (i.e., my scripts tried the correct build commands), only 57% produced identical ASTs as the pre-bundled artifacts. That means 43% didn't build, or produced a different output than the executables which were published.

I still need to manually inspect why things produced different outputs; but have noticed a trend of different variable names, and differently ordered statements. uglify-js minifies Javascript by renaming variables to be as small as possible, and can possibly produce random output. For differently ordered statements, it's possible different dependency versions were used, or a similar Javascript minifier is inherently non-deterministic!

While I had previously thought of an idea to help automate all of this across Github, I find that's unlikely to be scalable. Instead, I may take a look at Docker actions, and determine the common causes for differing ASTs between Javascript builds!

Notes on Github Actions, Reproduciblity, and Trusting Trust

2026-03-10T15:27:00Z

I am working towards a course project related to "trusting trust" attacks in the Github CI/CD ecosystem. The idea is that Github actions have a lot of pull in supply chains, and the publication method allows malicious developers to upload compromised actions artifacts.

For example, Github prefers that all Javascript actions are minified and compressed to the smallest size possible, recommending rollup.js to compile all dependencies into one large .js file. This file format is small and portable, but not so fun for auditors. Github's trust model assumes the build artifacts developers upload match the source, but there's nothing stopping someone from injecting obfuscated JavaScript into the release, compromising the integration or deployment of pipelines which make use of it!

The usual solution for linking source code to build artifacts is reproducible builds. To be done effectively, that requires the project maintainers to be extremely explicit in what source code, environment, and/or dependencies were used to produce their artifacts (named 'provenance attestations'). Under those circumstances, anyone can have some degree of certainty an artifact is legitimate by building it themselves starting from source under the same conditions and getting an output which is identical down to each individual bit.

Note this says nothing on whether the original source code is legitimate. It also is not true protection from 'trusting trust' attacks, because all of the independent builders could have used a compromised toolchain (i.e., a specially crafted compiler binary which always compiles in a backdoor). For the latter, you could bootstrap the full environment by compiling absolutely everything from source without use of a binary you didn't compile yourself. Nonetheless, reproduciblity is what I wanted to focus on.

Problem with reproducibility number two: if a Github actions maintainer truly wanted to compromise dependent projects by injecting malware into their bundled actions, they certainly would not release information on how they built their project. If the maintainer doesn't want to cooperate, it's difficult to discern if something is intentionally non-deterministic.

What is there to do?

Diffoscope is the standard tool for determining what differs between two sets of build artifacts. In the JavaScript example, it uses js-beautify to de-obfuscate bundled code and diff them in a readable way. In cases of unreproduciblity, it may be possible to feed this prettified output into further static analysis tools to access whether or not a payload has been introduced. Though, most papers I have read leave this as a manual process. Perhaps it would be good to leave to an LLM and assign a confidence level?

There are legitimate reasons why a piece of software isn't reproducible, but they can usually be solved. I've recently come across this in packaging software for GNU Guix (which supports both reproducible and bootstrappable builds), in which diffoscope also saved the day, and may write a follow up post. As for actions, the question I ask myself now is whether or not some heuristic could be automated and scaled up to cover a larger portion of the Github actions marketplace. For example, could we create a Github app to scan new commits from tracked repositories and upload potentially evil code snippets publicly?

Pinenote

2026-01-14T01:55:00Z

I bought an experimental eink tablet for almost $500, because I like to read a lot.

Maybe the smugness is over the top, given the second clause doesn't succeed in following from the first. Paper and books are cheap, and reading epubs is a simple enough task on any device. But the Pinenote solves a specific problem I have more elegantly than I think any other device could, in a way that makes me curious what else I can use it to solve.

To get closer to the point, I've purchased a fair share of physical books off of sites like thriftbooks or biblio. While I like Emacs, I only want to use it when I need to be at the computer. Times where this is not the case is almost never. Pleasure reading is one of my few hobbies that does not require a computer, but only if I own a physical copy of the text I want to read, which again is almost never, especially because I'm reading at any time is done on a whim; I don't have a backlog to borrow or purchase.

It may be wrong to consider using the Pinenote a 'break' from computers, but eink feels extremely low tech. It looks like paper, gives off as much light as paper, and requires as much power as a piece of paper to keep the display on (each would-be-pixel stores a state similar to NAND flash when a charge is applied). In the pinenote's case, it almost feels like paper. I only wish it turned like paper.

Most importantly, the distractions of a laptop or phone are not there; eink isn't nearly fast enough for watching videos, it doesn't display color, and the frontlight eats through enough battery I am motivated to leave it off in favor of an external light. I didn't expect this kind of boon, but have even started using the device to hand write out the plot of my fiction rather than typing. Most of my usage has been limited to koreader, xournalpp, and a browser for some light research while preparing materials. In this case, Emacs as a writing environment fails to provide the most important thing: a distraction free environment.

In just a few days, I'm already appreciative the screen of an eink is never blank even when off; it gives the impression the device is always ready and waiting for me to use it (I make a point to leave the device 'suspended' on my desk with whatever I was doing still perfectly visible).

Of course; one of the main benefits is that the Pinenote is a full GNU/Linux device, which is without many restrictions and familiar enough. I'm excited to try running GNU Guix on it (I have no experience with troubleshooting or developing for ARM yet) and seeing if it can be used as a input device or drawing tablet on my laptop. I am in no rush though, for now I'm satisfied with the preinstalled software suite and much anticipated reading.

Personal Filesharing

2025-02-17T00:00:00Z

I previously used pastebin/filesharing sites to send code and images over IRC. However, I no longer do; after piecing together my own.

The code to run the pastebin is small; it would be even smaller if I were more practiced in shell scripting:

One script on my laptop accepts a file name and an optional time specifier in hours. It generates a new file name with a few random characters piped out of /dev/urandom, concatenates it with a dash - character, and a unix timestamp representing CURR_DATE + USER_SPECIFIED_HOURS. It then uses scp to copy the given file to a public-facing directory on the remote server with the generated name (the directory listing itself cannot be viewed in a browser, though, since the pasted files are meant to be relatively hidden).

The second script on the remote server simply scans the directory every hour for files that are expired (based on the file name) and removes them.

Of course, hosting your own pastebin is a relatively common idea if you have a webserver and the need, (I even found some tools which set up an entire netcat server if you need a full web service). However, I myself had not yet seen anyone use filenames to get almost all of the functionality of file-hosting sites like 0x0.st. I reused the idea from Protesilaos Stavrou's Emacs 'denote' package, which also showcases the power and simplicity of embedding metadata in file names; since it remains very accessible to filtering with standard CLI tools.

I like being able to upload my files and share them from a space I still control. I only remembered to finish writing about this idea since it already had appreciated benefits in unexpected places; allowing me to share my University work through a link to my own site, rather than the suggested use of services such as google drive.

Sometimes it's easy to forget that browsers can play raw video or display pdfs when directed to a video file.