Polling Emacs Major Mode Data w/Minor Modes

After finishing my first major mode for use at work, I took a stab at writing a minor mode for a SomaFM mode I forked and modded. I’m new to writing code like this but happily, after some effort, I got that minor mode working. It simply polls the somafm-mode favorites file (songs favorited while listening to SomaFM in Emacs) and parses them to an Org file. It’s very simple and there’s much to do, but I instantly saw the possibilities.

This is actually a question to the community about the basic concept of polling data created by an Emacs major mode with a minor mode running in the background. I suspect it’s very common but as I’m just dipping my toes into coding modes, I’m seeing most of the ones I use every day from the outside; they do useful things and I’ve not peeked under the hood (outside code syntax-related ones) to see if this feature is behind any of that useful stuff.

What Emacs major modes utilize a listener that polls data files and then grabs the data for use elsewhere? Is that model a sensible one, in terms of a process actively running in the background in Emacs and “doing things”? It makes sense to me, as GNU/Linux is full of daemons “doing things” in the background, often supporting major applications that accept data from those processes. I just never thought about the model in context of Emacs before.

I realize I’m raising my newbie flag pretty high with this question, but I’m also really excited to write more modes. My first instinct is to be sure I “do it right” now that I’ve stumbled my way into writing something that actually works :slightly_smiling_face:

TIA

I’m answering my own question here a bit, but after some research I’m seeing the relationship between erc-mode and erc-querypoll-mode, erc-track-mode or ibuffer-mode and ibuffer-auto-mode where the observation mechanism is at play (or timing, if there is no event-driven mechanism). I guess where my thinking is really going is whether my combination is conventional or if there are changes that I should make.

somafm-mode

  • Primary station display and playback interface

somafm-favorites-org-mode

  • Watches favorite-song data
  • Detects changes
  • Updates an Org file
  • Starts and cancels its own timer or file watch

Looking at other major modes and their minor mode partners, it seems like I might be better off with:

  • Call the synchronization function directly when the major mode changes its data.
  • Use a package-specific hook (from within the major mode, used by the minor).
  • Use file-notify-add-watch when another process may modify the data file.
  • Use an idle or repeating timer as a fallback.

Anyway, i feel like I’m asking myself the right questions now, especially after reviewing other code. I just kind of jumped into this without much preparation and even though I have written scripts for single features for work purposes, I wanted to go further and bypass the requirements :wink: Sure, it works, but is it efficient?

I did see your post, but i wasn’t sure how to respond to your core issue.

Below, I mainly tried to identify projects that cover either:

  • using emacs-lisp to integrate with emacs’ core UI/UX features
  • some general aspect of desktop integation.

This is fairly long, but if you really want to dive into Emacs, hopefully there’s something here which will help.

alphapapa:

If you’re new to emacs-lisp, you should look at alphapapa’s code. His style uses a lot of Emacs’ common lisp functionality. It looks like you’re using some of that functionality.

alphapapa/emacs-package-dev-handbook

  • This resource covers a ton of packaging details. Alphapapa’s projects are usually based on fairly straightforward Makefile
  • Eldev’s comparison of emacs test frameworks. This resource also covers packaging/etc, but gives comparisons for different tools.

alphapapa/bufler.el

This is another common-lisp heavy example. It provides s-expression-based query language for deciding how to display buffers and organize them

alphapapa/org-ql and alphapapa/org-ql

org-ql provides an s-expression-based query language for other alphapapa packages. IIRC org-ql is intended to provide read-only features. It’s much nicer to work with than than the org-element API.

org-ql is the basis for org-super-agenda, which is a pretty big upgrade over the agenda features in org-agenda.

As a beginner, I had a hard time getting enough usage of org-agenda features to really expand beyond just the basic agenda display. I also work in too many different projects with too many TODO.org files

alphapapa/activities.el

This project helps you store “snapshots” of an emacs frame’s state without being too presumptive about how you plan to restore that state in the future.

  • Many of alphapapa’s projects rely on custom types of bookmarks and so they’re good examples of how to interact with those features. Here, your frame’s windows are serialized to a bookmark.
  • When I restore an activity, I get exactly what I expect: files, window arrangements, etc – whereas I never quite figured out how to restore an “emacs session.” That was always problematic
  • IIRC, running M-x desktop-read will attempt to restore all buffers that were open when you ran M-x desktop-save. I use this sometimes, but it doesn’t restore frames. It restores all open buffers as such, but doesn’t restore their state – IMO this is nice because emacs’ buffers can have weird stateful issues depending on their type/mode/specialness.

This package solved a lot of ergomic problems I had with Emacs. In hyprland, i have special workspaces for my dotfiles that I jump to quickly

# note: there were a few quirks i had to overcome to get this to work
workspace = special:$wsDotfilesEmacs, on-created-empty:[float] uwsm app -- doomclient -- -ce '(activities-resume (cdr (assoc "df�.doom.d" activities-activities)))'

bind = MOD3, $wsDotfilesEmacsKey, togglespecialworkspace, $wsDotfilesEmacs
bind = MOD3 SHIFT, $wsDotfilesEmacsKey, movetoworkspacesilent, special:$wsDotfilesEmacs

Other good reference projects:

For documentation, the magit projects are fairly simple. They build the docs site & PDFs from org-export.

magit/forge

uses common-lisp style defstruct with eieio for object-oriented programming and manages a sqlite store of repository data. This has an underlying project ghub that separates out the API access to forges.

  • using forge is somewhat complicated to set up. I encrypt my tokens with GPG and put them in ~/.password-store/api.github.com/username^forge.gpg for example.
  • if you’re installing packages with straight.el, then it clones the package sources and retains their .git folders. so these can be added to your forge database. With Doom Emacs’ build system, it may occasionally garbage collect or redefine remotes.
  • This means you can use/manage emacs packages… while also checking PRs/Issues and maybe contributing to them.

Git remotes based on Forgejo/Gitea don’t quite support Forge yet. But emacsmirror/agitjo.el can help with that: you don’t need to fork the project to submit a PR on Codeberg.

emacs-mirror/emacs lisp/auth-sources.el

This package ships with emacs. It defaults to netrc format, but supports other authentication sources including libsecret (d-bus)

If you’re using Doom Emacs, auth-secrets is already there. Other it requires a bit of setup:

(require 'auth-source-pass)
(auth-source-pass-enable) ; enable the 'pass source

;; manually examine the entries available from configured sources
(auth-source-pass-entries)

;; manually fetch source. careful with emacs features. there are a lot of caches.
;; you can use the 'no-littering package to consolidate your local caches & state files
(auth-source-pass-get 'secret "api.github.com/username^forge")

This is used for many packages that integrate with external sources

emacs-jupyter/jupyter jupyter-zmq-channel

ZMQ has been integrated into emacs’ core packages and most standard builds of Emacs include zmq support. Instead of custom URI protocols, this is how I would choose to integrate emacs with other programs while ensuring there’s some separation…

I kinda suck at emacs-lisp though lol. I’m assuming that you’re smarter than me, so this probably isn’t TMI.

ipython and jupyter expose a zmq interface and this project demonstrates how to interact with it.

  • It took me quite a bit of effort to set up emacs-jupyter and ob-jupyter. There’s quite a bit of metadata required for your org files and the workflow isn’t fantastic. However, it does work and the setup/functionality has improved a ton over the years.
  • IIRC it supports many of the standard jupyter necessities, including access to completion from the repl – you can open the REPL that your org-file is evaluating against.
  • The workflow problems are mainly downstream of (1) reproducibility and (2) deciding whether the source of truth is the notebook.org or an-ipykernel.ipynm

python, direnv, mise, etc

These are some misc notes on using emacs for development.

Almost out-of-the-box: emacs itself integrates very well with ipython and python-mode supports evaluation against a remote ipython kernel. Later on, as you setup projects, you can get clashes with uv, LSP, etc that make python more complicated if you’re not a pythonista.

With direnv, you can support many isolated environments. Using this with dependencies from Guix/NixOS can be a pain, but these days uv is native to those systems

I guess NixOS/Guix solved the python-build-standalone problem … nevermind).

Regardless, I have uv installed on NixOS via mise and everything seems to work well. I haven’t used it much though (and i still haven’t integrated it with Emacs.

IIRC emacsmirror/mise conflicts with the approach that purcell/envrc takes. It isn’t as transparent/comprehensive as purcell/envrc – that very comprehensively makes environment as transparent as direnv in the shell. 99% of your buffers will magically have the environment you expect – even if they are remote buffers over tramp /ssh:$host/path/to/.envrc!

I appreciate the comprehensive reply! While not new to Emacs Lisp I am fairly new to attempting package-style implementations (modes of any type). My introduction to stepping up was studying several language major modes and, of course, Arte Ebrahimi’s somafm.el SomaFM player for Emacs.

alphapapa has been a primary reference point, but I’m still working through the Dev Handbook. You identified two, bufler.el and activities.el, that I also honed in on that are providing the most help in terms of coding examples and solid functional implementations.

I struggled quite a bit with a file-notify-add-watch implementation and handling change attributes, but skimming through Doom Emacs code has given me ideas for better structure and usage of functions like that. Honestly, I’ve had more light bulb moments and ideas for future packages in the last couple years than I had in the decade prior! Standards at the companies I’ve worked for the last thirty years have all been IEEE- and CMMI-centric and formal methods are a part of my DNA. It’s honestly nice to read code that is a little lighter and I need to work on some relaxation in my own.

On a side note, I do have Magit installed and have been navigating it slowly, but I’m a diehard user of git on the command-line and generally have written my own .el tools for additional support. I suppose it’s time to to step out of the Renaissance! It’s the same with Python, too. I used it quite a bit in my sysadmin days (my move away from Perl) but once I moved into software testing I’ve focused on Emacs Lisp for tooling and documentation. It’s expanded my interest in Lisps and I tend to remain in that ecosystem as much as possible.

Thanks, again! Excellent references to chew on…