Guix Locale setup and how Guix loading works

Guix
1

Background

I’m new to Guix and Guile but not Lisp (I’m comfortable with Emacs Lisp and Common Lisp) or GNU/Linux. I love the Lispyness and declarative system configuration that Guix enables (along with many of it’s powerful features). While I love it in theory, figuring it out has not been easy for me. While I was able to get Guix System running on my Laptop a few months ago (last time I had time lol) I eventually came to the conclusion that Guix System was not a viable solution for me (too many of the things I use or am asked to use for my actual life and work aren’t packaged and working for Guix and I simply don’t have the time or energy (or in many cases skill) to figure out workarounds (VM’s, guix shell, elfpatching and craziness) or manually fix all the packages.

After coming to that conclusion, I put it on hold for a while as I got busy and kept on using my old laptop. Now I have some time again and want to get my new system running. To fix the problem with Guix System not supporting important things I need well enough, I decided to use a minimal installation of Arch as my base and get as many programs as I can installed with Guix and let other stuff fall through to other methods.

My ultimate goal (given I don’t foresee Guix improving enough in the near term) is to make my own Lispy system management framework that works as an abstraction over my system in Common Lisp (where I feel much more comfortable). While I don’t have the system engineering abilities to make a full solution like the Guix Dev’s I think that for the things I need I can (if I get the time) make a more reasonable programmable system framework that would then use Guix (and other tools) in the background for the things that are outside of my ability to implement on my own.

Problem

Despite this general setup having lots of promise for my use-case, I’m having some issues figuring out the Guix side of things again. Specifically, I can’t figure out how to get the locale stuff set up properly.

As recommended by the manual, I want to build just the primary locale that I use. I.e. in this form shown in the manual:

(use-modules (gnu packages base))

(define my-glibc-locales
  (make-glibc-utf8-locales
   glibc
   #:locales (list "en_CA" "fr_CA" "ik_CA" "iu_CA" "shs_CA")
   #:name "glibc-canadian-utf8-locales"))

While I was eventually able to figure out how to install the custom package, I put it in a separate file and ran guix package --install-from-file=locales.scm on it as shown in this blog post on cosarara.me - Setting up guix, and locales, when I run guix home reconfigure path/to/file.scm to actually apply it using my home configuration that has the name of my custom package in the specification->package form, it errors out and claims that the package is not defined.

I presume this means that I need to be manually loading all of the required files in my configuration and can’t assume anything to be inhereited from the guix-profile state.

This brings me to my main questions:

  1. What are working, clean ways of doing things like this? I’ve tried to find example configurations so I can see how the stuff in the manual actually fits together in practice, but most I’ve actually found haven’t been touched in years or implement lots of undocumented custom stuff on top of Guix.
  2. How can I get a better look at how these things actually function. If this were Common Lisp, I could just get in Emacs on my main computer and M-. around to at least get a clue. Maybe things would be even be documented lol. In Guile, though I have installed Geiser on my current computer, I can’t seem to actually load any of the Guix packages (despite also having a Guix installation there).

Thanks in advance!

Edit: I didn’t mention before, but I did figure this specific case out. The make-glibc-utf8-locales form returns a literal package record(?) so if you do it all in one file (or if I figure out the proper way to load if in a loaded file) you can then have it be appended with, cons’d onto, or spliced with the specification->package list. The more general questions of how loading works and how I would be able to figure these things out on my own still remain.

1 Like
2

Quick tangent, what are the top apps you’re missing? Also, if you’re comfortable adding random channels to your system, maybe you can find more packages here: https://toys.whereis.social/ (Also, Flatpaks work in Guix.)

I also have a hard time finding the actual code or documentation. The best luck I’ve had so far is to check codeberg directly or grep through the local checkout at ~/.cache/guix/checkouts/.

guix package … guix home

I could be totally wrong here, but this is triggering my spidey senses. I don’t think these two systems share anything. So anything you install with guix home won’t show up if you do guix package --list-installed and I’m guessing nothing is connected the other way either. My gut tells me you should use guix package XOR guix home—and it probably should be guix home for persistent desktop stuff. Ok, there’s a 3rd isolated option: guix shell + manifest.scm for dev tools or just guix shell pkg for temporary stuff.

Maybe you need guix home reconfigure -L path/to/custom/locale/thing home.scm? It would probably be helpful to post the actual command and error you’re seeing.

1 Like
3

Thanks for the reply and sorry for the delayed response.

Quick tangent, what are the top apps you’re missing?

The top one I ran into when I tried Guix System was Qtile which I am using as my WM right now (which is broken and out of date at the moment). If I had the time I would probably switch to StumpWM or something similar, but I don’t have the time to rewrite all my functionality at the moment. In general (but I didn’t get this far), my problem is compatibility with random scientific programs and things I’m asked to use for school. Some of these things are already a pain to even get working on Linux at all let alone a FHS incompatible one. I am aware of Guix HPC and some other related channels, but even then there just isn’t enough support for me to go all in yet (as much as I would love that). I thought about doing stuff with containers and VMs for all this stuff, but I just don’t think I have the bandwidth to figure those things out while I’m going (and it might be really difficult with some of the more hardware integrated stuff). Fiji and/or ImageJ are some other ones that I know aren’t available yet (and those are the easy cases because they’re open source at least). While I eventually want to get to the point that I can quickly package the things that I need and contribute to Guix, I don’t think that I can do that reasonably yet.

maybe you can find more packages here: …)

Yep, thanks for those resources. I knew about the Flatpaks, but I didn’t know about that second channel aggregator site.

I also have a hard time finding the actual code or documentation. The best luck I’ve had so far is to check codeberg directly or grep through the local checkout at ~/.cache/guix/checkouts/.

Ok, I’ll keep that in mind. Thanks for the tip!

I could be totally wrong here, but this is triggering my spidey senses. … I don’t think these two systems share anything. About guix package (profile) and guix home.

Yeah, that is what found out too. I guess I was kind of confused about that from reading the manual because they spend so much time talking about the command line stuff and then only a bit on the basics of home. I had assumed that the home thing was just another interface for the same thing, but apparently that was mistaken.

Maybe you need guix home reconfigure -L path/to/custom/locale/thing home.scm? It would probably be helpful to post the actual command and error you’re seeing.

I ended up just defining it in the home one and then splicing it into the package list generated from specifications->packages. My original mistake was assuming that defniint with the name makes it available to be referenced by the string name in specifications->packages.

I had:

(define my-locales
  (make-glibc-utf8-locales
    glibc
    #:locales (list ...)
    #:name "my-glibc-utf8-locales"))


(define packages
  (specifications->packages
   (list
    "my-glibc-utf8-locales"
    ...)))

Whereas the working config is the same for the definition, but dropping the “my-glibc-utf8-locales” string in the packages definition and switching that for adding my-locales to the list pointed to by packages .

4

Excuse me for chiming into the conversation, I may not be able to be of assistance or provide any resources, in fact I haven’t yet installed as many packages via the guix package manager without breaking a thing or two. Still, having seen that the participants in this thread express similar concerns, I wish to be here also as a wanna-be-user of guix, perhaps the most inexperienced in this entire forum, as someone who is also attracted by the Lispy-ness and the configurability, but sort of intimidated by certain aspects of the practicalities, specifically by how there appears to be a nameless cloud which shrouds the discovery of the guix landscape. Perhaps what beginners in guix urgently need, beyond an overview of the basic CLI commands, is a convenient help system not unlike that built into emacs, and an official entry point to the guix-specific constructs exposed in guile.

Such an entry point and such a help system may already exist, but then it has eluded me for now, in which case I would like to apologize for unfairly casting guix in a more or less beginner-distant light.

Admittedly, I haven’t read the guix manual carefully despite having a copy of it, but in my defense it should not be apparently the only route to discovery. This may only be the case for me or for some and maybe not all (or maybe it is a consensus? I really don’t know), but in fact, as someone who prefers a top-down approach to most topics, personally I often find that manuals are best reserved as (one of many) means for those occasions when one has already gained familiarity with the overall structure of a topic, and probably also, with the various ways he can study the topic further, and is now at the phase where he finds diving into the details and carrying out the fine-tuning desirable. Without these preparations, I would find drilling into the manual exceedingly slow and dull for everyday workflow at the very best. With this in mind, I personally don’t think that the guix manual (or any manual for its corresponding matter, as they come) could be friendly to an average beginner such as myself.

So, as a summary, I would be extremely grateful if we can stumble upon an entry point and a help system for guix as described earlier, a sort of parallel to the emacs’ help system.

5

Yep, I agree with you on that. In fact I found the same problem with Common Lisp and am working on building such a generalized help system (well a bit beyond that). Perhaps once I get that setup a similar approach could be applied (at least within other Lispy/easilly machine-introspectable languages). At least the Emacs interface should be quite reusable across languages leaving basically just the interface style to be ported to Guile (and or other scheme implementations).

That said while I have put a lot of thought into it, there is still lots of work that I need to do to implement it and that might take a while because I have so many other things I need to deal with.

For anyone interested in the approach, I’ll explain. The ultimate goal is to make an extensible system for gathering helpful information. Basically on the Emacs side something (theoretically anything from forms, to objects, to symbols at least in the Common Lisp implementation) is sent as an input to a main help function from a loaded package/system/module on the language side. Help for symbols is of course the main focus, but it is effectively a query so anything could be implemented.

Gathering information is a multi-step process:

  1. Basic information is collected by the default function, things like the name of the symbol, docstring, value, function signature, type, and any other information that can be consistently gathered from just with the basic language methods. This step is (to my knowledge) where helpful.el stands for Emacs Lisp. It is already a level above what I’ve seen using Sly/SLIME or Geiser.
  2. Using this information the main function will then try to call package/system/module specific functions (using a default symbol in the package/system/module a symbol is from, manually registered handler functions (which could be added by both the original author/maintainer or the community and provided in a separate package), and finally on the Emacs Lisp side (though this method is probably not the best to use in most cases). This second step is the main power of this package idea. It aims to give a abstract interface to any stuff people decide is relevant regardless of whether such information is stored by the language implementation, whether the package is still maintained or the maintainer wants to take the time to add such functionality, or the behind-the-scene implementation. The goal is to make help hackable by everyone (bypassing lots of organizational and standardization problems that typically plague things like this). This way the interface is standardized and everything else can be as custom and hacky as the provider wants to be. Information can be stored or generated any way you want as long as you can make a Lisp function that retrieves it.
  3. The final goal on top of all of this is making it possible (and easy) for users to contribute to/improve the documentation (and packages but that isn’t the main focus). I haven’t done as much work in this area yet, but I think the idea from elisp-demos is great. I want to also facilitate contribution in general by effectively adding as many “contribute here” kind of buttons as possible which try to take people to a relevant palace to extend any part of the information shown. Other possible ideas could be Q&A (probably facilitated through git) and an integration with IRC streamline asking questions there and contributing the question and answer.

Anyway, I think that a system like this would vastly improve the help system in lots of lisp languages. I think the biggest problems I’ve run into come from barriers to contributing documentation to the original project, and then barriers to accessing all the information that is provided in a consistent way when everyone has their own way of doing it (which I think is reasonable). The point of this project is to find a way to give maintainers the autonomy they deserve while giving others a uniform way to access and contribute to documentation.

1 Like
6

Wow, this is genuinely intriguing! Let me try to crudely rephrase the process of gathering information you have laid out, and see if I understand it correctly.

  1. The first step is to use a built-in facility of the language, hereby called the DEFAULT FUNCTION, to retrieve the definition of a symbol as a form, and extract from it information inherent in the symbol definition;
  2. The second step mainly consists of somehow allowing the authors / maintainers / users to augment and / or reinterpret the original definition of the symbol via an interface known as the MAIN FUNCTION, so that specialized and even personalized documentation can be appended.
  3. Um, I may not have grasped the division between the second and the third step, I think this is on me, I am not a programmer or a computer science student by training and have thus far no experience in collaboration on a codebase or a project. But from a rough reading, is the third part mainly addressing how users may more easily contribute to the documentation with less bureaucracy?

Excited to see where you will take it, and no rush of course, I for one will be watching your updates on this if you feel like sharing them.

7

I believe we need tutorials for this everyday workflows in the Guix blog or a wiki.

I copied the setup of the install disc with info and gpm on tty2 for my systems to have the Emacs like comfort of the documentation always available.

8

Pretty close, but a bit off.

You can think of the first two steps as searches. The first step just takes what you give it and finds out the basics of what it is and where it came from (if possible).

Here are some example first steps (occasionally simplified for didactic purposes): (help 1) determines that 1 is a number with the value 1 and gathers a useful printed representation of it (e.g. 1 (1 bit, #x1, #o1, #b1)), (help '+) determines that + is a symbol bound to a function that it gathers basic information about like function signature and package/module/system it is from, (help '(1 2 3)) is a cons and a proper list with a value (1 2 3), and so forth.

Step two is asking specialist help providers. For example if there is a function confusing from a package obscure the help function will see if there is a function special-help-name in the obscure package (the actual name will probably be borg-help in Common lisp at least) will call it with the same query and see what it has to say about the function (things that one might want to provide here are reference manuals, and other things that are not necessarily stored in the code directly). It will also check the query in relevant registered help providers (that allow maintainers to put the help function somewhere else or under a different name, or allow people to add their own help provider from outside of the package).

All the information aggregated from steps 1 and 2 will then be passed to the Emacs side to show a well formatted, hyperlinked, and quickly navigable help buffer (think something like helpful.el).

The third step is a bit different. The first two were describing how the information is queried and aggregated from various sources. The third however, describes how the the help system facilitates users contributing their knowledge (or others knowledge) so that it can be accessed by those query mechanisms. Basically, I want to make the process of contributing to and improving documentation as streamlined as possible.

The preliminary name of borg (or maybe borg-help) is potentially illustrative if you are familiar with the Borg from Star Treck: my goal is to assimilate as much Lisp knowledge as possible into the help system, and then provide a powerful, two-way interface to all of that knowledge through Emacs (rather than having it scattered all around the internet, code, and random people’s minds and only accessible if you know where to find it).

As a more technical description of the registration process, the basic idea is adding to an alist in a global variable. The form would be (match-form . help-function) where match-form describes when a function is relevant.

E.g. (:package obscure) would call the associated function when the package of the queried item is obscure. It could also be a predicate function that is called with all the information so far. This allows for arbitrary matching. Thus, it could even provide help on things in the context of a DSL. Of course this is a more niche case, but I think it is a lot of potential power for little to no additional cost in terms of complexity. As a basic example (though not necessarily justified in this case), (help '(lambda (x) (* x x))) could be set to do more than the base case (which would only recognize it as a list and not lambda syntax) and could evaluate it to make a function and describe it just as if it were associated with a symbol.

9

After your explanation and the examples, I think now I have a clearer mental picture of the structure you are working towards.

Does it align with your intended design to say that: the parsing of the exact definition of a symbol does factor into the general equation, but the more crucial mechanism, which pertains mainly to the second step, is how there is effectively a layer of “polymorphism” of the borg-help function which enables package-wise divergences; and somewhere along the second or the third step, there will be devised a uniform interface for users to inject stuff into, or augment, the existing (polymorphic) help function in a case-wise manner. It is the polymorphism and the adaptability of the help function which does the heavy-lifting, and the forms that are fed into it does play a basic but passive role?

Yes, the Borg metaphor is particularly vivid, I think it does speak to the philosophy which I perceive you are trying to bring forth. On a minor digression, glad to find a fellow Star Trek fan! And having now mentioned it, it has been a while since I last checked the latest series under the franchise, maybe I will do that today to indulge in a bit of thrill, haha.

10

On this I also concur, I would imagine it to make the critical operations and config paths more accessible, especially during vulnerable circumstances where one happens to be amidst the process of installation or to have wandered into a dire disaster situation from which he must recover.

Um, I’m currently not in the slightest familiar with the TTY and the info system to have an opinion of it, but your words do point to another direction, which seems to be to graft certain portions of the guix system image to the host operating system, maybe I will also look into this gradually. :smiley:

11

IMHO, it would also help new users - those who like the idea of Guix but haven’t yet wrapped their heads around the paradigm shift - get it by following tutorials on the new workflows.

I still feel like, I’m missing a couple of “AHA“ moments :wink:

here the part in my config.scm responsible to automatically getting into info on tty2:

		 (modify-services %desktop-services
				  ;; modify TTY2 to get info for quick references if I hose the GUI
				  (mingetty-service-type
				   config =>
				   (mingetty-configuration
				    (inherit config)
				    (auto-login
				     (match (mingetty-configuration-tty config)
				       ("tty2" "info")
				       (_ #f)))))
	    			  (delete gdm-service-type))))
12

Yes, in fact none of the stuff that I am personally making involves directly finding parsing source code related to definitions. Basically I do as much introspection as I can in a general/standardized way, and then provide ways for maintainers and users to extend the help system to do other things (e.g. show the relevant section of the manual, parse their own test cases and have them show as examples for functions, or really provide any information that they want). Ultimately my point is that Lisps are extremely powerful, so why not use the full power of Lisp for our help system instead of mimicking less powerful languages by showing a statically defined doc-string with maybe a bit of information on arguments all provided in a rigid, implementation dependent way by functions like describe (CL) or help (Scheme). We need tools that facilitate contributing and accessing help information generically from many sources just like Lisps provide tools for abstracting over the minutia of computing so you can focus on modeling the computation you intend.

Yes, the extensibility of the help function (talked about in the second part) is of course core to the whole idea. The second part makes a generic bridge between a source of information and the interface. Instead of requiring a standardized source format (as docstrings or documentation generators do) and extracting it with a central system, it lets maintainers/developers/hackers who know stuff about a system provide any information they want; thus, giving them freedom to handle the backed however they want/need to).

The third part, however, focuses not so much on extending the help functions themselves, but rather on providing ways for even less technical or motivated people to contribute to the development of documentation. This is facilitated by the generic provider interface, but goes a step further. Essentially since help information in this framework is basically anything you can output from a function that is displayable by the client, you can easily make a function that gets help from non-traditional sources. Thus my focus here is to enable interactive information sharing by streamlined contribution to a source that is provided to the main help function.

Again, an example of something similar that already exists is elisp-demos which adds a “Demos” section to Emacs help that pulls demos from a big Org file which people can contribute to (a process that is optionally streamlined by a button which takes you to the file and creates a new entry automatically. I want to make (and encourage the development of) similar tools that make it easy for people to contribute information.

You can think of the first part as traditional help system stuff, the second as providing generic, programmatic access to relevant help information sources (enabling systematic contribution), and the third as building, facilitating, and encouraging help sources that streamline incidental contribution.

Ultimately, I think that this has lots of potential in the Lisp ecosystem. A common theme in Lisp and adjacent communities is that our tool makes impossible/advanced things possible or sometimes easy, but things are often harder to find or figure out because we’re small and each busy doing things. Once you know about all of your tools it’s pretty amazing, but getting there takes lots more intentional effort than it does for popular stuff. Contrast that to popular things like JS or Python where there are enough people learning any thing related to it that teaching the language is a major activity and focus in and of itself to the point that it is hard not to find things. Obviously we can’t just get big™ or start having more time™, so the only way we can really improve is innovating in dissemination of information.

Yep, when I was developing this idea I quickly realised that calling it something generic and supposedly obvious like “helpful-cl” didn’t capture the scope of the project well enough. Like this isn’t a prettier or improved version of the default help system, it focuses on a completely different idea about how helpful information should flow from people who have it to the rest of the community who would benefit from it.

I actually haven’t seen any of the new stuff either; mostly just TNG and Voyager.

As another digression, I found this in the Common Lisp spec while researching for making this system in an example for describe-object:

 (defclass spaceship ()
   ((captain :initarg :captain :accessor spaceship-captain)
    (serial# :initarg :serial-number :accessor spaceship-serial-number)))

 (defclass federation-starship (spaceship) ())

 (defmethod describe-object ((s spaceship) stream)
   (with-slots (captain serial#) s
     (format stream "~&~S is a spaceship of type ~S,~
                     ~%with ~A at the helm ~
                       and with serial number ~D.~%"
             s (type-of s) captain serial#)))

 (make-instance 'federation-starship
                :captain "Rachel Garrett"
                :serial-number "NCC-1701-C")
=>  #<FEDERATION-STARSHIP 26312465>

 (describe *)
>>  #<FEDERATION-STARSHIP 26312465> is a spaceship of type FEDERATION-STARSHIP,
>>  with Rachel Garrett at the helm and with serial number NCC-1701-C.
=>  <no values>
2 Likes
13

This is definitely the impression the communities have given me as well. But to elaborate, I think the root of it all is not merely that “we are small and each busy doing things”, or perhaps it is but in a more fundamental way than your words directly entail — it is not just a lack of time or collective effort to maintain the documentation alone. I think that in a way, an ecosystem such as that of lisp, and even that of GNU/linux, is by their intrinsic nature (not merely unattended aspects of collective maintenance) against popularization or unification, because each of its members and participants is on a journey to discover some kind of structure in their mind and to actualize it around them, that they find resonate with their values and their views. Though some of us happen to pick up the same tools along the way, the structure each wishes to implement, including those that are already implemented and out there, is ultimately personalized. This results in a plethora of domain-specific or use-case-specific tools, and some (in fact, many) of these tools may have been (unintentionally) portrayed as more generic than they actually are, plus the unfortunate fact that no one individual can yet afford the time or the energy to reinvent the wheel from scratch, to avoid having to adapt to other people’s tools, eventually all these lead to what we now have, where both individuals and ecosystems have trouble navigating around, or even merely understanding, the landscape of each other, ultimately because they are not made to be as general as possible, but to serve specialized purposes and philosophies.

I am not sure whether this impression is reflective of the facts, nor do I consider such situation a bad one, it is rather a marvelous one, to think that we are in an age where different paradigms emerge to collide with each other, and that one has the opportunity to exercise his will via his own actions, to explore this strange scenery and to utilize it to complement his own ideals, to carve out his own trajectory.

But it is then also the case that propagation of information is faced with a fundamental obstruction.

Actually, having said these, the idea you are implementing becomes even more meaningful in my opinion: the underlying principle you have described is not just a help system for lisp dialects, the same idea could alleviate the pain of communicating information across different communities and different cultures.

14

Yeah that could be related as well. Though I hesitate to casually attribute it given I never got to compare it to alternate timelines lol. I also suspect that intuitively, but in some senses it is not particularly actionable. Like if it is supposedly a fundamental problem with the technology that implies resignation at some level—and I’m too stubborn to resign when it comes to powerful ideas. See this if you want a longer read that covers some more of the things I didn’t say: The Lisp “Curse” Redemption Arc, or How I Learned To Stop Worrying And Love The CONS.

On the point about these things being a ‘special interest’, I agree that is part of this (at least now), and I can’t and don’t expect everyone to care about these things as much as an enthusiast, but I do think there is room for improvement.

I will say that at least some of it is related to time though. Just going off the top of my head I have seen amazing things implemented in Lisps that I have never seen done anywhere near as well in other places, I think of things like Petalisp, SBCL, Sly/Slime, Symex.el, cl-autowrap, CLASP/CANDO and other project that truly blow my mind. Like projects that are basically a few people, completely surpassing what is possible in other attempts of the same things that have far more contributors and support. In other words, in addition to my own experience and theoretical ideas, it is clear that something about Lisps gives them enormous potential when you know what you want.

At the same time, I’m well (often painfully lol) aware that basic libraries for inter-operation with the rest of the world in “normal” use cases are often non-existant, completely undocumented, or incomplete.

I would say that when the difficulty/scale of what you are doing surpasses the annoyance of not having “batteries” and documentation (or when you are doing some of the many things that are well supported) you get the most out of Lisps. But that doesn’t change that lack of “batteries” and documentation really gets annoying sometimes.

Like I’m formally scientist and only a hobby/incidental computer system designer/programmer/system crafter. Some comes up when doing data science, and I love my system configuration for what I do, but to be honest I don’t want to have to dig deep into understanding D-Bus protocols, make my own wrappers for C libraries (even tho we have amazing tools to do it), or get into the details of IPC when I’m trying to make fancy system things. I know that I want to use those technologies, I could probably even do so in a hacky way with like cl-python or something for at least some of them to use their many libraries, but I really wish that CL just had what I needed and all to similar quality to those magnificent projects I see leading the world in specialized areas (also I admit I am kinda overly focused on building CL’s stuff so that other people don’t need to do what I have to).

Tl;dr there are amazing things, but lots of things for mundane tasks just doesn’t have much done yet.

Side note: Another project that I have started work on (yes, this is all yak-shaving for a much larger system-controlling program I’m making because I want Guix/Lisp power everywhere) is a DSL for defining wrappers for external programs. Unlike ruricolist/cmd: Utility for running external programs which focuses on making a convenient way to use one off shell stuff in CL, I want to make something like the cl-autowrap but for external programs (rather than for C libraries). Which all started when I found that CL does not have a portable way to make FIFO pipes :confused:. Anyways if you get anything from this, it is that I’m a bit crazy (but understandably) about wanting to do more things from Common Lisp (I want my system to be programmable, not just scriptable) and hopefully at least helping a bit to make the less flashy but important stuff (documentation, interop libraries) get the attention it needs.

2 Likes
15

Please pardon me, I may have to review the links you have shared here and more carefully formulate a reply tomorrow, it is a rather late hour in my region, I am not thinking very clearly and should really tuck in now. Good night.

Having skimmed through the articles in the links, um, what comes to mind immediately and probably immaturely is that, maybe Lisp dialects need an optional type system or something similar, to rigorously define what actually is the problem a Lisp hacker has solved during his own hacking (but would this risk mutating the very things that make Lisp Lisp?)? my current read is that the essence of the Lisp curse is not that each Lisp hacker has his own way of solving the same problem, but rather that each hacker has his own problems, and though very often the problems different hackers face may overlap and have similarities, they are ultimately different problems if we are to be precise, which is why different solutions to apparently the “same problem” are not interchangeable and the real hindrance to communication in the Lisp community? This is the current hunch.

16

Yeah, sorry. I didn’t give much context because I’ve probably spent too long thinking about this lol. That article was basically just a philosophical analysis of something related to a point you make. I explain a bit more below. The other stuff was just me giving some examples. You might read if you’re interested. But first; a tangent!

Common Lisp actually already has this and there are libraries you can use to basically take it as far as you want (like The Coalton Programming Language which basically is “we liked Haskell and Lisp so we implemented Haskell in and as a Lisp”).

Not exactly. The most general concept of the “Lisp curse” is that Lisp gives programmers so much power that technical problems in other languages (i.e. how do we even get this to work) become social issues (i.e. I want to do it this way and you want to do it this way how can we collaborate). Since you aren’t aware of it, you would probably want to skim this article: The Lisp Curse (it is linked to in the one I posted before) which is really the main description of the idea.

Tl;dr: Lisp is obviously immensely powerful, but that also removes a powerful reason that causes projects to actually get a large enough base to continue and reach their full potential.

The connection to what you said was linking the idea to what you said here:

I was linking this idea with that of the “Lisp Curse” idea (which goes a bit more into why Lisp specifically might be like this). This was to set the stage for introducing the second article which responds to the “Lisp Curse” idea and thus to some extent your own.

The ultimate point was: while there are probably some intrinsic factors to Lisp/Guix/ being more fragmented and difficult to get into than they could be, I think that ultimately it is nothing that can’t be overcome by working on the right things. It is a factor, but not an immutable destiny. Lisp is expressive and I think by building the right things (and of course not neglecting social aspects) that we can substantially improve the situation.

I don’t want to justify the situation; I want to improve it.

17

Um, I also accept the premise that one way to phrase the Curse is: the power of Lisp makes the emergence of abundant choices each favored by different hackers effortless, but the difficulty for different minds to agree upon a single choice stymies collaboration and standardized codebases. So the Curse is a social problem when viewed in this light, but social problems and technical problems are not necessarily mutually exclusive; furthermore I would argue that looking at the social aspects of a problem alone could make the problem itself and the prospect of a solution far less tangible than looking at the technical aspects, and that by trying to tease out the technical aspects of an apparently social problem, we can move from the justification onto the improvement more easily. So I was thinking, can we rephrase the Curse or retrace it, so that potential technical aspects of the Curse can be uncovered? Maybe it would then be more easily susceptible to scrutiny?

With this in mind, upon reading The Lisp Curse, my attention latches itself onto the following passage, for some reason I regard this as the passage to the technical aspect of the Curse:

What I notice the most is this lack of portability, and I am wondering if we can rephrase the Lisp Curse as the implicit omnipresence of said lack, for this could then suggest that, what is desired is perhaps some kind of “type system” — or more generally an interface or protocol (could these be the more appropriate terms here? or should “type” still be preferred over “protocol” because types can be mechanistically checked?) — for capturing and encoding the very conditions or properties emerging from the specific Lisp hacker’s own setup that enable his solution to work on his own setup (but probably not everybody else’s).

Such a type system or protocol should, ideally and by design, be capable of expressing both “the problem that the hacker, himself, is having” (the “specific problem”) and “related parts of the problem which would make the program more useful to others” (the “general problem”) as special cases, so that we have PROBLEMS = TYPES, and instead of drawing a line between “specific problems” and “general problems”, we just adhere to our natural inclination to define different problems / types on our own setups, and compose them accordingly if collaboration is needed.

Now comes a point where I am confusing myself, and it may feel like blabbering to you. It feels really vague to speak of all this, the only certain thing is that, usually types annotate expressions, here however the hypothetical “types” are supposed to annotate something else. My first thought was to say “here the hypothetical types annotate programs instead”, but on second thought, this should be immaterial for Lisp, because Lisp programs are also forms — this immediately prompts myself to doubt whether such a PROBLEMS = TYPES type system can really lift the Curse at all, since as you have pointed out, sophisticated type systems are already implemented for Lisp. But on second, second thought (XD), even though programs are also forms, are all forms the same sense of expressions that are annotated by the usual types? Or is it the case that, a program is also a form syntactically, but semantically is more than a form? I think to sort out these confusion I should first get my hands on an existing type system for Lisp.

To sum up the probably messy informal “reasoning” above, in which are surely many discontinuities, I think:

  1. We should rephrase “Lisp Curse is a social problem” as “Lisp Curse is a portability / composability problem”, and emphasize the latter as much as the former;
  2. Narrow down on the exact category (not sure whether the pun is appropriate here) of “PROBLEMS” underpinning the failure of composability;
  3. Devise a type system, probably not in the sense that “TYPES annotate EXPRESSIONS”, but in the sense that “TYPES = PROBLEMS” (assuming there is a difference between the two senses when it comes to Lisp), to lift the Curse: not by directly encouraging collaboration through more tools for maintenance and documentation alone, but also by trying to make precise the nature of the Curse and then navigate around it.

It is alright, that I did get XD, I think I was the one who failed to articulate clearly last night.

I would really like to apologize in advance if I harbor misconceptions regarding Lisp and type systems which pose as a serious hindrance. There are many things that easily arouse my interest but I currently excel in none; I am afflicted by this tendency to just dabble and scratch the surface of something, say A, before whatever circumstance shifts my focus to something else B, and it would be circling about for a long way before some unplanned immediate desire takes me back to A to just peel off another layer of A, and then the cycle continues. Also, I am currently not affiliated with an institutionalized program of education, and mostly arrange what I wish to learn at a very personal pace at home, I could be very much slower in progress, especially when it comes to depth and formalism. This is true for my current familiarity with the capacity of Lisp and for my (shallow) intuitive grasp of type systems and type theories as well, I just cannot resist participating in such interesting discussions, but there could be much redundancy or noise in the opinions I express. I am sorry if these raise ambiguity in our communication.

18

No, at least that isn’t what I’ve experienced. What I experience (and what they are talking about in the quote you are responding to) is how there are lots of half-done projects. While there are sometimes implementation specific things for some specific projects, generally CL code is very portable. Like there are literally programs that were written in the early days, or even programs from before CL (like written for Lisp machines and stuff) that all they need is recompile and maybe a tweak for some little bit to run perfectly.

The problem isn’t that their programs don’t generalize in a technical sense, it is that they don’t get the work to get them from a technical solution to a well-known, fully maintained, and documented technical solution.

The crux of the argument is that Lisps supposedly make it easier for a random programmer to make a solution, but because of this, they aren’t forced (as in other languages doing a similar things) to get collaborators, standardize, and otherwise build the community around what they are doing. They can just do it on their own for their own little project.

I think this is to some extent true, but as the other article pointed out, the predictions of the “Lisp curse” don’t play out in full, generally people do settle on a few standards for things.

But while the whole thing is not as bad as the original article portrays it, there are aspects of it that are real. The one that I think I do see the most is how many amazing projects there that seemingly lack the work needed on the social side.

Programs aren’t just for computers, they are for humans as well. And while you can have a perfectly good program for a computer with no comments, no docstrings, no manual, and no further maintenance after creation, the project that treats their program like that will tend to die.

A good metaphor is this: Imagine you have some robot, you get it to be more long lasting than people (it is immune to disease and biological aging after all), you make it strong, it doesn’t need sleep, and make it pretty smart. In some senses it is just completely superior to humans, much more reliable for many purposes, but unlike humans, it doesn’t reproduce. Even if it lasts longer and is better at lots of tasks, it isn’t a self sustaining system. Eventually it will break down and that will be the end. Humans on the other hand, despite lacking all sorts of technical advantages that robots might have, can reproduce and this is a big advantage that in many cases can outweigh enormous technical disadvantages.

Lisp programs are similar to that hypothetical robot. Lisp gives the power to make amazing things. The robot may be amazing and extensible and exactly what other people would want for something, but too often it is sitting apparently abandoned, maybe not fully assembled, with no user manual, and no one there to show you how to use it.

Ultimately the most real form of the “Lisp Curse” that I see is this: a social problem (i.e. not having the community support, documentation, and such that helps other people start using (and eventually contributing to) the program) that is made easier to fall into because Lisp is so expressive that things that things that would require the social aspect in other languages are things that one person can do in their spare time in Lisp.

The efficacy of a solution at solving the task it is made for is only a small part of the larger picture of what becomes popular/standard. Lisp is amazing at making really good programs. The bottleneck here is work on the communication side. While this is in some sense a social problem (i.e. assuming we all had enough time it could be solved if we all were disciplined enough to write more documentation and be more involved in our projects in the long term), even this has a technical aspect. Specifically, we can write (and document and promote) programs that facilitate the social things we need. That is precisely what I intend to do. I want to use the expressiveness of Lisp to try to facilitate the interaction and work that needs to happen for Lisp programs to live beyond their original creator.

Well good because communication and trying is one of the best predictors of ultimate ability. If you are consistently interested in something, you will tend to figure it out.

19

I think now I understand better the themes of the two articles and what you are conveying. I may have started out assigning a far greater weight to composability / portability in the technical sense than what was originally meant, which was in fact that even the deficiency in portability is due to the lack of a desire to reach out or to respond, to connect to other people’s use cases, i.e. a social problem. I do have a mild taste that programs are literature as much as they are artifacts, in the same sense as in the robot-people interaction metaphor you have devised to illustrate it, and though the artifact as a standalone object may last indefinitely, it is the literary aspect of a project that requires sustenance. I think I may have yet to relate this to the case of the Lisp Curse, being a beginner in Lisp and having not yet done as much hacking and also haven’t yet the experience of being part of a project. Maybe years later, when I look back upon our conversation and on the two articles, there will be a sense of complete experience and conception of the phenomena you describe. :smiley:

Accepting that the major factor is the social aspect, then I see no other way but yours exactly as you have put it, “use the expressiveness of Lisp to facilitate interaction”. I wish to be able to offer some fresh perspectives on how this would be approached specifically, but frankly I don’t think I have specific ideas right now on how to increase the social cohesion between Lisp hackers (or any group of developers), and even if I immediately think of something right here and now, you may have thought of them already. Having read your replies, I do sense the extent of determination and passion you possess on this aim even through the screen, and I wish that you will be achieve your vision elegantly. I will also bookmark this thread we have, to follow your progress and in case I can ever be of the least help in the future, .

And thank you for your encouragement, I may have only looked at my leaving something halfway, but it was until your reminder that I realize I have also always been coming back, and it may take longer than I initially expected for the progress to be obvious, but you just made me realize that I have been making advances all this time! And that I can see it if I simply look back to where I was a year, a month or even just a week ago, and maybe deep down I know it, so I never once stopped.

1 Like