I just found out about Piper TTS and this is hands-down the highest quality TTS I’ve seen for Linux in terms of voice quality. Piper TTS is all local as far as I can tell. Meaning it doesn’t pump the text up to some cloud service to do speech generation.
One more project added to the list I never get around to shortening. 
Nabu Casa (home assistant sponsor) has done a lot of great work for offline voice based computing as part of their “year of voice”. I’ve been following along but haven’t implemented it yet.
Another voice related project is whisper. It worked great when I wanted to extract some notes as text from an audio book. It runs much better on my 1080 GPU (year old info) than it did on my laptop.
@unruffled_lumiere Thanks!, I as not aware of piper, it sounds really nice.
I’ve been developing a package that integrates denote notes with the use of a TTS engine (festival and recently piper). You can try it by evaluating denote-say.el .
I need to add more functions to read parts of a buffer but right now you can use it with org files.
[ I had to add PYTHONPATH to be able to use piper. In emacs I also had to set it even though piper works on the shell.]
That sounds like you are headed in a direction very much like the one I would like to travel. I started dorking around with festival but… the voices… 
Good luck on the project!
Thanks!. The festival Spanish voice that I was using is not too bad but I agree that the default ones in English are terrible. I’m using piper now, and the voices are excellent.