GrooveStomp Profile Picture

VPS Learning

Today we learn how little I know about system administration.

I host most of my own “cloud” services on a VPS I rent from SSDNodes. Recently I purchased an upgraded KVM node for reasons of which I’ll get into later. Anyway, I ran into a hiccup while trying to start migrating from my older VPS.

└─ curl
curl: (56) Recv failure: Connection reset by peer


NOTE: First of all, I know the machine is reachable because I can SSH into it.

I tried to audit all my stuff from the VPS itself.

Nginx systemd unit:

$ sudo systemctl status nginx
[sudo] password for gs:
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-02-26 21:16:49 UTC; 24min ago
     Docs: man:nginx(8)
  Process: 877 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 840 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 883 (nginx)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─883 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─884 nginx: worker process

Feb 26 21:16:49 ssdnodes-83291 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 26 21:16:49 ssdnodes-83291 systemd[1]: nginx.service: Failed to read PID from file /run/ Invalid argume
Feb 26 21:16:49 ssdnodes-83291 systemd[1]: Started A high performance web server and a reverse proxy server.

Nginx enabled sites:

$ sudo ls /etc/nginx/sites-enabled/

$ sudo cat /etc/nginx/sites-enabled/default
server {
  listen 80 default_server;
  listen [::]:80 default_server;

  root /var/www/html;

  index index.html index.htm index.nginx-debian.html;

  server_name _;

  location / {
    try_files $uri $uri/ =404;

$ ls -lh /var/www/html
total 4.0K
-rw-r--r-- 1 www-data www-data 612 Feb 11 18:41 index.nginx-debian.html

And the firewall rules:$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
Nginx HTTP                 ALLOW       Anywhere
OpenSSH                    ALLOW       Anywhere
Nginx HTTP (v6)            ALLOW       Anywhere (v6)
OpenSSH (v6)               ALLOW       Anywhere (v6)

All of this looks good - no obvious issues.

I also checked iptables via sudo iptables -L and noticed a ton of crap in there.

Something else to note is that the old system is running on Debian 8 but this server is running on Debian 9.

Okay, long story short, SSDNodes ran this for me:

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload

I had actually seen a reference to this while searching for random stuff but it was correlated with Fedora, so I chose to ignore it. (Also, it was late and I was tired of stabbing at this here and there when I had a spare half hour or so over multiple days.)

Okay, so why did I upgrade to a KVM instance?

Primarily for FUSE support so I can use s3fs to mount an AWS S3 bucket locally as a block device and have Nextcloud Files write to S3 instead of local disk.


I finally took the opportunity to jump back into RISC-V, Go and QEMU as a followup to my previous exploration.1

I had a difficult time understanding exactly what was required to run my compiled Go program in QEMU. Let’s take a look at the instructions.2

Compile and run in qemu-riscv64 (which is expected to be in PATH):

$ GOARCH=riscv GOOS=linux go run ../riscvtest/add.go


$ GOARCH=riscv GOOS=linux go build ../riscvtest/add.go

I was confused by was the mention of qemu-riscv64 being in PATH.

The qemu-riscv instructions I followed3 didn’t make any particular mention of a standalone qemu-riscv64 executable, and there’s all those parameters for block devices and whatnot.

Well, if I just take a look at my riscv-qemu dir:

↳ ls | grep riscv

Aha! Something!

In fact, there is exactly a qemu-riscv64 executable:

↳ find . -name 'qemu-riscv64'

So now it’s a piece of cake to run the RISC-V executable built from Go:

↳ pwd

↳ GOARCH=riscv GOOS=linux go build ../riscvtest/add.go

↳ ~/code/riscv-qemu/riscv64-linux-user/qemu-riscv64 ./add
Aaron: 12

↳ echo $?

I modified the program to print out like that because I didn’t know offhand how to check the exit status of an executable. Turns out you just echo $? and it works!

But that’s not all. It wasn’t at all clear to me what was going on, so I started doing some searching and stumbled upon this4 explanatory post:

There are three families of targets in QEMU:

User-mode emulation, where QEMU provides an AEE (Application Execution Environment). In this mode, QEMU itself acts as the supervisor – in other words, when QEMU sees an ecall it will decode the syscall number/arguments, perform the system call, and then return to emulating instructions. QEMU’s user-mode emulation is thus tied to a specific supervisor’s ABI, with the only instance I’ve seen in RISC-V land being Linux.

Soft MMU, where QEMU provides an MEE (Machine Execution Environment). In this mode the entire software stack is run as if it was running on a full RISC-V system with QEMU providing emulated devices – in other words, when QEMU sees an ecall it will start emulating code at the trap vector. Here QEMU doesn’t need to know anything about the supervisor as the exact supervisor code is running.

Hardware virtualization, where QEMU provides a HEE (Hypervisor Execution Environment) while relying on hardware emulation to provide better performance. We don’t have this working on RISC-V yet (as of October 2018), but there are specifications in progress along with early implementation work.

If you see ecall instructions in userspace just magicly working, then you’re probably running in user-mode emulation.

Ah, very interesting!

That led me nearly directly to sections 35, 46 and 57 of the QEMU documentation, covering that StackOverflow post. Section 5, directly led me to finally understanding how to run my Go program built for RISC-V architecture.

So this is very interesting to me. As I understand it, QEMU is running in “user space emulator” mode, acting as the operating system, implementing system calls. It seems like the riscv-go page even mentions this8:

Spike plus pk support only a small subset of Linux syscalls and will not be capable of supporting the full Go runtime.

The RISC-V QEMU port supports a much wider set of syscalls with its “User Mode Simulation”. See Method 2 in the QEMU README for instructions.

However, I’ll be honest: I don’t know what “Spike plus pk” is, so I essentially just ignored that part on first read.

Unfortunately, it seems like this is a long ways from running a native Go executable on simulated RISC-V hardware. I think. I’m honestly not super clear on System Emulator mode vs. Guest Agent mode vs. User Space Emulator mode yet. I have a vague understanding but will need to explore more. I think running the Fedora image as I did before is closer to running a “true” RISC-V system.

Here’s the command:

qemu-system-riscv64 \
   -nographic \
   -machine virt \
   -smp 4 \
   -m 2G \
   -kernel bbl \
   -object rng-random,filename=/dev/urandom,id=rng0 \
   -device virtio-rng-device,rng=rng0 \
   -append "console=ttyS0 ro root=/dev/vda" \
   -device virtio-blk-device,drive=hd0 \
   -drive file=stage4-disk.img,format=raw,id=hd0 \
   -device virtio-net-device,netdev=usernet \
   -netdev user,id=usernet,hostfwd=tcp::10000-:22

So it’s running in System Emulator mode (I’m guessing, based on the executable invocation) which should be as close as we can get.

Hopefully I can put some more time into better understanding this stuff. It’s been interesting!


2riscv-go Quick Start
3qemu-riscv Wiki
4QEMU targets
5QEMU System Emulator
6QEMU Guest Agent
7QEMU User Space Emulator
8riscv-go QEMU

PostgreSQL Password Encryption

I’m working on a side project to gain more exposure to different kinds of projects in Go. This one is a pretty typical web application involving multiple authenticated users. For data storage I’m using PostgreSQL for its UUID type and cryptographic extensions. The cryptographic extensions are interesting, because of things like chkpass1.

My project right now is using chkpass for storing user passwords. Almost immediately after hooking this up I started wondering how chkpass actually works. Let’s take a look.

The encryption uses the standard Unix function crypt(), and so it suffers from all the usual limitations of that function; notably that only the first eight characters of a password are considered.

So already we see something special:

only the first eight characters of a password are considered


And let’s look at the documentation for crypt2:

key is a user’s typed password.

salt is a two-character string chosen from the set [a-zA-Z0-9./]. This string is used to perturb the algorithm in one of 4096 different ways.

By taking the lowest 7 bits of each of the first eight characters of the key, a 56-bit key is obtained. This 56-bit key is used to encrypt repeatedly a constant string (usually a string consisting of all zeros). The returned value points to the encrypted password, a series of 13 printable ASCII characters (the first two characters represent the salt itself). The return value points to static data whose content is overwritten by each call.

Warning: the key space consists of 2**56 equal 7.2e16 possible values. Exhaustive searches of this key space are possible using massively parallel computers. Software, such as crack(1), is available which will search the portion of this key space that is generally used by humans for passwords. Hence, password selection should, at minimum, avoid common words and names. The use of a passwd(1) program that checks for crackable passwords during the selection process is recommended.

The DES algorithm itself has a few quirks which make the use of the crypt() interface a very poor choice for anything other than password authentication. If you are planning on using the crypt() interface for a cryptography project, don’t do it: get a good book on encryption and one of the widely available DES libraries.

There’s a lot there that’s good to know. A short summary here based on my reading:

This is very neat, but reading this makes me lean towards not using it for password storage, especially with statements like:

If you are planning on using the crypt() interface for a cryptography project, don’t do it

I understand that “true cryptography” probably requires a different level of cryptographic security than authentication credentials in most cases, but I don’t like to think of it that way. In my opinion, if it’s not good enough for cryptography, then it’s not good enough for the protection of my users’ privacy. Not to mention, the cryptographic security of the password is fixed independently of the size of the password entered. That’s not good.

So what do I recommend?

I recommend encrypting with bcrypt3 4 in your language of choice and a per-row unique salt per password5. Store the encrypted password in a regular varchar column.

Also read Jeff Atwood’s post entitled “You’re Probably Storing Passwords Incorrectly”6.


4bcrypt vs. scrypt
5Where to store password salts
6You’re Probably Storing Passwords Incorrectly

qemu and RISC-V

It’s been a long time since I’ve posted here. So what’s up?

I’ve been working at Lendesk in a team lead / senior developer role and trying to pursue my interest in Go and Linux. This has been a very tricky path to navigate as work has abandoned any pretense of plans to adopt Go and all work machines are MacOS. Thankfully, I’ve been able to carve out some personal time to keep poking around in my area of interest.

I met up with my buddy Lucas for drinks the other day, and as we are wont to do, we discussed Linux, programming, security, privacy and other such things. This got my mind running and eventually I remembered RISC-V1.

A while back I bought a System76 laptop. Originally I had placed an order for a Purism Librem 15, but due to some miscommunication and problems with delivery, I wound up cancelling that order. Luckily, right after I had placed the order, System76 announced that they were removing Intel’s IME from all of their machines. Management-engine-freedom is a great step, but it’s not quite all the way to Libreboot support, or entirely open hardware. RISC-V is entirely open hardware. I’m a huge fan of this.

So, last night I decided that I want to try poking around in the RISC-V world and see what kind of damage I can do. All I managed to do is boot Fedora Linux on a version of qemu built with RISC-V support. The community has done a ton of work here and the process is very easy to get started with; just follow the docs at qemu.org2. I was hoping to start poking around with Go on RISC-V, but doing so requires a cross-compilation build as described on the riscv-go README3.

I will hopefully follow up again with this sometime soon and post appropriate updates here.


3riscv-go README

Android: Mainframe

After lunch today I made a visit to a local boardgame store just to see what was new. The local gamestore near my place is much cheaper, but has a drastically different selection, so it’s always nice to see what else is available.

There were two notable games on today’s visit:

I’ve seen mention of Agricola: Family Edition before, so that was neat to see in person. I watched a Tom Vasel review of it, and I think I would still prefer to play Agricola: All Creatures Big and Small.

Onto the other game; Android: Mainframe is completely new on my radar. A quick read on the back says:

This sounds absolutely fantastic!

I’ve read two text reviews on Boardgamegeek, and unfortunately this game looks like a dud. I think this is really interesting, because I’ve been seriously thinking about Android: Netrunner a lot lately. It’s a game that I think is really interestingly broken. I won’t go into all the details here; I’ll just say that I’ve written pages of notes on it and am trying to identify what I like in the original design.

I was excited to see this two player strategic boardgame in the Android universe because there seems to be a large crossover with my thinking on Android: Netrunner. Things I would have liked to have seen in this version:

A quick glance makes it obvious that none of this applies to Android: Mainframe. This game is simply a rethemed version of Bauhaus1 with some random elements added via your “powers” in your hand.

Bauhaus looks pretty fun, but I think the attempt to theme it and add random elements probably subverts the basic game. Oh well. I guess this is good motivation to keep doing my thinking and planning on an Android: Netrunner influenced game design of my own! :-)


1Bauhaus boardgame

Race for the Galaxy Android

Race for the Galaxy is out now on Android1.
This features Keldon Jones’ AI2 and is the best digital version I’ve seen. It translates wonderfully and is easily worth the ten bucks.


1Race for the Galaxy on Google Play
2Keldon Jones’ Race for the Galaxy AI

Editor Sizes

A few months ago I was starting to get irked at the relatively poor performance of Emacs on my primary development system. Emacs isn’t terrible, but it is noticeably slow compared to Vim. So, for the sake of curiosity I decided to look at some executable sizes. I started by just doing a simple:

ls -alh `which <editor>`

But that only captures the size of the executable on disk without regard for any shared libraries or dynamic memory allocations.

I can’t recall where, but I found a comment on some forum that suggests the following:

ps -Ao rsz,vsz,cmd | grep <editor>

This works well. rsz and vsz are Resident Size and Virtual Size, respectively. cmd gives the name of the executable. Output looks like this:

$ ps -Ao rsz,vsz,cmd | grep [s]t
8896  65592 /usr/local/bin/st

The resident size is how much memory is actually in use by the application and the virtual size is how much memory the application has requested. There’s usually a rather large delta there, and it’s kind of interesting to see. In both cases the size is given in bytes, so divide by 1024 to get Kibibytes and by 1048576 to get Mebibytes. (Kilobytes and Megabytes in common terminology.) I did an audit of various text editors running in terminal or gui mode and captured their resident memory sizes which I’m sharing below. Pay attention to the suffix because there are many orders of magnitude differences between the smallest and largest programs!

In no particular order:

Editor Name Resident Memory Size GUI or TUI Additional Information
emacs 57.99 MB GUI No local settings files
emacs 22.43 MB TUI With local settings files
emacs 12.77 MB TUI No local settings files
nano 1.77 MB TUI
uemacs/pk 916 KB TUI
jed 1.6 MB TUI
gvim 25.8 MB GUI
vim 25.8 MB TUI
atom 266 MB GUI
nice 1 MB TUI
micro 10.95 MB TUI
vis 3.88 MB TUI
TextAdept 25.58 MB GUI
TextAdept 6.58 MB TUI
jupp 3.37 MB TUI
mg 1.03 KB TUI Micro GNU/emacs, this is a portable version of the mg maintained by the OpenBSD team.

NOTE: TUI means Textual UI and you’ll often see “curses”, “cli”, “console” or “terminal” used to mean the same thing.

Don’t take these records as gospel; I’ve noticed differences on subsequent captures of various programs. Without a doubt each editor will perform differently depending on the specific file and how the editor represents said file in memory. Also, I generally don’t have a good understanding of Linux memory, so there are lots of ways these numbers could be non-representative. As a ballpark estimate, however, I suspect these numbers should suffice.

If you liked this, you might enjoy A Memory Comparison of Light Linux Desktops

Android Applications

I’ve been getting frustrated with my computing experiences lately. I won’t go into many details here, but my frustration has extended to my smartphone and has spurred me to investigate alternatives to many programs and services I’ve been using.

Provided for your benefit here is a list of several Android programs I’ve recently discovered. I’m not vouching for these yet, and may not even have much experience with them yet, but they are here and may serve you well.

Name Description On F-Droid?
Syncthing P2P File Syncing Yes
Ted Text Editor Yes
DAVdroid Caldav and Carddav client Yes
VLC Media Player Yes
Firefox Web Browser Yes
Anuto TD Game (Tower Defense) Yes
Document Viewer Views PDFs and other files Yes
Google Authenticator Two Factor Auth Yes
FreeOTP Two Factor Auth Yes
Hacker’s Keyboard Keyboard Yes
Wallabag Read-It Later/Pocket Alternative Yes
Antennapod Podcast Client Yes
Swiftnotes Notes Yes
Forecastie Weather Forecasting Yes
SkyTube Youtube Yes
Clip Stack Clipboard Manager Yes
OsmAnd Maps Yes
Torch Flashlight Yes
Focal Camera Yes
Vanilla Music Player Yes
Barcode Scanner Barcode Scanner Yes
Mathdroid Calculator Yes
Ghost Commander File Manager Yes
Gallery Photo Viewer Yes
Timer Clock, Stopwatch, Countdown Yes
Night Screen Screen Brightness/Coloration Yes
Bubble Comic Book Reader Yes
Android IMSI-Catcher Detector Yes
Yaairc IRC Yes
KeePassDroid Password Manager Yes
Termux Terminal Emulator Yes
aSQLiteManager SQLite DB Manager Yes
BubbleUPnP DLNA Client No
OWLR Foscam IP Camera Client No
Zarchiver Archive Expander No
Private Internet Access VPN No
Protonmail Email No
Signal Messaging No

HandmadeCon 2016 and Self Hosting

HandmadeCon 2016 is just around the corner!
I’m really excited to go!
The speaker lineup looks absolutely stellar this year.

Besides that, I’ve obviously been keeping myself busy for a few months here with nary an update. The most visible change to anyone who might read this blog is that I’ve completely revamped my hosting situation; both making great progress on my goal of being entirely self-hosted, and for serving up content via HTTPS to you.
The specific implementation by which I’ve accomplished this is pretty interesting and has been a great opportunity to learn. I have documented a large portion of what I’ve done and I hope to share that here.

Short and simple. Talk soon.


Have you ever heard of suckless1? I think I stumbled onto suckless sometime while reading K. Mandla2, though there’s a reasonable chance it might have happened before then. suckless immediately resonated with me because of their philosophy3. Their software also looks good, very accurately reflecting the idea of minimalism4; often requiring you to rethink how you typically interact with software to do what you want to do. Sometimes this process of re-evaluation is fruitful, sometimes maybe less so, but it’s always interesting to experience. They have their own Linux distribution, called stali5 which I really recommend reading about. I’ve found a couple of other distributions that are similar in various ways: CRUX6, Tiny Core Linux7 and Morpheus8. Morpheus in particular looks to be faithful to the suckless way. I think CRUX and Tiny Core Linux are both full GNU stacks, so they’re less interesting to me personally, but if I struggle to be productive with the more experimental stali and Morpheus, then I wouldn’t hesitate to give them a try.

There are multiple facets to minimalism. Morpheus and stali both adopt non-GNU tools like musl9, sbase10 and ubase11. (Though stali doesn’t officially adopt all of these yet.) I like the idea of this, because GNU tools are often huge, complicated pieces of software with all kinds of support, sometimes for legacy systems that are practically ancient. The idea is also novel to me. I had simply never considered using a runtime libc other than glibc before. Clang isn’t new, so the idea of using a non-GNU compiler isn’t quite as novel, but in practice it’s still not common. Have you ever thought about the organization of a Linux distribution? Prior to reading about stali, I hadn’t. Linux distributions ship with dynamic libraries in common locations and programs are built to dynamically load these libraries at runtime. This can cause some problems:

Armed with this new knowledge, I’m really curious to try building statically linked programs with a non-GCC compiler using something other than glibc. Tiny C Compiler (TCC)12 caught my eye, but I had issues doing what I wanted. I can’t recall exactly what I encountered. I also gave Clang a whirl13 but that similarly didn’t want to work for me. I did get something going with musl and GCC14, but that’s clearly just a piece of what I want.

Okay, sorry, minimalism. Another aspect here is system initialization. I guess there’s a conflict in the larger Linux community over systemd. I’ve never really been curious about init systems before and my only prior exposure was invoking systemctl or /etc/init.d/... or service [start|stop|restart] .... When I started exploring Manjaro Linux, I also saw two variants (regardless of the specific desktop environment being included): with systemd or with openrc. I had no idea what that meant at the time, so I didn’t learn anything from it. suckless link to information about systemd on their stali page and I learned a lot from that. I can’t say I actively dislike systemd because it’s in my Arch installation and it appears to be working well enough. However, I’m now curious about alternative init systems and especially in trying to pare things down to their simplest minimal subset. suckless have sinit and all the distributions I linked above use something other than systemd.

So all of these things are very interesting to me and I’ve been downloading images and trying to use alternative Linux systems for just the tiniest amount of time now. It’s a learning experience for sure. Both Morpheus and stali lack less, more and various other tools I’ve come to rely on. We’ll see how things continue.

2Motho ke motho ka botho
3Suckless Philosophy
4Lessism Over Minimalism
7Tiny Core Linux
12Tiny C Compiler
13How to compile C apps with musl and Clang
14GCC + musl