A panoramic view from the balcony of a hotel. There's a road next to the hotel, followed by a quite heavily forested area. The sun is setting behind a lonesome cloud in the background.

A Trip to Mozambique

I just got back from a week-long trip to Mozambique. Sometimes my experience with X-Road and managing the platform in Estonia lands me consulting gigs in exotic places. Exotic to me, anyway.

I was contacted by some friends/colleagues that I have worked with many times over the years and they wanted me to do X-Road training in Mozambique. Since I have way too much free time currently and I’ve never been to Africa-proper, I of course said yes.

I first had to do a lecture on the fundamentals of X-Road over Zoom, which went decently well. We had materials available from a previous session done a while ago in another country. I wish I had a little more time to prepare and redesign them in my own personal taste (it was A LOT of text and very few images/diagrams). But it was okay.

Then it was time for the hands-on training session in person. I had to get a vaccine for yellow fever because either I have lost my vaccine passport or I have never been vaccinated for it, I don’t really remember. Technically it takes 10 days for the vaccine to become active and the record in the passport to be valid but it never got checked. I travelled there with the owner of the company I agreed to help. The flights were okay. Tallinn - Frankfurt - Johannesburg - Maputo. The Frankfurt - Johannesburg stretch was a little over 10 hours which was kind of a drag, but I listened to a lot of podcasts and managed to get some sleep.

On the left of the table, there is a small salad bowl. In the middle some mashed potatoes, meatballs and sausages, with a mustard sauce bowl. On the right, a glass of iced tea.
When in Frankfurt.

We spent the first night in a Radisson Blu hotel next to the coast. The room was pretty nice. There were two TV channels that showed movies so I watched a bit of the 1997 Batman and Robin movie while working on the training materials for the first couple of days. And then to my surprise, the next movie they showed was Good Luck to You, Leo Grande, which is a lovely film.

The next morning we were driven to a lodge about an hour’s drive away from Maputo. Unfortunately, since we were on such a tight schedule, most of what I actually saw in Mozambique was from car windows. When we got out of Maputo, we were driving on dirt roads.

In the foreground ins a hotel bed. In the background, a chair, a sofa and in the corner an open shower with brown tiles on the wall.
Interesting open shower in my room at the lodge. There’s a bath with a showerhead in the room next to it.

When we arrived at the lodge, the first thing I checked was the network speed. It was about 2 Mbps up and down in the conference room where we would be hosting the training sessions. I quickly had my clanker rig up a more sophisticated network simulation test because we were expecting 20 participants and that speed might be okay for ssh access but would probably start bottlenecking web interfaces. The results of the simulation tests said as much, it was stable but slow. Fortunately the local folks rearranged some network settings and we ended up with about 40Mbps up and down, which was plenty for the training session. A somewhat unfortunate consequence of that network change was that there was no more internet access at the rest of the lodge, which meant that I would be spending my evenings and nights in the same conference room, preparing for the next day.

In a conference room, on table covered in red cloth, there is a Macbook laptop. Next to it a mechanical keyboard and gaming mouse. It is dark outside, seen through the door in the backkground.
My setup for the sleepless nights.
A pool surrounded by tropical trees. In the foreground you can see my leg and sneaker. A rare moment of pause.
The pool at the lodge I never had time to get into.

So here’s the setup we used for the workshop. We set up (~170) EC2 instances for all the participants in af-south-1, so each participant would have their own X-Road instance and add-ons. We had all participants provide us with their public keys for ssh access. And since we weren’t able to get enough public IPs from AWS, we set up NetBird VPN. We provided the materials for the workshop via a BookStack instance running on ECS. I also set up some scripts to dump and push the books via BookStack’s API so that I could edit the pages as markdown files on my local machine while retaining the capability for others to use the WYSIWYG editor in BookStack.

During the first training day my colleague, who was providing us with AWS support from Estonia, vibeslopped a progress tracker tool that turned out pretty useful. It basically installed a probe on every EC2 instance and based on the training plan checked whether installed packages, configuration values etc. matched the expected values and provided a dashboard where we could see everyone’s progress or unexpected values. There were quite a few cases where we were able to proactively catch some misconfigurations. At first it was a Python back-end with a vanilla HTML/CSS/JS front-end and the probe itself was a huge bash script. After the training week I migrated all the back-end stuff and the probe to modular Python code and the front-end to Svelte because the tool was genuinely useful and I wanted it to have a more maintainable codebase moving forward.

Each day I would project my screen to the wall with half the screen showing the materials and the other half my terminal. I’d then proceed to set up the components, explaining the nuance, design decisions, context and various learnings from my Estonian experience. The workshop sessions were really intense. Everybody was super engaged but because there were 20 participants, that meant I had to spend a lot of time running around and providing support to folks. At one point I even started having my clanker ssh into their machines and because I was running it from the training materials repo, it could easily debug the issues and provide clear steps for fixes. I’ve already added authentication to the progress tracker so the tracker could be provided to the trainees in the future and I think it might be useful to add a support agent to it. It would just need to be oriented to focus on training and not implementing fixes on behalf of trainees. But I think I’ll focus on improving the materials before reaching for that.

The participants were really friendly and warm. Even though the week was intense, I had a lot of fun. I just wish I had more time to get to know them. We had breakfast, lunch and dinner at the lodge’s restaurant and I got to hang out with some people but that time was limited and there was a small language barrier issue as well. And after dinners I had to spend my nights preparing for the next day since I didn’t have too much time to prepare beforehand. I also learned a lot about what to improve during each day. I got about 3-4 hours of sleep each night during the week.

After the last training day, we were driven back to the Radisson Blu hotel. I was surprised to see that those dirt roads we took while traveling to the lodge now had a full traffic jam and the street was filled with people either finishing up their workday or already in a weekend mood, casually strolling between the snail-paced queues of cars. I probably slept a little during the drive back. After checking in, we had dinner with a colleague in a nearby restaurant. The seafood was great and they had a live band that among other popular tunes played a jazzy version of Creep by Radiohead. After getting back to the hotel, I crashed hard.

The only opportunity for exploring Maputo was the following Saturday, but when I awoke to my alarm at 6 AM, my body protested profusely. So I only woke up later to grab breakfast and then continued to enjoy my slumber until we had to be driven to the airport.

I’ve got another trip there coming up in July. Hopefully I’ll be able to walk around more then.