The theme of this month has been “getting angry with tech” - or at least the people building that tech.

Digital signing in Romania

Institutions here in Romania have made great strides towards digitalization in the last 10-ish years or so, at least in most major cities. Rural areas are another story.

My local City Hall requires documents to be electronically signed using a “qualified electronic signature”, which basically means signed using a certificate from one of a handful of companies, which City Hall is then integrated with to check the validity of the signature. You unfortunately can’t use a “holographic signature” (meaning your regular handwritten signature scanned and attached to the document), only a qualified electronic signature from one of the authorized emitters.

In order to qualify as an authorized emitter, a company needs to pass a bunch of requirements and their process for emitting your certificate needs steps to verify your identity. I don’t know if a manual validation by a human is a requirement or just how they do things, but it is part of the current process.

Normally, this would be fine.

Increased digitalization is fantastic in my opinion and I would be absolutely willing to go through the extra hoops and pay for that certificate.

The issue is that there’s a single authority capable of emitting certificates where I live and there have been allegations of corruption in the bidding process and how competitors are authorized. I don’t know how much truth there is to those allegations, but several years after this system has been introduced there still is only one single authority available to emit certificates for my City Hall and their prices do seem like an opportunity for disruption by competitors.

Since there’s a single emitter that has a monopoly and acts as a gatekeeper to using any of the digitalized services that City Hall and various other institutions offer, they can basically set up an egregious set of payment options where you can buy a certificate for one day, one year or multiple years.

You might be tempted to go for the 1-day option, but:

  • since validation is manual, there’s no telling when it will be complete. Since the whole appeal of doing things online is that you can do so at your own schedule, the fact that you’re waiting for a manual approval that starts a one-day timer mostly invalidates the convenience of doing this online;
  • if anything is wrong with your paperwork it will take days to find out from City Hall. Any later corrections to a form will require resigning, which means buying yet another 1-day certificate;
  • you generally submit one form (filled out in City Hall’s’ website and exported as a PDF) and a varying amount of scanned documents proving various aspects of your request, such as ID or ownership (as signed PDFs). Any errors that come up will usually be with the form itself or missing documents. Why oh why do I need to sign a PDF that the portal itself generated, knowing exactly who I am? Seems a lot like an opportunity just to require the certificate for the step most likely to be redone;
  • a 1-year certificate is roughly the equivalent price of five 1-day certificates, so due to the points above you’re highly incentivized to just buy the 1-year option.

There’s no reason why more realistic options aren’t made available (3-day, 7-day etc.) except to squeeze more money out of users. That would just be capitalism at work - after all the emitter is a for-profit company, if I were able to go to their competition. But alas, it’s a monopoly, the “best” kind of capitalism. Sigh.

The one good thing I’ll say is that response times at the certificate authority have improved since last year, though that may just be because I had been vetted in the past.

Incentives for digital parking systems

While we’re on the topic of digitalization of public institutions, I wanted to bring up the topic of parking payment systems around where I live - specifically concerning “public paid parking spots”. If you park in one of these then you can pay online - for a few hours, a day or even a month. To do so there are two payment options that I’m aware of.

One of them is via the AmParcat app, which is a miserable piece of technology littered with ads for gambling and betting. It should be nuked from orbit.

The second option is via City Hall’s website, which has a clean, simple interface, and which has worked pretty well in my experience.

The problem is that the incentives here are clear - City Hall wants your money and makes payment as easy as possible, but they’re then incentivized not to help you to continue paying for your parking, as that would rob them of a chance to issue a ticket.

They have your e-mail and phone number as part of the purchase, yet they:

  • don’t remind you about parking expiring soon, even for the 1-month option;
  • don’t tell you that your parking has expired, even for the 1-month option;
  • don’t tell you that they’ve issued a ticket for expired parking, but they will leave a notice in your car window that if you don’t contact them in 5 days and provide contact information then they’ll give you further (and larger) fines;
  • don’t allow you to check the exact date and time of when your parking expires, instead only showing you that information in the payment confirmation window on their website. It’s not part of the confirmation e-mail you get and it’s not viewable on their portal. If you don’t screenshot that confirmation screen - tough!

What they do do however is make sure to physically scan all of the cars (many hundreds of them) in these parking spots daily, to make sure that they can catch people even after missing a single day (or less, if your parking expires right before their scan).

For the last point about not being able to view when your parking expires - I did eventually discover that if you attempt to purchase more parking, a pop-up shows up showing the exact time when your current parking will expire, but that’s so hidden away that I don’t think it excuses them in any way.

It would clearly be trivial for them to change their system in such a way where they encourage compliance rather than punishment, but there would presumably be less money in that.

Excel / Access debugging

I’ve spent a lot of time in the past couple of weeks debugging a variety of Excel and / or Access issues. Most of that work isn’t worth sharing here, with two exceptions.

At one point of my investigation I had an UPDATE query running through Access that would update an XLSX file by writing a value to a specific cell in a specific sheet, which would corrupt the whole file (later attempts to open it in Excel or work with it programatically would fail). ChatGPT told me that’s impossible, since the transaction would be reverted if unsuccessful, preventing the file from being corrupted. Thanks ChatGPT!

After much annoyance the breakthrough came when I started unzipping the XLSX files in order to inspect the XML content inside, diffing the working and corrupted folders using BeyondCompare. That’s when I noticed that while the data was the same (with the exception of the single value I wrote), the exact format of the XML tags was different. If I copied over lines from the corrupt version to the working version (rezipping as an XLSX at each step), I noticed that the file would either break immediately or sheets would start to disappear (depending on which lines I copied over).

Turns out - there are multiple ways of serializing XML data (such as, in this case, explicitly writing out the XML namespace or relying on a default namespace). Access / Excel will use whatever version they believe is best, without giving you any control over it.

Simultaneously, Access is (theoretically) better optimized than Excel, meaning that it will only write out (and thus reserialize) data that was changed, leaving the rest intact (unlike Excel, which decides to reserialize the whole file).

Taken together, what was happening was that Access was reserializing just some of the lines in the XMLs and mixing and matching different serialization methods would fail. What was actually corrupting the file was that the list of sheets was being fully reserialized by Access (despite nothing in that list changing), which meant Excel couldn’t read them, which meant Excel would refuse to open the file because it couldn’t find any sheets.

The lesson here is simple - don’t mix and match Excel and Access in your pipeline (too late for that, haha). If you must mix them, try to only write using Excel, while you can read data with either one, since that will ensure your serialization is completely regenerated whenever some of the data changes. Alternatively, you can also try creating a brand new file using Access and copying over all of the data from the existing file (in order to force a full reserialization), but at that point why even bother using Access?

I wish this section of this blog post ended here, but I had another fun issue during the same investigation.

It turns out some Python pipelines were breaking a XLSX file by removing named ranges from it.

Surprisingly, ChatGPT was actually super useful here. It correctly suggested that the issue was caused by the fact that parts of the Python pipeline were reading data using Pandas, but writing it via openpyxl. Apparently, Pandas doesn’t support named ranges and just silently drops them, which means that when openpyxl writes the data that was read using Pandas, it will remove the named ranges from the file.

The lesson here remains the same - don’t mix and match different technologies if you can avoid it!

I do wish Pandas would warn about dropping named ranges though… In any case, this time thanks ChatGPT in a non-sarcastic way - it actually saved me the time I would have spent figuring out why those ranges are being dropped.

Filen.io issues, local back-ups to sleep better at night

And now, for something totally different.

I switched a while back to Filen.io as my cloud storage provider.

While it mostly worked fine, I ran into an issue where it would sometimes fail to acquire the sync lock, meaning it refuses to sync local changes to the cloud. Waiting a while usually fixes it, but it can be super annoying if it happens towards the end of a work session, since when I move to a different PC I won’t have the latest files from the previous PC. Quite annoying.

A bigger issue was when it (for some reason) dropped the version history for a file that had been corrupted (I’m not sure what was corrupting the file, if it was related to Filen or something on my end). Extremely annoying since I didn’t really have any recent back-ups, nor did I have time at that moment to deal with that bullshit.

I ended up setting up a rsync cron job on my machine, backing up all Filen storage to a local folder with incremental history. It seems to be working just fine, but the fact that I had to do that in order to feel more secure about my files stored in the cloud is not great.

Hopefully the next time the “acquiring sync lock” issue hits me I’ll be able to debug further and see what’s going on, but I’m not sure what I could possibly do about file histories going missing on Filen’s side (except notify support, which I didn’t do this time because I was in a hurry to get things fixed - maybe next time).

The enshittification of Black Library

I’m a fan of Warhammer 40k and I’ve read a large chunk of the Horus Heresy series (skipping some novels here and there). I’ve usually preferred to purchase them directly from Black Library’s website, in order to give them all of my money and not have Amazon leech off of each payment (even before my quest to move away from US-based companies and technologies).

That will no longer be an option - Games Workshop is retiring the Black Library website, meaning you’ll only be able to buy books from their mobile app (MyWarhammer or something), with previous purchases being migrated from the old platform to the new.

That in itself makes no sense to me - I don’t know how expensive it is to maintain the existing website, but surely it can’t be expensive enough to justify giving 30% to Google or Apple for sales that could have gone through their website (though I’m guessing most sales didn’t go through their website and instead went through Amazon’s Kindle store), yet GW thought the decision made sense.

Which… okay. So, you can now only buy books from them directly via the mobile stores. If you want to read them on your Kindle or similar device - tough! For reading on an ebook reader Games Workshop direct you to instead purchase their books from stores like Amazon, Kobo etc.

So not only do I need to give money to Amazon (which I don’t want to do), but I can no longer purchase DRM-free books like I used to directly from Black Library.

You know what, FINE! I’ll buy it from Kobo (who are Canadian), strip the DRM and read on my Kindle. But guess what - that specific book isn’t available in Romania, meaning I’d now need to get a VPN in order to be able to read it. Thanks, but no thanks.

Black Library - it used to be pretty neat, but we can’t have that, can we?