If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.
As Drew mentions on his post, you can find a lot of system administrators struggling with this, just because they are sharing publicly source code. As it sounds: because they have decided to distribute open source.
I have been self-hosting my repositories since 2023 and although it is not really a forge, I managed to make it small scale and work for me by providing a web interface on git.usebox.net. It has “about pages” –rendering the README see for example the SpaceBeans page–, and together with email and RSS feeds to track releases, it just works.
I noticed some issues last year in my server, but I attributed it to a mistake in the setup of cgit, honestly thinking that it was less performant than I was expecting and just configured a cache. It is supported by the tool, but being optional I thought I probably don’t need that.
Setting the cache seemed to fix the issue, and I didn’t investigate further. Until I few weeks ago that I was reading on Mastodon how someone was having a hard time dealing with these bots, and I realised that it was probably happening to me but I wasn’t paying attention. And that was the case!
Essentially it is what Drew is describing –if much smaller in my case–, and for me it seems to be always coming from the same IPs owned by Alibaba Cloud –that seem to have their own “Generative AI” product–. The day I checked the logs, I had over 200,000 requests in 24 hours coming from only two IPs.
My first impulse was to check with whois who was the owner of the IPs and, because they are owned by a cloud company, block on the firewall the whole range. And I kept monitoring the situation for a couple of days.
Of course they kept coming, so I kept blocking. At some point there was a few /16, /15, and even some /14 ranges in my block list. That was already 681,504 IPs, all owned by the same cloud company.
Because I have better things to do –really–, I wrote a small script that will ban IPs if they make what I consider “an abusive number of requests in 24 hours”, and keep the ban until they stop the abuse for 2 complete days. I don’t think this should affect legit users, but if you experience any issues, please contact me to justify why you need that volume of requests!
I did this on principle, because my forge is very small and I can handle the load. It wasn’t strictly necessary for me to block these bad actors, but I know people that couldn’t spend time with the problem and had to make all those open source repositories private; which is in my opinion the tragedy of all this: we are sharing code with the rest of the world, and the abuse of these companies trying to make profit on it is ruining it all for everybody.
Spring is coming and we have more hours of sun, or light at least, so I’ve been suffering headaches and pain around my eyes –likely to be eye strain–. My solution for this has always been closing the curtain so I can keep doing my work –that requires staring to a computer screen–. Is not living in darkness, but I was avoiding bright light into my eyes.
I’ve been using a dark theme in my terminal and my editor, that is what I use most when I’m programming, since forever. Can’t put a date, but over 20 years ago. I always found the black background more comfortable, and my screens have already very low brightness.
And over the years the “dark themes” have become more popular, and now virtually everything from your computer to your phone can be done in a dark theme; some websites (like this blog!) even support both light and dark themes depending on the settings of the user –the browser chooses the right CSS–.
There is also an aesthetic consideration: it just looks better to me.
But I know as well that there is some scientific proof that dark themes aren’t really the best for people with astigmatism –and to some extent myopia–. Essentially because a dark mode requires your pupils to dilate, which can make it harder to focus on the screen, resulting as well on the foreground content bleeding into the dark background and making it hard to read –specially with small fonts–. And that can lead to eye strain.
For some time now I have my phone configured to use both dark and light themes, depending on the time of the day: during the day hours it will use a light theme, and during the night hours a dark theme. And turns out I can see the screen better during the day!
So I have been running an experiment with the work laptop –that I use during the day–, and changed everything from a dark theme to a light one. Because I’m using the excellent gruvbox-material, it was very easy to switch my editor.
Gruvbox Material (light version)
After a couple of days in which I didn’t close the curtains, I think I can feel the benefits: no eye pain or headaches! So I have decided to transition completely, including in my personal machine, with the caveat that it requires me now to ensure that I always have good lighting when I’m using the computer –specially at night–, so the bright light coming out of the screen is not causing other problems.
Perhaps Gurvbox is not the most popular theme, but is popular enough that you can find settings for most popular applications. In my case there is already a theme built-in in WezTerm, and it was easy to find a theme for tmux.
Then I had to set my desktop to use a light theme, and everything else changed, with the exception of my i3 theme, that for now I’m keeping with the Gruvbox dark theme because looks great and is a very small portion of the screen that won’t affect my eyes.
I’ve never been an advocate for dark themes –use whatever works for you and leave me alone!–, and this post is not me recommending a light theme. It works for me and I wanted to share the experience.
Many years ago I used to have a home server. It was connected to elxwifi, a metropolitan area network built on WiFi, and also to the Internet.
It was hosting my blog and a few more things, so it kind of made sense to provide some services to the local network. Like a good firewall with QoS –back then residential connections didn’t have much upload bandwidth–, HTTP proxy for caching, and a DNS resolver.
Re-captcha takes a pixel by pixel fingerprint of your browser, a realtime map of everything you do on the internet.
And Cloudflare Protection achieves a similar goal: when you are forced to “prove that you are human” is just because they don’t have enough tracking information about you, so… you could be a bot. Because that is what differentiates humans from robots these days?
And what about my Internet provider’s DNS resolver? Well, my provider –like many others– implements a DNS hijacking service, so if you try to resolve in your browser a domain that doesn’t exist, they redirect you to a landing page they own. This can be disabled, but we are back to trust –why is this opt-out?–.
I don’t have a 7x24 server at home, so today I’m not going to implement this for my whole local network, but I fancied the experiment with my machine.
Please take into account that this might not be a good idea for you. My PC never leaves my desk and it always uses my home connection, so the use case is not the same as if I was using a laptop on a coffee shop’s free WiFi. I would say using Mozilla’s DoH may be your best option!
Unbound is a validating, recursive, caching DNS resolver. It is designed to be fast and lean and incorporates modern features based on open standards.
My OS is Debian 12, so I just run:
sudo apt install unbound
The configuration is in /etc/unbound and there is a full commented example in /usr/share/doc/unbound/examples/unbound.conf.
I recommend reading the base configuration, but essentially Debian enables remote control in localhost that is handy to check stats and manage the service using the unbound-control tool as root.
I added a local.conf file in /etc/unbound/unbound.conf.d:
I can’t remember if I had to do anything else, but it is managed by systemd, so you can run the usual commands, starting with systemctl status unbound.
Then I had to make two changes to use the new resolver:
In Network Manager, I edited my wired connection –yes, I don’t use WiFi in this machine–, setting method “Automatic (DHCP) addresses only” and in DNS servers “127.0.0.1”. Then restart the connection to apply the changes. When all is done, your /etc/resolv.conf should be like this:
# Generated by NetworkManager
nameserver 127.0.0.1
In Firefox, open settings and search for “DNS”. In “Enable DNS over HTTPS using”, select “Off, use your default DNS resolver”.
And that should be all.
I don’t have any scientific proof, but browsing feels snappier, and I guess it makes sense because for cached name resolutions there is no need to go to Cloudflare at all!
After a bit of browsing you can run unbound-control stats_noreset (the regular stats clears them), and get something like:
There is no need to be an expert to more or less understand what these mean.
It was all very easy, and it took much more time writing this post than setting it up. This could be a good service to offer the local network, so perhaps I have found finally a good use for one of my Raspberry Pis!
Yesterday I published a minigame I made in JavaScript following the ideas of my 7yo son. He drew most of the graphics –I gave him a hand with the slimes’ sprites–, and I had never made a twin stick shooter type of game, so that was interesting. I haven’t played any games on the genre either, and I think I understand now why they can be very fun!
It is still more like a tech demo, but playable –and kind of fun for 10 minutes–, and it helped me to make some improvements on the “engine”. The development experience was OK, but the bits I don’t like about JavaScript are still there, including the hit and miss performance of canvas 2D on Linux.
I don’t see myself writing larger games in JavaScript, but for small prototypes or jam games –if I was still doing jams!–, it could be a good match.
This is a bit of a rant. You can skip it or, depending on you experience, you may come for the ride nodding along.
I have spent a couple of evenings improving my feed reading experience, and I’m not sure why it is so difficult in 2025 to read blogs like we did in the early 2000s. Was it bad back then? It is possible it was!
Let’s set the starting point:
I don’t want to use a cloud service to read my blogs. I only read blog in my main PC, never in my phone –I used to, but I realised I wasn’t really reading them–.
Although is not necessarily linked to the previous point, I don’t want to use a website either. Essentially because I don’t want to self-host a complex application I would have to maintain to just read blogs on my PC. I know I could just run some containers and whatnot on my PC, but that’s over-engineering it.
If I’m using a native application, I want it to be native. We used to have those applications, now we have Electron and Flutter, and probably others I don’t want to know about. Let websites be websites, and native applications, well… something else.
I thought it would be easy! There aren’t that many options currently open source, working on Linux, and still maintained: do you want to download and process content from the Internet with an application that has been untouched for years? Sounds fun!
I started using Liferea over 20 years ago, but a few months ago something happened –can’t remember what, it could have been me!–, and I decided to look around to see what was out there.
And I found NewsFlash. Although not exactly: what I found because the decadence of the search engines –topic for another day– is a lot of click-bait sites saying how amazing NewsFlash is. And it really looks good, I agree!
The easiest way of using the latest version is via its official flatpak, which I’m not the biggest fan of but you can’t have it all. And slowly but surely you realise that all those sites raving about NewsFlash haven’t used it for more than 10 minutes.
Don’t get me wrong. NewsFlash it is Open Source and it keeps improving with each release. I have reported bugs, the response times of the main developer are fantastic, and I believe it has a bright future ahead. It is just that currently it isn’t a good match for my needs, and I don’t have the time –or the skills, to be honest– to contribute and make it work for me.
Hopefully I’m not unfair or not too picky, but I hit bugs importing an OPML file –that I resolved editing the sqlite database by hand, fun!–, exporting to OPML doesn’t seem to overwrite an existing file correctly and can result on a corrupt file –seems to be fixed already!–, it doesn’t seem to be rendering the fonts I choose –probably Flatpak’s fault–, and other small paper cuts I can’t remember right now.
Oh, and the last straw was when it crashed my whole system. Can’t tell 100% what happened because the system didn’t respond and I had to power-cycle, but my I3 status was reporting 800MB free from the 32GB of RAM that I have on my PC. Not bad for an application written in rust!
At the end, I’m back with Liferea. I edited ~/.config/liferea/liferea.css to:
And with GTK on a dark theme, looks decent enough. I’m back to a happy place, and the tools I use work and I’m not frustrated any more.
Then perhaps we should take a look to the current state of the feeds out there:
RSS is still popular, although it has many limitations. And I know it because: I use RSS! At very least it needs some extensions from Atom, but it is common that the feed uses the post creation date in pubDate instead of when it was modified last time resulting on updates to the post never showing in the feed reader. I am conflicted on this one: is it the RSS’ fault or the reader’s limitation? I don’t know, but it is hard to detect this unfortunately. We should be using Atom instead because it makes distinction between published and updated.
It is not common, but some people –including me– like tinkering, and we end with bad implementations of RSS or Atom –e.g. the images won’t load on the feed readers–. You can contact the author, but let’s be real: people are busy and I didn’t get results, so no images on that blog.
Some feeds don’t include the full post. I don’t know why, but back in the day this was because the blog post had ads and those don’t show in the feed. NewsFeed deals with it beautifully and turns out Liferea can do it as well –although is not 100% consistent applying its theme, but it is close enough–. This “retrieve the post from the web” works also with the two previous problems, but then feels like we are failing a bit a syndicating content.
Finally, it is kind of difficult to find new interesting blogs.
Back in the early 2000s we had blogrolls and comments, and both were an excellent way to build your own community of blogs and find new content organically.
Unfortunately all that was ruined because Google’s dominance in the search space and ads, and their pagerank that lead to SEO and spam, and links had value beyond their original intent of linking content. Having comments on your blog wasn’t useful anymore, because it was mostly spam, and most of us removed the functionality.
Similarly people started to drop the blogrolls, because all that perceived value on the outgoing links. Which I don’t think it matters anymore because Google buried blogs deep in their search results, so you may as well have links. If I stopped having a blogroll is because the blogs I was reading disappeared, mostly. And I lost the community part around blogging.
Since about a year that my blogroll is back, and we’ll see about the community. Go and check it, you may find blogs that you like.
If you got to this point, I hope it wasn’t too bad. Despite all these arguably small things, I love it and I’m slowly building a list of blogs that I enjoy reading, and I’m even writing about it!
I’ve been thinking about this after Alex asked himself and blogged about the “Blog Question Challenge”:
Why did you start blogging in the first place?
What platform are you using to manage your blog, and why do you use it?
Have you blogged on other platforms before?
What’s your favourite post on your blog?
Any future plans for the blog?
I found interesting Alex’s answers because my first blog, that I closed in 2021 after 18 years, also didn’t start as a blog.
Back in 2003 I had a home server –my old Pentium 100Mhz initially– and I was part of a project to build a metropolitan area network using a wireless –I still have a presentation in PDF about the project; in Spanish–. My server was connected to a nearby elxwifi node (that’s how we called our network), and I offered some services. That included a website in which I shared news about my server and the project, in reverse chronological order –newer entries first–. I had no idea that it was a blog, but that was what started it all for me.
At some point the web was also available via the Internet, which meant a bit more attention than the handful of visits I would get via the wireless network, and because I was learning a lot of new things about open source and Linux, I started to write about that in my page. Then you read other blogs, start commenting on those, and the authors of those blogs comment on yours, and you build a small community. A social network if you will. We called it blogosphere.
I met a lot of interesting people thanks to my blog. Some of them even in person –there were meet ups called Beers and Blogs, for example–, with friendships that last more than 20 years now.
Because my initial approach was homebrew, I ended writing my own blogging software in PHP with MySQL as database to store the posts. I rewrote that a few years later in Python (with Redis to store the posts; during the NoSQL hype), because I was learning that language and that’s what you used to do back then. Today I don’t need any dynamic parts, so I’m using Hugo to manage this blog as a static site.
I don’t think I have a favourite post, but the last post in my old blog was emotive to write. Not every day you decide to end something that has been with you for so long.
I’m not sure what are my plans for this blog. I started trying to replace other social media –Twitter back then–, so I used it to write updates about game development and my projects, and I still do that; but in my experience, thematic blogs are very hard to keep in focus and they turn a bit personal blogs at the end. So I may write about anything I find interesting and I feel like sharing.
I guess I would like to go back to that version of the Internet, that includes writing a blog and participating in memes like this one! So I’m going to tag Oscar, that was one of those bloggers that started blogging by the time I started, and I’m interested in his answers to this challenge.
Well, I did it last year, sort of. Is not that I don’t like writing this type of post, because it is good to reflect on what happened in the year that ends, but I always have that feeling that I’m missing a lot of things. Anyway, let’s do some highlights at least.
Let’s start with gamedev: two games this year! Not bad for feeling a bit uninspired. Part of the success is perhaps because my 7 years old son want us –which is me– to make more games.
Alien Intruder (DOS 16-bit), that feels a bit too close to the end of the year, but to be fair it was almost finished for a while but I run out of steam in the last 10%.
Why I have been uninspired? I don’t know for sure. We have passed the 10 years mark of me releasing consistently at least one game every year (sometimes more!). I suspect I’m getting tired because it is a lot of effort. Not that much the actual work of programming, drawing the graphics, and writing the music; but the focus required to actually finish a more or less polished game, and release it –two different things, by the way–.
If found myself wondering if it was worth it, pushing forward for months to reach my own high standards, when it is a free game after all. Even if I do things for myself –and my sons, of course–, I believe it is human wanting to know that what you do kind of matters. There so much stuff out there, and could I be using my free time for other things? Absolutely.
I can hear you say: take a break! And you may be right, but I don’t want to go back to “not finishing things”. I suspect there is a risk of that if I stop.
As I mentioned recently, I would like to write more Haskell in 2025, and that would mean a bit less time for retro-gamedev. So that could be it: not that much a break as to do different things.
And that goes in hand with my old mini PC from 2015 (an Intel Core i3-5010U), dying on me. So I got a new one, this time a bit more powerful. Still not what you would consider a gaming rig, but is clearly not a potato like the old one, so… I can play modern-ish games now!
I suppose we can add that to the list of things I could be doing instead of spending all my nights working on games, isn’t it? Anyway, not a big factor, but I have enjoyed finishing Pyre –after I completed Transistor, I’m a big fan of Supergiant Games–. It is a beautiful game, with a great story, music and sound design. Very inspiring, even if in a way that I’m sure I can’t make games like that.
Because Lutris is awesome, we have been trying other games, but our big hit is Torchlight 2 “family co-op” in the local network. It is almost not age appropriate for the boys, but the kind of isometric perspective graphics with the fantasy theme makes it alright. We have completed it once, and we are having another go with different character classes. We are still playing some Minecraft, of course, but I prefer Torchlight 2!
The third big activity that has eaten a lot of my free time has been reading books. I’m still finishing about two books a month, although this year I’ve been reading authors that write very long books. Because I had a “not very good” book I wanted to read because it was the end of a series, and a “slow starter”; I didn’t manage to read 24 books this year. But 23 is still pretty good!
I have realised that finding what to read next is a big problem when reading two books a month. So when I discover a new author that I like, I tend to try other books by them, hoping that it would be as good. I did that last year, and it did work well, even if that meant reading a lot of Brandon Mull. This year I’ve been disappointed a few times, because it wasn’t as good –but it wasn’t bad either, so it is still a win–.
My highlights are the “Mistborn” series by Brandon Sanderson; probably the best fantasy I have read in many years to the point that I’m considering re-reading it in 2025. Another highlight, also by Sanderson, was “The Sunlit Man”. Only one book with almost no padding! Very recommended.
And that’s all. Would you believe that I’ve only been making games, playing some games, and reading books? Exactly, that is the feeling I was talking about!
So I finally released Alien Intruder yesterday, on Christmas day. Perhaps it should have happened a few weeks ago, but I was tired and couldn’t focus on writing the music. But that’s OK, it has been like a Christmas present –although I don’t think is the best date to release a game!–.
The blobs have a lot of personality
Although I sent a couple of emails to the usual people –what I call the press release–, the main “announcement” has been this post in Mastodon, and a few people said they liked the game, so it is being played. But you know, it is very unlikely that the news have reached to everybody that could be interested in an new DOS game in 2024. Any help spreading the word would be appreciated!
Checking some of the websites that cover DOS games beyond being a site to download “free” games, and at least check new titles, I was visiting DOS haven, and I liked they answer to why would someone make game for DOS nowadays?, which is essentially three reasons:
Because it’s fun, if you are into that sort of thing. Indeed.
Because it’s retro, and retro is cool! Perhaps I would say nostalgia plays a role here. Although I didn’t look at the DOS era like that, some people do.
Because thanks to DOSBox, you can run DOS programs almost anywhere: Windows, Linux, MacOS, smartphones, tablets, Raspberry Pi, you name it. Yes, this is a nice plus: games are as portable as DOSBox.
I’m not encouraging people to make DOS games as a for-profit adventure –there are plenty of limitations–, but playing the game on my RG35xx H is very nice and I didn’t have to port the game.
Anyway, I wanted to make a 16-bit DOS game, in the spirit of the shareware boom of early 90s. Although it isn’t shareware –and you don’t need to register the game!–, I was inspired by the feel of those titles –and I read Shareware heroes this year as well, an enjoyable book–. I know I have already released other DOS games, targeting both the age before and after, but I guess it doesn’t matter because it is 2024 anyway.
Making a game for the 286 with VGA graphics wasn’t perhaps the best idea, and I mentioned already how dificult was to write my own sound driver, but I enjoyed the process (it is fun, remember?). I may or may not make more DOS games, but after this experience I think that would be 32-bit –meaning 386 or better–. It is probably what makes more sense now that I have been there, done that.
As always, there’s a free download in the game’s website. I was approached by a publisher asking if I would like to make a big box release of the game. I’m not sure about this, but never say never. For now, just go an blast some cute aliens!
I streamed some sessions of game development with Haskell and I got to make a platform game with some reasonable functional code. I didn’t release a game because there wasn’t a game there, really; but despite that, I think I implemented enough pieces to be confident that I can finish a game in Haskell.
However, time flies when you are having fun, and my free time this year has gone mostly to The Heart of Salamanderland –released in May–, and my current project for DOS 16-bit “Alien Intruder” that should be ready in the next two weeks, if I manage to finish writing the music –seriously, it is hard to write decent enough tunes–.
And I haven’t written any more Haskell, despite revisiting the codebase a couple of times and missing writing Haskell. Why I miss it? I don’t know for sure, but besides being fun, I like how it makes me feel happy about writing code.
I don’t have much choice of languages when I’m making games for classic systems. If is a Z80 based machine, it is going to be assembler and C. If it is for DOS, it will be assembler and C. See the pattern?
For my tools I don’t think I’m going to stop using Python, because I am very productive with it. Being my tools, I don’t need to worry about packaging, distribution, or performance. So I started writing Python in 2010 and it still makes me happy in 2024, even if I’m not interested in using it everywhere like perhaps 10 years ago.
Then I write a lot of Scala professionally every day. Sometimes I even write it for fun, and I love it. But then, it runs on the JVM and I wanted something different, perhaps also functional like Scala, for my personal projects. Likely a compiled language that generates native binaries.
I put a significant amount of time in Go two years ago, and even dabbled a bit into Ebiten. I liked that after a while the language just disappeared, and what was left was the problem I was trying to solve. But I also found out that the language was very uninspiring, and some of Google’s practices put me off. So I moved on.
I refreshed my Javascript game a bit –but that wasn’t really what I was looking for–, I wrote some C with SDL2, and it was fun, but also too similar to what I’m doing when I’m making games for classic systems.
I even tried a Scheme, again, very inspired and motivated to like it. And it didn’t happen. Again. I’m just going to accept the fact that lisps aren’t for me and live a normal life.
But with Haskell was different. Writing the code made me happy. Evidently, it is a shame I didn’t have gameplay to support the code I wrote, but that is only my fault. “Alien Intruder” could be a PC game, and that would have been a nice project to write in Haskell. I know, like making a game is not challenge enough!
So that is one thing for next year: write more Haskell –hopefully with a PC game release!–.
More than a month since the last update? What happened!?
Well, “Alien Intruder” is progressing nicely: I have 25 stages ready –including playtesting–, and everything else is more or less in place.
The current TODO list is reasonable:
At least another tileset (two ideally), so we can get to 50 stages with some variability to the eye.
Likely another enemy, to make the stages 30 to 40 more interesting.
High score table, because this is an arcade game and chasing score is part of it.
The music, at least: the menu tune, in game, game over, and stage clear. I could also change the tune every 10 stages, but it depends on inspiration writing!
Some more testing!
So is not that much, is it? The engine has been finished for the last month and I have fixed a handful of bugs that necessarily would appear when designing and playtesting the stages, although the “problem” are usually those bugs that don’t present themselves as part of this process, and that’s why we need testing.
I still need to learn to use the Reality AdLib Tracker v2, although I went down a rabbit hole with the music player and I ended implementing a DRO player, so In theory I can use anything that generates AdLib audio and can be captured by DosBox.
There was also a lot of polishing, from improving the asset management –that looks like I almost reimplemented the WAD format–, better memory allocation and far pointer support –ah, isn’t DOS 16-bit fun?–, and small things like saving the joystick calibration to disk.
Anyway, I keep posting updates in my mastodon account –that, by the way, I have moved from SDF; but that is probably a story for a different day–.