I knew about SDF for years, a community coming from the early 90s, from the early BBS era –it started on a Apple IIe microcomputer– where you can have a free “UNIX Shell Account” (it is running NetBSD, so the “UNIX” part is true).

I even have vague memories of my own SDF account at some point, but the thing is that since the late 90s I’ve been using Linux, so having a shell account kind of lost the mystery at some point for me.

As part of my short trips around Gemini, I’ve ended in a couple of git hosting services I didn’t know about, like Sourcehut and tildegit. “tildegit” makes reference to the tildeverse.

“Tildeverse?” you may ask. Well, there’s a website for that:

we’re a loose association of like-minded tilde communities. if you’re interested in learning about *nix (linux, unix, bsd, etc) come check out our member tildes and sign up!

tildes are pubnixes in the spirit of tilde.club, which was created in 2014 by paul ford.

And that’s why I started mentioning SDF, because it looks pretty similar.

It is worth reading this post by Paul Ford where he explains how his tilde.club came about. Although there are a few references here and there about it being “not a social network”, for me, it’s all down to this quote form the post:

Tilde.club is one cheap, unmodified Unix computer on the Internet.

It reminds me very much of my own experiences early 2000s when I was involved in Free Software advocacy, as part of all the Linux User Group movement. A few of those groups, spawning sometimes from public newsgroups, formed large communities of self-hosted services with shared accounts.

My local group had an interesting twist on this, because we also formed a non-profit metropolitan area wireless network.

I guess everybody can have a Linux shell on a Raspberry Pi, but there’s something exciting about having an account on a public UNIX system, isn’t it? It is the community.

So tilde.club may be a social network after all.

PS: my sysadmin senses are constantly tingling when learning about these services. The old Linux User Groups systems relied heavily on trust, and I don’t know how these new communities deal with security, but it must be interesting for sure!

I think I have ready everything I wanted to support in my Gemini server, at least on a first version that is “useful”. I may add more things later, perhaps experimenting with some ideas like server-side support for gemlogs.

This is the feature list:

• Static files, including optional directory listings.
• IPv4 and IPv6.
• Configurable MIME types, or a built-in resolver.
• Virtual hosting, with SNI support.
• User provided certificates or auto-generated in memory (for development).
• Configurable SSL engine (e.g. TLS 1.2 and/or TLS 1.3), with configurable ciphers.

Other than the virtual hosting with SNI support, there’s nothing too exciting.

Although I have already tests for the tricky parts, I still need to write some more to validate a few branches, and to have some regression protection moving forward.

But then I found this Gemini diagnostics repo created by the author of Jetforce Gemini server, as a torture test for Gemini servers.

And turns out, my server was passing all tests but two!

The first issue was related to returning a “not found” response when the request contained an invalid UTF-8 sequence, instead of “bad request”.

This was easy to fix: I was trusting Akka’s decode method, and it replaces invalid sequences with the infamous �. Then I was checking for a resource on disk that obviously didn’t exist (hence the “not found” response).

The solution was to write my own decoder:

  val charsetDecoder = Charset.forName("utf-8").newDecoder()
  def decodeUTF8(value: ByteString): Either[Throwable, String] =
    Try(charsetDecoder.decode(value.toByteBuffer).toString()).toEither

Very simple. I included it in the input validation, and that test now passes.

The other issue is related to the TLS implementation provided by Akka, that doesn’t seem to provide a close_notify TLS signal before closing the connection.

I’m not a TLS expert, so I will investigate to see how important is this, what are the consequences, and if I can fix it or perhaps file a bug report to Akka.

So I’ll see to write some tests and I’m almost ready for a first release –that nobody will use, of course; look a Jetforce, it is very nice!–. I may deploy the service as well, and put some content on the Gemini space!

I got some help on Akka’s Gitter channel, and finally got to the bottom of the problem, and it wasn’t me!

It was all down to the TLSClosing parameter when creating the TLS listener. It uses IgnoreComplete, and that was breaking my code.

The explanation:

All streams in Akka are unidirectional: while in a complex flow graph data may flow in multiple directions these individual flows are independent from each other. The difference between two half-duplex connections in opposite directions and a full-duplex connection is that the underlying transport is shared in the latter and tearing it down will end the data transfer in both directions.

When integrating a full-duplex transport medium that does not support half-closing (which means ending one direction of data transfer without ending the other) into a stream topology, there can be unexpected effects. Feeding a finite Source into this medium will close the connection after all elements have been sent, which means that possible replies may not be received in full. To support this type of usage, the sending and receiving of data on the same side (e.g. on the Client) need to be coordinated such that it is known when all replies have been received. Only then should the transport be shut down.

So that’s why they had to change the behaviour on the TLS wrapper.

The thing is that you don’t need to use queues –most of the time, at least–, because Akka is smart enough to do the right thing if you define your flow correctly.

For example:

    Tcp()
      .bind(conf.address, conf.port) // not using TLS!
      .runForeach { connection =>
        logger.debug(s"new connection ${connection.remoteAddress}")

        val handler = Flow[ByteString]
          .via(
            Framing
              .delimiter(
                ByteString("\r\n"),
                maximumFrameLength = maxReqLen,
                allowTruncation = true
              )
          )
          .map(b => Request(b.utf8String))
          .map { req =>
              handleReq(connection.remoteAddress.getHostName(), req)
          }
          .take(1)
          .flatMapConcat(_.toSource)

        connection.handleWith(handler)
    }

I’m using framing to define what is a request, I map that into a Request, then my handler does its thing and maps that request into a Response.

At that point the element that is being handled by the flow is a response, so I say that I want to send back only one (using take), and I convert that response into a sequence of sources that are processed and converted into a stream of bytes that end in the client.

Akka is smart enough to keep the connection open until the response is sent, and then close automatically (I said I wanted to send back only one response).

And that’s managed with an internal status in the flow that is complete, and it was ignored by the TLS wrapper. So my connection was stuck after I had finished sending the response, no matter how I did it.

I found that using IgnoreCancel is what preserves the behaviour that I need:

IgnoreCancel means to not react to cancellation of the receiving side unless the sending side has already completed.

So basically, if the client has finished sending data, we don’t close until we have sent all our data (the completed signal), but the fact that the completed signal closes the connection is OK because Gemini protocol doesn’t support responses by the client.

This is a bit more complicated than what I’ve shown in the example, because it is possible that the client sends more data than the request. That happens, for example, if we try to access the Gemini server using curl –that speaks HTML and expects sending multi-line requests–. In those cases we have to drain that input (I’m using prefixAndTail, that splits one element from the input and provides a source that I can drain to essentially consume the data ignoring it).

I can’t tell how many different solutions I tried to this problem, and the issue was in the TLS layer. Anyway, looks like the server may work at the end!

I got to a blocker with my TCP server using Akka Streams.

Looking for some examples, I found a few references to this piece in the Streaming TCP section of the docs:

Closing connections is possible by cancelling the incoming connection Flow from your server logic (e.g. by connecting its downstream to a Sink.cancelled and its upstream to a Source.empty). It is also possible to shut down the server’s socket by cancelling the IncomingConnection source connections.

Which sounds complicated, but after a couple of days of looking at the problem, it doesn’t make a lot of sense. Neither did for this fellow programmer 5 years ago (his solution is around the server, and I think the answer should come from terminating cleanly the stream).

And you know it is a tough problem when the only examples of TCP servers you can find online are pretty much the ones you have in the Akka docs –if this wasn’t a learning exercise, I think I would rather use a different technology–.

The problem I think is that when you are processing the request, you are in the flow, and is not possibleto do anything from there to complete the stream. Not sure if there is a reason to force this by design.

Sounds to me like a common pattern: client makes a request, the server sends back a response, and then it closes the connection. With Akka streams you can’t do that from the place where you are processing the request, at least as far as I can tell.

So it goes for ever, unless the client (or the server in case of an exception; for example an idle timeout) closes the connection.

My current code uses a Source.queue because it is supposed to complete the stream when complete() is called on the queue.

    Tcp()
      .bindWithTls(conf.address, conf.port, () => createSSLEngine)
      .runForeach { connection =>
        logger.debug(s"new connection ${connection.remoteAddress}")

        val (queue, source) =
          Source
            .queue[ByteString](1024, OverflowStrategy.backpressure)
            .preMaterialize()

        val sink = Sink.foreach { b: ByteString =>
          handleReq(connection.remoteAddress.getHostName(), b.utf8String)
            .map(_.compact)
            .map(b => queue.offer(b))
            }
            .run()
            .onComplete {
              case Failure(error) => logger.error(error)("Internal error")
              case Success(_) =>
                logger.debug("setting queue to complete")
                queue.complete()
            }
        }

        val handler = Flow.fromSinkAndSource(sink, source)

        connection.flow
          .idleTimeout(conf.idleTimeout)
          .joinMat(handler)(Keep.right)
          .run()
      }

The terminology of Akka is very specific, but basically I connect a flow to the incoming flow, created from a sink and a source. The sink reads the input and adds the response to a queue, that is where the source gets the data to send back to the client.

This seems to work. But the call to complete() is not completing the stream and closing the connection. If it change it for a call to fail, it does work by cancelling the stream.

And this is where I am at. I asked for help on Akka’s Gitter channel, without answer so far, but at least someone solved the issue with queues, so I must be doing something wrong!

Update: it is important that you read the second part.

It is Thursday, and this week most of my gamedev has been basically “admin”. But that’s OK because I’m very happy with my Gemini server –that just today got SNI support implemented–, so no complaints!

What I have today is mostly news regarding Brick Rick: graveyard shift.

First, the cover art by na_th_an (of the legendary Mojon Twins group) it is looking amazing already. I’m looking forward to see it in colour!

And secondly, the big news that the game will be published in the 499 range of tfw8b.com.

I know “The Future Was 8-Bit” for many years now. I’ve been customer –their hardware for our 8-bit machines is great–, and I like them personally. So I’m happy that Brick Rick is going to be available in cassette on a lovely budget range.

Is Graveyard Shift a budget game? I don’t know if that distinction makes sense any more in new 8-bit games (see that I didn’t say homebrew?), but I like the style and for this game I wanted to try something different to the big box releases I’ve had with Poly Play so far.

So that’s all for this week. Not a lot, but at least a good peek of what is coming!

As part of programming a Gemini server, I’m dealing with some classic problems, such as directory traversal attacks:

A directory traversal (or path traversal) attack exploits insufficient security validation or sanitization of user-supplied file names, such that characters representing “traverse to parent directory” are passed through to the operating system’s file system API.

For example, if the content we are serving is in /var/gemini/, our server should only serve content from that directory. In other words, the following request should be illegal:

gemini://server.local/../../etc/passwd

That file should never be served, of course.

There are different ways of preventing this type of attack, the most common is:

• Get the root of our directory, in this example /var/gemini/.
• Add the requested path, and notmalize the result; basically removing relative path components (for example, the ..). In this example we would have:

/var/gemini/../../etc/passwd -> /etc/passwd

• If the resulting path doesn’t begin with the root directory, then the path is invalid.

It is very straightforward. But there’s one thing I don’t like: it leaks directory information out of the root directory.

For example, let’s say this URL is valid:

gemini://server.local/../../var/gemini/

It translates to exactly the root directory, and that’s a valid path. Which I guess is not that important, but besides the fact that the server is leaking the root directory path, that URL can’t be easily normalized and we could have multiple valid URLs for the same directory.

So I thought of an alternative way, that detects that URL as invalid, and all the valid URLs are easy to normalize via a redirect.

The basic idea is to split the path by the / separator. Then calculate for each path component a value based on:

• .. goes back one directory: value -1.
• . is the same directory, so no change: value 0.
• Any other component: value 1 (as we go forward one directory).

If the value for a path component is less than zero, we can say that the path is illegal and we can happily return a bad request response.

Let’s show some Scala code:

def validPath(path: String): Boolean =
    !path
      .split('/')
      .drop(1)
      .foldLeft(List(0)) {
        case (acc, "..") => acc.appended(acc.last - 1)
        case (acc, ".")  => acc.appended(acc.last)
        case (acc, _)    => acc.appended(acc.last + 1)
      }
      .exists(_ < 0)

This code is called with the path already decoded from the URL, and the maximum length has been checked as well.

If the URL is valid, as in we never “went out of the root directory”, we can continue, and check as well if the URL is not normalized and redirect to the normalized version.

Now we can safely normalize and redirect. For example, all the following requests are redirected to the same path (the root document):

gemini://server.local/./
gemini://server.local/directory/../
gemini://server.local/directory/another/../../

I found it all very interesting, despite being an old problem to solve.

What is the Gemini protocol?

Gemini is a new, collaboratively designed internet protocol, which explores the space inbetween gopher and the web, striving to address (perceived) limitations of one while avoiding the (undeniable) pitfalls of the other.

In my opinion, the Wikipedia page is more informative (if a bit technical):

The Gemini protocol is an application layer protocol for distributed hypertext information systems that provides access to simple, primarily textual documents in Gemini space. This is done with contemporary technologies such as TLS, thereby improving privacy and user agency relative to the Web. The protocol is being designed collaboratively and is not currently being standardized as an internet standard.

According to that wiki page, the project is quite new, even for Internet standards, as it was introduced in June 2019.

Last week I was looking for a project to tinker with fs2 streams, so I thought that may be writing a simple Gemini server could be perfect.

That didn’t go well, and I ended frustrated and rage-quiting fs2, but that’s a story for a different post. So I thought, what about Akka Streams? It is a completely different beast, but the docs are better and there is no need to deal with the tagless-final pattern, so I found the API quite pleasant to use.

After a couple of days of coding, I’m getting close to beta quality, and I have learnt a lot in the process.

It was inevitable that I went down the Gemini rabbit hole, and I tried some clients:

• Lagrange: probably the more flashy client, although the SDL rendering makes it feel like a weird desktop app.
• Amfora: a console client. Considering that Gemini is text-only, this one is excellent.
• Ariane: this one for Android. It is probably not as polished as the other two, but is nice to have Gemini in your phone I guess. (Edit 2022-05-17: this client is no more).

What I found in Gemini really surprised me.

It feels like an early Internet, as in a smaller place, with reminiscences of webrings from the 90s. There is less noise and more focused content. I’ve found forums, personal pages, and blogs.

The use of RSS is widespread and looks like is one of the main ways to discover content via aggregators (although there are at least two search engines). Some hosts provide mirrors and proxies to the world wide web, as a simplified version of the same content.

And because the name of the project comes from the second U.S. crewed spaceflight program, it all has its own flavour: pages are capsules, blogs are gemlogs (or glogs), and there is a CAPCOM (from capsule communicator) and a Spacewalk (both aggregators).

I tried one of the clients and, without thinking about it, I ended spending a good time reading random blogs. I mean, glogs.

I plan to continue with my server and, probably, release it at some point when I think it is production ready. And, who knows? I may put some content on the Geminispace.

It looks like it’s Thursday again, and that means gamedev update!

I spent probably too much time to get this blog and the website in order. Mostly getting the light/dark themes to behave, some CSS here and there, and after torturing a bit my Hugo templates, I think it is “perfect™” now.

And there was some time for gamedev as well, of course.

Brick Rick: Graveyard Shift

I finished the music, and posted a WIP of it. The music is not exactly like that any more, that’s the WIP spirit right there.

There was more testing. My good friend Antxiko is doing a great job and found some behaviour that had to be improved in the pause code, and that’s done. We will keep testing a bit more, although at this point I feel like playing the game just for fun because this is “release candidate” quality already.

The cover artist provided a first draft, and I love it. I’m looking forward to have something final so I can show it.

I’ve been told that it wasn’t intentional, but to me the cover art feels a bit less cartoony than the Amstrad CPC cover. The sprites look less chibi on the speccy, and I changed the style slightly, so it could be that or just a happy accident.

I had a chat with the other artist that is going to work on the loading screen, to double check what is needed. At the end it will be better if he has something that is as final as possible as it will be in the inlay (including the lettering of the title).

I’m not really in control of the times, but I suspect we could be 2 or 3 weeks away from the release. Watch this space!

Better Windows support in ubox MSX libs

~mk~, user of MSX Resource Center, has been trying to make the ubox MSX libs compile on Windows.

After some back and forth –I don’t have a Windows system to test the changes–, it looks like he managed to compile the libraries, the tools and the example game. Great success!

I released version 1.1.4, once again without testing the changes myself. At least they don’t break anything on Linux, so perhaps this is the release that solves all the problems on Windows (or it only requires minimal tweaks).

And that’s all for this week in gamedev.

This Saturday (Feb 13th) I will be attending (virtually) to Scala Love in the City, a conference for Scala developers.

As you can see, the schedule is very promissing. It all starts at 8:00AM GMT, which is not bad at all!

It’s been a while since the last time I attended a conference, and I mean pre-COVID times, but I usually watch some of the talks when they’re uploaded to YouTube or similar. This time I’ll try to attend in real-time. Even if it is online, I’m sure there will be a difference –I guess you can ask in the Q&A time, and there will be party–.

Personally, I’m interested in any talk that focus on Scala 3, and of course the keynote by Martin Odersky.

Any novice programmer that decides to make a game, usually has an oversized type of project that would like to tackle, specially as first project.

In my case it was an CRPG (Computer Role-Playing Game), and it is funny to see how things change a lot to essentially remain the same. I’ve seen newbies trying to make their first game, and it was a JRPG (Japanese style RPG). Then an MMORPG (Massively Multiplayer Online RPG). At some point, interest shifted to the FPS (First Person Shooter) genre. The White Whale changes, yet remains the same, unreachable.

Now that I’m more experienced and I’ve finished a few games, perhaps is time to make that CRPG; although not even now I’m sure that I’m capable of finishing such a project!

I’ve spent some time deconstructing the genre, and I read “Fundamentals of Role-Playing Game Design” by Ernest Adams, a bit by accident. I can’t remember when I bought the ebook.

Although the book didn’t tell me anything new, strictly speaking, it was a good overview of the genre, and it kind of encouraged me to give the idea another go.

At the end, I think I’m going to focus on the core gameplay, and then expand from there. Instead of having everything I think a CRPG should have, I’ll see how much of a subset is sufficient to recognise the genre.

I’ve settled in a few points:

• It will be dungeon focused, probably with no over-world or “city” view.
• As part of the dungeon exploration, there will be object/key/switch puzzles.
• I’m keen on a rogue-like component, so I don’t have to design the dungeon –and I can have more fun playing it–.
• There will be inventory, and some sort of management of it.
• I’d like it to be story heavy, or at least with an interesting story, via scripted scenes and NPC dialogue.

The only part that I’m undecided is the combat. I know the player will walk into a new room and there will be enemies, and there will be a fight.

For now I have this list of possibilities I like:

• JRPG turn based combat, party based. Limited actions: attack, magic, use item, defend, etc. See Bard’s Tale, FF or Dungeon Quest series.
• Turn based combat, party based and the environment matters. Same actions as before, but the characters move on a map that can be used strategically. See Ultima or Shinning Force series.
• Action RPG, with little skill involved. The player bumps into the enemies, and stats kick in. See Ys, and lots of rogue-likes use this mechanic.
• Skill based action RPG. Which is not really, really an RPG, but I guess you can make stats count. See 8-bit entries in the Zelda series.

Depending on what I choose, the implementation may vary a lot. For example, inventory management is quite different if you manage one single character or a party.

Another constraint is that it must be small, even if that means being less of an CRPG. Mostly because that would increase substantially my chances of finishing it.

I can start working on the points I know for sure I want in the game, and hope that, as I work on them, the rest will reveal itself naturally.

I have some nice Zelda-esque tiles for the MSX, so I may have decided the platform already!