Foenix index server_protocol

*** Foenix Server Protocol ***


    Foenix Server Common Protocol

    Draft 20231223.aa

    Craig Turner 2023

Project overview,

    Goal: design and implement a common protocol approach that will work for
    multiple games/applications.

    The approach must scale down to MTU of 255, in order to support the 8-bit

    Server side work will be done in python3 asyncio.

    Deploy initially to, later to a Ryzen box I have in London on
    FTTH that I can dedicate to this.

    The protocol is to be practical for client authors using C.

    The protocol is to be practical for client authors using Basic.

    The primary use-case should be for UDP clients, as conventional wisdom is
    that this is a better foundation for games.

    Protocols should be defined to support UDP fail-cases, such as lost
    messages in-flight.

    We need to keep options open for TCP support down the line. I cannot think
    of any accomodations we would need to support. Porting from TCP to UDP is
    problematic, but I cannot think of any hassles going in the other

    Similarly, we need to keep the door open to being able to support
    encrypted channels. It is not clear how current foenix hardware will
    support this. Perhaps through cartridge additions.


    m1. Bootstrapped codebase and release arrangements presenting an echo
    server (Done 20231006, see next section for evaluation services)

    m2. Draft protocol document to Rene, Gadget, and rest of community. (Done

    m3. Basic chat server with TCP support. (Done early oct 2023)

    m4. Chat server now also has a UDP server. (Done 20231111)

    m5. Guess-the-number server with reference client in python (via solent
    framework). The purpose of the work in this milestone is to develop an
    approach to binary UDP protocol design. (Done around 23 December 2023)

    m6. Multiplayer, arcade-based interaction.
        Purpose. Evolve the binary UDP model and create a multiplayer arcade
        interaction game that is fun to play. System should have a lobby, and
        should be able to run multiple games in parallel on the same server.

        Option. Driving game. Reluctance: a. I don't think per-frame would
        give an acceptable experience. b. We should be able to come up with
        something simpler, particularly something that is fun to play on a
        grid rather than an open field. c. Collision detection seems complex
        due to client/server sync concerns.

        Option. Boom UDP. Bomberman-like. All movement is strict grid-based,
        which could be represented on an ascii grid. Players enter a grid
        square, exit a grid square. Players can only move once they are
        settled on a grid square.

        Option. Tails. Snakes with fixed tail length operate inside a plain.
        Works for development purposes, but I don't think this would be fun.
        Snake seems to have limited play.

    First foenix client to games above.


    Endianess. There will be big-endian and little-endian versions of each
    server. We expect the 68k models will connect to BE servers and the 65x
    models will connect to LE servers.

    Form "something_id" names an integer enumeration.

    Form "something_h" names a string enumeration ('handle').

    There will be a standard set of datatypes, "dtype". This document
    describes these.

    There will be a simple DSL for describing protocols. This will be useful
    to client authors.

        Where you see cs, it means client-to-server.

        Where you see sc, it means server-to-client.

    It is up to each application to ensure their message size does not exceed
    MTU. Messages should be kept within 256 bytes, as this is the maximum that
    the 8/16-bit platforms can handle.

        We will need to discuss this further once we are looking at streaming
        music over the network. Perhaps we come up with a binary protocol
        optimised for each platform set.

    The protocol is designed for situations where there is a clear server and
    a client. UDP can support other arrangements, but there are no
    accomodations for those in this design.

    In order for apps to support UDP communication, the onus is on each app to
    design their message schema to support this. Some systems that communicate
    by UDP have the server keep a full history of messages for each client (or
    for a broadcast group) to allow clients to rerequest missed messages. This
    protocol does not place such responsibilities on the server.

    Server will be implemented in python3 asyncio ande deployed to linux. I
    have personal (and strictly non-work) IP available in this, and the
    platform allows me to move quickly and build sophisticated stuff.

    The codebase includes a design for a deployment, for multiple environments
    (uat, prod, etc) and port allocation to specific applications.

Outstanding matters,


        There are deployment complications around making the code available,
        to do with the keys of the ssh client. Those should not sit in github,
        and I would rather move the server code forward than spend time
        polishing deployment in this moment.

        License of server code. My intent is to make this code available
        either to Stefany for her business project, or else as open-source
        code. I have not settled on a model for this.

        License of this document. I have no problems with people building
        derived works from this document, but at this time ask to be credited
        in any derived works. (i.e. standard copyright assumptions, but with
        no charge for use.)

    Checksum algorithm,

        I have not worked out how this works.

    Third party application developers

        It is not yet clear how other people will write game servers that can
        deploy into this model. One option would be to have the code as a
        monorepo, with defined administrators.

    More advanced deployment considerations,

        Want a substrate that allows me to deploy an array of server apps from
        a single command.

        Seek to design deployment so that players are not booted off their
        existing processes at time of a restart, but get to finish their game,
        whilst new clients connect to a new server. May need a dual host
        blue/green arrangement to support this, with a proxy in front of it.


        Should be possible to stream foenix application content to foenix
        users via this server system. Have not put much thought into this yet.

        Once we have the basics of this system in place, we could ask people
        from other retro communities if they would like to implement a client.


        It would be /awesome/ to be able to stream music from the foenix
        server out to clients. I also have some ideas about a role for jingles
        in Foenix-Cup. I have formal musical training (theory, singing) but
        low experience in audio programming. Could use a partner who could
        work with me on this. Basically, I think we will just need a data
        structure that we could stream out. Could start with a melody, and
        then move to polyphony, custom samples.

        Would appreciate someone I could work with one-on-one for this, in
        parallel to the milestones given below.


        Once we are talking about sending authenticaiton details over the
        internet, we will need an encryption story. It is straightforward for
        me to implement on the server-side, but there will need to be some
        story about this on the Foenix side. One option would be to have a
        piece of bridging hardware between Foenix devices and the internet
        that has a build processor that sorts this out. i.e. Foenix device
        would think it has a TCP connection, but it would really be running
        over something like stunnel between the bridging device and the
        server. There is an option here for Stefany to get into the business
        of selling stunnel devices to people in the retro community who want
        to be able to talk to Foenix servers or webservers.

Available demonstrations,

    HTTP server

        Comments you place here will be retained in the linux process.

    SSH server
    ssh -p 2323
    Password is 'admin'

        This is a ssh shell into the same process as the HTTP server. If you
        type ls you can see a "say" and "show" command. See the way the text
        in the HTTP server is the same as the text you can interact with in
        the ssh shell. It is my intent that this ssh shell should become the
        command interface for the foenix server into the future, because it is
        much faster to build administration functionality against a CLI like
        this than having to create web pages for each new action. (Also, it is
        cooler.) I suspect we will do other application funcitonality through
        this system. For example, I can imagine implementing in ASCII a track
        generator for Foenix-Cup, with the tracks them being available on the

        Note that foenix clients will never interact with this ssh server.
        Rather, this is a control interface for server admins who want to
        affect the foenix server.

        If you are on Windows and have not used ssh before, download the
        program Putty, and then use that to connect to the details above. If
        you are on a mac, you will already have ssh configured. Open a
        Terminal, and then type the syntax above.

    Echo server
    nc -vv 2327

        This simply echoes bytes back to you. May be useful for someone who is
        playing with foenix serial-to-network equipment.

    Emphasis: all the links above connect to a single linux process. It is not
    separate software instances. Ssh is into the process, not into the Linux.

Types used in foenix server protocol

    dtype_h     size    desc
    --          --      --
    u8          1       uint8
    arr_u8      *       leading byte gives length.
    mask_u8     1       each bit is a flag
    u16         2       uint16
    arr_u16     *       leading byte (u8) gives length.
    u32         4       uint32
    str         0-256   leading byte gives length ('pascal string')
    arr_str     *       leading uint8 gives array length, then each string
                        within is a pascal string.
    arena       *       untyped mem, can occur as last field in the packet.

If you care about perfection,

    Strings should be encoded as UTF-8, with the length representing the
    length of the UTF-8 string, not the number of bytes.

If you do not care about perfection,

    Do the easy thing of treating the length given by the leading character as
    the number of bytes, and unpacking characters as 7-bit ascii. You will
    probably get away with it without ever suffering ill-effects.

    If you get problems with strings in the future, ask us about this in the
    forums and people who know about UTF-8 will be able to guide you to the
    'perfection' solution above.

Structure of envelope,

    type    name                size
    --      --                  --
    u16     env_mtype_h         2
    u16     env_server_secret   2
    u16     env_client_secret   2

UDP protocol notes

UDP protocols,

    Guess-the-number protocol

    Be robust in handling duplicate message receive.

    Be generous in performing duplicate message send.


        The client opens a three-way handshake with the server. This
        highlights routing problems between sides, and allows an exchange of
        secrets which would frustate subsequent efforts to do spoofing.

        Tha handshake also has both sides agree on the name of the protocol.
        This acts as a defensive measure against situations where a client
        connects to the port of a server that speaks a different protocol.

        Handshake should never handle authentication. That design would make
        it vulnerable to timing attacks that spoof logon. Where auth is
        needed, handle it as a separate exchange of messages that happens
        after connection/greet.

Equipment notes,

    Whenever you order foenix units, ask Stefanie to install a feather with
    the SLIP firmware loaded.

    Separately, Gadget recommended these devices for connecting Foenix to
    network in a way that uses the serial interface, and you should also
    consider getting one of these. Note that you will probably need to do
    firmware updates to that, which may steer you to install crappy software
    on your machine. (I refuse to do this on computers I care about) 

        You may also need a serial cable extender due to practical matters.
        When I got my theoldnet unit it did not plug into the foenix because
        both the computer and the plug have standoff screws that run into one
        another. The solution to this is to get a serial extender cable. Be
        careful not to order a null modem cable when you need a serial
        extender. Be careful to get a unit with one male and one female

    If you can get a null modem cable (plus gender-changer, probably) you
    could hook your foenix directly to a linux box and supply connectivity via
    SLIP from there.