10. Internal network

Fig. 10.1 Graph of the logical (internal) Ethernet of REDAC. See main text for explanation.

REDAC features an internal IPv4 network somewhat similar to a typical home subscriber network (Fig. 10.1). This network has the following physical components:

Mikrotik Switching Router

Off-the-shelf rack mount router including a 24port front facing ethernet switch. The first port of the switch is dedicated to the external uplink while all other ports are bridged to an internal network. The device is introduced in section The Mikrotik Router.

Super Controller (x86 Server)

An off-the-shelf low power low noise low depth server with IPMI out of band remote managament facilities. The device can also be used as a terminal server interactively. For further details see section The Super Controller Server.

Hybrid Controllers (Teensy Microcontrollers)

The network contains dozens of microcontroller units (MCU), each of them connected with 100Base-TX to the switch and centrally managed by the super controller. These devices provide/require no direct network access by system operators.

This section will present the particular components in detail and explain relevant configuration steps.

Note

At the time of reading this, you should have the Confidential Credentials given at system handover next to the Public Information on your REDAC installation ready at your hand. Note that the authentification systems of REDAC internal network structure are scoped and not unified/synchronized. That means that each device has a local user and password configuration and no single login exists. Do not confuse the scope of the credentials. credentials listed with the REDAC public application level authentification, which are managed by OpenID and discussed in the section Authentification.

10.1. Initial network setup/access

By default, the router behaves as a IPv4 DHCP client at the external interface. If you have a DHCP server running in the outer network at your control, this default configuration might be suitable for you. In this case, you can access the super controller server by means of the TCP ports exposed (by NAT in the router). You can also take the server as an intermediate to configure the router. Note that the router configuration interface (web, ssh) is only accessible from the internal network.

Tip

By default, the REDAC Router behaves similiar to a IPv4 home subscriber setup (think of a Fritzbox in Germany): There is a restrictive firewall allowing no ingoing traffic, doing masquerading and exposing thus only a single IP address at the outer network. System operators are encouraged to enter the internal network with their equipment, in particular for setup and maintenance.

If you cannot make use of REDAC acting as a DHCP client but need to have it a given and fixed address, you need to get access to the internal network, which requires physical access. There are two principal ways how to do that:

• Connect your notebook/computer to the internal network or

• Connect a monitor and keyboard/mouse to the super controller and make use of it as a standalone computer.

Note

The neccessary ports (USB and VGA) for connecting a terminal to the server are located at the front of the rack. If there are no free USB slots, you can plug out some USB devices while not using the Analog part of REDAC. Make sure to connect all USB devices back after having finished the administrative work, or use a USB hub if neccessary.

For connecting your own computer to the internal REDAC network, there exist two options:

• Either you connect the Wifi hotspot opened by the router. Keep the SSID and WPS2 password credentials at hand in order to connect.

• Or you connect physically with a network cable to one of the free ports on the switch.

In both ways, your computer should be assigned an address by the REDAC internal DHCP server. If you cannot run a DHCP client on your computer, you may choose a fixed address freely in the range 192.168.104.5 - 192.168.104.9 with subnet mask 255.255.255.0 and gateway 192.168.104.1. The gateway acts also as DNS server.

If you, instead, choose to make use of the super controller as an interactive computer, you must have the username and password credentials at hand to login at the desktop greeter. After successful login, the linux computer will launch a desktop environment which allows you to start the usual applications such as a web browser or a terminal. If you do not have a mouse available, hit CTRL + ALT + F2 or F3 in order to switch the virtual desktop to a TTY instance which allows you to use the linux shell without a graphical user interface.

Warning

Keep in mind that the direct access to the internal REDAC network is only intended for system administrators and not regular users. The reason is that ordinary users should not communicate directly with the microcontrollers. Otherwise, the Super Controller is bypassed and is prevented from doing his work correctly, messing up proper system managament.

The same is true for the Super Controller graphical interface! If you want to provide your users a console like desktop access in the same room, for instance, please setup your own computers which are put outside of REDAC.

10.2. The Mikrotik Router

The Mikrotik router runs the proprietary operating system RouterOS by Mikrotik. It is rich in features and the supplier provides a detailed documentation available at https://help.mikrotik.com/docs/. If you are not familiar with this technology, please read the section Getting started in the linked Mikrotik documentation.

The REDAC ships with the mikrotik model type CRS326-24G-2S+RM. It features a managed 24 port gigabit switch next to the routing capabilities.

10.2.1. Relevant settings

The router comes with REDAC-specific default settings (given below) which only use a tiny fraction of all functions. Virtually any professional grade router available on the market, such as Cisco, Linksys, Juniper and other can do the same. Note that the MCUs only have a fast ethernet uplink (100mbit) and thus modern multi-gigabit powerhorses are not neccessary. The Mikrotik is therefore a sufficient and energy saving choice.

The following router configuration options must not be changed by the systems operator:

Internal ethernet configuration

All ports are bridged except port 1, which belongs to the external port. The wifi (activated by default) is also bridged to the internal network.

Internal network DHCP server and IP configuration

There is a single subnet 192.168.104.0/24 with DHCP enabled. The router has the internal IP 192.168.104.1/32. There is no IPv6 active in the REDAC internal network.

TCP port forwarding (NAT) for the super controller

The super controller has a fixed IP address by the mikrotik DHCP server. TCP ports 22, 80, 443 are dst-nat to the super controller.

In contrast, the system operator is invited to change not only, but in particular the following default options:

• Changing the behaviour at the external interface, called ether1, in particular changing the IPv4 configuration or enabling IPv6.

• Changing or disabling the Wifi AP

• Changing (sharpening or softing) the default firewall

• Changing the access credentials

• Making use of advanced functions such as VPN services.

10.2.2. How to access the REDAC Ethernet/IP Routing Switch

When being connected to REDACs internal network, access the mikrotik web browser based graphical user interface by pointing your browser to http://192.168.104.1. Your browser should show you a login mask where you have to fill out the given credentials. At Fig. 10.2 you can see a typical screenshot of the web browser based graphical user interface of the router after having logged in successfully .The page shown allows to turn of the DHCP client behaviour of the REDAC router.

Fig. 10.2 Mikrotik RouterOS Web Browser GUI screenshot

Alternatively, the router provides an SSH console which allows you to display and edit the router settings without a graphical user interface. This comes in handy in particular when you access the system with the linux console (VGA/keyboard directly connected to super controller) without having a USB mouse at hand.

some@computer $ ssh admin@192.168.104.1
...
[admin@REDAC1-EthRouter1-MikroTik] > /ip/address/print
Flags: D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
#   ADDRESS             NETWORK        INTERFACE
0   192.168.104.1/24    192.168.104.0  bridge
1   10.110.0.26/32      10.110.0.0     wg-tinybridge
2 D 192.168.100.168/24  192.168.100.0  ether1

10.2.3. Mikrotik default configuration

The device default configuration is shown in the following listing. It can be obtained by typing /export compact into the console. Save this listing to a text file and you can restore the device configuration at any time:

# 2025-01-08 20:00:42 by RouterOS 7.13.5
# software id = 1SRY-CS15
#
# model = CRS326-24G-2S+
# serial number = HGC09YWZXTY
/interface bridge
add admin-mac=D4:01:C3:80:2D:EB auto-mac=no comment=defconf name=bridge
/interface list
add name=WAN
add name=LAN
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=dhcp ranges=192.168.104.10-192.168.104.250
/ip dhcp-server
add address-pool=dhcp interface=bridge name=dhcp1
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge comment=defconf disabled=yes interface=ether1
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=ether11
add bridge=bridge comment=defconf interface=ether12
add bridge=bridge comment=defconf interface=ether13
add bridge=bridge comment=defconf interface=ether14
add bridge=bridge comment=defconf interface=ether15
add bridge=bridge comment=defconf interface=ether16
add bridge=bridge comment=defconf interface=ether17
add bridge=bridge comment=defconf interface=ether18
add bridge=bridge comment=defconf interface=ether19
add bridge=bridge comment=defconf interface=ether20
add bridge=bridge comment=defconf interface=ether21
add bridge=bridge comment=defconf interface=ether22
add bridge=bridge comment=defconf interface=ether23
add bridge=bridge comment=defconf interface=ether24
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
/interface list member
add interface=ether1 list=WAN
add interface=bridge list=LAN
/ip address
add address=192.168.104.1/24 interface=bridge network=192.168.104.0
/ip dhcp-client
add interface=ether1
/ip dhcp-server network
add address=192.168.104.0/24 dns-server=192.168.104.1,8.8.8.8 gateway=192.168.104.1 netmask=24
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
/system clock
set time-zone-name=Europe/Berlin
/system identity
set name=REDAC1-EthRouter1-MikroTik
/system note
set show-at-login=no
/system routerboard settings
set boot-os=router-os enter-setup-on=delete-key

Note that this listing does not include any default passwords, i.e. it will not change the device users or wifi access.

10.3. The Super Controller Server

The Super Controller is the name both for a certain kind of software service and the physical server built into REDAC. This section will only cover the physical server and its operating system. For applications running on this server, see for instance Services, Authentification and in general the overview of REDAC-specific software in the developers manual.

10.3.1. Server Hardware

REDAC ships with a 1U low noise Mini-ITX server based on the SuperMicro Xeon Broadwell SoC server-grade mainboard X10SDV-4C and equipped with 32GB RAM and 1 TB SSD. The low depth enclosure has a single power supply and front facing I/O, making it easy to connect a KVM (monitor/keyboard) or USB stick. Most imporant is the IPMI feature which ensures out of band / lights of managament of the server.

This server was chosen due to its compact form factor and not for high availability. In principle virtually any other rack mountable server is suitable to fulfill the job, in particular any product made by large server companies such as Dell, HP or Supermicro.

10.3.2. Server Operating System

The Super Controller server is shipping with the Ubuntu Server GNU/Linux distribution and is primarily supposed to be managed via SSH. Access is possible straight from outside the REDAC network as the SSH port is exposed by the router. It is up to the system administrator to also expose the IPMI or to lock down the ports if no remote managament is needed or realized in other fashions.

In general, administrators are expected to be able to manage a Linux server and it is outside the scope of this manual to discuss how to obtain graphical access from remote or how to upgrade the operating system. The same applies with basic monitoring and health checks at an operating system level. The system comes in standard configuration for Ubuntu Server, which means for instance that security updates for system packages are installed automatically if internet access is available.

10.3.3. Login and usage of the server

There are a number of differnet ways to access the server: Either you connect physically (see also Initial network setup/access) and use the standard VGA terminal (getty login) or graphical login (X11 login greeter). Or you connect via network, primarily using SSH. We have the lightweight Xfce desktop environment preinstalled which includes common tools to get the everyday job done in the graphical user interface.

Via SSH, you have a number of options: Either you use plain SSH, which is just fine for any advanced Linux user and in particular operator. Or you combine SSH with graphical forwarding (i.e. ssh -x). The most advanced way is to use the free terminal server/remote desktop software X2go which is preinstalled. Again, Xfce should be chosen as the installed GUI. A typical configuration screen is shown in figure Fig. 10.3, whereas figure Fig. 10.4 is shown a typical running session.

Fig. 10.3 X2Go session initialization screenshot, showing the configuration options.

Fig. 10.4 Graphical XFCE session running on the REDAC login server, showing a web browser and a terminal.

In order to have a better experience at the (remote) desktop, we recommend to disable the compositor (3D features such as windows dropping shadows). We recommend to use xfce4-terminal as the terminal application and firefox-esr as the web browser.

A powerful feature of X2Go is that it allows you to interrupt and resume sessions the same way as terminal multiplexers such as GNU screen and tmux allow for the terminal. However, the session is by default not shared with the direct terminal access session.

10.4. How user access works from outside

This section explains how general network access works, exemplaric for the Anabrid Operations:

First, the domain redac.anabrid.com resolves to the fiber internet uplink of the anabrid Ulm labs. The router has a Destination-NAT firewall rule which forwards incoming traffic on TCP ports such as 80 (HTTP) or 443 (HTTPS) to the REDAC SuperController within the REDAC internal network. This is possible because on-site, the REDAC internal network is accessible from outside. If this was not the case, the REDAC itself (i.e. the The Mikrotik Router) should do a similar DNAT port forarding, given that it has a single external IP address by default.

Warning

DNAT port forwardings and reverse proxying to IP addresses can be fragile if the IP addresses of the relevant devices or services are not static. These situations do happen in practice and result in errors which can be hard to trace in particular if administrators are not familiar with the setup. For instance, as a design choice in this network, the IP addresses are managed by permanent, yet principally dynamic DHCP service entries. In particular, devices with multiple network interfaces or changing MAC addresses are subject to different IP addresses if network cables are plugged into other interfaces or MAC addresses are changed manually. On an application level, TCP ports can easily become blocked by stalled server processes where some “clever” server code decides to listen at some other address, rendering the whole system unreachable. In order to be able to debug such problems quickly when they arise, administrators are urged to learn the route of user requests from outside to the inner parts of REDAC and back.

On the SuperController server, there is the caddy webserver running. It provides easy access to Let’s Encrypt certificates and reverse proxying to relevant services. The caddy webserver is steered by the configuration file /etc/caddy/Caddyfile (see https://caddyserver.com/docs/caddyfile for syntax explanation) which has the following exemplaric content:

# Caddyfile for the SuperController Webserver

(error_handler) {
        handle_errors {
                respond "{http.error.status_code} - {http.error.message} -- hint, check system availability at https://status.anabrid.net/status/redac1"
        }
}

redac.anabrid.com {
        root * /home/caddy-redac.anabrid.com
        file_server

        redir / /ui

        redir /manual.pdf https://anabrid.dev/docs/redac-manual/redacmanual.pdf

        redir /auth  https://auth.redac.anabrid.com/

        redir /api /api/
        reverse_proxy /api/* 127.0.0.1:8081  #  redaccess code = main REST interface

        import error_handler
}

auth.redac.anabrid.com  {
        reverse_proxy 127.0.0.1:8080  #  keycloak = main Authentification
        import error_handler
}

stats.redac.anabrid.com  {
        reverse_proxy 127.0.0.1:3000  #  grafana
        import error_handler
}

jupyter.redac.anabrid.com {
        reverse_proxy 192.168.102.251:8080  #  *EXTERNAL* jupyter notebook server
        import error_handler
}

# ...

This example shows how the Caddy server distributes incoming requests based on their domain name or path. Given the current permeability between the Ulm lab network and the REDAC internal network, it is easy to reverse proxy also network services which are not physically hosted on the same server.