.. _opnetwork: Internal network ================ .. _fig-internal-ethernet: .. graphviz:: :caption: Graph of the logical (internal) Ethernet of REDAC. See main text for explanation. digraph EthernetDiagram { graph [bgcolor=white]; node [fillcolor="#bfc1d7", shape=box, penwidth=0, style=filled, fontname="Arial"]; edge [color="#2b307b", penwidth=2]; subgraph cluster_internal { label="REDAC-internal Ethernet network"; fontname="Arial"; labeljust=l; labelloc=t; style=dashed; color=black; Router [label="Mikrotik Switching Router"]; Node1 [label="SuperController Server", fillcolor="#eaeaf2"]; Node2 [label="MCU_1", fillcolor="#eaeaf2"]; Node3 [label="...", fillcolor="#eaeaf2"]; Node3 [label="MCU_N", fillcolor="#eaeaf2"]; Router -> Node1; Router -> Node2; Router -> Node3; } Internet [label="Outside Internet", shape=cloud, fillcolor="#bfc1d7", penwidth=0]; Internet -> Router; } REDAC features an internal IPv4 network somewhat similar to a typical home subscriber network (:numref:`fig-internal-ethernet`). This network has the following physical components: Mikrotik Switching Router Off-the-shelf rack mount router including a 24port front facing ethernet switch. The first port of the switch is dedicated to the external uplink while all other ports are bridged to an internal network. The device is introduced in section :ref:`opsetup-mikrotik`. Super Controller (x86 Server) An off-the-shelf low power low noise low depth server with IPMI out of band remote managament facilities. The device can also be used as a terminal server interactively. For further details see section :ref:`opsetup-server`. Hybrid Controllers (Teensy Microcontrollers) The network contains dozens of microcontroller units (MCU), each of them connected with 100Base-TX to the switch and centrally managed by the super controller. These devices provide/require no direct network access by system operators. This section will present the particular components in detail and explain relevant configuration steps. .. note:: At the time of reading this, you should have the :ref:`op_credentials` next to the :ref:`op_identifiers` ready at your hand. Note that the authentification systems of REDAC internal network structure are scoped and not unified/synchronized. That means that each device has a local user and password configuration and no single login exists. Do not confuse the scope of the credentials. credentials listed with the REDAC *public* application level authentification, which are managed by OpenID and discussed in the section :ref:`opauth`. .. _opinitialaccess: Initial network setup/access ---------------------------- By default, the router behaves as a IPv4 DHCP client at the external interface. If you have a DHCP server running in the outer network at your control, this default configuration might be suitable for you. In this case, you can access the super controller server by means of the TCP ports exposed (by NAT in the router). You can also take the server as an intermediate to configure the router. Note that the router configuration interface (web, ssh) is only accessible from the internal network. .. tip:: By default, the REDAC Router behaves similiar to a IPv4 home subscriber setup (think of a Fritzbox in Germany): There is a restrictive firewall allowing no ingoing traffic, doing masquerading and exposing thus only a single IP address at the outer network. System operators are encouraged to enter the internal network with their equipment, in particular for setup and maintenance. If you cannot make use of REDAC acting as a DHCP client but need to have it a given and fixed address, you need to *get access to the internal network*, which requires physical access. There are two principal ways how to do that: * Connect *your* notebook/computer to the internal network or * Connect a monitor and keyboard/mouse to the super controller and make use of it as a standalone computer. .. note:: The neccessary ports (USB and VGA) for connecting a terminal to the server are located at the front of the rack. If there are no free USB slots, you can plug out some USB devices while not using the Analog part of REDAC. Make sure to connect all USB devices back after having finished the administrative work, or use a USB hub if neccessary. For connecting your own computer to the internal REDAC network, there exist two options: * Either you connect the Wifi hotspot opened by the router. Keep the SSID and WPS2 password credentials at hand in order to connect. * Or you connect physically with a network cable to one of the free ports on the switch. In both ways, your computer should be assigned an address by the REDAC internal DHCP server. If you cannot run a DHCP client on your computer, you may choose a fixed address freely in the range ``192.168.104.5 - 192.168.104.9`` with subnet mask ``255.255.255.0`` and gateway ``192.168.104.1``. The gateway acts also as DNS server. If you, instead, choose to make use of the super controller as an interactive computer, you must have the username and password credentials at hand to login at the desktop greeter. After successful login, the linux computer will launch a desktop environment which allows you to start the usual applications such as a web browser or a terminal. If you do not have a mouse available, hit ``CTRL + ALT + F2`` or ``F3`` in order to switch the virtual desktop to a TTY instance which allows you to use the linux shell without a graphical user interface. .. warning:: Keep in mind that the direct access to the internal REDAC network is **only** intended for system administrators and not regular users. The reason is that ordinary users should not communicate directly with the microcontrollers. Otherwise, the Super Controller is bypassed and is prevented from doing his work correctly, messing up proper system managament. The same is true for the Super Controller graphical interface! If you want to provide your users a console like desktop access in the same room, for instance, please setup your own computers which are put *outside* of REDAC. .. _opsetup-mikrotik: The Mikrotik Router ------------------- The Mikrotik router runs the proprietary operating system *RouterOS* by Mikrotik. It is rich in features and the supplier provides a detailed documentation available at https://help.mikrotik.com/docs/. If you are not familiar with this technology, please read the section *Getting started* in the linked Mikrotik documentation. The REDAC ships with the mikrotik model type ``CRS326-24G-2S+RM``. It features a managed 24 port gigabit switch next to the routing capabilities. Relevant settings ................. The router comes with REDAC-specific default settings (given below) which only use a tiny fraction of all functions. Virtually any professional grade router available on the market, such as Cisco, Linksys, Juniper and other can do the same. Note that the MCUs only have a fast ethernet uplink (100mbit) and thus modern multi-gigabit powerhorses are not neccessary. The Mikrotik is therefore a sufficient and energy saving choice. The following router configuration options must not be changed by the systems operator: Internal ethernet configuration All ports are bridged except port 1, which belongs to the external port. The wifi (activated by default) is also bridged to the internal network. Internal network DHCP server and IP configuration There is a single subnet ``192.168.104.0/24`` with DHCP enabled. The router has the internal IP ``192.168.104.1/32``. There is no IPv6 active in the REDAC internal network. TCP port forwarding (NAT) for the super controller The super controller has a fixed IP address by the mikrotik DHCP server. TCP ports ``22, 80, 443`` are ``dst-nat`` to the super controller. In contrast, the system operator is invited to change not only, but in particular the following default options: * Changing the behaviour at the external interface, called ``ether1``, in particular changing the IPv4 configuration or enabling IPv6. * Changing or disabling the Wifi AP * Changing (sharpening or softing) the default firewall * Changing the access credentials * Making use of advanced functions such as VPN services. How to access the REDAC Ethernet/IP Routing Switch .................................................. When being connected to REDACs internal network, access the mikrotik web browser based graphical user interface by pointing your browser to ``http://192.168.104.1``. Your browser should show you a login mask where you have to fill out the given credentials. At :numref:`fig-starting-digital_mikrotik-screenshot` you can see a typical screenshot of the web browser based graphical user interface of the router after having logged in successfully .The page shown allows to turn of the DHCP client behaviour of the REDAC router. .. _fig-starting-digital_mikrotik-screenshot: .. figure:: starting_digital_mikrotik_screenshot.png :scale: 50 % :alt: screenshot of a browser Mikrotik RouterOS Web Browser GUI screenshot Alternatively, the router provides an SSH console which allows you to display and edit the router settings without a graphical user interface. This comes in handy in particular when you access the system with the linux console (VGA/keyboard directly connected to super controller) without having a USB mouse at hand. :: some@computer $ ssh admin@192.168.104.1 ... [admin@REDAC1-EthRouter1-MikroTik] > /ip/address/print Flags: D - DYNAMIC Columns: ADDRESS, NETWORK, INTERFACE # ADDRESS NETWORK INTERFACE 0 192.168.104.1/24 192.168.104.0 bridge 1 10.110.0.26/32 10.110.0.0 wg-tinybridge 2 D 192.168.100.168/24 192.168.100.0 ether1 Mikrotik default configuration .............................. The device default configuration is shown in the following listing. It can be obtained by typing ``/export compact`` into the console. Save this listing to a text file and you can restore the device configuration at any time: :: # 2025-01-08 20:00:42 by RouterOS 7.13.5 # software id = 1SRY-CS15 # # model = CRS326-24G-2S+ # serial number = HGC09YWZXTY /interface bridge add admin-mac=D4:01:C3:80:2D:EB auto-mac=no comment=defconf name=bridge /interface list add name=WAN add name=LAN /ip hotspot profile set [ find default=yes ] html-directory=hotspot /ip pool add name=dhcp ranges=192.168.104.10-192.168.104.250 /ip dhcp-server add address-pool=dhcp interface=bridge name=dhcp1 /port set 0 name=serial0 /interface bridge port add bridge=bridge comment=defconf disabled=yes interface=ether1 add bridge=bridge comment=defconf interface=ether2 add bridge=bridge comment=defconf interface=ether3 add bridge=bridge comment=defconf interface=ether4 add bridge=bridge comment=defconf interface=ether5 add bridge=bridge comment=defconf interface=ether6 add bridge=bridge comment=defconf interface=ether7 add bridge=bridge comment=defconf interface=ether8 add bridge=bridge comment=defconf interface=ether9 add bridge=bridge comment=defconf interface=ether10 add bridge=bridge comment=defconf interface=ether11 add bridge=bridge comment=defconf interface=ether12 add bridge=bridge comment=defconf interface=ether13 add bridge=bridge comment=defconf interface=ether14 add bridge=bridge comment=defconf interface=ether15 add bridge=bridge comment=defconf interface=ether16 add bridge=bridge comment=defconf interface=ether17 add bridge=bridge comment=defconf interface=ether18 add bridge=bridge comment=defconf interface=ether19 add bridge=bridge comment=defconf interface=ether20 add bridge=bridge comment=defconf interface=ether21 add bridge=bridge comment=defconf interface=ether22 add bridge=bridge comment=defconf interface=ether23 add bridge=bridge comment=defconf interface=ether24 add bridge=bridge comment=defconf interface=sfp-sfpplus1 add bridge=bridge comment=defconf interface=sfp-sfpplus2 /interface list member add interface=ether1 list=WAN add interface=bridge list=LAN /ip address add address=192.168.104.1/24 interface=bridge network=192.168.104.0 /ip dhcp-client add interface=ether1 /ip dhcp-server network add address=192.168.104.0/24 dns-server=192.168.104.1,8.8.8.8 gateway=192.168.104.1 netmask=24 /ip firewall nat add action=masquerade chain=srcnat out-interface-list=WAN /system clock set time-zone-name=Europe/Berlin /system identity set name=REDAC1-EthRouter1-MikroTik /system note set show-at-login=no /system routerboard settings set boot-os=router-os enter-setup-on=delete-key Note that this listing does not include any default passwords, i.e. it will not change the device users or wifi access. .. _opsetup-server: The Super Controller Server --------------------------- The Super Controller is the name both for a certain kind of software service and the physical server built into REDAC. This section will only cover the physical server and its operating system. For applications running on this server, see for instance :ref:`opservices`, :ref:`opauth` and in general the overview of :ref:`REDAC-specific software ` in the developers manual. Server Hardware ............... REDAC ships with a 1U low noise Mini-ITX server based on the SuperMicro Xeon Broadwell SoC server-grade mainboard ``X10SDV-4C`` and equipped with 32GB RAM and 1 TB SSD. The low depth enclosure has a single power supply and front facing I/O, making it easy to connect a KVM (monitor/keyboard) or USB stick. Most imporant is the IPMI feature which ensures out of band / lights of managament of the server. This server was chosen due to its compact form factor and not for high availability. In principle virtually any other rack mountable server is suitable to fulfill the job, in particular any product made by large server companies such as Dell, HP or Supermicro. Server Operating System ....................... The Super Controller server is shipping with the *Ubuntu Server* GNU/Linux distribution and is primarily supposed to be managed via SSH. Access is possible straight from outside the REDAC network as the SSH port is exposed by the router. It is up to the system administrator to also expose the IPMI or to lock down the ports if no remote managament is needed or realized in other fashions. In general, administrators are expected to be able to manage a Linux server and it is outside the scope of this manual to discuss how to obtain graphical access from remote or how to upgrade the operating system. The same applies with basic monitoring and health checks at an operating system level. The system comes in standard configuration for *Ubuntu Server*, which means for instance that security updates for system packages are installed automatically if internet access is available. .. _oplogin: Login and usage of the server ............................. There are a number of differnet ways to access the server: Either you connect physically (see also :ref:`opinitialaccess`) and use the standard VGA terminal (``getty`` login) or graphical login (``X11`` login greeter). Or you connect via network, primarily using SSH. We have the lightweight `Xfce `_ desktop environment preinstalled which includes common tools to get the everyday job done in the graphical user interface. Via SSH, you have a number of options: Either you use plain SSH, which is just fine for any advanced Linux user and in particular operator. Or you combine SSH with graphical forwarding (i.e. ``ssh -x``). The most advanced way is to use the free *terminal server/remote desktop software* `X2go `_ which is preinstalled. Again, Xfce should be chosen as the installed GUI. A typical configuration screen is shown in figure :numref:`fig-x2go-starting`, whereas figure :numref:`fig-x2go-running` is shown a typical running session. .. _fig-x2go-starting: .. figure:: x2go-client.png :alt: screenshot of the x2go GUI X2Go session initialization screenshot, showing the configuration options. .. _fig-x2go-running: .. figure:: x2go-running.jpg :alt: screenshot of the x2go GUI Graphical XFCE session running on the REDAC login server, showing a web browser and a terminal. In order to have a better experience at the (remote) desktop, we recommend to disable the *compositor* (3D features such as windows dropping shadows). We recommend to use ``xfce4-terminal`` as the terminal application and ``firefox-esr`` as the web browser. A powerful feature of X2Go is that it allows you to interrupt and resume sessions the same way as terminal multiplexers such as `GNU screen `_ and `tmux `_ allow for the terminal. However, the session is by default *not* shared with the direct terminal access session. How user access works from outside ---------------------------------- This section explains how general network access works, exemplaric for the :ref:`opanabrid`: First, the domain ``redac.anabrid.com`` resolves to the fiber internet uplink of the anabrid Ulm labs. The router has a Destination-NAT firewall rule which forwards incoming traffic on TCP ports such as 80 (HTTP) or 443 (HTTPS) to the REDAC SuperController within the REDAC internal network. This is possible because on-site, the REDAC internal network is accessible from outside. If this was not the case, the REDAC itself (i.e. the :ref:`opsetup-mikrotik`) should do a similar DNAT port forarding, given that it has a single external IP address by default. .. warning:: DNAT port forwardings and reverse proxying to IP addresses can be fragile if the IP addresses of the relevant devices or services are not static. These situations *do* happen in practice and result in errors which can be hard to trace in particular if administrators are not familiar with the setup. For instance, as a design choice in this network, the IP addresses are managed by permanent, yet principally dynamic DHCP service entries. In particular, devices with multiple network interfaces or changing MAC addresses are subject to different IP addresses if network cables are plugged into other interfaces or MAC addresses are changed manually. On an application level, TCP ports can easily become blocked by stalled server processes where some "clever" server code decides to listen at some other address, rendering the whole system unreachable. In order to be able to debug such problems quickly when they arise, administrators are urged to learn the route of user requests from outside to the inner parts of REDAC and back. On the SuperController server, there is the `caddy webserver `_ running. It provides easy access to `Let's Encrypt `_ certificates and reverse proxying to relevant services. The caddy webserver is steered by the configuration file ``/etc/caddy/Caddyfile`` (see https://caddyserver.com/docs/caddyfile for syntax explanation) which has the following exemplaric content: :: # Caddyfile for the SuperController Webserver (error_handler) { handle_errors { respond "{http.error.status_code} - {http.error.message} -- hint, check system availability at https://status.anabrid.net/status/redac1" } } redac.anabrid.com { root * /home/caddy-redac.anabrid.com file_server redir / /ui redir /manual.pdf https://anabrid.dev/docs/redac-manual/redacmanual.pdf redir /auth https://auth.redac.anabrid.com/ redir /api /api/ reverse_proxy /api/* 127.0.0.1:8081 # redaccess code = main REST interface import error_handler } auth.redac.anabrid.com { reverse_proxy 127.0.0.1:8080 # keycloak = main Authentification import error_handler } stats.redac.anabrid.com { reverse_proxy 127.0.0.1:3000 # grafana import error_handler } jupyter.redac.anabrid.com { reverse_proxy 192.168.102.251:8080 # *EXTERNAL* jupyter notebook server import error_handler } # ... This example shows how the Caddy server distributes incoming requests based on their domain name or path. Given the *current* permeability between the Ulm lab network and the REDAC internal network, it is easy to reverse proxy also network services which are not physically hosted on the same server.