REDACProtocol

Documentation for REDACProtocol. A packge which defines a the protocl to communicate with REDAC computers.

Types of messages

There are 4 types of messages.

  • Request sent from the client running library to a REDAC
  • Responeses from the server to the client answering a request
  • Errors from the server to the client communcation about problem to a request
  • Notifications send out by the server to the client

Each request has one and only one response or error.

sequenceDiagram Client->>REDAC: Request 0: Create session REDAC->>Client: Reply 0: Session created Client->>REDAC: Request 1: Acquire resources REDAC->>Client: Reply 1: Resources aquired Client->>REDAC: Request 2: Start run Client->>REDAC: Request 3: Acquire way to much resources REDAC->>Client: Reply 2: Run started REDAC-->>Client: Notification with run data REDAC->>Client: Error 3: Resouces unavaible REDAC-->>Client: Notification with run data REDAC-->>Client: Notification with run data

Enveloping messages

Messages exchanged between the server and the client are always wrapped in an envelope which carries metadata about the messages. See:

REDACProtocol.RequestEnvelope — Type
struct RequestEnvelope{MType<:AbstractREDACMachine, MsgType<:AbstractREDACRequest} <: REDACProtocol.AbstractREDACEnvelope

  • id::Base.UUID: Unique identifier under which the reply is extected

  • msg::AbstractREDACRequest: A message which contains the data to process the request. An instance of a subtype of AbstractREDACRequest

  • type::Symbol: Type of the message. Reply usually has the same type.

source
REDACProtocol.ResponseEnvelope — Type
struct ResponseEnvelope{MType<:AbstractREDACMachine, MsgType<:AbstractREDACResponse} <: REDACProtocol.AbstractREDACEnvelope

  • id::Base.UUID: Unique identifier noting which previous message this replies

  • msg::AbstractREDACResponse: A message which contains the results of the processed request. An instance of a subtype of AbstractREDACResponse

  • type::Symbol: Type of the message. Usually the same type as the request.

  • success::Bool: Correctly processed request always have success == true

source
REDACProtocol.NotificationEnvelope — Type
struct NotificationEnvelope{MType<:AbstractREDACMachine, MsgType<:AbstractREDACNotification} <: REDACProtocol.AbstractREDACEnvelope

  • msg::AbstractREDACNotification: Contains server send information not associated with any particular request. An instance of a subtype of AbstractREDACNotification

  • type::Symbol: Type of the notification.

source
REDACProtocol.ErrorEnvelope — Type
struct ErrorEnvelope{MType<:AbstractREDACMachine, MsgType<:AbstractREDACResponse} <: REDACProtocol.AbstractREDACEnvelope

  • id::Base.UUID: An error raised in response the request with the same id.

  • type::Symbol: Matches type of the message it is responding to

  • success::Bool: Always false

  • error::String: Human readable error which should be shown to the user.

source

This format is unambigious if one knows one is a client or is a server.

Parsing messages

The protocol uses JSON for transport. This is accomplished by representing the format as StructTypes.jl and using JSON3 for serialisation and deserialisation. As this library can be used both for clients and for end points presenting as REDACs the library has provides means to parse a message as a client or a REDAC by using these two exported constants which are parametrized on the type of machine.

For machine independ functionality

We observe that a client has no problem parsing this Error message sent while a server complains bitterly. 

julia> using JSON3, REDACProtocol;
julia> amsg = "{\"id\":\"1538250c-a380-4596-a766-5b794e6dee63\",\"type\":\"start_session\",\"success\":false,\"error\":\"The service is being rebooted\"}";
julia> JSON3.read(amsg, MatchTypeClient)ERROR: MethodError: no method matching AbstractRecievedEnvelope()
julia> JSON3.read(amsg, MatchTypeREDAC)ERROR: UndefVarError: `MatchTypeREDAC` not defined

Creating readable error messages for this was not a priority, as only a developer can encounter these by accident. It is the developers responsibility to throw a more human readable error.

The internal parsing machinery

If you are not interested in understanding what happens when you add new messages then you can skip to the protocol specification. For parsing this in an extendable manner we rely on both Julias type hierarchy and on a different hierarchcy maintained by StructTypes.jl. The Julia type hierachy start with:

REDACProtocol.AbstractREDACEnvelope — Type
abstract type AbstractREDACEnvelope <: REDACProtocol.AbstractComparable

As per protocol definition messages send via the REDAC protocol need to be wrapped in such an envelope. We distinguish between Requests, Responses, Notifications and Errors.

source

Introduction and REDAC side parsing

Let's explore how the server side parsing works:

julia> MatchTypeREDACERROR: UndefVarError: `MatchTypeREDAC` not defined

JSON3.jl relying on StructTypes.jl tries to interpret the content of a message as a RequestEnvelope. To do it creates a new hierachy of subtyping independt of Julias own type hierarchy. For that it checks the subtypekey of the associated type. Reads that field in the JSON object then looks the value in the of Indexable thing stored in type associated subtypes. Once the subtype is looked an attempt is made to construct that subtype given the message. The associated source code for RequestEnvelope is:

StructTypes.StructType(::Type{RequestEnvelope}) = StructTypes.AbstractType()
StructTypes.subtypekey(::Type{RequestEnvelope}) = :type
@generated StructTypes.subtypes(::Type{RequestEnvelope}) = :(return $(Dict{
    Symbol,
    DataType,
}()))

This returns the same dictionary which maps symbols to datatypes every time. Through macros we append to these Dicts to add new message types to handle.

This diagram notes this pattern as

flowchart TD subgraph REDAC side parsing CSM[Client sent message matched by RequestEnvelope] ---> s["`RequestEnvelope{T} where T <: AbstractREDACRequest`"] end

Client side parsing

The client can't know whether the next message is (in) a ResponseEnvelope, a NotificationEnvelope or a ErrorEnvelope. Thus we use the same AbstractType mechanism again. Our entry point is:

julia> MatchTypeClientAbstractRecievedEnvelope
flowchart TD subgraph Client side parsing SSM["Server sent message matched by AbstractRecievedEnvelope"] -->|msg <: AbstractREDACResponse| d["`AbstractRecievedEnvelope{T} where T <: AbstractREDACResponse`"] SSM -->|msg <: AbstractREDACNotification| r["`NotificationEnvelope{T} where T <: AbstractREDACNotification`"] d -->| success| w["`ResponseEnvelope{T} where T <: AbstractREDACResponse`"] d -->| !success| q["`ErrorEnvelope{T} where T <: AbstractREDACResponse`"] end

Depending if the type of the message is associated with a notification we can build notification object right away depending on the type noted in the type field of the envelope which a NotificationEnvelope is guaranteed to have. Otherwise it is possible that the message could be either a ResponseEnvelope or an REDACProtocol.ErrorEnvelope. So we check the success field to determine this. We make heavy use of parametric types here.

StructTypes.StructType(::Type{AbstractRecievedEnvelope{T}}) where 
  {T <: AbstractREDACResponse} = StructTypes.AbstractType()

Every AbstractRecievedEnvelope{T} where T can be any subtype of AbstractREDACResponse is treated as an AbstractType by StructTypes.jl. Every AbstractRecievedEnvelope{T} checks the success field to decide whether a ResponseEnvelope{T} or a ErrorEnvelope{T} should be created.

function StructTypes.subtypekey(::Type{
    AbstractRecievedEnvelope{T},
}) where {T <: AbstractREDACResponse}
    :success
end
function StructTypes.subtypes(::Type{
    AbstractRecievedEnvelope{T},
}) where {T <: AbstractREDACResponse}
    (; Symbol("true") => ResponseEnvelope{T}, Symbol("false") => ErrorEnvelope{T})
end

As notfications are subtypes of AbstractREDACNotification they don't go through this check, this check wouldn't even make sense as notifications don't have a success field. Instead they directly said to be structs in which case JSON3.jl relying on StructTypes.jl tries to interpret the content inside the msg field of the JSON as the matching notification object.

function StructTypes.StructType(::Type{T}) where {T <: AbstractREDACNotification}
    StructTypes.Struct()
end

See

StructTypes.StructType(::Type{REDACProtocol.AbstractRecievedEnvelope{M}})

All this machinery is maintained by using the macros to register your function.

@register_antiphon sets up a request-response pair and also convinently maintains the look up structure to get the expected response type from a request type.

Understanding the last two sections should enable developers to add new messages to the client and server.

Messages

This is an alphabetically sorted list of all stuff where there are docs strings for but not yet further described. Consult the python documentation for now.

REDACProtocol.MatchTypeClient — Type

Using this constant as MatchTypeClientAbstract{MachineType} enables to parse the JSON arriving at the client to the correct Julia types.

source
REDACProtocol.MatchTypeHc — Type

Using this constant as MatchTypeHCAbstract{MachineType} enables to parse the JSON arriving at the Hybridcontroller to the correct Julia types.

source
StructTypes.StructType — Method

An AbstractRecievedEnvelope could either be a notification or a AbstractExpectedEnvelope. Both errors and responses are AbstractExpectedEnvelope.

source
StructTypes.StructType — Method
StructType(
    _::Type{REDACProtocol.AbstractExpectedEnvelope{M<:AbstractREDACMachine, T<:AbstractREDACResponse}}
) -> StructTypes.AbstractType

Handles success dependent interpretation as a ResponseEnvelope or ErrorEnvelope. The success field must be set otherwise we crash.

source
REDACProtocol.getname — Method
getname(D::DataType)

A helper function. Looks up type string for a message using StructTypes dispatch data.

source
REDACProtocol.RunDataMessage — Type
struct RunDataMessage <: AbstractREDACNotification

  • entity::REDACProtocol.Path: from which entity

  • id::Base.UUID: Which run this data is for.

  • data::Vector{Vector{Float64}}: Data from the run.

source
REDACProtocol.RunStateChangeMessage — Type
struct RunStateChangeMessage <: AbstractREDACNotification

  • id::Base.UUID: Which run changed state

  • new::RunState: Current state

  • old::RunState: Previous state

  • runflags::Union{Nothing, REDACProtocol.RunFlags}

  • t::Int64

source
REDACProtocol.StartRunRequest — Type
struct StartRunRequest <: AbstractREDACRequest

  • id::Base.UUID: Starts a run under this UUID.

  • config::REDACProtocol.RunConfig: Configuation of the run.

  • daq_config::Union{Nothing, REDACProtocol.DAQConfiguration}

source
REDACProtocol.Path — Type
struct Path <: REDACProtocol.AbstractComparable

  • board::String: MAC Address which identifies the carrier board

  • cluster::Union{Nothing, String}

  • block::Union{Nothing, String}

  • funct::Union{Nothing, Int64}

source
REDACProtocol.RunConfig — Type
struct RunConfig <: REDACProtocol.AbstractComparable

  • halt_on_external_trigger::Bool

  • halt_on_overload::Bool

  • ic_time::Int64

  • op_time::Int64

source
REDACProtocol.RunFlags — Type
struct RunFlags <: REDACProtocol.AbstractComparable

  • externally_halted::Bool

  • overloaded::Union{Nothing, Vector{REDACProtocol.Path}}

source

Useful stuff for developers

Here are some utilities helpful for developers.

REDACProtocol.getname — Function
getname(D::DataType)

A helper function. Looks up type string for a message using StructTypes dispatch data.

source

Index