Control protocol

The control protocol transmits messages via its Transport layer from one Component to another. The Message layer is the common language to understand commands, thus creating a remote procedure call.

Transport layer

The transport layer ensures that a message arrives at its destination.

Protocol basics

Socket Configuration

Each Coordinator shall offer one ROUTER socket, bound to an address. The address consists of a host (this can be the host name, an IP address of the device, or “*” for all IP addresses of the device) and a port number, for example *:12345 for all IP addresses at the port 12345. The default port number is 12300.

Components shall have one DEALER socket connecting to one Coordinator’s ROUTER socket.

Coordinators shall have one DEALER socket per other Coordinator in the Network. This DEALER socket shall connect to the other Coordinator’s ROUTER socket.

Note

While the number of DEALER sockets thus required scales badly with the number of Connectors in a LECO Network, the scope of the protocol means that at most a few Coordinators will be involved.

Communicating with a Coordinator, messages must be sent to a Coordinator’s ROUTER socket. Only for acknowledging a Coordinator sign-in, it is permitted to send a message to a Coordinator’s DEALER socket.

Naming scheme

Each Component must have an individual name, given by the user, the Component name. Component names must be unique in a Node, i.e. among the Components (except other Coordinators) connected to a single Coordinator. A Coordinator itself must have the Component name COORDINATOR.

Similarly, every Node must have a name, the Namespace. Every Namespace must be unique in the Network.

A Component name or a Namespace must be a series of printable ASCII characters (byte values 0x20 to 0x7E), without the character “.” (byte value 0x2E).

As each Component belongs to exactly one Node, it is fully identified by the combination of Namespace and Component name, which is globally unique. This Full name is the composition of Namespace, “.”, and Component name. For example N1.CA is the Full name of the Component CA in the Node N1.

The receiver of a message may be specified by Component name alone if the receiver belongs to the same Node as the sender. In all other cases, the receiver of a message must be specified by the Full name.

The sender of a message must be specified by Full name, except for the sign_in message, when the Component name alone is sufficient.

Message composition

A message consists of 4 or more frames.

1. The protocol version (abbreviated with “V” in examples).

2. The receiver Full name or Component name, as appropriate.

3. The sender Full name.

4. A content header (abbreviated with “H” in examples).

   1. UUIDv7 as a conversation id

   2. A three byte message id

   3. A one byte message type. A value of 0 means “not defined”, a value of 1 means JSON encoded.

5. Message content: The optional payload, which can be 0 or more frames.

Directory

Each Coordinator shall have a list of the Components connected to it. This is its local Directory.

They shall also keep a list of the addresses of all Coordinators, they are connected to.

Additionally, they shall maintain a global Directory, which is a Coordinator’s copy of the union of the local Directories of all Coordinators in a Network.

Conversation protocol

In the protocol examples, CA, CB, etc. indicate Component names. N1, N2, etc. indicate Node Namespaces and Co1, Co2 their corresponding Coordinators.

Here the Message content is expressed in plain English and placed in the Content frame, for the exact definition see Message layer.

Note

TBD: How to show the encoded content in the examples?

In the exchange of messages, only the messages over the wire are shown, the connection identity used by the ROUTER socket is not shown.

Communication with the Coordinator

Signing-in

After connecting to a Coordinator (Co1), a Component (CA) shall send a sign_in message (see Coordinator) indicating its Component name. The Coordinator shall indicate success/acceptance with a result response (according to JSON-RPC), giving the Namespace and other relevant information, or reply with an ERROR, e.g. if the Component name is already taken. In that case, the Coordinator may indicate a suitable, still available variation on the indicated Component name. The Component may retry signing in with a different chosen name.

After a successful handshake, the Coordinator shall store the Component name in its Directory and shall ensure message delivery to that Component (e.g. by storing the (zmq) connection identity with the local directory). It shall also notify the other Coordinators in the network that this Component signed in, see Coordinator coordination. Similarly, the Component shall store the Namespace and use it from this moment on, to generate its Full name.

If a Component does send a message to someone without having signed in, the Coordinator shall refuse message handling and return an error.

sequenceDiagram Note over CA,N1: Name "CA" is still free participant N1 as N1.COORDINATOR CA ->> N1: V|COORDINATOR|CA|H|sign_in Note right of N1: Connection identity "IA" Note right of N1: Stores "CA" with identity "IA" N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|result Note left of CA: Stores "N1" as Namespace Note over CA,N1: Name "CA" is already used CA ->> N1: V|COORDINATOR|CA|H|sign_in N1 ->> CA: V|CA|N1.COORDINATOR|H|ERROR: The name is already taken. Note left of CA: May retry with another Name Note over CA,N1: "CA" has not send sign_in Note left of CA: Wants to send a message to CB CA ->> N1: V|N1.CB|CA|H|Content Note right of N1: Does not know CA N1 ->> CA: V|CA|N1.COORDINATOR|H|ERROR: Component not signed in yet! Note left of CA: Must send a sign_in message<br> before further messaging.

Heartbeat

Heartbeats are used to know whether a communication peer is still online.

Every message received counts as a heartbeat.

A Component should and a Coordinator shall send a pong request message (see Actor) and wait some time before considering a connection dead. A Coordinator shall follow the Signing out for a signed in Component considered dead.

Note

TBD: Heartbeat details are still to be determined.

Signing out

A Component should send a sign_out message (see Coordinator) to its Coordinator when it stops participating in the Network. The Coordinator shall acknowledge the sign-out with a result message and remove the Component name from its local Directory. It shall also notify the other Coordinators in the network that this Component signed out, see Coordinator coordination.

sequenceDiagram CA ->> N1: V|COORDINATOR|N1.CA|H|sign_out participant N1 as N1.COORDINATOR N1 ->> CA: V|N1.CA|N1.COORDINATOR|H|result Note right of N1: Removes "CA" with identity "IA"<br> from local Directory Note right of N1: Notifies other Coordinators about sign-out of "CA" Note left of CA: Shall not send any message anymore except sign_in

Communication with other Components

The following two examples show how a message is transferred between two components CA, CB via one or two Coordinators.

Coordinators shall route the message to the corresponding Coordinator or connected Component.

sequenceDiagram alt Full name CA ->> N1: V|N1.CB|N1.CA|H| Give me property A. else only Component name CA ->> N1: V|CB|N1.CA|H| Give me property A. end participant N1 as N1.COORDINATOR N1 ->> CB: V|N1.CB|N1.CA|H| Give me property A. Note left of CB: Reads property A CB ->> N1: V|N1.CA|N1.CB|H| Property A has value 5. N1 ->> CA: V|N1.CA|N1.CB|H| Property A has value 5.

sequenceDiagram CA ->> N1: V|N2.CB|N1.CA|H| Give me property A. participant N1 as N1.COORDINATOR Note over N1,N2: N1 DEALER socket sends to N2 ROUTER participant N2 as N2.COORDINATOR N1 ->> N2: V|N2.CB|N1.CA|H| Give me property A. N2 ->> CB: V|N2.CB|N1.CA|H| Give me property A. Note left of CB: Reads property A CB ->> N2: V|N1.CA|N2.CB|H| Property A has value 5. Note over N1,N2: N2 DEALER socket sends to N1 ROUTER N2 ->> N1: V|N1.CA|N2.CB|H| Property A has value 5. N1 ->> CA: V|N1.CA|N2.CB|H| Property A has value 5.

Prerequisites of Communication between two Components are:

• Both Components are connected to a Coordinator and signed in.

• Both Components are either connected to the same Coordinator (example one), or their Coordinators are connected to each other (example two).

The following flow chart shows the decision scheme and message modification in the Coordinator Co1 of Node N1. Its Full name is N1.Coordinator. nS, nR are placeholders for sender and recipient Namespaces. recipient is a placeholder for the recipient Component name. iA is a placeholder for the connection identity of the incoming message and iB that of N1.Recipient. Bold arrows indicate message flow, thin lines indicate decision flow. Thin, dotted lines indicate decision flow in case of errors. Placeholder values are written in lowercase, while actually known values begin with an uppercase letter.

flowchart TB C1([N1.CA DEALER]) == "V|nR.recipient|nS.CA|H|Content" ==> R0 C0([nS.COORDINATOR DEALER]) == "V|nR.recipient|nS.CA|H|Content" ==> R0 R0[receive] == "iA|V|nR.recipient|nS.CA|H|Content" ==> CnS{nS == N1?} CnS-->|no| RemIdent CnS-->|yes| Clocal{CA in <br>local Directory?} Clocal -->|yes| CidKnown{iA is CA's identity?} CidKnown -->|yes| RemIdent Clocal -.->|no| E1[ERROR: Component not signed in yet!] ==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Component not signed in yet!"| S S[send] ==> WA([N1.CA DEALER]) CidKnown -.->|no| E2[ERROR: Component not signed in yet!]==>|"iA|V|nS.CA|N1.COORDINATOR|H|ERROR: Component not signed in yet!"| S RemIdent[remove sender identity] == "V|nR.recipient|nS.CA|H|Content" ==> CnR CnR -- "is None" --> Local CnR{nR?} -- "== N1"--> Local Local{recipient<br>==<br>COORDINATOR?} -- "yes" --> Self[Message for Co1<br> itself] Self == "V|nR.recipient|nS.CA|H|Content" ==> SC([Co1 Message handling]) Local -- "no" --> Local2a{recipient in local Directory?} Local2a -->|yes, with Identity iB| Local2 Local2[add recipient identity iB] == "iB|V|nR.recipient|nS.CA|H|Content" ==> R1[send] R1 == "V|nR.recipient|nS.CA|H|Content" ==> W1([Wire to N1.recipient DEALER]) Local2a -.->|no| E3[ERROR: Receiver is not in addresses list<br>send Error to original sender] ==>|"V|nS.CA|N1.COORDINATOR|H|<br>ERROR: N1.recipient is unknown"|CnR CnR -- "== N2" --> Keep Keep[send to N2.COORDINATOR] == "V|nR.recipient|nS.CA|H|Content" ==> R2[send] R2 == "V|nR.recipient|nS.CA|H|Content" ==> W2([Wire to N2.COORDINATOR ROUTER]) subgraph Co1 ROUTER socket R0 end subgraph Co1 ROUTER socket R1 S end subgraph "Co1 DEALER socket <br>to N2.COORDINATOR" R2 end

Coordinator coordination

Coordinators are the backbone of the Network and need to coordinate themselves.

Coordinator sign-in

A Coordinator joins a Network by signing in to any Coordinator of that Network. The sign-in/sign-out procedure between two Coordinators is more thorough than that of Components. During the sign-in procedure, Coordinators exchange their local Directories and addresses of all known Coordinatos. They shall sign in to all Coordinators, they are not yet signed in. The sign-in might happen because the Coordinator learns a new Coordinator address via Directory updates or at startup. The sign-out might happen because the Coordinator shuts down.

Similarly to Component sign-in, the Coordinator shall refuse a sign-in request with an ERROR, if it is already connected to a Coordinator with the same Namespace as the requesting Coordinator’s Namespace.

These are the sign-in/sign-out sequences between Coordinators, where address is for example the host name and port number of the Coordinator’s ROUTER socket.

sequenceDiagram participant r1 as ROUTER participant d1 as DEALER participant r2 as ROUTER participant d2 as DEALER Note over r1,d1: N1 Coordinator<br>at address1 Note over r2,d2: N2 Coordinator<br>at address2 Note over r1,d2: Sign in between two Coordinators Note right of r1: shall connect<br>to address2 activate d1 Note left of d1: created with<br> name "temp-NS" d1-->>r2: connect to address2 d1->>r2: V|COORDINATOR|N1.COORDINATOR|H|<br>coordinator_sign_in Note right of r2: stores N1 identity r2->>d1: V|N1.COORDINATOR|N2.COORDINATOR|H|result Note left of d1: DEALER name <br>set to "N2" d1->>r2: V|N1.COORDINATOR|N2.COORDINATOR|H|<br>add_nodes(Coordinator addresses)<br>record_components Note right of r2: Updates global <br>Directory and signs <br>in to all unknown<br>Coordinators,<br>also N1 Note over d1,r2: Mirror of above sign-in procedure activate d2 Note left of d2: created with<br>name "N1" d2-->>r1: connect to address1 d2->>r1: V|COORDINATOR|N2.COORDINATOR|H|<br>coordinator_sign_in Note right of r1: stores N2 identity r1->>d2: V|N2.COORDINATOR|N1.COORDINATOR|H|result Note left of d2: Name is already "N1" d2->>r1: V|N2.COORDINATOR|N1.COORDINATOR|H|<br>add_nodes(Coordinator addresses)<br>record_components Note right of r1: Updates global <br>Directory and signs <br>in to all unknown<br>Coordinators Note over r1,d2: Sign out between two Coordinators Note right of r1: shall sign out from N2 d1->>r2: coordinator_sign_out Note right of r2: removes N1 identity d2->>-r1: coordinator_sign_out Note right of r1: removes N2 identity deactivate d1

Note

Note that the DEALER socket responds with the local Directory and Coordinator addresses to the received Acknowledgment.

If a not signed in Coordinator tries to sign out from a second one, the latter one does ignore that message. If a Coordinator tries to sign out, but the message arrives via a different identity, the sign-out is rejected.

Coordinator updates

Each Coordinator shall keep an up-to-date global Directory with the Full names of all Components in the Network. For this, whenever a Component signs in to or out from its Coordinator, the Coordinator shall notify all the other Coordinators regarding this event. For that, the Coordinators sends the other ones the full directory, i.e. all Components and Coordinators connected to the Coordinator. The other Coordinators shall update their global Directory according to this message (add or remove an entry).

Note

TBD: These updates have to be determined.

On request, Coordinators shall send the Names of their local or global Directory, depending on the request type.

For the format of the Messages, see Message layer.

Message layer

The message layer contains the actual information exchanged between Components. As LECO is about controlling experiments, the message layer has to transmit commands, that is calling procedures remotely.

We use the JSON-RPC standard to encode these remote procedure calls (RPC) and the responses. We further use the OpenRPC standard to describe the possibly callable methods of a Component.

Therefore, a Component MUST execute remote procedures according to JSON-RPC and return an appropriate response. A Component MUST also offer a list of all possibly callable methods in accordance with OpenRPC.

For such a RPC message, the first content frame MUST consist in a JSON-RPC compatible content, for example a single request object or a batch of request objects.

For the definitions of methods see:

• List of control protocol methods
  • Component
  • Coordinator
  • Actor
  • Polling Actor
  • Locking Actor

Errors

Every error has a code and a message. Additionally they may have a data field with more information.

According to JSONRPC, applications can define error codes between -32000 and -32099. LECO defines the following errors.

Routing errors

Errors related to routing (mainly emitted by Coordinators). Their error codes are in the range of -32090 to -32099.

code	message	data	description
-32090	Component not signed in yet!	Name of the Component	If a Component did not sign in.
-32091	The name is already taken.	Name of the Component	A Component tries to sign in, but another Component is signed in with the same name
-32092	Node is unknown.	Name of the Node	The Node to which the message should be sent, is not known to this Coordinator.
-32093	Receiver is not in addresses list.	Name of the receiver	The Component to which the message should be sent, is not known to this Coordinator.

Locking errors

Errors related to locked Resources

code	message	data	description
-32050	Resource locked!	-	The resource is locked by another component.