10 Data Logging
This chapter is based on the projectlet:
- 08_blog
10.1 Projectlet Goals
Thus far the projectlets depended on somewhat informal means of communications ie text based logging. As the applications grow in size and scope, we will find dedicated compute engines performing a small set of related tasks, several of them collaborating to achieve the overall system objective. An insulin delivery device for example may deploy a dedicated pump driver to deliver insulin while using another to interact with the user. In such applications text based communications is no longer adequate and the designs have to resort to binary formats. With text based logging, precious cycles have to be expended to convert raw data to a text form - usually requiring more space. A floating point variable representing say flow rate is typically 32 bits in raw form while converted to text could be much larger.
In this projectlet, we convert the USART based framework used earlier for logging into a binary data communication framework and use this to transmit the onboard sensor data digitized and transmitted to a host.
10.1.1 Protocol for binary data communications
Having taken the road to binary transmission, we now face a whole new set of design decisions. Particularly as the applications grow in size and scope, so do the development teams and the need to implement a disciplined approach is paramount.
A review of the blog post will be very beneficial.
In this projectlet, we adapt the nanoprotobuf library, being a lightweight implementation of the protobuf framework.
10.1.2 Data Integrity, Framing and Synchronization
The objective for this projectlet is to integrate the nanoprotobuf library and the USART framework to achieve data transmission. The receiver is assumed to be a high level platform such as linux and a simple utility is developed for illustrative purposes. Complications of framing, and a full featured interaction framework is left for a future projectlet.
10.2 Implementation
10.2.1 Payload Handling
As indicated above, nanoprotobuf is the tool for payload handling; the initial step being the definition of the payloads the controller wants to transmit and the commands it expect to receive.
File : blog.proto
0002 | syntax = "proto2";
0003 |
0004 | message SensorsOnBoard {
0005 | required int32 Vbat = 1 ; // raw digitized value
0006 | required int32 Temp = 2 ;
0007 | required double Vbatf = 3 ; // converted into voltage
0008 | required double Tempf = 4 ;
0009 | }
0010 |
0011 | message AnalogInput {
0012 | required int32 Raw = 5 ; // raw digitized value
0013 | required double Voltage = 6 ; // converted into voltage
0014 | }The above defines two different types of payloads SensorsOnBoard and a generic AnalogInput. The definitions focus on the payload only, leaving out framing, transport error detection and such concerns. This definition is of course language agnostic, supporting C, C++, go etc. Ada is not in this list however at this time!
10.2.1.1 Application Interface
This projectlet develops the shims necessary to enable the embedded application to be able to convert raw data into a protobuf and vice versa.
File : getset.ads
0019 | function SetSensorsOnBoard(buffer: System.Address ;
0020 | buflen : Int ;
0021 | vbat : Int ;
0022 | temp : Int ;
0023 | vbatf : Interfaces.C.double ;
0024 | tempf : Interfaces.C.double )
0025 | return int
0026 | with Import => True , Convention => C , External_name => "SetSensorsOnBoard";
0027 |
0028 | function GetSensorsOnBoard(buffer: System.Address ;
0029 | buflen : Int ;
0030 | vbat : out Int ;
0031 | temp : out Int ;
0032 | vbatf : out Interfaces.C.double ;
0033 | tempf : out Interfaces.C.double )
0034 | return int
0035 | with Import => True , Convention => C , External_name => "GetSensorsOnBoard";10.2.1.2 Framing and Data Integrity
The protobuf buffer then is enveloped into a packet format before it can be handled by a suitable transport mechanism.
File : toolkit-messages.ads
0010 | -- Application message types should be chosen not to clash with
0011 | -- standard message types - identified as #8x#
0012 | subtype Message_Type is Storage_Element ;
0013 | Msg_Sync : constant Message_Type := 16#80# ;
0014 | Msg_DateTime : constant Message_Type := 16#81# ;
0015 |
0016 | subtype Special_Bytes is Storage_Element ;
0017 | packet_begin : constant Special_Bytes := 16#55# ;
0018 | packet_escape : constant Special_Bytes := 16#aa# ; -- future
0019 | packet_end : constant Special_Bytes := 16#00# ; -- future
0020 |
0021 | -- Payload is nanoprotobuf encoded payload.
0022 | -- Checksum follows the Payload
0023 |
0024 | -- +-------------+
0025 | -- | #55# | <- Implicitly transmitted
0026 | -- | MTYPE |
0027 | -- | LENGTH | <- No overheads. Payload only
0028 | -- | PAYLOAD | <- Begin Checksum calc here
0029 | -- | .. |
0030 | -- | .. | <- End of Checksum calc
0031 | -- | CHECKSUM |
0032 | -- +-------------+
0033 |
0034 | TYPE_IDX : constant Storage_Offset := 1;
0035 | LEN_IDX : constant Storage_Offset := 2;
0036 | PAYLOAD_IDX : constant Storage_Offset := 3 ;The message buffer definition below enables the above specification. In particular it contains Suspension_Objects that enable the application to synchronize the operation with the transport mechanism.
File : toolkit-messages.ads
0121 | type Buffer is tagged limited
0122 | record
0123 | Status : Status_Type := Buffer_Empty ;
0124 | Storage : Storage_Array( 1..MAX_SIZE );
0125 | Stored : Storage_Count := 0 ;
0126 | Next_Out : Storage_Offset := 0 ;
0127 | Next_In : Storage_Offset := 0 ;
0128 | Reception_Complete : Suspension_Object;
0129 | Transmission_Complete : Suspension_Object;
0130 | Error_Status : Error_Conditions := No_Error_Detected;
0131 | end record ;
0132 | function CalcChecksum( This : in out Buffer ) return Storage_Element ;
0133 | procedure CalcChecksum( This : in out Buffer );
0134 | function VerifyChecksum( This : in out Buffer ) return Boolean ;
0135 | Senders of data wait on Transmission_Complete.
File : toolkit-datalogs.adb
0065 | loop
0066 | Logger.Get (nxtmsg);
0067 | toolkit.messages.Set (Outgoing, nxtmsg.all);
0068 | Serial_Io.Binary.Send (DataLogCom, Outgoing'unchecked_access);
0069 | Outgoing.Await_Transmission_Complete;
0070 | end loop;Command handlers wait on Reception_Complete.
File : toolkit-datalogs.adb
0083 | loop
0084 | Incoming.Clear;
0085 | Serial_Io.Binary.Receive (DataLogCom, Incoming'access);
0086 | Incoming.Await_Reception_Complete;
0087 | mt := Incoming.Content (Integer (toolkit.messages.TYPE_IDX));
0088 | mlen := Incoming.Content (Integer (toolkit.messages.LEN_IDX));
0089 | toolkit.logs.logger.Put_Line
0090 | ("Message Type : "
0091 | & hex.Image (Unsigned_8 (mt))
0092 | & " Length "
0093 | & hex.Image (Unsigned_8 (mlen)),
0094 | source => myname);
0095 | end loop;In both cases, these are performed by helper tasks and the application interfaces with them through messaging to these tasks.
10.2.2 Transport
The transport of text used in the logging support toolkit.logs and its backend serial_port.nonblocking (directly copied from ADL) is adapted for handling binary data packets. In particular the notion of line termination does not lend itself to this requirement. The following amendments are necessary:
Message Exchange Protocol
- Packets have a starting indicator packet_begin, followed by a message type and then the packet length. Message Type is not relied upon for any application in this version though it could be used for administrative reasons (eg. heartbeat). The payload follows the packet length.
- The Last octet of the packet contains a checksum of the payload.
- There is not attempt to handle occurrences of the packet_begin value within the payload.
- The receiver waits for a packet_begin, followed by the message type and packet lengths. Then octets are assembled into a packet for application usage
The weaknesses identified above are not insurmountable and are left as a future enhancement.
10.2.2.1 Hardware Interface
The approach follows the familiar pattern in embedded systems - leveraging the Interrupts generated by the USARTs. Transmissions proceed by starting with one character and every completion event triggers the transmission of the next character to be transmitted. Similarly octet arrivals generate an interrupt handled by retrieving the octet and then going back to wait for the next input.
The package **serial-io.binary” implements the data send/receive as well as error detection.
File : serial_io-binary.adb
0030 | procedure Handle_Transmission is
0031 | Next_Out : Storage_Element;
0032 | Empty : Boolean;
0033 | begin
0034 | Outgoing_Msg.Get (Next_Out, Empty);
0035 | if Empty then
0036 | Device.Disable_Interrupts (Source => Transmission_Complete);
0037 | Outgoing_Msg.Signal_Transmission_Complete;
0038 | Outgoing_Msg := null;
0039 | else
0040 | Device.Transmit (UInt9 (Next_Out));
0041 | end if;
0042 | end Handle_Transmission;The UART pins when monitored using a scope or a tool such as pulseView might exhibit:
Similar handler for reception:
File : serial_io-binary.adb
0046 | procedure Handle_Reception is
0047 | Received_Byte : constant Storage_Element :=
0048 | Storage_Element (Device.Current_Input);
0049 | begin
0050 | Incoming_Msg.Append (Received_Byte);
0051 | if Incoming_Msg.PacketComplete then
0052 | loop
0053 | -- wait for device to clear the status
0054 | exit when not Device.Status (Read_Data_Register_Not_Empty);
0055 | end loop;
0056 | Device.Disable_Interrupts (Source => Received_Data_Not_Empty);
0057 | Incoming_Msg.Signal_Reception_Complete;
0058 | Incoming_Msg := null;
0059 | end if;
0060 | end Handle_Reception;the whole cycle initiated like so:
File : serial_io-binary.adb
0084 | procedure Start_Sending (Msg : not null access Buffer) is
0085 | begin
0086 | Outgoing_Msg := Msg;
0087 | Device.Enable_Interrupts (Parity_Error);
0088 | Device.Enable_Interrupts (Error);
0089 | Device.Enable_Interrupts (Transmission_Complete);
0090 | Handle_Transmission;
0091 | end Start_Sending;with an analogous operation for packet reception.
10.3 Diagnostic Toolset
Host applications
The top level projectlet collection under 98_Utils contain a series of utilities designed to exercise the facilities as they are being developed.
For example 98_Utils/03_logger contains the projectlet capture that receives text log transmission on a host.
98_Utils/blog similarly contains the projectlet capture designed to recive binary packets in the above format and unpack the protobuf data and display.
MType 10 Len 19 08fc0a10a40819000000000000124021cdcccccccc4c4f4056 Checksum : 56 >
Vbat 1404 Vbatf 4.50E+00 Temp 1060 Tempf 6.26E+01
MType 10 Len 19 08840b10a508190000000000001240210000000000804f4017 Checksum : 17 >
Vbat 1412 Vbatf 4.50E+00 Temp 1061 Tempf 6.30E+01
MType 10 Len 19 088a0b10a508190000000000001240210000000000804f401d Checksum : 1d >
Vbat 1418 Vbatf 4.50E+00 Temp 1061 Tempf 6.30E+01
MType 10 Len 19 08fe0a10a508190000000000001240210000000000804f4090 Checksum : 90 >
Vbat 1406 Vbatf 4.50E+00 Temp 1061 Tempf 6.30E+01
MType 10 Len 19 08f80a10a70819000000000000124021cdcccccccccc4f40d5 Checksum : d5 >
Vbat 1400 Vbatf 4.50E+00 Temp 1063 Tempf 6.36E+01In most systems these packets will be written to external files for later analysis using an appropriate tool. Julia and R will be used in this effort for illustration.
10.4 Development Insights
While USARTs are used in this projectlet, systems are just as likely to use I2C, SPI or even Ethernet as the physical and datalink layer disciplines. nanoprotobuf is of course agnostic and thus will grow with us. USARTs pose a particular challenge of synchronization ie the different controllers may start and stop independently and there has to be enough hints in the transport protocol to be able to identify the start and end of a packet. This projectlet again glossed over such complications for a simple solution leaving a more robust design for later.
For integrity this projectlet implements a checksum algorithm which again is not robust enough for safety critical applications. A CRC of 16 or even 32 bits might be needed if not a whole hash such as md5. Of course if TCP/IP is a choice - the data integrity concerns are correspondingly reduced.