The Python Hybrid Controller interface

PyHyCon – a Python Hybrid Controller interface.

Note that the IO::HyCon Perl 5 module is the reference implementation that is maintained by the HyConAVR firmware author (Bernd). You can find the IO::HyCon Perl module at https://metacpan.org/pod/IO::HyCon

While this implementation tries to be API-compatible with the reference implementation, it tries to be minimal/low-level and won’t implement any client-side luxury (such as address mapping). It is the task of the user to implement something high-level ontop of this.

Especially, the following tasks are implemented by different modules which can but do not needed to be used:

  • Connection managament: HyCon just assumes a file handle, but different connection types are proposed in the connections module.

  • Autosetup: PyHyCon is plain python and has no dependency, for instance on YAML. There is autosetup which implements the “autosetup” functionality of Perl-HyCon.

  • High level functionality is implemented on top of HyCon and not within. See for instance :cls:`fpaa.fpaa` for an abstraction which can generate HyCon instructions and is aware of the circuit design at the same time.

The hycon module also includes an interpreter for the HyCon serial stream “protocol”. See replay for further details.

Logging and Debugging

There are several ways to inspect what HyCon is doing. One of the simplest is to activate logging on INFO level:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> # proceed here as usual, i.e.: hc = HyCon(...)
hycon.HyCon.ensure(var, **q)[source]

This is our assert function which is used widely over the code for dynamic parameter checking. q stands for query. The function will return silently if the query is fullfilled and raise a ValueError otherwise. Examples for success:

>>> ensure(42, eq=42)
>>> ensure("foo bar", re="fo+.*")
>>> ensure(17, inrange=(0,20))
>>> ensure("x", within="xyz")
>>> ensure("bla", length=3)
>>> ensure("blub", isa=str, length=4, re="b.*")

And in case of failure: >>> ensure(3, within=[1,2,9]) Traceback (most recent call last): … ValueError: Got var=3, but it is none of [1, 2, 9].

class hycon.HyCon.expect(**q)[source]

ensure delayed and on stereoids: Can be initialized with a query (but with further options) and then called with a HyConRequest. Will check the response and also allows return value mapping, for instance with regexpes or by splitting. Example:

>>> R = HyConRequest("dummy")
>>> R.response = "1,2,3"
>>> E = expect(split=",", type=int)
>>> print(list(E(R)))
[1, 2, 3]
hycon.HyCon.wont_implement(reason)[source]

Will not implement: Returns a function which raises NotImplementedError(reason) when called.

class hycon.HyCon.HyConRequest(command, expected_response=None)[source]

A HyConRequest models a single request and response cycle. It stores the ASCII command emitted by the HyCon and can save a expected response future/promise (see expect class). A HyConRequest can only be made once. If you want to do it several times, you have to (deep) copy the instance.

write(hycon)[source]

Run this request. Can only be executed once. Can be chained.

read(hycon, expected_response=None, read_again=False)[source]

Read response from HyCon. If read_again is given, will read several times. Can be chained.

class hycon.HyCon.HyCon(fh, unidirectional=False)[source]

Low-Level Hybrid Controller OOP interface, similar to the Perl Hybrid controller.

This is a minimalistic implementation which tries to implement all neccessary checking of input/output request/reply structure correctness, but won’t do any high level support for applications. Users are assumed to write such code on themselves. The PyFPAA library is an example for a high level “frontend” against HyCon, which includes a circuit understanding, etc.

query(*args, **kwargs)[source]

Create a request, run it and check the reply

command(*, help=None, **kwargs)[source]

Return a method which, when called, creates a request, runs it and checks the reply

ic()

Switch AC to IC-mode

op()

Switch AC to OP-mode

halt()

Switch AC to HALT-mode

disable_ovl_halt()

Disable HALT-on-overflow

enable_ovl_halt()

Enable HALT-on-overflow

disable_ext_halt()

Disable external HALT

enable_ext_halt()

Enable external HALT

repetitive_run()

Switch to RepOp

single_run()

One IC-OP-HALT-cycle

pot_set()

Activate POTSET-mode

single_run_sync()[source]

Synchronous run (finishes after a single run finished). Return value is true if terminated by ext. halt condition

set_ic_time(ictime)[source]

Sets IC (initial condition) time in MILLISECONDS.

set_op_time(optime)[source]

Sets OP (operation mode) time in MILLISECONDS

get_data()[source]

Supposed to be called when a read out group is defined and the machine is in (synchronous) OP mode.

read_element_by_address(address)[source]

Read any machine element voltage. Expecting 16-bit element address as integer.

set_ro_group(addresses)[source]

Defines a read out group, expects addresses to be an integer list of 16-bit element addresses.

read_ro_group()

Query for currently set read out group

read_digital()

Read digital inputs

digital_output(port, state)[source]

Set digital output pins of the Hybrid Controller

set_xbar(address, config)[source]

Exactly {self.XBAR_CONFIG_BYTES*2} HEX-nibbles are required to config data.

read_mpts(**kw)

Not implemented because because it is just a high-level function which calls pot_set and iterates a list of potentiometer address/names.

set_pt(address, number, value)[source]

Set a digital potentiometer by address/number.

read_dpts()[source]

Asks the Hybridcontroller for reading out all DPTs in the machine (also DPT24 modules). Returns mapping of PT module to list of values in that module.

get_status()[source]

Queries the HybridController about it’s current status. Will return a dictionary.

get_op_time()

Asks about current OP time

reset()

Resets the HybridController (has no effect on python instance itself)

Connection managers

Connection or “backends” for the PyHyCon.

The HyCon.HyCon class requires a file handle to be passed. Usually, file APIs are cursed in many languages (also python), but you can get your way out with the following examples and also classes in this module.

Tested or “proven” connection interfaces are:

  • tcpsocket: A small adapter for the socket() python builtin.

  • human: A small dummy adapter which prints to the interactive user terminal session and expects commands from there (the naming is ironically pointing to the human acting as actual Hybrid controller hardware endpoint).

Somewhat experimental but known to work is especially for unidirectional access:

  • StringIO.StringIO: Circumventing file access by reading from/to strings.

  • sys.stdout for just dumping HyCon-generated instructions

Note

All functions in this module do some progress reporting if you enable python logging facilities. Do so with

>>> import logging
>>> logging.basicConfig(level=logging.INFO)

Usage Examples

The following examples are suitable to be run in an interactive python REPL to explore the APIs.

Using PyHyCon with a microcontroller “simulator”

>>> from hycon import HyCon
>>> ac = HyCon(human())                                                                                            
>>> ac.set_ic_time(1234)                                  
<< Sending [C001234] to uC
[type reply of uC]>> T_IC=1234
HyConRequest(C001234, expect(eq: T_IC=1234), self.executed=True, response=T_IC=1234, reply=T_IC=1234)

Using PyHyCon only for writing firmware commands

>>> import hycon, sys
>>> ac = hycon.HyCon(sys.stdout, unidirectional=True)
>>> ac.set_ic_time(234)
C000234HyConRequest(C000234, expect(eq: T_IC=234), self.executed=True, response=n.a., reply=n.a.)

Such a unidirectional approach can be interesting when generating bitstreams, for larger integration tests, etc.

Using PyHyCon over TCP/IP

>>> sock = tcpsocket("localhost", 12345)                 
>>> ac = HyCon(sock)                                     
>>> ac.reset()                                           
>>> ac.digital_output(3, True)                           
>>> ac.set_op_time(123)                                  
>>> ac.set_xbar(0x0040, "0000000210840000781B")          

This setup is particularly interesting when connecting network-transparently to actual hardware. The target TCP server is expected to route the contents to a serial port/USB UART without introducing buffering. Examples for this kind of stub servers are given at networking-hc_.

Using PyHyCon over Serial

>>> fh = serial("/dev/ttyUSB0", 115200)                     
>>> ac = HyCon(fh)                                          
>>> ac.digital_output(3, True)                              
>>> # etc.

You are encouraged to use the serial class, which uses PySerial under the hood and does the clearing/resetting of the stream for you (something which is more cumbersome over serial then over TCP).

If you really want, you can also use PySerial directly:

>>> import Serial from serial                               
>>> fh = Serial("/dev/ttyUSB0", 115200)                     
>>> ac = HyCon(fh)                                          

Note that this approach suffers from binary/string conversions, but you could probably wrap open(fh) in some text mode.

If you (also) do not like PySerial, you can connect to a char device on a unixoid operating system with vanilla python (this example is kind-of-untested):

>>> import os
>>> fd = os.open("/dev/ttyUSB0", os.O_RDWR | os.O_NOCTTY)      
>>> fh = os.fdopen(self.fd, "wb+", buffering=0)                
>>> ac = HyCon(fh)                                             

In this case, you certainly want to set the connection parameters (baud rate, etc.) by ioctl, for instance in before on your linux terminal using a command like stty -F /dev/ttyUSB0 115200, or with stty ospeed 115200 and stty ispeed 115200 on Mac. Furthermore, when using this approach, consider writing a small wrapper which runs fh.flush() after writing.

hycon.connections.repeated_reset(fh)[source]

This routine tries to clear output buffers of the hycon UART by sending repeated reset instructions and waiting until the reply “RESET” appears on the line. Doing this, it implements the HyCon protocol, but the HyCon code does not deal with connection issues, which is why this function is aprt of connections.

Calling this function on beginning the setup is recommended for direct serial connections.

You can also call to this method with the :fun:`HyCon.HyCon.repeated_reset()` shorthand.

This function returns True when the connection suceeded, else False.

class hycon.connections.human[source]

Dummy IOWrapper for testing HyCon.py without the actual hardware

class hycon.connections.tcpsocket(host, port)[source]

Wrapper for communicating with HyCon over TCP/IP. See also HyCon-over-TCP.README for further instructions

write(sth)[source]

Expects sth to be a string

class hycon.connections.serial(port, baudrate, **passed_options)[source]

Small wrapper for making the use of PySerial more handy (no need for extra import)

Autosetup features

The autosetup module of the hycon package is the python implementation of the similar named feature of the Perl HyCon library.

It is used to setup a hybrid controller based from a YAML file which includes a mapping from names to computing element and potentiometer addresses and a problem description containing information about timing, potentiometer values (coefficients) and a readout group of interest. It can also describe the configuration of an XBAR module.

The idea of this YAML file is to describe the analog circuit as complete as possible, to keep the steering hycon code in perl (or python, respectively) short. Furthermore, it brings some kind of highlevel description of the circuit, since many parts of the circuit are given names.

This idea is some intermediate idea to the pyFPAA code which I wrote. It can be seen as an alternative high-level frontend to pyHyCon. Remember, the idiom of pyHyCon is to provide only a lowest level API for interaction with the hardware hybrid controller.

About the history of this code: Bernd started to write his auto-setup code at 25-DEC-2019. I started to write my pyFPAA code at the same time. Roughly one year later, where most of the time was spent at other stuff, I now port parts of Bernds auto-setup code to python in order to be able to use the same YAML files.

class hycon.autosetup.DotDict[source]

Small syntactic sugar: Dot notation to access to dictionary attributes, which is especially handy for deeply nested dicts. There are plenty of similar library for python around, but this implementation is only five lines (yes, five). The following usage example is longer then the implementation:

>>> a = { "b": 42, "non-identifier": 3, "foo": { "bar": { 3: 123 } }}
>>> a = DotDict(a)
>>> a.b
42
>>> a.foo.bar
{3: 123}
>>> a["non-identifier"]   # traditional __getitem__ access is still possible
3
>>> a.foo.bar[3]          # especially hand for non-pure-ascii identifiers
123
>>> DotDict(DotDict(a)).b # DotDict can be applied repeatedly without loss of functionality
42
>>> c = DotDict()
>>> c.foo = "b"
>>> c                     # also works for setting, not only reading
{'foo': 'b'}
>>> c.bar = DotDict()
>>> c.bar.baz = "bla"     # Limitation for nested setting: create nested DotDicts first.
class hycon.autosetup.PotentiometerAddress(address, number)[source]

Stores a potentiometer address, which is a tuple of a (typically hex-given) bus address of the hardware element and an element-internal number. Example:

>>> a = PotentiometerAddress(0x200, 0x20)
>>> b = PotentiometerAddress.fromText("0x200/20")  # FIXME: Is number really base 16?
>>> a == b
True
>>> a.address   # Don't forget that python standard numeric output is in decimal
512
>>> a.toText()
'0x200/20'
classmethod fromText(text)[source]

Parses something like 0x200/2 to (0x200, 2). Will also accept 0200/2 as hex.

hycon.autosetup.autosetup(hycon, conf, reset=True)[source]

hycon is expected to be an instance of HyCon. conf is expected to be a dictionary.

If you want to load from a YAML file, use the yaml_load function.

TODO: XBAR support not yet implemented.

hycon.autosetup.autoconnect(conf)[source]

Opens a file handle to the target position found in the YAML file. Follows the same rules as the perl routine, i.e. looks for serial or tcp key and connects according to the parameters.

Example serial port configuration:

serial:
  port: /dev/cu.usbserial-DN050L21
  bits: 8
  baud: 250000
  parity: none
  stopbits: 1
  poll_interval: 10
  poll_attempts: 20000

Note that we only take into account port and baud rate, since everything else looks standard and the pySerial port cannot deal with an integer stopbit 1 but expects something like serial.STOPBITS_ONE. As a note to the future, https://tools.ietf.org/html/rfc2217.html is supported by pySerial and should be adopted by the YAML definition.,

Example TCP port configuration:

tcp:
    addr: 192.168.31.190
    port: 12345
    connection_timeout: 2
    timeout: 0.1
    poll_interval: 10
    poll_attempts: 2000
    quick_start: False

Again, we only take into account the IP address and the TCP port, everything else is ignored for the time being.

class hycon.autosetup.AutoConfHyCon(conf)[source]

Syntactic sugar to provide a “setup” method similar to the perl HyCon API.

TODO: Should also provide other methods for high level value read and set access.

conf can either be a string holding the YAML filename or a dictionary (holding the configuration content, i.e. parsed YAML file).

get_data_by_name()[source]

Get readout group data handily labeled by name

set_pt_by_name(name, value)[source]

Set a digital potentiometer by name

read_element_by_name(name)[source]

Reads element by name

read_dpts_by_name()[source]

Asks the Hybridcontroller for reading out all DPTs in the machine (also DPT24 modules). Returns single map of DPT name to value (as float).

read_ro_group_by_name()[source]

Returns an OrderedDict of read-out group elements, with names

Protocol Replay features

A HyCon command stream interpreter.

Will spill out LISP-like commands which can be fed into the hycon again. This allows for replaying, which is helpful for a number of special scenarios such as:

  • Offline-validating a HyCon instruction stream

  • Man-in-the-middle inspecting an HyCon instruction stream

  • Validating the correctness of high-level HyCon instructions (such as emitted by the PyFPAA or autosetup codes)

The code basically implements a character-by-character tokenizer/parser. It is built based on a simple mapping datastructure which assigns each one-letter command the respective PyHyCon method name. Furthermore, method arguments can be read and converted.

It would be nice to join HyCon.py and replay.py to a single file which translates between the serial protocol and the OOP API calls. The transformation is quite trivial, but now we have a lot of code doing nothing of bigger interest.

The ordering follows the AVR Ino code.

hycon.replay.delayed(static_method)[source]

This is a decorator for a static method in a class. It will make the function body “delayed”, i.e. when calling the function, a future/promise/delay/deferred element is returned. Some parameter bounding (closure) happens: The decorated arguments are evaluated early while the later execution expects only a single reader argument. Example to follow the logic:

>>> f = lambda a,b,c,d: print(a,b,c,d)
>>> g = delayed(f)
>>> h = g(1,2,3)
>>> h(4)
4 1 2 3
class hycon.replay.consume[source]

Consume is an ugly namespace and not a class, actually. The basic idea of these (static!) functions is to be called delayedly with a function as it’s argument which acts like the io.IOBase.read() function, i.e. advances an internal cursor (side effect) and returns n characters from the stream. Crude Example:

>>> tokenizer = [consume.exact("test"), consume.decimals(3), consume.exact("foo"), consume.hex(2)]
>>> test = io.StringIO("test123fooAA")
>>> reader = test.read
>>> [ token(reader) for token in tokenizer ]
['test', 123, 'foo', 170]
number(digits, base, multiply=1)[source]

Reads a number with #digit digits in some base. Can perform a multiplication afterwards.

>>> consume.number(8,16)(io.StringIO("deadbeef").read)
3735928559
>>> consume.number(2,10,multiply=2)(io.StringIO("42").read)
84
list(split, end, digits, base)[source]

Reads a list of numbers. Limitations: * Always expects end token to come * All numbers must have same number of digits (and same base) * Cannot handle empty list or and won’t accept end-of-file before end token.

Examples:

>>> consume.list(split=",",digits=1,base=10,end=".")(io.StringIO("1,5,2,3,9.").read)
[1, 5, 2, 3, 9]
>>> consume.list(split=":",digits=2,base=16,end=";")(io.StringIO("5a:88:ff:ff;").read)
[90, 136, 255, 255]
class hycon.replay.HyConRequestReader(stream_or_string, mapping={'?': ('NOT_IMPLEMENTED', 'Prints help'), 'A': 'enable_ovl_halt', 'B': 'enable_ext_halt', 'C': ('set_ic_time', <function delayed.<locals>.decorated.<locals>.deferred>), 'D': ('digital_output', <function delayed.<locals>.decorated.<locals>.deferred>, True), 'E': 'single_run', 'F': 'single_run_sync', 'G': ('set_ro_group', <function delayed.<locals>.decorated.<locals>.deferred>), 'L': ('NOT_IMPLEMENTED', 'Locate a computing element'), 'P': ('set_pt', <function delayed.<locals>.decorated.<locals>.deferred>, <function delayed.<locals>.decorated.<locals>.deferred>, <function delayed.<locals>.decorated.<locals>.deferred>), 'R': 'read_digital', 'S': 'pot_set', 'X': ('set_xbar', <function delayed.<locals>.decorated.<locals>.deferred>, <function delayed.<locals>.decorated.<locals>.deferred>), 'a': 'disable_ovl_halt', 'b': 'disable_ext_halt', 'c': ('set_op_time', <function delayed.<locals>.decorated.<locals>.deferred>), 'd': ('digital_output', <function delayed.<locals>.decorated.<locals>.deferred>, False), 'e': 'repetitive_run', 'f': 'read_ro_group', 'g': ('read_element_by_address', <function delayed.<locals>.decorated.<locals>.deferred>), 'h': 'halt', 'i': 'ic', 'l': 'get_data', 'o': 'op', 'q': 'read_dpts', 's': 'get_status', 't': 'get_op_time', 'x': 'reset'})[source]

Converts HyCon “configuration strings” to high level API calls. This can be seen as the inverse operation to calling the HyCon.

Instances of this class act as iterator. It will consume the incoming stream character by character (or the whole string, if given as a string).

Example:

>>> instructions = 'C000100c015000P0200000204P0300030000G0362;0363;0220;0221;0222;0223.'
>>> commands = list(HyConRequestReader(instructions))
>>> for c in commands: print(c)
('set_ic_time', 100)
('set_op_time', 15000)
('set_pt', 512, 0, 0.19941348973607037)
('set_pt', 768, 3, 0.0)
('set_ro_group', [866, 867, 544, 545, 546, 547])
>>> replayed = io.StringIO()
>>> hc = HyCon(replayed, unidirectional=True)
>>> replay(hc, commands)
>>> replayed.getvalue() == instructions
True
hycon.replay.replay(hycon, commands)[source]

Given commands a list of tuples, this will mostly act like operator.methodcaller on them. If no arguments are given, the tuple can be omitted.

Basic example:

>>> class WannaBeHyCon:
...     def toot(self,x): print("too(%s)ooot" % x)
...     def bar(self): print("this is bar")
...     def buz(self,a,b,c): print(f"a*b = {a*b} but what is {c}")
>>> replay(WannaBeHyCon(), [ "bar", ("toot", "fuz"), ("buz", 1,2,3) ])
this is bar
too(fuz)ooot
a*b = 2 but what is 3

The command format is produced by the HyConRequestReader and thus can be fed into this replay function:

>>> replay(HyCon(sys.stdout, unidirectional=True), HyConRequestReader("xiohaARt"))
xiohaARt

This works for almost any useful instruction stream.