Virtualization VGA Passthrough Concepts

VGA Passthrough Concepts and Terminology

I want to run Windows CAD on my Linux box without dual-booting.  Realistically, that means running a VM with GPU passthrough.  Under the current state of things, that means a separate VGA monitor.  That in turn means passing my USB keyboard and mouse back and forth between the VM and the host.  That's complicated.

**Update 2018:** The Looking-Glass project provides low-latency screen transfer.  Think of it as no-lag VNC.  Alterate UI no longer necessary, although mouse integration is still sketchy.

**Update 2017**: Used a modern PCI-E Nvidia adapter, and it mostly just works with LibVirt and OVMF/UEFI.  There are a few hoops to jump through to fix the Code 43.

• This  tutorial worked, but doesn't run under libvirt, so it doesn't support using virsh to automate a hot USB passthrough
• Adding the qemu arguments to my libvirt definition didn't work at all, but merits further exploration
• Adding the PCI device in virt-manager resulted in the pci device being present, but inoperable with a Code 10.  However, it actually started working after I force-installed the NVidia driver.  I don't seem to be able to dual-head with the virtual display adapter though.

Here's the (editorialized) terminology list I wish I'd had before setting off on this journey.

Hypervisor - The VM software.  Think of type 2 as emulation software that runs in your OS, perhaps using hardware acceleration (EG, VirtualBox).  Type 1 is like an OS container that runs on the bare metal (eg Xen and KVM).  Qemu-KVM is essentially type 2, but uses the KVM kernel modules to get many of the advantages of a type 1.

VirtualBox - Oracle's type 2 hypervisor.  Runs on Windows, OSX, and Linux.  I could not get VGA passthrough working on it.

Qemu - an open-source hypervisor.  Most of the VGA passthrough guides for it walk you through constructing a very hairy Qemu commandline.  That's unfortunate, because it precludes using the Libvirt management gui (below).

Passthrough - providing the VM with direct access to hardware.  In our case, this is a PCI, PCIe, or USB device.

VGA Passthrough - Providing the VM with hardware access to a display device.

GPU Passthrough - Mostly (incorrectly) used synonymously with VGA passthrough.  However, it also includes the concept of passthrough of a GPU that is *not* the display device, something that isn't possible on any of the Linux Hypervisors AFAIK.

Composing - This is the best word I have seen for the concept of passing through a rendering GPU which is something other than the display device.  In theory, you might passthrough a PCIe GPU for rendering your CAD model, and compose the display to your virtual display device.  None of the Linux hypervisors I've looked at can handle this.

Libvirt - a virtualization API, providing a consistent management interface for pretty much every linux hypervisor out there except VirtualBox.

USB passthrough - super easy on VirtualBox.  Super easy on Qemu, if you define it in the commandline. 

USB passthrough hotplug - This is built into the gui in Virtualbox.  Impossible in Qemu, without libvirt.  If you want to hotplug a usb device to a running libvirt VM, you need to use virsh, basically a commandline front end for libvirt.  I never found a way to do it without libvirt  (meaning if you ran your qemu VM from the commandline, you're out of luck).

http://rolandtapken.de/blog/2011-04/how-auto-hotplug-usb-devices-libvirt-vms-update-1

HMI/SCADA software for xLogic PLCs

ProView

I haven't actually used this yet, but it looks very promising.  Totally open-source, has process editors and multi-trends, and has build packages specific for linux, android, and raspberry pi.

RapidSCADA

This is a .NET/mono-based web interface.  Partly open-source, partly some (very cheap) licensed modules, it's relatively simple to follow their cookbook to get a bare-bones setup working.

In spite of my criticisms, of the options I've tried, it's the best combination of flexibility and easiness.

Criticism:

It's built with .NET, but in theory the service (but not the builder / config tools) should run on linux (and Raspberry Pi!) with the mono libraries.  I'm having some difficulty, but it might just be that my mono is 3.10 instead of 3.28.

The user interface is not very friendly.  I found myself having to repeat the same operation dozens of times, when I really wanted to multi-select and do the operation on all of them simultaneously.

The table creator feature is awesome, but the graphic editor is really basic (and does not come with a symbol library AFAIK).  It's really not a useful HMI without a multi-trend (which you have to pay for).

It's also not really SCADA without the automation plugin (which you have to pay for).

Overview:

The package consists of several distinct modules:

• Server - basically a database with some data flow functionality.  Has an "admin" gui, which does not run on linux.
• Comm - basically a collection of drivers, it interfaces between the Server and your devices.  Divided into a service and a gui (which always starts minimized to the tray!).
• Webservice - provides a user-interface to the server.  Consists of:
• Table: provides hourly summaries of all of your input points
• Chart: provides a graph of one point
• ChartPro: provides a graph of several points (paid)
• Schema: A graphic that displays differently based on input values.
• Table / Schema editors: creates interface elements for the web service.

Gotchas:

• The communicator front end always opens minimized to the tray.
• Follow the Modbus TCP recipe precisely, without deleting any extraneous points.  Do not change anything until that works. 
• Change the "chart gap" setting on the web service configuration page to 120s, or at least play with it until there is no gap between your points.

xLogic EasySCADA

XLogic supplies this development environment for free.

I don't recommend it, there's no way to test/debug without compiling and installing.   Also, the registration key needs to be generated again for every client computer, and that gets old really fast.

Overview

The basic intent of the software is to create a Windows HMI application that connects to your PLC network.  It appears to support USB RS485 and Modbus RTU, but I only ever used it to connect directly to my EXM's IP interface.

The only real advantage it has is you don't have to look up the Modbus address of the register you're interested in, you can just use xLogic's (semi-) descriptive name.

Notes

• You have to install their (bundled) Mysql6 and connector first.    
• When compiling, the "publisher id" field must be numeric.  
• Save the publisher ID.  You'll need it again later when you generate a registration key to install your HMI application.

OpenScada

First off, don't be fooled.  The openscada.org project has been dead for a couple of years.  All active development is on oscada.org, and appears to mostly be the work of one incredibly prolific developer.

If I'd persevered, I would've ended up with a SCADA server that would run on a Linux-based Raspberry Pi, with a touchscreen interface AND a web interface.  That's pretty cool.  The software appears incredibly versatile.  It is very modular, and there are modules for pretty much any kind of input or output device I've ever heard of.

Unfortunately, that versatility comes at the price of extreme complexity.  It took me most of two days to get a very basic HMI up showing a single modbus register, and I didn't really feel like I understood what I was doing.  It looked like every single field I wanted to add to the HMI was going to require a half-hour of tedius re-entry of long address strings.

Also unfortunately, the huge reams of very wordy documentation appear to have been written by a garrulous Ukranian, google-translated to Russian, and then to English (possibly by way of German).  I really didn't understand the intent of what I was reading.

AdvancedHMI

At the opposite end of the spectrum from OpenScada, AdvancedHMI makes heavy use of Visual Studio / .net / visual basic, and is very drag-and-drop.  It is almost entirely undocumented, but there's an active user community, and my only question was answered within a couple of hours.

The end result will probably only run on a windows platform, but it's so easy to get working that it's really hard to hate.

Gotchas:

This may  actually be a problem with the xLogic PLCs: register 1024 on xLogic, I had to instruct AdvancedHMI to access 41025.  With some help from the forums, I got this by

• prepending a 4 -- Modbus addresses starting in the 40,000 range  are the ones which are accessed using modbus command 0x3.
• adding 1 -- xLogic's register address reference is 0-based, but apparently all modbus addressing is 1-based.

Energy Monitor front ends

I have an ongoing energy monitoring project -- I'm archiving meter consumption data for a couple of clients, and would like to make it more usable.  I'm currently in the process of evaluating visualization front-ends.

Energy Star Portfolio Manager

Provides a REST api, and no other way to bulk upload data.  The only way to upload CSV data is to paste it into a preformatted excel spreadsheet.

Does not provide any useful visualization with anything finer grained than monthly billing data.

It may be useful to upload monthly electricity, gas, and water data if I have it all in one place.  If I can find a simple example, I'll take a crack at it.

OpenEnergyMonitor

This project has a lot of promise.  It consists of some hardware for sale, pre-configured to talk to the project's open-source PHP server software.  You have your choice of hosting the data on your own server (which has some hiccups), or for no charge on the project's server (which didn't immediately work for me).

My server

I was able to get the software installed and my data imported on my linux workstation with a minimum of fuss.   Some of the features didn't immediately work because redis.so and dio.so aren't installed by default and have no ubuntu packages.  Pretty simple -- `sudo pecl install dio-beta` worked just fine.

Project Server

Unfortunately, the same upload tools would not work to upload to the project server (the uploads failed silently with a success message!), and none of the live examples from the project website are currently working.

Conclusions

Most of the project's energy seems to be (understandably) going into making their for-profit products work with the software, so there isn't a lot of effort left over to fully document the API for edge-cases like my own. Asking on the (very active) community forum returned suggestions that the system was not designed for displaying historical data.  I belive that, so I'll be looking elsewhere.

Processing.js

The great-granddaddy of data visualization, this comes down to rolling my own data visualization website using some pretty awesome tools.  Ultimately, I'm not a front-end guy, so making this into something that can be customer-facing will take a lot more work than I want to invest. 

Networking XLogic PLCs

My house is run with XLogic PLC controllers.  The manuals are thorough, but are written in terrible Chinglish.  Which is unfortunate, because the UIs are almost completely incomprehensible.

All "manual" references in these notes are to the X-Messenger manual and xLogic's "eSms Config" software unless indicated otherwise.

WIFI PLC - Network Module

My network includes one EXM-12 series module with ethernet and wifi.  It was a real bear to understand the unintuitive setup process.  It'll go better for you if you think of all the TCP networking as being located in a stand-alone external module with its own CPU, which talks to the PLC over a 9600 baud serial interface.

Web Interface

 The device has a mostly useless web interface.  You can use it for two things:

• change WIFI between AP (Serve as a wifi access point) and STA (Connect to your existing AP).  You should use a serial cable instead.
• You can also use it to screw up some other settings:
• the serial connection to the PLC.  Must remain at 9600 N 8 1.
• the ethernet port
• If you reset the device from the panel->set->network menu, it sets the network module's serial speed to 57600, which is incompatible with the PLC.  You *must* connect by wifi and change it back to 9600 before you can make any serial changes to the network module.

Ethernet  

• I couldn't make it work at all, not even respond to ping.
• The LEDs on the ethernet are hijacked to indicate WIFI status.  Nothing to do with ethernet.  (pg 356)
• Open/Closed in this context means the ethernet port is active or disabled.
• Possibly the ethernet port is only useful for Modbus TCP -- I didn't test this.

WIFI Modbus/TCP

• Initial configuration is AP mode.  Connect to it at http://10.10.100.256, credentials admin / admin.
• You should probably change wifi settings with eSMS config and a serial cable
• If you're feeling lucky, you can use the web interface to change it to STA mode and have it connect to your AP.  If you screw up the cryptic settings for security, you won't be able to access the unit over WIFI anymore, and you'll need to fail over to changing settings with the config software
• MAC field is optional.

eSms Config

Use an onboard RS232 port if you're using the serial cable.  I had very little luck with USB-RS232 converters [update 2019: I found a genuine PL2303 device that worked, albeit not very well].  If you own the xLogic USB cable you'll probably be in better shape than me, but I couldn't bring myself to pay US$25 shipping on such a simple part.  Anyhow, once you have a physical serial connection, the process looks like this:

1. Tools->Configuration
2. Select RS232 port and EXM address
3. Click "connect to EXM"
4. Check that you have a valid connection 
• Use "get time from PLC" (toolbar L-circle-up-arrow)
5. Tools->Wifi Param
6. On every screen in Wifi Param,  click "READ" to see current state.
7. If you make changes, also click "WRITE".
8. I have no idea what the apply button does.

The left (Program) port is Com0.  The right (Extend) port is Com1.

HMI / SCADA development

See my later blog entry.

ICBC - Cost of short-term Autoplan policy

ICBC doesn't seem to publish anything about the cost of a short-term auto insurance policy.  I couldn't get a straight answer out of the Autoplan agent about the total amount of fees that are non-refundable.

TLDR: Up to about 5 days, buy a trip permit.  Above that, buy a year and cancel.

---

Early Cancellation

• refund is pro-rated by day, except for nonrefundable:
• $30 early cancellation fee
• $18 plate is forfeited
• Some other costs.
  
• EG: Imagine your total yearly charge is $1000 
• That includes $48 in non-refundable charges
• refundable premiums will be $952.
• Final cost will be $48 + $2.61 / day.

Short-Term policy

• Is not cheaper than cancelling a full-year policy
• premiums are pro-rated to the day
• some of the fees are not pro-rated (?)
• there is an additional $30 or $40 short-term fee 

Trip Permit

Length of Period	$1 million Liability only	$5 million Liability only	$5MM, Collision & Comprehensive
one day	$31	$36	$43 (most common permit)
2 days	$48	$55	$70
3 days	$64	$71	$92
one week	$138	$150	$189
15 days	$238	$254	$310

Basil experiment: 440nm blue LED grow light

Growing Basil under 450nm LED

Summary:

Grew basil under a 50W LED at 450nm, at a distance of about 15 cm.  Growth was markedly inferior to plants grown on a window ledge in the September BC sun.

Problem:

Apparently blue wavelengths are optimal for promoting leafy growth, centered at around 430 nm.

Apparatus:

A bit of searching on Aliexpress turned up this 50W LED and matching driver at 450nm.

• LED - U$4.50
• Driver - U$6.50

It came without instructions, but it wasn't hard to figure out the polarity of the terminals.

The driver is rated for 30-34V, but only made about 28V.  It was still insanely bright, but got hot enough to melt the solder off after about 2 minutes.  Obviously it needed a heat sink.  That got me thinking about how to safely assemble the whole thing.

A 4.5" device box is just the thing to contain the driver.  It's fireproof, and it's already got 1/2" knockouts.

The driver didn't have mounting holes for standoffs, and this was a quick-and-dirty experiment, so I just made some clips out of strapping and screwed it onto a scrap of plywood.    I needed a buck power supply to generate 12V for the heat sink fan, but fortunately there was still enough room on the plywood.

Light intensity wasn't very high relative to sunlight, so I put the LED about 10cm above the basil.  The basil seemed to be browning a little after a week, possibly from excess IR, so I raised it 5cm.

Experiment:

I ran the LED on a lamp timer for 14 hours per day, for two weeks, on a tray of four recently propagated basil plants, watered with capillary ceramic watering spikes.

My control was two more cuttings from the same batch, placed on a south-facing kitchen window ledge, getting most of the day's late-september sun.

Results:

I didn't do any rigorous measurement, and one of the LED plants died.

The control basil had almost twice as much growth as the LED basil.

Conclusion:

The basil mostly didn't die, but wasn't very happy under the monochromatic light.

I should try another experiment in deep winter, doing actual growth measurements, and comparing against both the (very poor) BC winter sunlight, and a full-spectrum T5 grow light.

KOdi keymap for USB remote "X10 lola"

Had remarkably a lot of trouble getting my old X10 RF  remote set up to work with Kodi.  Linux keymapping isn't quite as trivial as it once was.  Here are the steps I followed:
 
1. find the device vendor with lsusb.    Mine was 0BC7.

2. Find all the device's modalias files with
    find /sys -name *modalias | xargs grep -i 0BC7

3. Use the one that is a single string.  Mine was the file at
    /sys/devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/modalias

My modalias string looks like this.  We want everything before the 'e', because we want it to apply to all versions.
    input:b0003v0BC7p0002e0100-e0,1,2,k110,111,113,114,r0,1,amlsfw

4. Using heavy reference to this Arch wiki page,
    https://wiki.archlinux.org/index.php/Map_scancodes_to_keycodes
    /usr/include/linux/input.h

I create a hwdb file in /etc/udev/hwdb.d/10-x10.hwdb.  It contains:

Here's the HWDB file

evdev:input:b0003v0BC7p0002*
 KEYBOARD_KEY_00=esc    #    setup
 KEYBOARD_KEY_2c=favorites    #    learn
 KEYBOARD_KEY_02=power    #    power
 KEYBOARD_KEY_03=tv    #    tv
 KEYBOARD_KEY_04=audio    #    audio
 KEYBOARD_KEY_31=tv2    #    cable
 KEYBOARD_KEY_05=pc    #    pc
 KEYBOARD_KEY_06=vcr    #    x10
 KEYBOARD_KEY_0b=up    #    ch+
 KEYBOARD_KEY_08=volumedown    #    vol-
 KEYBOARD_KEY_0a=mute    #    m
 KEYBOARD_KEY_09=volumeup    #    vol+
 KEYBOARD_KEY_0c=down    #    ch-
 KEYBOARD_KEY_0d=1    #    1
 KEYBOARD_KEY_0e=2    #    2
 KEYBOARD_KEY_0f=3    #    3
 KEYBOARD_KEY_10=4    #    4
 KEYBOARD_KEY_11=5    #    5
 KEYBOARD_KEY_12=6    #    6
 KEYBOARD_KEY_13=7    #    7
 KEYBOARD_KEY_14=8    #    8
 KEYBOARD_KEY_15=9    #    9
 KEYBOARD_KEY_17=0    #    0
 KEYBOARD_KEY_16=numeric_star    #    add/del
 KEYBOARD_KEY_18=numeric_pound    #    a/b
 KEYBOARD_KEY_19=fn_f1    #    red – album
 KEYBOARD_KEY_1b=fn_f2    #    yellow – artist
 KEYBOARD_KEY_21=fn_f3    #    green – genre
 KEYBOARD_KEY_23=fn_f4    #    blue – track
 KEYBOARD_KEY_1c=pageup    #    pg up
 KEYBOARD_KEY_20=pagedown    #    pgdown
 KEYBOARD_KEY_1a=up    #    thumb up
 KEYBOARD_KEY_1d=left    #    thumb left
 KEYBOARD_KEY_1f=right    #    thumb right
 KEYBOARD_KEY_22=down    #    thumb down
 KEYBOARD_KEY_1e=enter    #    thumb press
 KEYBOARD_KEY_01=fn_f5    #    purple – playlist
 KEYBOARD_KEY_2d=fn_f6    #    orange – playing
 KEYBOARD_KEY_2e=home    #    t top
 KEYBOARD_KEY_2f=end    #    e end
 KEYBOARD_KEY_07=find    #    find
 KEYBOARD_KEY_30=select    #    select
 KEYBOARD_KEY_2b=previoussong    #    scan -
 KEYBOARD_KEY_24=rewind    #    rewind
 KEYBOARD_KEY_25=playpause    #    play
 KEYBOARD_KEY_26=fastforward    #    fast fwd
 KEYBOARD_KEY_2a=nextsong    #    scan +
 KEYBOARD_KEY_27=record    #    rec
 KEYBOARD_KEY_28=stop    #    stop
 KEYBOARD_KEY_29=pause    #    pause