evanjbrunner

Raspberry-pi and the Silicon Labs C2 Programming Interface

2014-10-29T07:00:00Z

what? why would anyone do this?

cause I’m cheap and it seemed marginally possible

I recently started a project, and decided that it necessitated connecting a collection of C8051F30x devices to a raspberry-pi. I don’t own a device capable of programming the 8-bit micros. Ultimately what I (::cough::cough:: google) discovered was that the kernel has had support for the C2 interface since 2008 for various reasons.

Silicon labs has a wide array of micro controllers that support a programming interface simply refered to as C2:

..a two wire serial communication protocol designed to enable in-system programming and debugging on low pin-count [devices].¹

The C8051F30 QFN has a 4×4mm footprint combined with a seemingly impossible number of peripherals. The combination of small size, low pin count, a fast multiplexed ADC, and at least 3 PWM outputs couldn’t be matched (as far as I found) from the likes of Microchip, Amtel, or Fairchild.

how I did it

summary

It took some further probing around to figure out that the c2port-core.ko (as it used to be called) needed an associated client. The included c2port-durmar2150.c is one such example, but is not useful for the Broadcom BCM2835 (raspberry pi).

Coincidentally I’d found a gpio-bit-banging implementation in a random openwrt github repo. I’m not really sure why it’s there (and it’s not very well documented) but I was able to splice all this into the official raspberry-pi linux repo – and create a functional set of modules.

I’ve setup a fork of the r-pi linux repo that contains my changes. This patch created from commit c588a3851 is the quickest way of adding the necessary changes to an arbitrary repo or branch of your own preference.

exercise

NOTE: if you’re serious about going down this road, do yourself a favor and proceed in a Linux environment (specifically I used Ubuntu 12.04.5). I’ve got a mac, and I’ve tried the do-everything-in-OSX (you can find help all over the web), but OSX is not Linux. Every step you take, OSX will remind you of that – and you’ll need help. Run a VM.

Start with a vanilla Raspbian (Debian Wheezy) image from raspberrypi.org.

Boot it up, setup SSH with some keys (or not).. etc.

Clone a linux repo (for the demo I’ll use the original the raspberrypi/linux repo):


user@host:~/prj$ git clone https://github.com/raspberrypi/linux.git

Apply the patch:


user@host:~/prj/linux$ git clone https://gist.github.com/f7c56965e6b21f7a9877.git
user@host:~/prj/linux$ git am --signoff  c2port-gpio.patch

Get a cross-compile toolchain²:


user@host:~/prj$ git clone https://github.com/raspberrypi/tools.git
user@host:~/prj$ export CCPREFIX=~/prj/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
user@host:~/prj$ ${CCPREFIX}gcc --version
arm-linux-gnueabihf-gcc (crosstool-NG linaro-1.13.1-4.8-2014.01 - Linaro GCC 2013.11) 4.8.3 20140106 (prerelease)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

build a configuration that inherits from the rpi:


user@host:~/prj/linux$ #fetch the config from the raspbian build
user@host:~/prj/linux$ scp pi@raspberrypi:/proc/config.gz .
user@host:~/prj/linux$ gunzip -c config.gz  .config
user@host:~/prj/linux$ #update it 
user@host:~/prj/linux$ ARCH=arm CROSS_COMPILE=${CCPREFIX} make oldconfig
scripts/kconfig/conf --oldconfig Kconfig
*
* Restart config...
*
*
* C2PORT
*
Silicon Labs C2 port support (C2PORT_CORE) [N/m/y/?] (NEW) m
C2 port support generic GPIO (C2PORT_GPIO) [N/m/y/?] (NEW) m
#
# configuration written to .config
#

Build the kernel


user@host:~/prj/linux$ #build the kernel with n threads where n=[number of cores]+1
user@host:~/prj/linux$ ARCH=arm CROSS_COMPILE=${CCPREFIX} make -jn

Mount the pi filesystem at root, with root permissions³


user@host:~/prj/linux$ sudo mkdir /mnt/pi
user@host:~/prj/linux$ sshfs root@raspberrypi:/ /mnt/pi
root@raspberrypi's password:

Build and install the new modules


user@host:~/prj/linux$ ARCH=arm CROSS_COMPILE=${CCPREFIX} \ 
 INSTALL_MOD_PATH=/mnt/pi DEPMOD=/bin/true make -jn modules_install

Slap the new kernel into place


user@host:~/prj/tools/mkimage$ #decompress the image
user@host:~/prj/tools/mkimage$ ./imagetool-uncompressed.py ../../linux/arch/arm/boot/zImage
user@host:~/prj/tools/mkimage$ #move it to the pi
user@host:~/prj/tools/mkimage$ mv kernel.img /mnt/pi/boot/

Restart the pi⁴


user@host:~/prj/$ ssh pi@raspberrypi
pi@raspberrypi:~/$ sudo reboot

Insert the new modules⁵⁶


pi@raspberrypi:~/$ #This enables the c2port on brdcm GPIO ports 17 and 18
pi@raspberrypi:~/$ #which are rpi-GPIO ports 11 and 12 respectively.
pi@raspberrypi:~/$ sudo modprobe c2port-gpio c2d=17 c2ck=18 
pi@raspberrypi:~/$ #for a quick debug output, mount the debugfs
pi@raspberrypi:~/$ #enabled in the default .config: CONFIG_DEBUG_FS=y
pi@raspberrypi:~/$ sudo mount -t debugfs none /sys/kernel/debug 
pi@raspberrypi:~/$ sudo cat /sys/kernel/debug/gpio
GPIOs 0-53, bcm2708_gpio:
 gpio-16  (led0                ) out hi
 gpio-17  (c2port data         ) in  lo
 gpio-18  (c2port clock        ) in  lo 
pi@raspberrypi:~/$

Reach out and touch a microcontroller


pi@raspberrypi:~/$ su root
Password:
root@raspberrypi:/home/pi# echo 1 /sys/class/c2port/c2port0/access
root@raspberrypi:/home/pi# cat /sys/class/c2port/c2port0/dev_id
10
root@raspberrypi:/home/pi# cat /sys/class/c2port/c2port0/rev_id
0

finishing touches


pi@raspberrypi:~/$ #insert udev rule to mod ownership of c2port dev
pi@raspberrypi:~/$ sudo echo "SUBSYSTEM=="c2port", ACTION=="add", PROGRAM="/bin/sh -c 'chown -R pi:pi /sys%p'""  /etc/udev/rules.d/98-c2port.rules
pi@raspberrypi:~/$ #automatically load module on boot
pi@raspberrypi:~/$ sudo echo "c2port-gpio c2d=17 c2ck=18"  /etc/modules

that-last-thing-that-should-probably-be-the-first-thing

specify device flash size

It’s important to note that the c2port-gpio has the flash block size, and number of blocks hard-coded into the driver. To make your specific device work correctly, you’ll have to get the flash size from the device specifications and edit the driver before you build. Specifically I did the following for the C8051F30:


--- a/drivers/misc/c2port/c2port-gpio.c
+++ b/drivers/misc/c2port/c2port-gpio.c
@@ -67,7 +67,7 @@ static void gpio_c2port_c2ck_set(struct c2port_device *dev, int status)
 
 static struct c2port_ops gpio_c2port_ops = {
 	block_size	:512,		/* bytes */
-	blocks_num	:30,		/* total flash size: 15360 bytes */
+	blocks_num	:15,		/* total flash size: 7680 bytes */
 	access		:gpio_c2port_access,
 	c2d_dir		:gpio_c2port_c2d_dir,
 	c2d_get		:gpio_c2port_c2d_get,

¹ AN127 Flash Programming via the C2 Interface

² The easiest way to get a toolchain is to clone it from the raspberrypi github account (as I did in the demo). There are more formal instructions on how to use the provided binary or build your own toolchain using crosstool-ng in the Embedded Linux Wiki. Also, there are instructions on how to enable kernel debug.

³ The pi doesn’t have a root password set by default. To create one:

pi@raspberrypi:~/$ sudo passwd root

⁴ If the pi refuses to boot, your kernel image is bad. Start over.

⁵ If depmod didn’t run on reboot, modprobe may claim the module was not found. To fix this quick run depmod --all.

⁶ Actual port mappings can be read from the schematic (found here), or from the world’s most insane table.

Layers for PDFs in R

2013-01-10T08:00:00Z

useless personal anecdote re: why layers
tldr

R: an implementation of the S language

My fiance was recently working on her thesis (a survey of Crangon crangon in the Westfjords) and wanted to use R do do some statistical analysis, [and / or] some spatial mapping.

Once she got into it, she quickly became frustrated. I dismissed her frustration as the expression of a standard learning curve, and would help her overcome particularly sticky spots now and again when I had the time. Never having used R myself, but hearing so much about it my curiosity was piqued. After I finished what I’d most urgently been trying to complete: I started to till into spatial mapping with R; to lend a hand.

Having made it this far what I’ve learned is that:

R is a quagmire of unintuitive arguments and functions that don’t operate in any uniform or expected way. There is no idiomatic R.
These unpredictable functions with any number of arbitrary arguments in whatever format are actually altogether poorly documented. The most practical way to get answers on obscure problems is to search mailing lists, or stackoverflow.
R is not inherently good with resources: big datasets will consume enough to slow down everything your doing
R is not fast:¹ there is often nothing to gain from having multiple processors – R Base is not threaded (you can add packages, and will it to be so: parallel, multicore, etc). Providing software for parallel or high performance computing (HPC) with R was not a development goal², but rather something that it was born into.

Nevertheless, by the time you’ve fully realized the drawbacks: you’ll be so far into your first project with R that you can’t turn back. You’ll proceed diligently to cobble together your own little [fiefdom / house of cards / specific selection of tools] to do exactly what you want – and nothing else.

And it’s just about then that you come back to: hey – this R thing is happening.

geospatial mapping in R

She had given me some shapefiles, and I quickly figured out I could load shapefiles from the web (GADM) simple enough


#load map from GADM
load_iceland=function() {
  #setup connection iceland map 
  if (!("iM" %in% ls())) {
    con - url(
      "http://www.filefactory.com/file/6dhsa8xf9mpz/n/ISL_adm2_RData")
  }
  #load map from GADM
  # NOTE: if variable name changes, and this function 
  # no longer works print(load(con)) to find new variable name
  load(con)
  close(con)
  return(gadm)
}

and plot them using spplot(), part of the sp package. I could even overlay them (by way of spplot()+spplot()) if I used another package latticeExtra.

Then I got that bug that everyone gets – I want [more / better] data. Fortunately (or not) I was able to find it. I found more detailed coastline and demographic shapefiles from OpenStreetMap.org via the OpenStreetMap package (credit to this guy), and then ?.dem files from the Advanced Spaceborn Thermal Emission and Reflection Radiometer (ASTER) a Japanese device on the Terra Satellite.

It was about this time that I discovered rgdal (a set of bindings in R for the Geospatial Data Abstraction Library GDAL). To load rgdal, first you have to build GDAL – which is a tool external to R that allows the manipulation of “raster geospatial datasets”, like ?.dem files. GDAL in and of itself turned out to be an enormously useful tool for manipulation of the data external to R. Once I thought I had something pretty swell I’d load it up:


if (!("iWater" %in% ls())  LOAD_FULL) {
  iWater - readOGR(dsn="./iceland_shapefiles", layer="iceland_water")
}
if (!("iCoast" %in% ls())  LOAD_FULL) {
  iCoast - readOGR(dsn="./iceland_shapefiles", layer="iceland_coastline")
}
if (!("iEle" %in% ls())  LOAD_FULL ){
  iEle - readOGR(dsn="./iceland_shapefiles", layer="iceland_elevation")
  iEle$uniform - as.factor(rep(c(1), length(iEle$elev)))
}

which is pretty straight forward but for the last line there where I create a column in iEle (uniform) that lets me plot all the elevation lines as one color (as all the rows have the same value==1). If the variables are already in the workspace, I don’t load them, and if I want to force them to not load I toggle LOAD_FULL. Loading this data set takes a noticeable few seconds.

I fooled around for quite some time with plotting rasterized spatial data in R, and none of it came off all that pretty. I never really got that right.

But once you do get something right, you’re going to want to export it in a way that’ll look pretty on your webpage – or in a word file, or (heavens forbid) a PowerPoint presentation, etc. The best way to export plots from R is in a vectorized format (as apposed to raster) such as SVG, or PDF. In these formats the display device will be able to recreate the figure dynamically at whatever resolution is required because the file specifies shapes, instead of pixels.

But you knew that.

So I proceeded to export my whole new nifty stack of spatial plots to a PDF. Popping it open I was sort of unsurprised to find that the layers I’d constructed with the use of latticeExtra weren’t preserved. So I googled aimlessly on, and eventually found the following exchange:

Dear all, Is it possible to create a pdf file with layers using the pdf() device in R? Many thanks for your help!

-n00b: Christoph

No. Is it possible to specify layers in the R graphics language or any device? (From what I understand by ‘layers’, no.)

-humbly: The author of pdf()

This is (I have found) a pretty standard open source guru-type response: the “it’s not a bug – it’s a feature” response. Nothing is more infuriating. What it really means is one of the following in order of ascending likelihood:

I have no interest in implementing that feature for you.
I have no idea how to do that, but I’m not going to admit that to you.
I have no interest in even trying to figure out if I know how to do that (ie I have too much else to do to get to that)

Quite sure there was no solution in the works, nor would anyone with more experience or insight pursue such nonsense: I got on it directly.

Layers in `pdf()`? easy-peasy.

anatomy

pdf() is implemented as what I would eventually come to interpret as an extension. There’s a R function that references a quick one-off C library that implements an interface to the graphical device library. It’s a quazi API for graphical devices, but it’s purely sequential – and omits what could have been executed more intuitively by passing a XML-like tree object. They went they way they did for the reasons they had at the time, but it makes implementing the interface a bit like trying to make a finger-painting while your hand is asleep.

Whatever comes out of the interface has to be dealt with, and even within the calling structure there are arbitrary attempts to clean up repetitive output. But this is that sort of thing where you get the repetitive output anyway. It’s mucked up so well that you can’t even tell what you’re supposed to be able to dismiss.

Anyhow, the implementation of pdf() works well enough. Many of the essential objects and their dictionaries are actually statically assigned ids and so forth – which tumbles things around again in a sort of bothersome way, but alas – I succeeded. PDF layers (as per the ISO 32000-1 specification) are implemented as “Optional Content Groups”. I finagled a way within the scope of the original implementation to shuffle the tags and objects into a suitable conformation. Took me a bit longer than I would have expected (as I should have expected) as certain dictionaries that were marked ‘optional’ in the spec absolutely weren’t.

In the midst of wrestling with this I found a simple, free (favorite kind) tool that helped me greatly:

PDF Object Browser by canoo : A java webstart type application that lets you browse a PDF tree. The Canoo webisite has since been completely re-done, and the original link to the page for the applicaiton no longer works. For now though, (by downloading and running the file via the provided link) it seems the application is still available with an expired security certificate. It was good while it lasted.

then I tested it.

The layering I’d implemented into pdf() seemed to work with readers like Adobe Acrobat, and Ubuntu’s default Document Viewer – but not with the CS5 suite I had access too. I was gunning for layers not because I wanted to print PDFs with optional layers (although that’s nice), but I wanted PDFs that I could load into Illustrator (Ai) and have all the objects for each figure organized by layer. Then I could do wonderful-cool things like join all the segments that described the coastline and create clipping layers and shadow effects… really useful and practical stuff!

It turned out, I was exactly half way to nowhere. Ai and the other Adobe CS5 products are stubbornly and knowingly not complaint to the PDF specification: they don’t honor Optional content as layers. Instead, as I would learn, when you build layers in Ai, and then save to a PDF – the PDF specification is so pliant to an excess of data within the structure that they embed an Illustrator ?.ai file right there within it. Wikipedia refers to it as a dual-path approach. Awesome. Why write one file, when you can write two, one within another?! As far as I could figure out it’s a compressed postscript format that at one time was probably the grandfather of the PDF format itself.

There was no way I was going to convince the sloppy little pdf() to write a format I had no formal description of.

At this point I sadly tried to do something with ?.svg, but – very sad. Not even workable.

Then I had an epiphany: I realized that Ai had to honor clipping layers. In order to create a clipping layer you need two objects (an object to clip, and an object to specify the clipping) and in Illustrator those objects have to be grouped. pdf() was hierarchy agnostic up and to this point – every clipping layer was promptly created, and then closed as soon as possible to retain a constant object depth. Sometimes it would try to exit a dozen times or so – sometimes it would create the clipping layers multiple times over: once with something necessary inside, the next time with nothing. It was all rather noisy – but fortunately it had no care for the depth of the tree – it always wanted to be creating at the root.

So I tricked it – every time I open an Optional Content group I bump the tree up one layer, and every time I leave a group I bump down one. I force the clip at the media size to so it has absolutely no effect on the content.

end
top

what to expect

Adobe CS# products load the new layers from pdf() as superfluous clipping groups. They’re unnecessary, but the objects being grouped within them makes it easy enough to reorganize the content to the intended state within 2 minutes. Otherwise, in viewers such as Acrobat and Document Viewer (in Ubuntu) the layers will just load up naturally: as layers. I haven’t set it up so you can appropriately name the layers, but that could be done if I get a sense that people are really that excited about it.

usage

All I can provide right now is a patch for (2.16.0 Unsuffered Consequences). If you know what to do with that: good on you (you have to download the R source, patch it, build it, and install it). If you know what you’re doing it’s not all that difficult, if you don’t – maybe throw some comments down below so we can help everyone out – so I know where people commonly have problems.. (if you’re that motivated). I’ve used it on 32 and 64 bit linux machines – I doubt that it will work much differently on any other architecture. Nothing I changed was architecture specific, but what I’m not sure of is if there are other PDF engines in there. The extension that I edited was a rudimentary reinvent-the-wheel type of deal, and I was surprised to find that it wasn’t just a library. Some systems may load a library(?) if it’s available at compile time – maybe I just didn’t have that configured. I don’t really know – I’m not overly familiar with the grDevices available in R, I just thought I’d hack this together and see if it made a stir. If it does and there’s legitimate interest in putting it into the next release – well, we’ll just figure that out as it comes.

Pretty straight forward to use:



pdf("file/name",onepage=TRUE)
[ PLOT STUFF ]
dev.off()

It won’t much like you if you try to set the version 1.5 (optional content is only valid 1.5 and after), nor if you try to set onefile=FALSE at the same time. Otherwise it doesn’t bite.

¹ julialang.org

² M. Schmidberger, M. Morgan, D. Eddelbuettel, H. Yu, L. Tierney, U. Mansmann (2009) “State of the Art in Parallel Computing with R” Journal of Statistical Software

JSON requests with HttpURLConnection in Android

2012-07-17T07:00:00Z

In the midst of doing something quazi-irrational, I stumbled over the need to create and send JSON requests to a web-server in an android context. Software should be oriented towards the inevitable dominance of a better future, and thus I built the mechanics of this new toy to be dependent on HttpURLConnection, as advised in the Android developer blog:

For Gingerbread and better, HttpURLConnection is the best choice. Its simple API and small size makes it great fit for Android. Transparent compression and response caching reduce network use, improve speed and save battery. New applications should use HttpURLConnection; it is where we will be spending our energy going forward.

-Jesse Wilson (of Dalvik team, but now square)

HttpURLConnection has no nifty little JSON request function, and googling around for a bit didn’t turn up anything too exciting or functional.

The solution is simply to incrementally build a tailored request:



//can catch a variety of wonderful things
try {
  //constants
  URL url = new URL("http://myhost.com/ajax");
  String message = new JSONObject().toString();

  HttpURLConnection conn = (HttpURLConnection) url.openConnection();
  conn.setReadTimeout( 10000 /*milliseconds*/ );
  conn.setConnectTimeout( 15000 /* milliseconds */ );
  conn.setRequestMethod("POST");
  conn.setDoInput(true);
  conn.setDoOutput(true);
  conn.setFixedLengthStreamingMode(message.getBytes().length);

  //make some HTTP header nicety
  conn.setRequestProperty("Content-Type", "application/json;charset=utf-8");
  conn.setRequestProperty("X-Requested-With", "XMLHttpRequest");

  //open
  conn.connect();
  
  //setup send
  OutputStream os = new BufferedOutputStream(conn.getOutputStream());
  os.write(message.getBytes());
  //clean up
  os.flush();
  
  //do somehting with response
  is = conn.getInputStream();
  String contentAsString = readIt(is,len);
} finally {  
  //clean up
  os.close();
  is.close();
  conn.disconnect();
}

note: the streams don’t need to be buffered if they are generally expected to be dealt with in one go, but large messages should be buffered to alleviate device memory load.

The aim is to build something similar to the following HTTP request ¹:

POST /ajax HTTP/1.1
Accept: */*
Content-Type: application/json;charset=utf-8
X-Requested-With: XMLHttpRequest
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.0.4; sdk Build/MR1)
Host: myhost.com
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 123

{"hash":{"attribute":"123"},"attribute":"value", …}

The important attributes are:

content-Type: application/json;charset=utf-8: tell the server to expect a JSON Object

X-Requested-With: XMLHttpRequest: pretend this request was created with an AJAX appropriate API

Content-Length: 123: the byte length of the message

{"hash":{"attribute":"123"},"attribute":"value", …}: the body of the request is a properly formatted JSON Object

The content length figure may appear particularly innocuous, but it is a lurking evil. The tendency when setting up HttpURLConnection is to lean towards using setChunkedStreamingMode(int) as you don’t have to worry about figuring out the precise message length before you send it. This does not work because the Transfer-Encoding: chunked mechanism apparently inserts additional characters (non-JSON [formatting / parsing] friendly) into the message body which can cause the server to fail on interpreting the object – resulting in failure, or even worse: inconsistent JSON interpretation. For example¹:

POST /ajax HTTP/1.1
Content-Type: application/json;charset=utf-8
X-Requested-With: XMLHttpRequest
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.0.4; sdk Build/MR1)
Host: myhost.com
Connection: Keep-Alive
Accept-Encoding: gzip
Transfer-Encoding: chunked

70
{"data":{"password":"123","email":"ejb"},"method":"account.sign_in",…
0

as interpreted on a rails test bench, has the parameters:

Started POST "/ajax" for localhost 2012-07-16 23:48:36 -0700
  Processing by AjaxController#post as */*
  Parameters: {"ajax"=>{"action"=>"post", "controller"=>"ajax"}}
Completed 200 OK in 8ms (Views: 7.2ms | ActiveRecord: 0.0ms)

which is entirely incomplete relative to intention. When the request is assembled (as above) with setFixedLengthStreamingMode(…) the result better fits expectations:

Started POST "/ajax" for localhost at 2012-07-17 01:15:45 -0700
  Processing by AjaxController#post as */*
  Parameters: {"data"={"password"="[FILTERED]", "email"="ejb"}, "me…
Completed 200 OK in 6ms (Views: 4.8ms | ActiveRecord: 0.0ms)

¹ Requests were captured by pointing an app at nc -l