Archive for the ‘howto’ Category

Migrating to GIT with Reposurgeon

Friday, April 18th, 2014

I’ve recently worked a lot with reposurgeon, a tool by Eric S. Raymond to do surgery on version control data. With this tool it is possible to migrate from almost any version control system to almost any other version control system — although these days the most feature-complete system is GIT which is the recommended and best supported target system (and the only one I’ve tested). It comes with a migration guide, the DVCS migration HOWTO.

Beyond just converting data, reposurgeon can be used to clean up artefacts, the simplest of which is to reformat commit comments to conform to established standards on GIT commit messages.

My conversion of the history of the pyst project, a python library to connect to different network interfaces of the Asterisk telephony engine, is a good example of what can be done with reposurgeon. The project, originally started by Karl Putland with version control at the time in CVS, was later taken over by Matthew Nicholson who used Monotone for version control. When I took over maintenance in 2010, I used Subversion. So we had three separate source code repositories in different version control systems. No effort was ever made to convert the version history from one system to another, so each new maintainer imported the last release version into the new version control system and continued from there.

Fortunately, Monotone has a GIT export feature and reposurgeon can natively read the other two formats. So I used separate reposurgeon scripts to clean up the three repositories and then used the reposurgeon graft command to unite them into one. The Subversion repo started with release 0.2 but there had been some commits after 0.2 in Monotone (which were later merged in Subversion) so the commits after 0.2 were put on a branch in the new repository. The last step then was to write out the resulting repository in GIT fast-export format and import into a GIT repository.

What artefacts did I clean up? Let me give two examples, both of which are problems I have when using Subversion.

Subversion doesn’t have the concept of a tag like other version control systems have. Instead tags are emulated by copying the to-be-tagged content to a new location in the repository, effectively creating a new branch, it’s just a naming convention that this is called a tag.

The first example deals with last-minute changes when doing a release: It’s not possible in Subversion to really remove a tag as in other systems. So when doing a release in GIT and something in the release process doesn’t work (not having created a release yet), as long as we didn’t push our changes to the public repository we can still move the release tag. This isn’t possible in Subversion. So I frequently have the situation that a release tag isn’t just one commit but several and the changes are either merged from the trunk to the tag-branch or the other way round from the tag-branch to the trunk. An example of a commit on a tag (r22 “fix PACKAGE definition for SF release”) and subsequent merge back to trunk can be seen in the following illustration created with the graph command of reposurgeon. The tag, originally Subversion commit r21 has already been tagified by reposurgeon. But the commit on the tag is now on the branch V_0_3.

First example

This can be fixed with reposurgeon with the following commands:


debranch V_0_3 trunk
[/^V_0_3\//] paths sup
:70 unmerge
:70 tagify --canonicalize
tag emptycommit-23 delete
tag V_0_3-root rename V_0_3
tag V_0_3 move :69

This puts the branch V_0_3 back onto the trunk and creates a new subdirectory V_0_3 there. Then this subdirectory is removed with the paths reposurgeon command. We then make the merge commit a normal commit with only one predecessor with unmerge and create a tag from this new commit which is possible because it doesn’t change any files. Finally we delete the tag just created and rename and move the V_0_3 tag.

The second example involves accidental commits on a release tag. This frequently happens to me when using Subversion for doing a release and happens as follows:

  • Create new tag by copying to a subdirectory in tags
  • Switch to this new tag using svn switch
  • Do the release
  • Forget to switch back to trunk
  • Come back later, do some accidental commits on the tag
  • Merge accidental changes back to trunk
  • Revert the changes on the tag

An example of this problem can be seen in the following figure.
Second example

This example is from my svnpserver project and shows a series of commits on the V_0_4 tag. Just before the next release I noticed, merged the commits from the tag to trunk, and reverted the erroneous commits on the tag. This is fixed with reposurgeon as follows:

:14063 delete
debranch V_0_4 trunk
[/^V_0_4//] paths sup
tag V_0_5-root rename V_0_5
tag V_0_5 move :14061
:14062 unmerge
:14062 tagify --canonicalize
tag emptycommit-4734 delete

First we delete the commit that reverts the changes on the tag. Then we move the commits from the tag to trunk and remove the resulting path prefix V_0_4. The new V_0_5 tag is moved to the last commit on what was previously the last commit on the tag because we’re going to eliminate the merge-commit next: First we make the merge commit a normal commit by removing the earlier ancestor using unmerge. The last step is to convert this commit into a tag (which is possible because it now doesn’t modify anything) and remove that resulting tag.

Modifying history is usually a bad idea when converting repositories. After all, the version control system is here to preserve the history. My rule is to remove artefacts of the used version control system that would never have occurred with another system. All the problems above would have been avoided by using, e.g., GIT in the first place: With GIT we can simply move a tag (if we haven’t pushed yet) and the erroneous commits on the tag could never have happened because we don’t have to switch branches for doing a release with GIT, so forgetting to switch back from the branch is not possible.

So by using reposurgeon we now have a GIT repository for pyst that spans the entire history of the project united from three different version control systems in use over the duration of the project.

SIGALRM blocked forever — by init

Friday, September 23rd, 2011

I’m working on an embedded Linux system which allows to chose the root filesystem to boot from (flash card or NAND flash) early in the boot process. Now I was trying to get ppp (for a GSM/GPRS connection) working. But the chat-script hangs forever, it does not get a timeout. Turns out, chat uses alarm(2) to wait for a timeout. After quite some time of debugging I found out that SIGALRM is blocked (look for SigBlk in /proc/pid/status, this is the mask of blocked signals for process with pid).

After some googling I came across a blog entry (look for September 5, 2011) that describes a bug in bash: Bash, when calling “read” with a timeout, will install a signal handler for SIGALRM and longjump out of the signal handler, leaving SIGALRM blocked forever.

The boot-script runs as init (with pid 1) and therefore will leave SIGALRM blocked for all children. My immediate workaround is to read the filesystem to boot in a sub-shell…seems the bug is fixed with newer versions of bash.

LILO “junk in compressed archive” and Grub2 “out of disk” error

Monday, January 10th, 2011

I’ve recently upgraded a Debian/Asterisk Installation on an older Soekris Net 4501 embedded hardware. After the upgrade the device didn’t boot anymore. The installed LILO bootloader produced the error message:

Initramfs unpacking failed: junk in compressed archive

and later failed with a kernel panic


VFS: Unable to mount root fs on unknown-block(0,0)

Googling for the error message found a blog entry that indicated a missing LILO option “large-memory” — which wasn’t the problem in my case, the option was already present.
So I booted into GRML and installed grub2 instead of LILO. After a reboot, Grub2 ended up in rescue mode with the error message:

error: out of disk.
grub rescue>

I could display the partitions with ls and also get a listing from my root partition but trying to list other directories produces the “out of disk” error again:


grub rescue> ls
(hd0) (hd0,msdos1) (hd1) (hd2) (hd3) 
...
grub rescue> ls (hd0,msdos1)/
./ ../ lost+found/ var/ etc/ media/ initrd.img usr/
...

grub rescue> ls (hd0,msdos1)/boot

error: out of disk.

So my diagnosis was that the BIOS of the Soekris box is unable to address the whole (in this case 60GB) harddisk. The install had only worked before because the kernel and the /boot directory where within the BIOS-accessible area on the disk. So there are still BIOSes out there that don’t support large harddisks — I didn’t find out until now what the limit of the Soekris BIOS is. The fix after this diagnosis was easy: Shrink the root filesystem using resize2fs, create a new root partition and copy the shrunken filesystem there, resize2fs to the new partition size. Create a new small /boot partition at the start of the disk and copy the contents of the old /boot directory there. Of course this is only possible with a working rescue system, my rescue system of choice is a GRML netboot setup which enables me to quickly boot any x86-based system that supports network boot.
Lesson learned: For small X86 embedded hardware it still makes sense to have a small /boot partition.

Unix Domain Sockets

Tuesday, November 9th, 2010

I recently had to find a solution for a communication problem: An application running on a web-server should update configuration files that are only readable by a privileged user and these should not be directly writeable by the web-server user.
So the idea was to write an update-server running under the privileged account which receives update requests (and can perform additional checks) from the unprivileged web server user.
One of the checks I wanted to make was that only the web-server user (www-data on debian) should be able to send update requests. So I had to find out the user sending a request via the Unix-domain socket. Google found a nice socket howto on Henning Makholm’s blog which told me most of what I needed to know: “so I ended up just checking the EUID of the client process after the connection has been accept()ed. For your reference, the way to do this is getsockopt() with SO_PEERCRED for Linux”.
But one issue was remaining: I didn’t need a SOCK_STREAM socket but wanted to send datagrams to the other side (and didn’t want to fiddle with implementing my own datagram layer on top of a stream socket). With normal SOCK_DGRAM datagram sockets there is no connection — and therefore I can’t determine the user sending the datagram from the other side of the socket.
Looking further I discovered that Linux has connection-oriented datagram sockets for quite some time under the name SOCK_SEQPACKET. With this type of socket you first connect() to the other side and then you send a datagram. Since now there is a connection the trick with SO_PEERCRED as described above works, too.
Code for Server (python):


from socket import socket, SOCK_SEQPACKET, AF_UNIX, SOL_SOCKET
from struct import unpack
try :
    # Not implemented in python 2.6, maybe higher
    from socket import SO_PEERCRED
except ImportError :
    SO_PEERCRED = 17 # Linux
sock = socket (AF_UNIX, SOCK_SEQPACKET)
path = '/path/to/socket'
try :
    os.remove (path)
except OSError :
    pass
sock.bind (path)
conn, adr = self.sock.accept ()
ucred = conn.getsockopt (SOL_SOCKET, SO_PEERCRED, 12)
pid, uid, gid = unpack ('LLL', ucred)
if uid... check uid:
    error...
    conn.close ()

data = conn.recv (4096)

Code for client (python):


from socket import socket, SOCK_SEQPACKET, AF_UNIX
s = socket (AF_UNIX, SOCK_SEQPACKET)
s.connect ('path/to/socket')
s.send (.....)
s.close ()

Skypes Flux Capacitor: UDP

Wednesday, October 6th, 2010

I recently wrote about the broken network obfuscation code (aka Flux Capacitor) of Skype published by Sean O’Neil. At the time I wasn’t able to decrypt UDP packets. Now I’ve looked a little more closely into the Vanilla Skype documentation — which also includes some code to decrypt Skype credentials on harddisk. This code contains a CRC implementation called CRC32. I had wrongly asumed that CRC32 of Skype would be the same as the crc32 implementation of pkzip, Ethernet, png, the POSIX cksum command etc. which is listed as “crc32″ in the CRC article on Wikipedia and which is standardized in e.g. IEEE 802.3.
The crc32 from the standards above inverts all the bits of the seed before using it (it uses an XOR mask of 0xFFFFFFFF) and does this again before returning the result of the CRC computation to the caller. But it uses the same polynomial as skype. So we can use an existing standard CRC implementation (e.g from the zlib library) as follows for computing the skype CRC:

def skype_crc (s, seed = 0xFFFFFFFF) :
    return (crc32 (s, seed ^ 0xFFFFFFFF)) ^ 0xFFFFFFFF

With this crc implementation I’m now able to also decrypt UDP (see updated code) packets. I’ve shown this some days ago at my talk @linuxwochenende, for slides see my events page.

Linuxwochenende 2010

Tuesday, September 21st, 2010

Dieses Wochenende ist wieder Linuxwochenende im metalab, ich werde einen Vortrag zu Skype und dem vor kurzem veröffentlichten Flux Capacitor code und einen zweiten zu Traffic Shaping unter Linux halten.

Skypes Flux Capacitor has been released

Wednesday, August 25th, 2010

Skype uses an obfuscation layer for making it harder for others to analyze the Skype network traffic. This obfuscation layer has been called “Flux Capacitor” by the authors of the 2006 publications “Silver Needle in the Skype” and “Vanilla Skype”, Philippe Biondi, Fabrice Desclaux, and Kostya Kortchinsky, see the Wikipedia article on Skype for references about their publications. They discovered this flux capacitor code when reverse engineering the Skype binary program but didn’t dare to publish it for fear of others attacking the Skype network. They hinted at all sorts of bad things that were possible when knowing the Skype prototocol, the least of which is scanning networks behind firewalls (Skype is well known to be able to connect to the internet even in the presence of firewalls).
Now this Flux Capacitor code has been released by a group called “Skype Reverse Engineering Team” in a blog post Skype’s Biggest Secret Revealed by Sean O’Neil claiming that the code was already leaked and used by spammers. But they didn’t tell us how to use that code and defer further information to the next Chaos Communication Congress 27C3.
I certainly hope that the security holes in Skype are not that bad as feared by others. Some good can come out of it: maybe we’ll see a free Skype client in the not-too-far future. A good plan now would be to write a Wireshark dissector for Skype so that we can analyze the network traffic. Unfortunately the authors didn’t tell us yet how to call that code to de-obfuscate Skype traffic. But there is plenty of information in the aforementioned publications. So I’ve written a Makefile to create a shared library from the now released code and a Python wrapper that decrypts a single packet from a Skype network dump called SkypeIRC.cap published on the Wireshark page. This at least proves that the code works — the first several bytes of a Skype TCP stream decrypt to a known value.
Maybe others want to use this as a starting point before more is released by the authors at the Chaos Communication Congress 27C3.

How to get smartcards or crypto-tokens running on Debian Linux and Windows

Saturday, December 5th, 2009

This is a step-by-step Howto on installation and usage of the necessary commands to get smartcards or crypto-tokens working on Linux — and on Windows.
Smart card software on Linux consists of a driver for the card reader also called card terminal and a driver for the smartcard or token. There are two projects, OpenSC and MUSCLE. Both projects produce drivers for card terminals (for OpenSC this sub-project is called OpenCT) and for various cards. There are configuration options to make one project see the drivers of the other and vice versa.
I’m using OpenCT (card terminal) and OpenSC (smart card) software because these have worked better for me — except for the pcscd daemon from the MUSCLE project that might be needed for some applications. If you’re using software that depends on a PC/SC interface, you’ll also need to install the pcscd daemon (comes as a debian package).
Manufacturers are continuously improving their smartcards and the applications that run on the card. Up to now this has created an ever-changing pile of middleware to talk to these cards. An idea to improve the situation was a programmable card: Put a Java interpreter on the card and provide applets (so-called “cardlets”) for applications. The idea claims this would improve the situation for middleware on the host. Unfortunately this isn’t quite true: The Java virtual machine on smartcards is subject to change. So we have cards with a recent JVM and old cards with an ancient (1.1 from 2001) JVM. This means the applet needed to make the card work has to be cross-compiled to an older JVM depending on the card used. And there are still a number of host configuration issues.
The situation is worsened by the fact that many manufacturers of smartcards provide cards with their proprietary cardlet in the ROM of the card. This means these cards cannot be used with the applet that matches your middleware. And we’re back to the start and have to write a driver for the proprietary (this time java) middleware as before.
If you’re using Java-based smartcards you need to get a card that can accept a new applet and has the necessary developer keys published. Usually these cards are called “developer version” or similar. You then can use the MCardApplet from the MUSCLE plugins project. I recommend to build the applet yourself — after all this is a security application where your security depends on the software running on the card. There is a section below for preparing a java card with the appropriate applet.
I got the eToken PRO 32k (4.2B) working with OpenSC but not the eToken PRO 72k (Java). The first uses Siemens CardOS, the number in parentheses denotes the version number. Unfortunately these old CardOS based tokens will be discontinued, and Siemens seems to have announced end of life status for its CardOS.
The 72k eToken from Aladdin uses Java and needs a Java applet. The OpenSC project claims that the Java token works with OpenSC, but it didn’t for me. Maybe the token wasn’t a developer version that can accept an applet. The token also didn’t work under windows (although it worked with the Aladdin software there, i.e., I could initialize it) and I had a recent OpenSC binary release for windows from October 2009.
From the OpenSC documentation: “For programmable java cards: the support is for the card with the specific applet – if you have a different applet on your card, it might not be supported.”
First steps for non-Java cards or Java-cards with MCardApplet installed (for installation of the applet, see below):
Install smartcard software and card terminal framework:


apt-get install pcscd opensc libengine-pkcs11-openssl
The following NEW packages will be installed:
  libengine-pkcs11-openssl libp11-0

If your smartcard reader is not supported by pcscd, you should install the openct package in addition to the packages above. You can then run pcscd on top of openct.
After installing these programs, we need to configure pcscd to correctly use our reader. Configure /etc/opensc/opensc.conf to use the pcscd method, search for reader_drivers inside the app default stanza of the config file and modify this to:

reader_drivers = pcsc;

For CCID compatible readers without openct installed, the reader should be visible after plugging in the reader into the USB port:

Readers known about:
Nr.    Driver     Name
0      pcsc       Gemplus GemPC Key 00 00

If the card reader is not directly supported by pcscd (but is supported by openct) we need to configure /etc/reader.conf.d/openct, a config file of pcscd, to use OpenCT as the input method, for this the file should contain:

FRIENDLYNAME     "OpenCT"
DEVICENAME       /dev/null
LIBPATH          /usr/lib/openct-ifd.so
CHANNELID        0

after modifying this file on Debian (on non-Debian systems you’ll probably have to modify /etc/reader.conf directly) we have to run:

update-reader.conf

which creates a new /etc/reader.conf. Now stop and restart openct and pcscd in the following order:

/etc/init.d/pcscd stop
/etc/init.d/openct restart
/etc/init.d/pcscd start

When everything went OK, you should see a pcscd based reader using opensc-tool:

opensc-tool -l
Readers known about:
Nr.    Driver     Name
0      pcsc       OpenCT 00 00

Add user to “scard” group in /etc/group and log in again to have the permissions to access the card. This is necessary if you want to use a reader provided by openct without going through pcscd.
If all else fails, you can still use openct without pcscd and set reader_drivers in /etc/opensc/opensc.conf to:

reader_drivers = openct;

Then a reader should be visible like this:


opensc-tool -l
Readers known about:
Nr.    Driver     Name
0      openct     Aladdin eToken PRO 64k
1      openct     OpenCT reader (detached)
2      openct     OpenCT reader (detached)
3      openct     OpenCT reader (detached)
4      openct     OpenCT reader (detached)

Now we should be able to read the card, if there is only one token in the USB port and no other smartcard readers are installed, we can leave out the -r option (for specifying the reader to use):


cardos-info
3b:f2:18:00:02:c1:0a:31:fe:58:c8:09:75
Info : CardOS V4.2B (C) Siemens AG 1994-2005
Chip type: 123
Serial number: 27 37 c0 09 2b 18
Full prom dump:
33 66 00 45 CB CB CB CB 7B FF 27 37 C0 09 2B 18 3f.E....{.'7..+.
00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
OS Version: 200.9 (that's CardOS M4.2b)
Current life cycle: 32 (administration)
Security Status of current DF:
Free memory : 1024
ATR Status: 0x0 ROM-ATR
Packages installed:
Ram size: 4, Eeprom size: 32, cpu type: 66, chip config: 63
Free eeprom memory: 27686
System keys: PackageLoadKey (version 0xfe, retries 10)
System keys: StartKey (version 0xff, retries 10)
Path to current DF:

1.   Using Java-based Smartcards
Java-based smartcards need some preparation to work with OpenSC. First you should make sure that you have a card that permits you to upload your own applet (cardlet) to the card. This usually means you need a so-called developer version. Avoid cards that have a proprietary manufacturer applet in read-only memory! You also usually need some development keys to upload a cardlet to the card.
For the following instructions I’ve used the Gemalto TOP IM FIPS CY2 (Cyberflex Access 64k v2) which is an old card that has a Java virtual machine version 1.1 from 2001 (!) but the card is well established and will probably be available for some time.
For the card reader I’ve used the Gemalto USB Shell V2 (GemPC Key), a CCID compatible reader. When buying a new reader, you should use one that follows the CCID specification from the PC/SC Workgroup, this ensures that OpenCT will support it. The card reader I’m using can read ID-000 format cards, also called SIM cards. These are small form-factor cards that are nice for authentication purposes. The Cyberflex card I’m using is available in ID-000. If a card isn’t available in ID-000, chances are you can make it fit using a service similar to this SIM cutting service.
Preparing the card for use with OpenSC is a two-step process. The first step is to obtain the necessary applet for the card. The second is to upload the applet to the card and initialize the card with a default PIN. Note that 2048 bit keys may not work with MCardApplet with the current software versions, see Some notes on key sizes.

1.1.   Obtaining the MCardApplet for your card
I recommend building your own version of the applet from source. Get the source code from svn://svn.debian.org/muscleplugins/trunk/MCardApplet using the source code control tool subversion:


svn co svn://svn.debian.org/muscleplugins/trunk/MCardApplet

I had to make several changes to use Java 1.6 for cross-building for the ancient Java VM that runs on the card. My Cflex.properties looks like this:

CARD_NAME=Cflex
JAVA_BUILD_HOME=/usr
JC_HOME=${basedir}/depends/jc212
API_JAR=${JC_HOME}/lib/api21.jar
API_EXPORT_FILES=${JC_HOME}/api21_export_files
CAPTRANS=${basedir}/depends/jc212/bin/captransf.jar
VM_SOURCE_VERSION=1.2
VM_TARGET_VERSION=1.1
BOOTCLASSPATH=${basedir}/depends/jdk1.2.2/depends/jdk1.2.2/lib

In particular, I’m using the native java environment installed on debian lenny. This lives in /usr. Otherwise you should follow the excellent building instructions in the file INSTALL. Then I’ve modified the javac in the target named compile in the common.xml ant build file as follows:

<target depends="precompile" name="compile">
    <mkdir dir="${OUTPUT_DIR}"/>
    <javac debug="on"
        verbose="on"
        fork="true"
        executable="${JAVA_BUILD_HOME}/bin/javac"
        srcdir="${APPLET_SRC}"
        destdir="${OUTPUT_DIR}"
        target="${VM_TARGET_VERSION}"
        source="${VM_SOURCE_VERSION}">
        <bootclasspath>
            <pathelement location="${BOOTCLASSPATH}"/>
        </bootclasspath>
        <classpath>
            <pathelement location="${API_JAR}"/>
            <pathelement location="."/>
            <pathelement path="${java.class.path}"/>
        </classpath>
    </javac>
</target>

I’ve added the target and source options. These refer to the specific java virtual machine version we’re cross-building for. I’ve also added the bootclasspath to get the classes that are VM specific from the old java development package.
This allows me to cross-build the applet for an old version of the java virtual machine without running the old java development environment which does no longer run on recent versions of Linux due to library incompatibilities.
I also had to change all the backslashes in common.xml to forward slashes. It looks like the java compiler can handle the backslashes in path names, but the other tools cannot.
Before building you should also look through CflexCapabilities.properties for any capabilities you want to enable which are not enabled in the default configuration. I had to enable -DWITH_RSA_2048 and -DWITH_SIGN for example.
For the impatiant I’m offering a pre-compiled version of the applet. I’ll update this text with the build instructions soon.

1.2.   Uploading the applet to the card
As a preparation for this step, you should install the pcsc daemon and the pcsclite development packages on debian:


apt-get install pcscd libpcsclite-dev

For uploading the applet to the card, I recommend using gpshell, a tool from the globalplatform project which depends on the globalplatform library. Unfortunately these are not yet packaged as Debian packages, so obtain the sourcecode to both, gpshell and the globalplatform library . Unpack these packages and build using the normal process:

./configure
make
make install

For the globalplatform library you need the development libraries for libpcsclite, available as the Debian package libpcsclite-dev which should be installed before the ./configure step above.
If you installed the globalplatform library without root privileges, you have to run ldconfig as root to make the new library available in the shared library cache.
Check that your smartcard services are running (see above for how to do that) and verify that you see your card reader (or token) using:


opensc-tool -l

The gpshell tool interfaces to the card via pcscd, so you should see something similar to the following:

Readers known about:
Nr.    Driver     Name
0      pcsc       Gemplus GemPC Key 00 00

Now we can proceed to upload the applet using gpshell. We need the following applet upload gpshell script, put this into the file applet_install.gpshell, note that lines terminated with “” need to be concatenated, gpshell currently doesn’t understand continuation lines:

enable_trace
establish_context
card_connect
select -AID a000000003000000
open_sc -security 1 -keyind 0 -keyver 0 
   -mac_key 404142434445464748494a4b4c4d4e4f 
   -enc_key 404142434445464748494a4b4c4d4e4f
delete -AID a00000000101
delete -AID a000000001
delete -AID a0000003230101
delete -AID a00000032301
install_for_load -pkgAID a000000001 -nvCodeLimit 16000 
   -sdAID a000000003000000
load -file CardEdgeCflex.ijc
install_for_install -instParam 00 -priv 02 -AID a00000000101 
   -pkgAID a000000001 -instAID a00000000101 -nvDataLimit 32000
card_disconnect
release_context

and run the script with gpshell:

gpshell applet_install.gpshell

Finally we will have to set a default PIN for the card using opensc-tool:

opensc-tool -s 00:A4:04:00:06:A0:00:00:00:01:01 -s 
B0:2A:00:00:38:08:4D:75:73:63:6C:65:30:30:04:01:08:
30:30:30:30:30:30:30:30:08:30:30:30:30:30:30:30:30:
05:02:08:30:30:30:30:30:30:30:30:08:30:30:30:30:30:
30:30:30:00:00:17:70:00:02:01

This sets the PIN to eight zeros, “00000000”. After this procedure the card can be used like a normal PKCS#15 based card with OpenSC.

1.2.1.   Some notes on using the gpshell script with other cards
Many of the magic numbers in the gpshell script depend on the card in use and on the applet that is uploaded. An AID is an applet-ID. The open_sc command has nothing to do with OpenSC, but opens a secure channel to the card. The parameters are the developer keys of the card. In our example, the key is used twice and the hex-bytes denote the string:

@ABCDEFGHIJKLMNO

The AID used for the secure channel is different for other brands of cards. I was able to find out this AID using the following AID gpshell script:

enable_trace
establish_context
card_connect
open_sc -security 1 -keyind 0 -keyver 0 
   -mac_key 404142434445464748494a4b4c4d4e4f 
   -enc_key 404142434445464748494a4b4c4d4e4f
get_status -element 80
card_disconnect
release_context

This displays for the Gemalto TOP IM FIPS CY2:


List of applets (AID state privileges)
a000000003000000        7       0

The delete commands remove older versions of the applet — there also was a version that had another applet ID — from the card before uploading the new version. Think of the applet living in a package named a000000001 and the applet named a00000000101. The install_for_load command establishes the package inside the a000000003000000 package. Then the install_for_install command installs the applet into non volatile memory after upload.
Note that these numbers are hard-coded into the applet that is being uploaded. This is defined during compile-time and for the MCardApplet this can be configured in the common.xml ant-file. When packaging the compiled java classes, the build process generates .cap files which contain the applet ID in binary format, notably the Applet.cap file contains the AID of the applet.
So if these numbers are changed when building a custom applet, be sure to adapt the upload commands, too. A warning here: All tools that depend on the MCardApplet expect the given AID, so changing the AID would incur a change of all tools or at least their configuration.

2.   OpenVPN Preparation: Step by step process with high-level tool
This section describes how to initialize a token, create a user key and certificate using tools that come with OpenVPN. This was only tested on Linux since the certificate handling scripts for OpenVPN are more advanced on Linux.
I had to patch the pkitool of OpenVPN to use the pkcs15-init command instead of the pkcs11-tool command for initializing the token and creating keys. The reason is that Aladdin limits their tokens to use one key for only one purpose (encryption or signing). The pkcs15-init command allows the specification of the key purpose while the pkcs11-tool command does not.
In the following we assume you are in the easy-rsa directory of OpenVPN, and you have initialized the configuration by reading the configuration in vars. It’s also always a good idea to have a backup of all the OpenVPN keys before starting.
Initializing the token for first use (or re-using an already formatted token) THIS DESTROYS ALL DATA ON THE TOKEN:


./pkitool --pkcs15-init 0 "Thomas Mustermann"
New User PIN.
Please enter User PIN:
Please type again to verify:
Unblock Code for New User PIN (Optional - press return for no PIN).
Please enter User unblocking PIN (PUK):
Please type again to verify:

If the token was already initialized, the procedure would also ask for the old password.
Now we can check that everything worked, by looking at the PINs:


pkcs15-tool --list-pins

Now we can proceed to generate a key pair and a certificate, this takes a long time:


./pkitool --pkcs15 /usr/lib/opensc-pkcs11.so 0 45 "user cert" client27

When everything went OK, we should be able to display keys, public keys, and certificates on the token:


pkcs15-tool --list-keys
pkcs15-tool --list-public-keys
pkcs15-tool --list-certificates

3.   What is needed for Windows
I’ve first installed the latest OpenVPN installer, only this package has the necessary TAP network driver for Linux. Only the TAP driver needs to be installed.
I’ve used a compiled version of OpenVPN that comes bundled with OpenSC from the binary windows repository, I’ve used the latest

opensc-i686-w32-mingw32-007-*.*

files (from Oct 2009). For 64bit Windows (AMD64) the


opensc-x86_64-pc-mingw32-003*.*

files should be used. Everything unpacked from these files should be installed to


C:ProgramsOpenVPN

I’ve also created a configuration directory under that directory called config. In config there are the necessary certificates (root ca) and hash keys and the client.ovpn configuration file.
When using my OpenVPN admin script, Python for windows from python.org and the Python WIN32 package are needed. In addition my rsclib library needs to be installed. For installing rsclib, unpack the .zip file and run:

setup.py install

from a command prompt. The admin script in pyovpn.zip has the same installation procedure.
After installing everything, basic working of the smart card can be verified with the OpenSC tools:


opensc-tool -l
cardos-info
pkcs15-tool --list-certificates

3.1.   Aladdin Token
For using the Aladdin eToken with Windows, the smart card tools of the manufacturer have to be installed — these include the driver for accessing the tokens. I’ve used:


PKIClient-x32-5.00.msi

from


eToken PKI Client 5.0 SP1 Windowsx32.zip

3.2.   Gemalto USB Shell Token V2
The driver for the Gemalto USB Shell Token reader can be downloaded free of charge from the Gemalto website.

4.   Using OpenVPN with SmartCard
OpenVPN uses a PKCS#11 provider library, on Debian this is /usr/lib/opensc-pkcs11.so, to access the smart card. We can show which certificates are on the card by issuing:

openvpn --show-pkcs11-ids /usr/lib/opensc-pkcs11.so
The following objects are available for use.
Each object shown below may be used as parameter to
--pkcs11-id option please remember to use single quote mark.

Certificate
      DN:             /C=AT/ST=AT/L=Weidling/ ...
      Serial:         03
      Serialized id:  OpenSCx20Project/PKCSx20 ...

Now OpenVPN can be started with the smartcard. I’m using the following additional config entries for OpenVPN — pkcs11-providers is set to the path of the PKCS#11 provider library:

pkcs11-providers "/usr/lib/opensc-pkcs11.so"
pkcs11-id-management
management 127.0.0.1 4711
management-query-passwords
management-hold

For Windows the provider library becomes (assuming the OpenSC tools where installed to C:ProgramsOpenVPNbin):


pkcs11-providers "C:\Programs\OpenVPN\bin\opensc-pkcs11.dll"

The other parameters are the same as for other operating systems.
Note also that the askpass option of OpenVPN does not work for querying the token password. In my experiment I had to give the token password to OpenVPN via the management interface using the config option management-query-passwords.
This configuration tells OpenVPN to open the management interface on Port 4711 of localhost. It will ask for the passphrase of the Aladdin token on that port. Additionally we wait (management-hold) until a management program has opened the management interface and told OpenVPN to proceed.
The pkcs11-id-management tells OpenVPN to accept the pkcs11-id to use via the management interface. With a little intelligence in the management interface we can avoid having the pkcs11-id in the configuration file (which would then be different for each user).
OpenVPN can be asked via the management interface about a listing of all pkcs11-ids. If there is only one certificate on the card, we feed the id of the only certificate back to OpenVPN when it asks about the pkcs11-id. This can be done without user intervention.
Unfortunately the feature that OpenVPN can ask the smartcard for all the certificates depends on the Token being present when OpenVPN is started. An alternative is to specify the pkcs11-id` directly in the configuration file. If this is specified, the management interface will ask for the token if it isn’t present when starting OpenVPN. The parameter to the pkcs11-id config parameter is the Serialized id from the command output above.
Since I have not found any graphical user interface programs for OpenVPN that can deal with asking the user for the token passphrase, I’ve written a little command-line python script which can be run on both, Windows and Linux, and will ask the user to insert the token and specify the token passphrase.
OpenVPN can be started with the script by issuing the command:


pyovpn

in a command prompt window. The command will start openvpn and proceed to ask for necessary token passwords.

4.1.   Low-level usage of the management interface
With the management-query-passwords option, OpenVPN will ask the password from the management interface. After starting the OpenVPN daemon and connnecting to the managment interface on the defined port (e.g., using telnet) we see the following message:

>INFO:OpenVPN Management Interface Version 1 -- type 'help' for more info
>PASSWORD:Need 'OpenSC Card (New User) token' password

The password can then be entered using the password command of the managment interface:


password 'OpenSC Card (New User) token' sehrgeheimespasswort

OpenVPN answers with:


SUCCESS: 'OpenSC Card (New User) token' password entered, but not yet verified

5.   Key Revocation
Revoke a certificate:


openssl ca  -config $KEY_CONFIG -revoke keys/02.pem

(re-) generate key revocation list (CRL):


openssl ca  -config $KEY_CONFIG -gencrl -out keys/crl.pem

6.   OpenVPN Preparation: Step by step process with low-level tools
This section describes the low-level tools as used by the patched pkitool from OpenVPN. It does not create a certificate with OpenSSL. This is for documentation purposes only (to understand what goes on behind the scenes) and was my first approach to getting started with OpenSC.

  • Erase the card — if you already have a Security Officer PIN installed, you need this for erasing the card:
    
    pkcs15-init -E
    
  • Create PKCS#15 structure (option –no-so-pin specified not to create security officer PIN and user PUK) on the card — note that the keys need to be between 6 and 8 characters long and should be numeric if you intend to use a keypad for key-entry (which applies mostly to smartcards not to USB tokens). Also note that if you omit –no-so-pin you should keep the Security Officer PIN secure — only with it can the token be re-formatted:
    
    pkcs15-init --create-pkcs15 --no-so-pin
    
    New Security Officer PIN (Optional - press return for no PIN).
    Please enter Security Officer PIN:
    Please type again to verify:
    Unblock Code for New User PIN (Optional - press return for no PIN).
    Please enter User unblocking PIN (PUK):
    Please type again to verify:
    
  • The steps above can be rolled into one command:
    
    pkcs15-init -E --create-pkcs15 --no-so-pin
    
  • Add an authentication (user) ID to the token with corresponding PIN and PUK:
    
    pkcs15-init --store-pin --auth-id 01 --label "User Name"
    
    New User PIN.
    Please enter User PIN:
    Please type again to verify:
    Unblock Code for New User PIN (Optional - press return for no PIN).
    Please enter User unblocking PIN (PUK):
    Please type again to verify:
    
  • Check that everything worked, by looking at the PINs:
    
    pkcs15-tool --list-pins
    
    PIN [Security Officer PIN]
        Com. Flags: 0x3
        ID        : ff
        Flags     : [0xB2], local, initialized, needs-padding, soPin
        Length    : min_len:6, max_len:8, stored_len:8
        Pad char  : 0x00
        Reference : 1
        Type      : ascii-numeric
        Path      : 3f005015
    
    PIN [User Name]
        Com. Flags: 0x3
        ID        : 01
        Flags     : [0x32], local, initialized, needs-padding
        Length    : min_len:4, max_len:8, stored_len:8
        Pad char  : 0x00
        Reference : 3
        Type      : ascii-numeric
        Path      : 3f005015
    
  • Now we can generate an RSA key on the card. Note that we could also import a PKCS-12 key onto the card, but the more secure option is to let the card generate the key (so the key will never be available outside the card). The split-key option actually generates two key-pairs, one for encryption and one for signing. There are some labelling options to attach names to the generated keys but these aren’t needed if you want only one key. You could specify an application profile with the –id option, if this isn’t given the default ID 45 (authentication purposes) is used. The command will ask for the Security officer PIN, then for the User PIN, then again for the Security officer PIN (!):
    
    pkcs15-init --generate-key rsa/2048 --auth-id 01 --split-key
    

    Note that with Java smartcards the keysize 2048 might not work with the current version of the tools, see Some notes on key sizes.

  • Now we can verify that the key was actually stored on the card:
    
    pkcs15-tool --list-keys
    
    Private RSA Key [Private Key]
        Com. Flags  : 3
        Usage       : [0x4], sign
        Access Flags: [0x1D], sensitive, alwaysSensitive, neverExtract, local
        ModLength   : 2048
        Key ref     : 16
        Native      : yes
        Path        : 3f005015
        Auth ID     : 01
        ID          : 45
    
  • List public keys:
    
    pkcs15-tool --list-public-keys
    
    Public RSA Key [Public Key]
        Com. Flags  : 2
        Usage       : [0x4], sign
        Access Flags: [0x0]
        ModLength   : 2048
        Key ref     : 0
        Native      : no
        Path        : 3f0050153048
        Auth ID     :
        ID          : 45
    
  • For the following steps we will need a minimum openssl config file:
    
    openssl_conf            = openssl_init
    
    [ openssl_init ]
    engines                 = engine_section
    
    [ req ]
    default_bits            = 2048
    default_keyfile         = privkey.pem
    distinguished_name      = req_distinguished_name
    
    [ req_distinguished_name ]
    countryName                     = Country Name (2 letter code)
    countryName_default             = AT
    countryName_min                 = 2
    countryName_max                 = 2
    stateOrProvinceName             = State or Province Name (full name)
    stateOrProvinceName_default     = Austria
    localityName                    = Locality Name (eg, city)
    localityName_default            = Vienna
    0.organizationName              = Organization Name (eg, company)
    0.organizationName_default      = example.com
    organizationalUnitName          = Organizational Unit Name (eg, section)
    organizationalUnitName_default  = IT-Department
    commonName                      = Common Name (eg, server's hostname)
    commonName_max                  = 64
    emailAddress                    = Email Address
    emailAddress_max                = 40
    
    [ engine_section ]
    pkcs11 = pkcs11_section
    
    [ pkcs11_section ]
    engine_id = pkcs11
    dynamic_path = /usr/lib/engines/engine_pkcs11.so
    MODULE_PATH = /usr/lib/opensc-pkcs11.so
    init = 0
    
  • Generate a certificate request (CSR) from this key using openssl:
    
    CLIENT=newclient
    openssl req -days 3650 -new -out $CLIENT.csr -config openssl.cnf 
      -engine pkcs11 -keyform engine -key 0:45 -sha1
    

    This will ask for all the certificate parameters. Alternatively these can be specified using the -subj option of openssl (this is broken into several lines but should be assembled into one line) with the following parameter:

    /C=AT/ST=Austria/L=Vienna/O=example.com/OU=IT-Department
    /CN=$CLIENT/emailAddress=user@example.com
    
  • You can view the contents of the CSR using:
    
    openssl req -in $CLIENT.csr -noout -text
    
  • Now sign the certificate request with whatever tools you are using. With the pkitool of OpenVPN this would become (after having copied the certificate request newclient.csr to the keys directory):
    
    pkitool --sign newclient
    

    The message about not being able to access newclient.key can be ignored.

  • You may want to delete old certificates from the token:
    
    pkcs15-init --delete-objects cert --id=45
    
  • Then import the new certificate onto the token:
    
    pkcs15-init --store-certificate newclient.crt --id=45
    
  • Reading a certificate from the token and output with openssl:
    
    pkcs15-tool --read-certificate 45 | openssl x509 -noout -text
    


6.1.   Some notes on key sizes
Using 2048 bit keys should generally be possible with newer cards. I had some problems with these key sizes using my Java smartcard, though.
After uploading the applet to the card, it is possible to create a key with either 1024 or 2048 bit. This indicates that the card is able to handle the large keysize. When trying to generate a CSR using openssl the command failed with:

[opensc-pkcs11] iso7816.c:99:iso7816_check_sw: No precise diagnosis
[opensc-pkcs11] muscle.c:745:msc_compute_crypt_process: returning with:
                Card command failed
[opensc-pkcs11] muscle.c:840:msc_compute_crypt: returning with:
                Card command failed
[opensc-pkcs11] card-muscle.c:749:muscle_compute_signature:
                Card signature failed: Card command failed
[opensc-pkcs11] sec.c:53:sc_compute_signature: returning with:
                Card command failed
[opensc-pkcs11] pkcs15-sec.c:273:sc_pkcs15_compute_signature:
                sc_compute_signature() failed: Card command failed
20808:error:0E06D06C:configuration file routines:NCONF_get_string:no value:
conf_lib.c:329:group=req_attributes name=unstructuredName_min
20808:error:0E06D06C:configuration file routines:NCONF_get_string:no value:
conf_lib.c:329:group=req_attributes name=unstructuredName_max
20808:error:8000A005:Vendor defined:PKCS11_rsa_sign:General Error:
p11_ops.c:97:
20808:error:0D0C3006:asn1 encoding routines:ASN1_item_sign:EVP lib:
a_sign.c:276:

Worse, after reinitializing the card in this state with:


pkcs15-init -E --create-pkcs15 --no-so-pin
pkcs15-init --store-pin --auth-id 01 --label "User Name"

and trying to generate a smaller key, this fails with:


pkcs15-init --generate-key rsa/1024 --auth-id 01 --split-key
[pkcs15-init] iso7816.c:102:iso7816_check_sw: Unknown SWs; SW1=9C, SW2=03
[pkcs15-init] muscle.c:557:msc_generate_keypair: returning with:
              Card command failed
[pkcs15-init] card.c:678:sc_card_ctl: returning with:
              Card command failed
[pkcs15-init] pkcs15-muscle.c:272:muscle_generate_key:
              Unable to generate key
[pkcs15-init] pkcs15-muscle.c:273:muscle_generate_key: returning with:
              Card command failed
Failed to generate key: Card command failed

I was able to fix this by re-downloading the applet onto the card. This also happens the other way round, when generating a 1024 bit key first and trying — after reinitializing the card with pkcs15-init — to generate a 2048 bit key.
So I recommend to stick with 1024 bit keys with the current software versions (debian lenny).

PGAPy: Genetische Algorithmen mit Python

Sunday, October 25th, 2009

[Dies ist der Abstract meines Vortrags am Linuxwochenende im Metalab, die alte Version des Abstracts ist inzwischen von der Linuxwochen Homepage verschwunden…]

PGAPy ist ein Python-Wrapper für PGAPack, eine der vollständigsten Bibliotheken für genetischen Algorithmen. Die Python-Bibliothek eignet sich für eigene Experimente mit genetischen Algorithmen, aber auch für die Implementierung von kompletten Anwendungen.

Der Vortrag gibt eine kurze Einführung in genetischen Algorithmen mit Beispielen in Python. Vorgestellt wird u.a. ein Programm zum automatischen Erzeugen der bekannten Sudoku Zahlenrätsel. Dabei wird schrittweise die Bewertungsfunktion für ein Sudoku entwickelt.

Configuring Asterisk to log CDR records via ODBC to a remote MS-SQL

Tuesday, August 18th, 2009

In the following I’ll describe how to setup asterisk to log via ODBC to a remote Microsoft SQL server — I needed this for a client. I’m using a Debian server, file location may differ for your brand of linux distribution.
The following packages exist for debian, the ones needed are marked with “NEEDED”
A good introduction to the Free TDS implementation of ODBC is the userguide of FreeTDS

Debian FreeTDS packages

  • freetds-common – configuration files for FreeTDS SQL client libraries: NEEDED
  • freetds-dev – MS SQL and Sybase client library (static libs and headers)
  • gda2-sybase – FreeTDS backend plugin for GNOME Data Access library for GNOME2
  • libct4 – libraries for connecting to MS SQL and Sybase SQL servers, needed only for sqsh for testing, gets automatically installed with sqsh.
  • libdbd-freetds – Freetds database server driver for libdbi
  • libsybdb5 – libraries for connecting to MS SQL and Sybase SQL servers
  • sqlrelay-freetds – SQL Relay FreeTDS (Sybase and MS SQL Server) connection daemon
  • sqsh – commandline SQL client for MS SQL and Sybase servers depends on libct4: Good for testing
  • tdsodbc – ODBC driver for connecting to MS SQL and Sybase SQL servers “This package includes the ODBC driver for FreeTDS, for use with UnixODBC or iODBC.”: NEEDED

FreeTDS Needs either unixodbc or iodbc, both are ODBC implementations for Linux/Unix. Asterisk is built against unixodbc.
iodbc packages:

  • iodbc – GTK+ config frontend for the iODBC Driver Manager
  • libiodbc2 – iODBC Driver Manager
  • libiodbc2-dev – iODBC Driver Manager (development files)

unixodbc packages:

  • unixodbc – ODBC tools libraries depends on odbcinst1debian1: NEEDED
  • unixodbc-bin – Graphical tools for ODBC management and browsing
  • unixodbc-dev – ODBC libraries for UNIX (development files)

common packages for debian:

  • odbcinst1debian1 – Support library and helper program for accessing odbc ini files: NEEDED

Asterisk depends on unixodbc which should be already installed, so we install the following packages (ignoring any warnings about already-installed packages):


apt-get install unixodbc sqsh tdsodbc

Other Software using ODBC

  • Python seems to have its own access module for MS-SQL:
    python-pymssql – Python database access for MS SQL server and Sybase

Configuraton for Asterisk Logging

  • /etc/freetds/freetds.conf
    
    [global]
            # TDS protocol version
    ;       tds version = 4.2
    
            # Whether to write a TDSDUMP file for diagnostic purposes
            # (setting this to /tmp is insecure on a multi-user system)
    ;       dump file = /tmp/freetds.log
    ;       debug flags = 0xffff
    
            # Command and connection timeouts
    ;       timeout = 10
    ;       connect timeout = 10
    
            # If you get out-of-memory errors, it may mean that your client
            # is trying to allocate a huge buffer for a TEXT field.
            # Try setting 'text size' to a more reasonable limit
            text size = 64512
    
    [logserver]
            host = 172.23.23.4
            port = 1433
            tds version = 8.0
    
  • /etc/odbcinst.ini
    
    [FreeTDS]
    Description = FreeTDS ODBC driver for MSSQL
    Driver = /usr/lib/odbc/libtdsodbc.so
    Setup = /usr/lib/odbc/libtdsS.so
    
  • /etc/odbc.ini
    
    [ODBC Data Sources]
    logserver = MSSQL Log-Server for Asterisk
    
    [logserver]
    description = MSSQL Log-Server for Asterisk
    driver      = /usr/lib/odbc/libtdsodbc.so
    servername  = logserver
    language = us_english
    trace = no
    tracefile = /root/mssql.trace
    
  • /etc/asterisk/cdr_odbc.conf
    
    [global]
    dsn=logserver
    username=asterisk
    password=VERYSECRET
    loguniqueid=yes
    dispositionstring=yes
    table=cdr              ;"cdr" is default table name
    usegmtime=no             ; set to "yes" to log in GMT
    
  • Test using isql

    # isql logserver asterisk "VERYSECRET" -v
    +---------------------------------------+
    | Connected!                            |
    |                                       |
    | sql-statement                         |
    | help [tablename]                      |
    | quit                                  |
    |                                       |
    +---------------------------------------+
    SQL> select * from cdr;
    [output of current cdr table]
    

Impressum/Kontakt