Planet SLUG

description Planet SLUG - http://planet.slug.org.au/
language en
link http://planet.slug.org.au/

Simon "Horms" Horman: An Introduction to Open vSwitch

http://horms.org/pleb_blossom/permalink/2011/2011-06-03T11_17_23.shtml

Yesterday afternoon I made a presentation introducing Open vSwitch at Linux Con Japan. This is an update on the presentation of the same title that I made at the Netfilter Workshop 2011.

slides

James Purser: Cultural Cringe 2011

http://jamespurser.com.au/blog/cultural-cringe-2011

"So enough about us, what do you think about us?"

Okay, how many of you reading this have watched question time in federal parliament?

Right.

How many of you think that question time is in fact a useless piece of perfomance theatre designed not to ask genuine questions, but to score points and futher political goals.

Right (please note I am assuming that there are large numbers of people agreeing with the above statements. I can't actually see who's raising their hands).

So given that this state of affairs has been going on for years and that in fact question time has become the political equivelant of an american sit-com (complete with laugh track, crap acting and a predictable plot), why is it suddenly news when someone from outside says this?

Iain Dale, a political blogger and radio presenter is over from the UK doing a bit of speachifying and presenting on things politechnical. On arrival he decided to watch a bit of question time to compare it against the UK system.

He was gobsmacked, and he tweeted it:

"As Dame Edna might say, I say this in a caring manner, but Question Time in the Australian Parliament is absolutely shameful today."

And of course no one was surprised. Most of the replies he got from Australian followers agreed with him and that, really was that.

Except it wasn't.

The local media jumped on his comments as if he was the child who dared declare the emperor's clothing was a little minimalist. This despite the fact that every sitting day you can get hundreds of people declaring exactly the same sentiments on twitter or hell you could go down to the pub and get pretty much the same views expressed in much more earthy tones.

I've met with Iain (I was invited to a dinner with him on Tuesday night) he's an interesting guy with a passion for politics, and he does have interesting ideas about politics and technology. However the only difference between his comments about question time and many, many others is the fact that he comes from far away.

Of course the Iain Dale fuss isn't the only example of cultural cringe that we've been subjected to over the past couple of weeks. The Economists series of articles spawned dozens of self-important articles about how Australia is percieved in the world, despite the fact that by and large, the Economist essentially stated what we already know.

For crying out loud, as a country we're 111 years old, you'd think by now that we'd have outgrown the desperate need for outside validation, but apparently not.

Lindsay Holmwood: On assholes, ideas, and actions

http://feedproxy.google.com/~r/AuxesisMusings/~3/SaA5V2cixgk/

Almost 2 weeks ago Rusty Russell wrote about assholes in the Open Source community.

Some people intepreted Rusty's post as a tacit toleration of assholes, with Matt Zimmerman commenting on the greader post I shared:

If we didn't tolerate assholes in our community, we would still be producing great software.

I don't believe Rusty's point is that "tolerance of assholes is a necessary evil". His point is that people of all walks of life are flawed, and the tech community is far from being a utopian exception.

The distinction is between ideas and actions.

I have no problem with people holding ideas or beliefs I find offensive or plain wrong. It's that lack of homogeny that makes life interesting and keeps me thinking. It becomes an issue if said person takes action on those ideas and that action hurts other people.

Hence, if you're being an asshole to me or someone else I won't tolerate that and will call you out on it. If you believe that the world is flat or that fasting cures cancer I'll vigorous argue against your position but I won't wield the ban hammer on the work you do.

James Polley: Running multiple instances of Chrome on Mac/Linux

http://feedproxy.google.com/~r/zhasper_slugworthy/~3/q0Rq67PpdaE/

Sometimes it’s handy to be able to have multiple browser instances open at once. For instance, Google’s Multiple Login only allows me to have 3 accounts signed in at once, which isn’t enough for me to have all the personal accounts I want to check plus my work account. Even if it could, I like to keep my personal and work search and browsing histories separate, so that it’s easier for me to find something I vaguely remember seeing recently.

When doing web development, it’s often handy to have one browser signed into the site as an admin, another signed in as a regular user, and one not signed in. Chrome’s “Incognito Window” feature can help with one of these, but you can’t have two Incognito windows at the same time (at least, not on Mac/Linux — I hear tell that the Windows version may have supported multiple incognito sessions at some point, but I don’t know if that’s still the case)

So.

I’ve created a little script. I call it chrome and it lives in ~/bin on all my machines. It detects the platform and calls the appropriate binary.

More importantly, it takes one (optional) parameter, which it uses to figure out which profile to run.

I usually start my day by running this script twice: once as chrome work and once as chrome personal. The order is significant, as clicking on urls in other applications will result in them being opened in the first profile that ran. So, while I’m at work I want most things to open in the work profile; if I’m not working I want a different default behaviour.

If you don’t pass a parameter, the script will invoke the default profile — the one that gets used if you don’t specify a profile at all.

I’ve put the script on github for your amusement and pleasure (and hardcore forking action).

James Purser: Carbon Cate a smoke screen

http://jamespurser.com.au/blog/carbon-cate-smoke-screen

What do the following headlines have in common?

'Carbon Cate' Blanchett tells Aussies to pay up over carbon charge

Screams the Daily Telegraph

Will you listen to Carbon Cate?

Asks the Herald Sun

Cate Blanchett under fire for new carbon tax commercial

Declares the Adelaide Advertiser

Yup, thats right:

Someone asked why people were bothering getting worked up over such a cut dried example of newspaper creptitude. We know it's going to happen, it's not going to affect us, so why should we get all het up?

If we don't who will?

 

This is the same sort of crap that we saw during the election where we'd spend days talking about whether Julia Gillard should have worn earrings, or discussing why she doesn't carry a flapping hand bag. We could be having a serious debate about the most effective way to deal with carbon and being able to transition to a post hydrocarbon economy, but instead we're bitching about how much money an actress has, and whether this makes her unsuited to talk about a policy.

To make this round of "attack the person not the policy" even more moronic, Cate Blanchett actually appears to practice what she preaches:

So there you have it. The actress who News Ltd has declared to be "out of touch" and "elitist", and have trashed for daring to tell "average australians" (sorry are those the ones earning more than $150,000?) that a carbon tax might actually be a good idea, is actually someone who knows what they're talking about.

And meanwhile the policy debate sinks slowly into the mire, just as was intended in the first place.

Pia Waugh: Twitter Weekly Updates for 2011-05-29

http://pipka.org/blog/2011/05/29/twitter-weekly-updates-for-2011-05-29/

Andrew Cowie: Using tinc VPN

http://blogs.operationaldynamics.com/andrew/software/research/using-tinc-vpn.html

We’ve been doing some work where we really needed “direct” machine to machine access between an number of staff and their local file servers. The obvious way to approach this sort of thing is to use a Virtual Private Network technology, but which one?

There are a lot of VPN solutions out there. Quite a number of proprietary ones, and of course the usual contingent of “it’s-free-except-that-then-you-have-to-pay-for-it”. In both cases, why anyone would trust the integrity of code they can’t review is quite beyond me.

We’ve used OpenVPN for some of our enterprise clients, and it’s quite robust. Its model excels at giving remove users access to resources on the corporate network. Technically it is implemented by each user getting a point-to-point connection on an internal network (something along the lines of a 10.0.1.0/30) between the user’s remote machine and a gateway server, and then adding routes to the client’s system to the corporate IP range (ie good old 192.168.1.0/24). That’s fine so long as the assumption is that all the servers on the corporate network have the gateway as their default route, then reply packets to 10.0.1.2 or whatever will just go do default and be sent back down the rabbit hole. Gets messy with things like Postgres if your remote developers need access to the databases; in the configs you do need to add eg 10.0.1.0/24 to the list of networks that the database will accept connections from.

Anyway, that’s all fairly reasonable, and you can set up the client side from NetworkManager (install Debian package network-manager-openvpn-gnome) which is really important too. Makes a good remote access solution.

Peer to Peer

But for our current work, we needed something less centralized. We’re not trying to grant connectivity to a remote corporate network; we’re trying to set up a private network in the old-fashioned frame-relay sense of the word — actually join several remote networks together.

Traditional VPN solutions route all the traffic through the secure central node. If you’ve got one system in NSW and another in Victoria, but the remote access gateway is in California, then despite the fact that the two edges are likely less than 50 ms away direct path, all your traffic is going across the Pacific and back. That’s stupid.

A major complication for all of us was that everyone is (of course) stuck behind NAT. Lots of developers, all working remotely, really don’t need to send all their screen casts, voice conferences, and file transfer traffic into the central corporate network just to come all the way out again.

The 1990s approach to NAT implies a central point that everyone converges to as a means of getting their packets across the port address translation boundary. Things have come a long way since then; the rise of peer-to-peer file sharing and dealing with the challenges of internet telephony has also helped a great deal. Firewalls are more supportive and protocols have evolved in the ongoing attempt to deal with the problem.

Meet tinc

So the landscape is different today, and tinc takes advantage of this. According to their goals page, tinc is a “secure, scalable, stable and reliable, easy to configure, and flexible” peer-to-peer VPN. Uh huh. Because of its peer-to-peer nature, once two edges become aware of each other and have exchanged credentials, they can start sending traffic directly to each other rather than through the intermediary.

$ ping 172.16.50.2
PING 172.16.50.2 (172.16.50.2) 56(84) bytes of data.
64 bytes from 172.16.50.2: icmp_req=1 ttl=64 time=374 ms
64 bytes from 172.16.50.2: icmp_req=2 ttl=64 time=179 ms
64 bytes from 172.16.50.2: icmp_req=3 ttl=64 time=202 ms
64 bytes from 172.16.50.2: icmp_req=4 ttl=64 time=41.6 ms
64 bytes from 172.16.50.2: icmp_req=5 ttl=64 time=45.4 ms
64 bytes from 172.16.50.2: icmp_req=6 ttl=64 time=51.3 ms
64 bytes from 172.16.50.2: icmp_req=7 ttl=64 time=43.3 ms
64 bytes from 172.16.50.2: icmp_req=8 ttl=64 time=42.3 ms
64 bytes from 172.16.50.2: icmp_req=9 ttl=64 time=44.2 ms
...
$

This is with the tincd daemons freshly restarted on each endpoint. The first packet clearly initiates edge discovery, key exchange, and setup of the tunnels. It, and the next two packets, are passed across the Pacific to the central node. Ok, fine. But after that, the tunnel setup completes, and both edge nodes have been informed of the peer’s network addresses and start communicating directly. Nice.

See for yourself

Watching the logs under the hood confirms this. If you run tincd in the foreground then you can specify a debug level on the command line; I find “3” a good setting for testing:

# tincd -n private -D -d3
tincd 1.0.13 (May 16 2010 21:09:47) starting, debug level 3
/dev/net/tun is a Linux tun/tap device (tun mode)
Executing script tinc-up
Listening on 0.0.0.0 port 655
Ready
Trying to connect to einstein (1.2.3.4 port 655)
Trying to connect to newton (5.6.7.8 port 655)
...

If you give it SIGINT by pressing Ctrl+C then it’ll switch itself up to the exceedingly verbose debug level 5, which is rather cool. SIGQUIT terminates, which you can send with Ctrl+\. If you’re not running in the foreground (which of course you’d only be doing in testing),

# tincd -n private -kINT

does the trick. Quite handy, actually.

Performance is respectable indeed; copying a 2.8 MB file across the Pacific,

$ scp video.mpeg joe@einstein.sfo.example.com:/var/tmp

gave an average of 31.625 seconds over a number of runs. Doing the same copy but sending it over the secure tunnel by addressing the remote machine by its private address,

$ scp video.mpeg joe@172.16.50.1:/var/tmp

came in at an average of 32.525 seconds. Call it 3% overhead; that’s certainly tolerable.

Setup

Despite my talking above about joining remote networks, an important and common subcase is merely joining various remote machines especially when those machines are both behind NAT boundaries. That’s our in-house use case.

The tinc documentation is fairly comprehensive, and there are a few HOWTOs out there. There are a few gotchas, though, so without a whole lot of elaboration I wanted to post some sample config files to make it easier for you to bootstrap if you’re interested in trying this (install Debian package tinc).

tinc has a notion of network names; you can (and should) organize your files under one such. For this post I’ve labelled it the incredibly original “private“. Note that when you specify host names here they are not DNS hostnames; they are just symbolic names for use in control signalling between the tinc deaemons. Flexibility = Complexity. What else is new. Obviously you’d probably use hostnames anyway but administration of the tinc network doesn’t need to be co-ordinated with people naming their laptop my-fluffy-bunny or some damn thing. Anyway, on system labelled hawking I have:

hawking:/etc/tinc/private/tinc.conf

    Name = hawking
    AddressFamily = ipv4
    ConnectTo = einstein
    ConnectTo = newton
    Interface = tun0

Note that I’ve got an Interface statement there, not a Device one. That’s a bit add odds with what the documentation said but what I needed to make it all work. Only one ConnectTo is actually necessary, but I’ve got one server in California that is reliably up and one in Victoria that is not so I just threw both in there. That’s what your tincd is going to (compulsively) try to establish tunnels to.

hawking:/etc/tinc/private/hosts/hawking

    Subnet = 172.16.50.31/32

Somewhat confusingly, you need a “hosts” entry for yourself. Above is what you start with. Each host also needs a keypair which you can generate with:

# tincd -n private -K4096

with /etc/tinc/private/rsa_key.priv getting the private key and the public key being appended to the hosts/hawking file:

    Subnet = 172.16.50.31/32

    -----BEGIN RSA PUBLIC KEY-----
    MIICCgKCAgEAzSd5V91X6r3NB3Syh2FV8/JC2M7cx6o2OKbVzP6X5SFPI1lEH1AD
    7SfIlQF4TE++X8RcpJaBi4KjMS/Ul36Tuk75eKA18aNTBoVqH/ytY0BipQvJ6TUd
    BEkCjYrOUHFYOQn8MxQzziG6nk9tvhTWS0yKCNbd68e5i9uyKOem3R/pJsd/Kh9V
    wdVB51Wxs1Sv07OYmGYyRmGWh450wBNEmQfPHmM60Yh6uoQNJ0Ef41k1ZcswWcfO
    0jp9EOvbW/ZCdBW6teIYZ3GMuMB/cFj0Dw2fx6dHNHZVZrPcivt0cuOG8L4jNoHj
    HQUGuzMrpDN8N1ymM/eDlx+kBFYreKiEYGoWWqlZPNoY+bCekMrNf6Sr9bBwbj23
    xmY1jf6v1LkxGtOi4wWJfbU4xaMnquIRQe6FtB4LHp29l2SYWcpZnjuLcZ4ZoZLQ
    WK4bb0bUCAI/eYb19JRnfKEwS9MhYaQhZLWAJ3xyOt9u/Kk9KV7vWApxR1f5e2KT
    77A446eQU5aedm8nBDbd+WHqTdklAQ7SdRyYmbD8PoXBd3DGP6dFiURVTy8Wn4gz
    Bn7PMI3zmhfCMtwq/3A/xfyjQY3qesGCmKUwTno3fhv1DScS0rS9TRxZfyxlaOB1
    qjtlU79VhI0UKlha2Fv4XLshQ5dYEutpatpij0NzPYlwiQFphFQKStsCAwEAAQ==
    -----END RSA PUBLIC KEY-----

These are the public identifiers of your system and indeed the remote system in your ConnectTo statement must have a copy of this in its hosts/ directory. For nets of servers we maintain them in Bazaar and share them around using Puppet. Central distribution brings its own vulnerabilities and hassles; for very small team nets we just share around a tarball :).

You don’t need the /32, it turns out, but I left it in here to show you that tincd is effectively trading around network route advertisements, not host address.

hawking:/etc/tinc/private/tinc-up

    #!/bin/sh
    ifconfig $INTERFACE 172.16.50.31 netmask 255.255.255.0

This gets run when the net comes up. You can do all kinds of interesting things here, but the really magic part is assigning a broader /24 network mask than that given the interface in the hosts/hawking file. That means this interface is the route to the network as a whole (not just to a single-attached host on the other side of a point-to-point tunnel, which is what OpenVPN does, leaving the default gateway to sort it all out). Lots of other ways to wire it of course, but one /24 in RFC 1918 land is more than enough. I’ve even heard of some people using Avahi link-local networking to do the addressing.

I could have hard coded tun0 there, I suppose, but they supply some environment variables. Much better.

Now for the California node:

einstein:/etc/tinc/private/tinc.conf

    Name = einstein
    AddressFamily = ipv4
    Interface = tun0
    Device = /dev/net/tun

That one I did need a Device entry. Not sure what’s up there; it’s a server running Stable, so could just be older kernel interfaces. Doesn’t matter.

Note again though that the tinc.conf file doesn’t have a public IP in it or anything. Bit unexpected, but hey. It turns up in the hosts files:

einstein:/etc/tinc/private/hosts/einstein

    Address = 1.2.3.4
    Subnet = 172.16.50.1/32

    -----BEGIN RSA PUBLIC KEY-----
    MIICCgKCAgEAqh/4Pmxy5fXZh/O7NkvebFK0OP+YD8Ph7JvK8RsUn75FY3DXjCCg
    VNRR+kRhnVoKVJcIAuvW7Tbs4fovWELOJbbUbKea8G+HANCgOY5F0rkJVtIAcTCL
    Jg1OelAfhF6yHV4vVgcawafWiMF2CtprveHomCnOwCbGuTDwTBqaUBZ9IOLzU2bx
    ArVA2No9Ks+xaaeSHejYoii3+WT58HUccntmIYkcdBa0uKZSis1XLUwdT7Evr1Ew
    K54RyMMEPC0MUziYZhAA0Qqpz79EzLXAGgQeuFxLjPoW/NbAD0PEBmsdmI5odprp
    t9Tx11v/UuhK2fszYKjM+DF2pYxxrKlOyus58zx5KKJQjjrzazrru5Ny0DNf/E6Y
    uB2kUtt7TCmoZg2CLAbIkyGJEiK+Wy2x2mabGDgicIs422XVslz2EODSI3qqF+f6
    gu+h/vYvjZxglYrL0SxTRV7wkUc+o9OVXMMYPazgPIkwnBeLrEhGL8GS4wDIYu4G
    E89m9UBE0fhVPJyw4QSfdeJZ4PgpJk6SG/7koVsJqr9EZOLp53K7ipnPylUKaRLD
    mcarvoDO6ybCuHUVUsLuzZZStSG8JEEe/8jb/Ex7UNBzJ14Nglqtu0aUZ/tzkrdS
    nPFFhdIwlUctM7sWKVfBugEkWjs3sR+XRVsCjxMrpZX0lXzcw9vhu60CAwEAAQ==
    -----END RSA PUBLIC KEY-----

This file must be on every system in the net (that has a ConnectTo it) — it’s how the edges know where to call. So the same file is copied to hawking:

hawking:/etc/tinc/private/hosts/einstein

    Address = 1.2.3.4
    Subnet = 172.16.50.1/32

    -----BEGIN RSA PUBLIC KEY-----
    MIICCgKCAgEAqh/4Pmxy5fXZh/O7NkvebFK0OP+YD8Ph7JvK8RsUn75FY3DXjCCg
    VNRR+kRhnVoKVJcIAuvW7Tbs4fovWELOJbbUbKea8G+HANCgOY5F0rkJVtIAcTCL
    Jg1OelAfhF6yHV4vVgcawafWiMF2CtprveHomCnOwCbGuTDwTBqaUBZ9IOLzU2bx
    ArVA2No9Ks+xaaeSHejYoii3+WT58HUccntmIYkcdBa0uKZSis1XLUwdT7Evr1Ew
    K54RyMMEPC0MUziYZhAA0Qqpz79EzLXAGgQeuFxLjPoW/NbAD0PEBmsdmI5odprp
    t9Tx11v/UuhK2fszYKjM+DF2pYxxrKlOyus58zx5KKJQjjrzazrru5Ny0DNf/E6Y
    uB2kUtt7TCmoZg2CLAbIkyGJEiK+Wy2x2mabGDgicIs422XVslz2EODSI3qqF+f6
    gu+h/vYvjZxglYrL0SxTRV7wkUc+o9OVXMMYPazgPIkwnBeLrEhGL8GS4wDIYu4G
    E89m9UBE0fhVPJyw4QSfdeJZ4PgpJk6SG/7koVsJqr9EZOLp53K7ipnPylUKaRLD
    mcarvoDO6ybCuHUVUsLuzZZStSG8JEEe/8jb/Ex7UNBzJ14Nglqtu0aUZ/tzkrdS
    nPFFhdIwlUctM7sWKVfBugEkWjs3sR+XRVsCjxMrpZX0lXzcw9vhu60CAwEAAQ==
    -----END RSA PUBLIC KEY-----

Ok, you get the idea with the public keys, but I wanted to emphasize the point it’s the same file. This is what you need to share around to establish the trust relationship and to tell E.T. where to phone home.

The Address entry in the hosts/einstein files spread around is what tells edge nodes which have been configured to ConnectTo to einstein where the real public IP address is. You can use DNS names here, and could play dynamic DNS games if you have to (sure, further decentralizing, but). If you have a few machines capable of being full time central supernodes then you’ll have much better resiliency.

You do not, however, need to share a hosts/ file for every other node on the net! If laptop penrose is already connected in to einstein and has been assigned 172.16.50.142 say, and hawking joins einstein and tries to ping .142, the central node einstein will facilitate a key exchange even though neither hawking nor penrose have each others’ keys, and then get out of the way. Awesome.

And finally, this all works over further distributed topologies. When new nodes join the new edges and their subnets are advertised around to the rest of the net. So if central nodes einstein and curie are already talking, and sakharov joins currie, then traffic from our hawking will reach sakharov via eintstein and currie, and in fairly short order they will have handled key exchange, step out of the way, and hawking will be communicating with sakharov direct peer to peer. Brilliant.

Nothing stopping you from share around (or centrally managing out-of-band) the hosts/ files with the Subnet declarations and the public keys, of course; it’ll save a few round trips during initial key exchange. Up to you how you manage the trust relationships and initial key distribution.

For completeness,

einstein:/etc/tinc/private/tinc-up

    #!/bin/sh
    ifconfig $INTERFACE 172.16.50.1 netmask 255.255.255.0

No surprises there.

Applications

Using tinc to cross arbitrary NAT boundaries has turned out to be supremely useful. I have successfully used this from within my office, over 3G UTMS mobile broadband, at internet cafes around Australia, in airport lounges in the States, and even from beach-side resorts in Thailand. In all cases I was able to join the private network topology. In fact, I now just leave tincd running as a system daemon on my laptop. When I need to talk to one of the file servers, I ping, and it’s there.

One surprising benefit was in getting voice-over-Jabber running again. We had some horrible regressions with audio quality during the Maverick release series of Ubuntu Linux. At one point in our diagnostics we found that the STUN algorithms for local and remote candidate IP detection were preferentially choosing localhost virtual bridges with lower route metrics than the default gateway resulting in routing loops. We brought up tinc and since both parties were on 172.16.50.x, Empathy and Jingle chose those as the “best” network choice. Packet loss problems vanished and the audio quality really improved (it didn’t finally get resolved until we got a Natty base system, tore out the Unity stuff, and got GNOME 3 and Empathy 3.0 on board via ppa:gnome3-team/gnome3 but that’s a separate issue). And as a side-effect we’ve got some ice on our voice channel. Excellent.

I’ve since read about a number of other interesting applications. A frequent use case is not needing encryption. While most people would interpret the “private” in virtual private network as meaning “secure”, in the old days it just meant a custom routing and network topology layered over whatever the underlying physical transport was. One crew running a large farm of servers on cloud provided infrastructure struggled to enable their various distributed nodes to find and talk to each other. So they disabled the encryption layer but used tinc as a means to facilitate do IP-over-IP tunnelling and giving their sys admins a stable set of (private) addresses with which to talk to the machines. They gave a talk at FOSDEM [their slides here] about it.

Also at FOSDEM was a talk by the “Fair VPN” effort, who are looking at improving efficiency of the network when the number of nodes scales into the thousands. Some nodes are “closer” than others so presumably they should be used preferentially; you don’t really need to discover information about every other node in the network on joining, and so on. The fact that they were able to use tinc as a research platform for this is fascinating and a nice kudo.

Next steps

So I’m pretty pleased with tinc, obviously. We’ve had a very positive experience, and I wanted to put a word in. If you’re involved in network engineering or security hardening, then I’m sure they’d welcome your interest.

It would be outstandingly cool if we could work out a NetworkManager plugin to set this up on demand, but that can wait for tinc version 1.1 or 2.0. I gather they’re working on making the key exchange and configuration easier; what I showed above is obviously well thought out and flexible, but there’s no denying it’s a bit cumbersome; there are a fair number of little knobs that need to be just right. A fire-and-forget daemon cross-product with some form of automatic addressing would be brilliant. But on the other hand, when you put network and security in the same sentence there’s a limit to how much you want to happen without any direct influence over the process. As it stands now tinc strikes a good balance there, and is entirely suitable for an environment managed by properly competent sysadmins.

AfC

Updates

  1. Turns out I was wrong about needing the Interface statement. After Dan’s post I tried it without one and tincd worked fine. Then I remembered why I’d done it that way — without an Interface statement the network interface was named for the tinc net label, private in this case. Preferring tun0, I went back to manually forcing it for my own personal aesthetic benefit.

Andrew Cowie: Force Pidgin online

http://blogs.operationaldynamics.com/andrew/software/gnome-desktop/force-pidgin-online.html

The NetworkMananger 0.9 series has made some changes which break current Pidgin. After I installed network-manager 0.8.999 Pidgin won’t connect, stalling with “waiting for network connection”.

Turns out there is a workaround in Pidgin: you can force it to ignore what it thinks network availability by running it as:

$ pidgin -f

There’s no GUI way in gnome-shell to edit a launcher at the moment, fine; The old “edit menus” trick didn’t seem to work either. So to do that manually:

$ cp /usr/share/applications/pidgin.desktop .local/share/applications/
$ vi .local/share/applications/pidgin.desktop

And change the Exec line to:

Exec=pidgin -f

It won’t take effect until the desktop recaches things. Reload gnome-shell by typing “restart” in the Alt+F2 run dialog and you’ll be on your way.

I’m sure upstream will catch up with the NetworkManager changes in due course but I can live without network availability detection for now and this gets me back online.

I’ve been using Empathy for instant messaging for a long time, but I still love Pidgin for IRC. Go figure. So two clients it is.

AfC

Comments

Adam Williamson writes that Fedora’s Pidgin package has correctly bound to NetworkManager 0.9 and submitted patches upstream. Cool.

Sonia Hamilton: Spain – the country that breaks the EU?

http://soniahamilton.wordpress.com/2011/05/23/spain-the-country-that-breaks-the-eu/

From the Wall Street Journal:

MADRID—Weekend elections that threaten to drive Spain’s ruling Socialist party from power in several regions and cities also promise a potentially nasty surprise: the revelation of piles of undisclosed debt in local governments that could undercut the country’s drive to avoid an international bailout

And in Spanish (El Pais): Iceland arrives in Spain at 15/May demo

Arrancó primero Nolesvotes, una iniciativa que insta a evitar llenar las urnas de papeletas de PP, PSOE y CiU acusándoles de aprovecharse de la ley electoral para perpetuarse en el Parlamento con unos “niveles de corrupción en España alarmantes”. … “¡De mayores queremos ser islandeses!”, clamó uno de los animadores de la organización durante la marcha del pasado domingo 15 de mayo ante una columna de jóvenes y no tan jóvenes, padres y niños, estudiantes y trabajadores, parados y jubilados.

And of course the Socialists got wiped out by the PP (right wing) in the weekend elections:

El tsunami del 22-M dejó al PSOE con su peor resultado de la historia en unas elecciones municipales (27,8% de votos), casi 10 puntos y dos millones de votos menos que el PP, que logró duplicar la distancia que sacó aquel histórico 1995 que preludió la derrota socialista en las elecciones generales del año siguiente tras 14 años de poder.

Given that more (empty) houses were built in Spain during the boom years than England, France and Germany combined, and there’s approximately 20% unemployment rate (and up to 50% for those in their 20′s and 30′s), and that the Spanish economy is about four times the size of Greece, the “Spanish situation” may be the straw that breaks the camel’s (EU’s) back.

There’s an old saying that “Europe ends at the Pyrenees” (ie the mountain ranges that divide France and Spain). Perhaps the saying will become “Europe ends at the Pyrenees, Italian Alps, and the mountains of Greece”…


Pia Waugh: Twitter Weekly Updates for 2011-05-22

http://pipka.org/blog/2011/05/22/twitter-weekly-updates-for-2011-05-22/

Lindsay Holmwood: Getting consistent media source access on Boxee with CIFS

http://feedproxy.google.com/~r/AuxesisMusings/~3/48u4lnLJy48/

Although a bit late to the party, I've been using Boxee at home for watching movies and TV shows for almost a year now. Its indexing engine is pretty accurate, and the killer feature has to be the Boxee remote app for the iPhone.

I've tried using XBMC but found it suffers from the common open source problem of exposing all its knobs and dials to make everything infinitely tweakable for the end user (also known as: the KDE user experience). Although the free Boxee release is drifting towards an unmaintained state now Boxee are focusing on the Boxee Box, it's simple and stable enough to use that everyone in my family can get their heads around it in under a few minutes.

The one bug that's consistently annoyed me has been disappearing CIFS shares when a configured media source (a file server) exists on a different subnet to the Boxee itself. This only happened recently when we moved house and put our file server on a separate subnet that's not DHCP serviced.

I scratched my head for a little while on how to solve this. The obvious solution is to shorten the DHCP lease range and put the file server on the same subnet as the Boxee machine, but I'm cautious of what other CIFS bugs lurk beneath the surface, and it feels vastly more complicated than it should be.

The simplest solution I could find was to pass off handling CIFS shares to Linux and add a plain old directory as a media source:

# /etc/fstab: static file system information.
#
# <file system>         <mount point>   <type>  <options>           <dump>  <pass>
proc                    /proc           proc    nodev,noexec,nosuid 0       0
/dev/sda1               /               ext4    errors=remount-ro   0       1
/dev/sda5               none            swap    sw                  0       0
//jules/videos          /media/videos   cifs    guest               0       0

This bypasses Boxees dodgy CIFS code completely, and has the added benefit of abstracting away the source from Boxee, making it vastly easier to shuffle file shares and disks around without Boxee getting upset.

Scott Sinclair: Linux World Map

http://www.dedoimedo.com/computers/linux-world-map.html

Quite well done, in the style of the XKCD Community Map. Big Enough to have as a new desktop Wallpaper too.

Scott Sinclair: IP sets in 2.6.39

http://ipset.netfilter.org/

Apply rules in iptables against a table, instead of adding a rule for every host that applies.

Will simplify rules about 300%. Now if only we could get a full working model of PF in Linux.

Simon "Horms" Horman: SH-Mobile ARM zboot presentation

http://horms.org/pleb_blossom/permalink/2011/2011-05-20T14_58_27.shtml

Hikari This afternoon I made a short presentation at Japan Technical Jamboree 37 the work that I have been doing to allow booting Linux directly on the SH-Mobile ARM platform. This is an update on the presentation on the same topic in January at MobileFOSS Miniconf, part of linux.conf.au Brisbane 2011.

slides, etc...

James Purser: Wollongong in 50 Years

http://jamespurser.com.au/blog/wollongong-50-years

Okay, so the Property Council of Australia's rep in the Illawarra David Lang has called for a discussion on future population targets for the Illawarra region, with his first target on the table being 500,000 by 2050.

The Greens have come back and said no way, it will make Wollongong look like the Goldcoast with 20 storey high apartment blocks blighting the skyline.

To be honest I think both of them have issues.

Firstly, you can't talk about a population target without talking about a vision for the region. Population is but one part of any plan for the Illawarra, and while it is a huge part of the plan it should not be dealt with in isolation.

Secondly if the Greens think that the Illawarra isn't going to hit 500,000 people by the middle of the century I have bad news for them. According to the 2006 census, the Wollongong Local Government Area is tracking a 2.6% annual population growth which gives us almost a million people by 2050. Even if we lower the rate to 1% over the next 39 years we're looking at around 490000 people.

Any discussion has to work from the point that Wollongong and the Illawarra is going to grow, we're simply too close to Sydney for anything else to happen. 

So with that in mind what do we want Wollongong to look like in 2050? Do we want to sacrifice arable farming land so that we don't end up with a coastal skyline resembling the Gold Coast? Are centres like Dapto going to turn into cities in their own right? What sort of infrastructure are we going to need to support whatever vision we agree on for a larger Wollongong?

Is Wollongong even going to continue to be the steel city or will we suffer the same fate as Newcastle? 39 years is a long time but we need to start planning for it now.

James Purser: They do not like it up em

http://jamespurser.com.au/blog/they-do-not-it-em

Dear Media,

Please, get over yourself.

If you want to to play in the political sphere (as opposed to simply reporting it), then you have no right to whinge when someone starts to treat you just like any other political player.

If your editor is, say, going to call for the destruction of a political party, and your media organisation is going to run a consistantly negative line about said party, then you are going to have to expect the following:

Now you could accept the above and move on, knowing full well that these are the natural consequences of diving into the political swamp, or you could splash about lobbing great steaming piles of prose and column inches in a vain effort to try and pretend that you still retain some sense of objective credibility.

Oh and one more thing, get used to the proles taking you to task in real time. Thanks to twitter and facebook, people can pull your dross apart quicker than you can blink and they're not going to stop. This will mean both that the debate will be live and that you are going to have to do a hell of a lot better when it comes to fact checking and subbing your news.

The media world is changing and if you lot don't evolve, someone else is going to replace you.

Scott Sinclair: TermKit - Interesting Idea

http://acko.net/blog/on-termkit

But it's probably not going to work in reality. Would be curious to see it built upon a bit more tho.

Worth checking out for the read.

Sonia Hamilton: Ubuntu 10.10 (Maverick Meerkat) Download

http://soniahamilton.wordpress.com/2011/05/18/ubuntu-10-10-maverick-meerkat-download/

Opinions around the latest Ubuntu (11.04) range from “interesting” (as in the curse “may you live in interesting times”) to “it sucks”.

Anyway, there no longer appears to be a link to the previous version (10.10 Maverick) on the main Ubuntu page – marketing droids seem to have finally taken over Ubuntu. It can be downloaded from here.

No rocket science – I just don’t want to have to look for the location in future.


Scott Sinclair: Using nginx rewrite's to remove the file extension and still work with PHP-FastCGI

http://www.nullis.net/weblog/2011/05/nginx-rewrite-remove-file-extension/

I had been debating trying out nginx for a while, and at the end of last week a project I work on had it's web server go down. I decided this was a good time as any to play with nginx when getting the new box up.

One of the rules we had on the old Apache setup however, was that we re-wrote URL's to remove their .php extension. We achieved this thru a simple rule using mod_rewrite. The rule was:

RewriteCond /var/www/site/%{REQUEST_URI}.php -f
RewriteRule ^/(.+)$ /$1.php [L]

So wanting to re-implement this in nginx, I did a bit of reading of he nginx wiki, and while the examples on the rewrite module are limited, there was enough information to point out that the following would do what we wanted:

rewrite ^/([a-z]+)$ /$1.php last;

However, this rule doesn't play nicely with PHP-FastCGI.

I read a lot of articles, and most of these suggested that the best way to work around this, was to make all requests go to an intermediary PHP script, and have it handle all page requests. This seems a bit silly, and doesn't seem to solve the problem, but work around it.

So to get this to work, we had to look at how PHP-FastCGI works. The default setup for PHP-FastCGI looks something like this:

location ~ \.php$ {
    include /etc/nginx/fastcgi_params;
    fastcgi_pass  127.0.0.1:9000;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/site$fastcgi_script_name;
}

So what this does, is when a page matches the .php extension, it will send it off to FastCGI for processing, and the second last line calculates the full path to the PHP script to run.

Now the problem with my rule is that $fastcgi_script_name is never going to contain the php script name, as because we are rewriting the URL, $fastcgi_script_name doesn't seem to update correctly.

Now, nginx does provide you with the set command in the config file, to set a variable, however, it appears you can't set all variables with it (probably a wise thing), but because we just use it in a string, we can just use another variable we have control over.

But we can't just do a rewrite and set the variable, we have to an if block.

After a bit of playing, this is what I came up with. It does the trick, and works as expected for us, but your mileage may vary.

location / {
    set $page_to_view "/index.php";
    try_files $uri $uri/ @rewrites;
    root   /var/www/site;
    index  index.php index.html index.htm;
}

location ~ \.php$ {
    include /etc/nginx/fastcgi_params;
    fastcgi_pass  127.0.0.1:9000;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/site$page_to_view;
}

# rewrites
location @rewrites {
    if ($uri ~* ^/([a-z]+)$) {
        set $page_to_view "/$1.php";
        rewrite ^/([a-z]+)$ /$1.php last;
    }
}

This works by setting a new variable $page_to_view, and setting the default variable to "/index.php" (i.e. the page to view, if the request is for the root folder, e.g., http://example.nullis.net/).

Then when we get a page that means our rewrite condition, we set it to the page name we expect, and then this is used when setting the FastCGI script name. This means when they go to http://example.nullis.net/info, it actually gives you info.php, without having a proxy PHP page, or multiple rewrite rules.

Given I couldn't find a good example elsewhere, I figured it would be worth documenting this for other people. Let me know if you have a better way to go about this, as I'm not sure this is the best way, but it's definitely better.

If you do have suggestions, then please contact me.

Pia Waugh: Twitter Weekly Updates for 2011-05-15

http://pipka.org/blog/2011/05/15/twitter-weekly-updates-for-2011-05-15/

James Purser: Angry Birds in Chrome all HTML 5 all the time! Except not

http://jamespurser.com.au/blog/angry-birds-chrome-all-html-5-all-time-except-not

So one of the big announcements at this years Google IO was the release of Angry Birds for Chrome. For those of you not in the know, Angry Birds is a game that started out on the iPhone and has come to represent the sort of simple, yet addictive game that has been so successful on mobile devices. Using a catapult you launch a series of birds with different talents at constructions filled with green pigs who have stolen the birds eggs.

Anyway, I was very impressed because it appeared that they had reimplemented Angry Birds entirely using the open HTML5 standard. From what I could see on a cursory glance it was Javascript all the way and all was good.

Except not. Turns out that there is still a bit of flash hanging around. If you disable flash in your chrome you'll be presented and refresh your Angry Birds page you'll be asked to install it. This is because buried down below the javascript and goodness there is this little code segment:

 

<obect id="gwtVoices100" type="application/x-shockwave flash"data="http://chrome.angrybirds.com/angrybirds/gwt-voices.swf">
<param name="FlashVars" value="id=gwtVoices1000">
</object>
 
I don't know what it's meant to do but the app won't work without it.
 
It is slightly disappointing, however I should point out that all the game mechanics appear to pure JS. I would like to know what they felt they couldn't re-implement in HTML5/JS though.

James Purser: Proposed Wollongong Wards Announced

http://jamespurser.com.au/blog/proposed-wollongong-wards-announced

Well, Wollongong Council has announced the proposed new wards and put them up for comment on the Council website.

As you can see from the map (warning its absolutely huge) they've essentially divided the Wollongong LGA in half and the carved a chunk out for the Wollongong metro area, leaving both Ward 1 and Ward 3 with large underpopulated areas.

Let's break down each ward.

Ward 1 (Northern/North West Wollongong):

Population: 46,304

Major towns: Helensburgh, Stanwell Park, Clifton, Wombarra, Austinmer, Thirroull, Bulli, Woonona, Corrimal, Balgownie

Ward Quota: 11576

Map: link

Ward 2 (Metro Wollongong):

Population: 46,263

Major Towns: Wollongong, Figtree, Mount Kembla, Unanderra

Ward Quota: 11565

Map: link

Ward 3 (Southern/South Western Wollongong):

Population: 42610

Major Towns: Dapto, Port Kembla, Primbee, Windang, Berkeley

Ward Quota: 10652

Map: link

Just keep in mind that these aren't set in stone, though as far as I can see, there isn't anything that's really begging to be changed.

So there you have it, in slightly less than 4 months, we're going to be going to the polls. That's not a lot of time to be getting the resources together to run a proper campaign, on the other hand, with the move to proportional representation does open things up for smaller groups and independents.

Matthew Palmer: How to Frustrate Your Train Driver

http://www.hezmatt.org/~mpalmer/blog/general/how_to_frustrate_your_train_driver.html

  1. Find a shift where the meal break is later than average. These are especially common in the afternoon, where there's a lot of trains to drive during the peak to get everyone home. Make sure to execute your fiendish plan on the last run before the meal break.
  2. Buy a big helping of hot chips, making sure to load up on the delicious condiments like malt vinegar.
  3. Sit right behind the driver's cab with your deep fried basket of deliciousness. With a battery operated fan, you can increase deliciousness penetration considerably, but it's something of a "nuclear option".
  4. Put your ear to the cab door to hear the loud growling sounds coming from the driver's stomach.
  5. ...
  6. PROFIT!

The only thing saving me from bursting out the cab door screaming "The chips! I wants them!" was the fact I had a nut bar in my bag. They were particularly delicious smelling chips.

Dave Airlie: glibc - inconsistent interfaces due to arrogance

http://airlied.livejournal.com/74826.html

To start off, I don't actually mind arrogant people as long as they back their attitude up with some semblance of sanity, however arrogance without ability pisses me off, and it seems that its the number 1 trait to be a maintainer of glibc.

Today someone pointed out F_SETOWN_EX to me which I can use to fix a problem in the X server, however trying to use gettid made me realise why these guys are considered such fuckwits.

http://sourceware.org/bugzilla/show_bug.cgi?id=6399

WTF? glibc exposes 3 interfaces that need the result of gettid passed to them and internally relies on gettid to implement raise properly and it won't expose the interface. Notice in the bug he never addresses the problems, just closes it with "CLOSED ARROGANCE".

Other recent glibc fuckwittery includes the whole overlapping memcpy issues in F14 breaking flash for everyone, and trying to remove the rpc implementation a week before the F15 final release freeze, so that if the package had gotten in and packages that needed a rebuild for say a security fix would mean linking the fixed package against a different RPC library than the original which could cause untold pains.

So really if glibc could DIACF already thanks.

Ian Wienand: PLT and GOT - the key to code sharing and dynamic libraries

http://www.technovelty.org/linux/pltgot.html

(this post was going to be about something else, but after getting this far, I think it stands on its own as an introduction to dynamic linking)

The shared library is an integral part of a modern system, but often the mechanisms behind the implementation are less well understood. There are, of course, many guides to this sort of thing. Hopefully this adds another perspective that resonates with someone.

Let's start at the beginning — - relocations are entries in binaries that are left to be filled in later -- at link time by the toolchain linker or at runtime by the dynamic linker. A relocation in a binary is a descriptor which essentially says "determine the value of X, and put that value into the binary at offset Y" — each relocation has a specific type, defined in the ABI documentation, which describes exactly how "determine the value of" is actually determined.

Here's the simplest example:

$ cat a.c
extern int foo;

int function(void) {
    return foo;
}
$ gcc -c a.c 
$ readelf --relocs ./a.o  

Relocation section '.rel.text' at offset 0x2dc contains 1 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000004  00000801 R_386_32          00000000   foo

The value of foo is not known at the time you make a.o, so the compiler leaves behind a relocation (of type R_386_32) which is saying "in the final binary, patch the value at offset 0x4 in this object file with the address of symbol foo". If you take a look at the output, you can see at offset 0x4 there are 4-bytes of zeros just waiting for a real address:

$ objdump --disassemble ./a.o

./a.o:     file format elf32-i386


Disassembly of section .text:

00000000 <function>:
   0:	 55			push   %ebp
   1:	 89 e5                	mov    %esp,%ebp
   3:	 a1 00 00 00 00       	mov    0x0,%eax
   8:	 5d                   	pop    %ebp
   9:	 c3                   	ret

That's link time; if you build another object file with a value of foo and build that into a final executable, the relocation can go away. But there is a whole bunch of stuff for a fully linked executable or shared-library that just can't be resolved until runtime. The major reason, as I shall try to explain, is position-independent code (PIC). When you look at an executable file, you'll notice it has a fixed load address

$ readelf --headers /bin/ls
[...]
ELF Header:
[...]
  Entry point address:               0x8049bb0

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
[...]
  LOAD           0x000000 0x08048000 0x08048000 0x16f88 0x16f88 R E 0x1000
  LOAD           0x016f88 0x0805ff88 0x0805ff88 0x01543 0x01543 RW  0x1000

This is not position-independent. The code section (with permissions R E; i.e. read and execute) must be loaded at virtual address 0x08048000, and the data section (RW) must be loaded above that at exactly 0x0805ff88.

This is fine for an executable, because each time you start a new process (fork and exec) you have your own fresh address space. Thus it is a considerable time saving to pre-calculate addresses from and have them fixed in the final output (you can make position-independent executables, but that's another story).

This is not fine for a shared library (.so). The whole point of a shared library is that applications pick-and-choose random permutations of libraries to achieve what they want. If your shared library is built to only work when loaded at one particular address everything may be fine — until another library comes along that was built also using that address. The problem is actually somewhat tractable — you can just enumerate every single shared library on the system and assign them all unique address ranges, ensuring that whatever combinations of library are loaded they never overlap. This is essentially what prelinking does (although that is a hint, rather than a fixed, required address base). Apart from being a maintenance nightmare, with 32-bit systems you rapidly start to run out of address-space if you try to give every possible library a unique location. Thus when you examine a shared library, they do not specify a particular base address to be loaded at:

$ readelf --headers /lib/libc.so.6
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
[...]
  LOAD           0x000000 0x00000000 0x00000000 0x236ac 0x236ac R E 0x1000
  LOAD           0x023edc 0x00024edc 0x00024edc 0x0015c 0x001a4 RW  0x1000

Shared libraries also have a second goal — code sharing. If a hundred processes use a shared library, it makes no sense to have 100 copies of the code in memory taking up space. If the code is completely read-only, and hence never, ever, modified, then every process can share the same code. However, we have the constraint that the shared library must still have a unqiue data instance in each process. While it would be possible to put the library data anywhere we want at runtime, this would require leaving behind relocations to patch the code and inform it where to actually find the data — destroying the always read-only property of the code and thus sharability. As you can see from the above headers, the solution is that the read-write data section is always put at a known offset from the code section of the library. This way, via the magic of virtual-memory, every process sees its own data section but can share the unmodified code. All that is needed to access data is some simple maths; address of thing I want = my current address + known fixed offset.

Well, simple maths is all relative! "My current address" may or may not be easy to find. Consider the following:

$ cat test.c
static int foo = 100;

int function(void) {
    return foo;
}
$ gcc -fPIC -shared -o libtest.so test.c

So foo will be in data, at a fixed offset from the code in function, and all we need to do is find it! On amd64, this is quite easy, check the disassembly:

000000000000056c <function>:
 56c:		 55			push   %rbp
 56d:		 48 89 e5             	mov    %rsp,%rbp
 570:		 8b 05 b2 02 20 00    	mov    0x2002b2(%rip),%eax        # 200828 <foo>
 576:		 5d                   	pop    %rbp

This says "put the value at offset 0x2002b2 from the current instruction pointer (%rip) into %eax. That's it — we know the data is at that fixed offset so we're done. i386, on the other hand, doesn't have the ability to offset from the current instruction pointer. Some trickery is required there:

0000040c <function>:
 40c:	 55			push   %ebp
 40d:	 89 e5                	mov    %esp,%ebp
 40f:	 e8 0e 00 00 00       	call   422 <__i686.get_pc_thunk.cx>
 414:	 81 c1 5c 11 00 00    	add    $0x115c,%ecx
 41a:	 8b 81 18 00 00 00    	mov    0x18(%ecx),%eax
 420:	 5d                   	pop    %ebp
 421:	 c3                   	ret    

00000422 <__i686.get_pc_thunk.cx>:
 422:	 8b 0c 24		mov    (%esp),%ecx
 425:	 c3                   	ret

The magic here is __i686.get_pc_thunk.cx. The architecture does not let us get the current instruction address, but we can get a known fixed address — the value __i686.get_pc_thunk.cx pushes into cx is the return value from the call, i.e in this case 0x414. Then we can do the maths for the add instruction; 0x115c + 0x414 = 0x1570, the final move goes 0x18 bytes past that to 0x1588 ... checking the disassembly

00001588 <global>:
    1588:       64 00 00                add    %al,%fs:(%eax)

i.e., the value 100 in decimal, stored in the data section.

We are getting closer, but there are still some issues to deal with. If a shared library can be loaded at any address, then how does an executable, or other shared library, know how to access data or call functions in it? We could, theoretically, load the library and patch up any data references or calls into that library; however as just described this would destroy code-sharability. As we know, all problems can be solved with a layer of indirection, in this case called global offset table or GOT.

Consider the following library:

$ cat test.c
extern int foo;

int function(void) {
    return foo;
}
$ gcc -shared -fPIC -o libtest.so test.c

Note this looks exactly like before, but in this case the foo is extern; presumably provided by some other library. Let's take a closer look at how this works, on amd64:

$ objdump --disassemble libtest.so
[...]
00000000000005ac <function>:
 5ac:		 55			push   %rbp
 5ad:		 48 89 e5             	mov    %rsp,%rbp
 5b0:		 48 8b 05 71 02 20 00 	mov    0x200271(%rip),%rax        # 200828 <_DYNAMIC+0x1a0>
 5b7:		 8b 00                	mov    (%rax),%eax
 5b9:		 5d                   	pop    %rbp
 5ba:		 c3                   	retq   

$ readelf --sections libtest.so
Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
[...]
  [20] .got              PROGBITS         0000000000200818  00000818
       0000000000000020  0000000000000008  WA       0     0     8

$ readelf --relocs libtest.so
Relocation section '.rela.dyn' at offset 0x418 contains 5 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
[...]
000000200828  000400000006 R_X86_64_GLOB_DAT 0000000000000000 foo + 0

The disassembly shows that the value to be returned is loaded from an offset of 0x200271 from the current %rip; i.e. 0x0200828. Looking at the section headers, we see that this is part of the .got section. When we examine the relocations, we see a R_X86_64_GLOB_DAT relocation that says "find the value of symbol foo and put it into address 0x200828.

So, when this library is loaded, the dynamic loader will examine the relocation, go and find the value of foo and patch the .got entry as required. When it comes time for the code loads to load that value, it will point to the right place and everything just works; without having to modify any code values and thus destroy code sharability.

This handles data, but what about function calls? The indirection used here is called a procedure linkage table or PLT. Code does not call an external function directly, but only via a PLT stub. Let's examine this:

$ cat test.c
int foo(void);

int function(void) {
    return foo();
}
$ gcc -shared -fPIC -o libtest.so test.c

$ objdump --disassemble libtest.so
[...]
00000000000005bc <function>:
 5bc:		 55			push   %rbp
 5bd:		 48 89 e5             	mov    %rsp,%rbp
 5c0:		 e8 0b ff ff ff       	callq  4d0 <foo@plt>
 5c5:		 5d                   	pop    %rbp

$ objdump --disassemble-all libtest.so
00000000000004d0 <foo@plt>:
 4d0:   ff 25 82 03 20 00       jmpq   *0x200382(%rip)        # 200858 <_GLOBAL_OFFSET_TABLE_+0x18>
 4d6:   68 00 00 00 00          pushq  $0x0
 4db:   e9 e0 ff ff ff          jmpq   4c0 <_init+0x18>

$ readelf --relocs libtest.so
Relocation section '.rela.plt' at offset 0x478 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000200858  000400000007 R_X86_64_JUMP_SLO 0000000000000000 foo + 0

So, we see that function makes a call to code at 0x4d0. Disassembling this, we see an interesting call, we jump to the value stored in 0x200382 past the current %rip (i.e. 0x200858), which we can then see the relocation for — the symbol foo.

It is interesting to keep following this through; let's look at the initial value that is jumped to:

$ objdump --disassemble-all libtest.so

Disassembly of section .got.plt:

0000000000200840 <.got.plt>:
  200840:       98                      cwtl   
  200841:       06                      (bad)  
  200842:       20 00                   and    %al,(%rax)
        ...
  200858:       d6                      (bad)  
  200859:       04 00                   add    $0x0,%al
  20085b:       00 00                   add    %al,(%rax)
  20085d:       00 00                   add    %al,(%rax)
  20085f:       00 e6                   add    %ah,%dh
  200861:       04 00                   add    $0x0,%al
  200863:       00 00                   add    %al,(%rax)
  200865:       00 00                   add    %al,(%rax)
        ...

Unscrambling 0x200858 we see its initial value is 0x4d6 — i.e. the next instruction! Which then pushes the value 0 and jumps to 0x4c0. Looking at that code we can see it pushes a value from the GOT, and then jumps to a second value in the GOT:

00000000000004c0 <foo@plt-0x10>:
 4c0:   ff 35 82 03 20 00       pushq  0x200382(%rip)        # 200848 <_GLOBAL_OFFSET_TABLE_+0x8>
 4c6:   ff 25 84 03 20 00       jmpq   *0x200384(%rip)        # 200850 <_GLOBAL_OFFSET_TABLE_+0x10>
 4cc:   0f 1f 40 00             nopl   0x0(%rax)

What's going on here? What's actually happening is lazy binding — by convention when the dynamic linker loads a library, it will put an identifier and resolution function into known places in the GOT. Therefore, what happens is roughly this: on the first call of a function, it falls through to call the default stub, which loads the identifier and calls into the dynamic linker, which at that point has enough information to figure out "hey, this libtest.so is trying to find the function foo". It will go ahead and find it, and then patch the address into the GOT such that the next time the original PLT entry is called, it will load the actual address of the function, rather than the lookup stub. Ingenious!

Out of this indirection falls another handy thing — the ability to modify the symbol binding order. LD_PRELOAD, for example, simply tells the dynamic loader it should insert a library as first to be looked-up for symbols; therefore when the above binding happens if the preloaded library declares a foo, it will be chosen over any other one provided.

In summary — code should be read-only always, and to make it so that you can still access data from other libraries and call external functions these accesses are indirected through a GOT and PLT which live at compile-time known offsets.

In a future post I'll discuss some of the security issues around this implementation, but that post won't make sense unless I can refer back to this one :)

Nick Jenkins: Economic reform of government housing distortions

http://blog.nickj.org/2011/05/10/economic-reform-of-government-housing-distortions/

Expensive housing is entirely counter-productive. Nobody celebrates when petrol goes up, or when electricity prices rise. So why celebrate when house prices rise so much, that your children have to pay relatively 3 times as much of their income for their first house, as you did for yours?

Unaffordable housing creates a social divide between classes, and between the old and the young. House prices have risen massively since 1985, and right now we have the mother of all housing affordability problems in Australia, and quite possibly one of the world’s few remaining housing bubbles (along with Hong Kong and possibly Canada).

Much of this problem originates with government distortions of free markets, which have made housing a particularly attractive form of investment for borrowed money, which in turn has driven up prices.

Tonight is Australia’s federal budget night, so today seems like an appropriate day on which to talk about those government taxes & incentives & rules.

I would like to see 5 changes made by government in relation to housing, and the rules and taxes governing it, to remove these distortions and get Australian housing closer towards being a free and efficient market, with affordable and sustainable prices:

  1. Eliminate the First Home Owners Grant. This would save the taxpayer in the region of $700 million per year, whilst also removing a market distortion that has achieved the exact opposite its stated aim (by making homes demonstrably less affordable to new buyers).
  2. End the negative gearing tax subsidy. Currently this subsidy costs the Australian taxpayer $4.8 billion per year. “Very few other ”advanced” economies are as generous in their tax treatment of geared investments as Australia is. In the United States, investors can only deduct interest incurred on borrowings undertaken to purchase property or shares up to the amount of income (dividends or rent) earned in any given financial year; any excess of interest expense over income (as in a ”negatively geared” investment) must be ”carried forward” as a deduction against the capital gains tax payable when the asset is eventually sold. In Australia, by contrast, that excess can be deducted against a taxpayer’s other income (such as wages and salaries) thereby reducing the amount of tax otherwise payable on that other income.”
  3. A dollar’s income should be taxed the same, whether it is earned from wages, from interest, from dividends, or from asset price rises. The current tax system penalises productive work far more than passive speculation, as the highest rates are imposed on wages and income from savings; lower rates are imposed on income from investments. In a fair tax system, the 50 % CGT reduction should be eliminated, and CGT would apply to all assets equally, including the family home. All forms of income should be taxed at the individual’s marginal rate. Removing the 50% CGT reduction alone would save the Australian taxpayer close to $10 billion per year.
  4. The “price guides” given by real estate agents for auctions in almost all cases are significantly under the final sale price, and almost always under the reserve, wasting buyer’s time and money. Also known as “under-quoting”, and it is “the difference between the guide given to buyers, and an agent’s sale estimate as written on the sale authority contract. It’s hard for buyers to prove under-quoting because they don’t have access to the contract.” A far more transparent process for sale by auction would be for the reserve price to have to be disclosed 1 or 2 weeks prior to auction. This would still leave the market to determine prices, but would let buyers know what the seller’s lower bound was, prior to having to spend money on building inspections and solicitors, only to then find out that the property was never in their price range. Agents who do not publish the reserve in time or who do not call the property on the market at the reserve price would have their real estate licenses revoked.
  5. The money saved from the above (over $15 billion per year) spent on improving public infrastructure and transport, and reducing or eliminating stamp duty.

In a perfect world, some or all of these reforms would be addressed in tonight’s budget. However our current politicians, from both major parties, are weak-willed poll-driven career-politicians, who are afraid of offending current home-owners. So instead of doing what’s in the economy’s and future home-owner’s best long-term interests, I fully expect more of the same. The longer the current incentives are left in place, the worse the problem will be for Australia when it inevitably corrects.

Update: Just after finishing writing this, I saw Steve Keen make some very similar suggestions today, with good graphs showing private and government debt levels.

Sonia Hamilton: Books are better?

http://soniahamilton.wordpress.com/2011/05/09/books-are-better/

I’ve noticed that most of the time I’m happy with ebooks/webpages for learning a new technology. But when it’s something difficult or something I really want to grok, I like to print things out or best of all buy a book (and then curl up on the couch and just read and annotate).

Maybe I’m not the only one – O’Reilly Radar has an article about a study that suggests university students learn better with paper books. (Let’s ignore the fact that O’Reilly are book publishers…).


Sonia Hamilton: Solaris – max of 100 cron jobs

http://soniahamilton.wordpress.com/2011/05/09/solaris-max-of-100-cron-jobs/

I stumbled across a limitation feature behaviour of Solaris over the weekend – by default it only allows you to run a maximum of 100 cron jobs at once. I suppose this to protect against “shooting oneself in the head”, but aaaarghhh… Reminds me why I love Monday mornings.

Anyway, the file to edit is /etc/cron.d/queuedefs, see man queuedefs.

Thanks to Solaris Tips for pointing me in the right direction.


Sonia Hamilton: git amend

http://soniahamilton.wordpress.com/2011/04/28/git-amend/

A nice git option I stumbled upon today – amend.

Typical scenario – you’ve done a local check-in, then notice there’s a typo in your code comments. Fix up the typo, then use - – amend to re-check-in, without having to do a reset/revert. For example:

git add foo.file
git ci -m "added the frob feature to the whiz-bang"
...
notice and fix typo in foo.file
...
git add foo.file
git ci --amend

When you do the amend check-in, your editor will pop up, allowing you to re-edit the commit message if desired.

Obviously, if you’ve done a commit to a remote/public repository, amend has issues, similar to reset.

Also, I have entries like this in my .gitconfig, hence “git ci” rather than “git commit”.

[alias]
        ci = commit
        co = checkout
        st = status
        br = branch

James Purser: Malaysia, Confusion About Teh Gays and more

http://jamespurser.com.au/blog/malaysia-confusion-about-teh-gays-and-more

Okay this is going to be a bit of a rant so here goes:

Let's Play Refo Swap

What.The.Hell?

No, seriously, let me see if I get this straight. In order to deal with the problem of over crowded detention centres and an increasingly stressed system, our national leaders have decided to send 800 asylum seekers (people who have already entered the process of applying for refugee status), off to Malaysia, a country that while full of truly excellent people, nevertheless has a much larger problem with asylum seekers (90,000 at last count), is NOT a signatory to the Refugee treaties and has a real human rights problem in dealing with asylum seekers.

In return we as a country will be accepting 4000 people that have been assessed as being genuine refugees.

Oh and on top of that it's going to cost Australia 300 million dollars on top of what is already spent on processing asylum seekers under our laws and the treaties we've signed up to.

This isn't a solution, it's a desperate attempt to appear to be doing something while not actually doing anything worthwhile. This is a once off deal while the Government frantically trawls around the pacific (leaving Naeru well alone) to find someone to host their "Regional Processing" centre.

gah!

People do get a little confused

According to Victorian state Liberal member Geoff Shaw, being gay and expecting to be able to show the love for your partner is akin to demanding the right to travel at 150k/h on the roads. This is a new one and does show some originality, for which I can only applaud Geoff however, sadly, he very quickly falls back into the same tired old lines equating a relationship between two loving adults with demanding the right to abuse children.

All that before he even got to the meat of the issue he was supposed to be dealing with (the ability of religious organisations to discriminate even in non religious activities).

Sigh.

Of course this is apparently the same man who objects to the now traditional "Welcome to country" ceremonies (where the original indigenous inhabitants of the land being used are acknowledged) on the grounds that it all belonged to God anyway.

This is getting old. We've been hearing the same tired old arguments for a while now (about 150 years by my count), and yet the doomsayers have proven to be unerringly wrong. When does the message start getting through that allowing gay people to live like everyone else is not going to cause the flame death of the universe.

Denial is not just a river in egypt

This one is specifically for Wollongong people. The news that former Wollongong Council General Manager Rod Oxley intends to run for Lord Mayor in the September election is both not surprising an awesome in its ability to ignore reality.

For those of you not in the know, Rod Oxley was the General Manager of Wollongong Council during the scandal that lead to the elected Council being sacked due to "systemic corruption". While Oxley was never charged himself, the Independent Commission Against Corruption found that he had engaged in "conduct that was liable to allow, encourage or cause the occurrence of corrupt conduct", a damning indictment of his management of an organisation as large and important as Wollongong Council.

Rod is as free to run for Council as anyone else in Wollongong, I'm certainly not trying to take that away from him in any way shape or form. However let's be honest, his chances are slim, and I'm pretty sure he's going to be hard pressed to vote any significant voting block.

Pia Waugh: Twitter Weekly Updates for 2011-05-08

http://pipka.org/blog/2011/05/08/twitter-weekly-updates-for-2011-05-08/

James Purser: Wollongong and Shellharbour Elections

http://jamespurser.com.au/blog/wollongong-and-shellharbour-elections

So, true to his word, Barry O'Farrell has submitted the legislation that will be required to bring forward the election of Councillors for the Wollongong and Shellharbour Local Government Areas.

However there are a number of changes coming to both Councils.

First up, because the elections are going to be brought forward a year, elections for both Councils in 2012 have been cancelled. This means that Councillors will have a 5 year term to bring both Councils back into line with the rest of the state.

Secondly the Premier has decreed a number of changes to the way both Councils select and elect their Councillors.

While there isn't an official change to the way voting is going to occur, the reduction/elimination of wards is going to introduce a defacto proportional voting system. This should open things up a little so that the two big parties don't lock everything up.

Oh and we also have a date for the election. September 3rd. That's the first Saturday in September, so if you're thinking about running for Wollongong or Shellharbour Council, you'd better get started soon, there's only 4 months to go.

Bruce Badger: Yes to AV?

http://openskills.blogspot.com/2011/05/yes-to-av.html

In the UK today we get to vote in a referendum on whenther AV should be adopted in place of first past the post in elections for MPs.

I thought of two situations where AV would be useful to me.

They key is I can submit more information with my single vote using AV.

Some have suggested that AV is a poor compromise and that we should really be aiming to have PR, but I disagree strongly with that. I very much want to retain directly elected MPs who represent specific areas rather than moving to the PR model where I would choose a party and the party would pick the MP(s) who represent me.

As for the information-free mud slinging we have seen by the major parties leading up to this referendum, well, I think we'd be better off just ignoring that.

So, AV seems to let me say more in my single vote while still ending up with a single MP to represent my area in parliament. One person, one vote, one MP. I think I like that.

Sonia Hamilton: LISP – downloading SICP videos, JRuby

http://soniahamilton.wordpress.com/2011/05/05/lisp-downloading-sicp-videos-jruby/

I’m busy learning LISP/Scheme at the moment (in order to later move on to Clojure). And, improving my programming learning LISP is :-)

I’m following along with the MIT SICP videos, but got tired of manually downloading each video. So I wrote a quick little script to download all the videos – get_sicp. Next, a script for the UC Berkeley SICP videos (getting through the flash player should be a learning experience).

I’ve also been getting good mileage out of JRuby. I was complaining a few weeks ago about not being able to run Ruby on production servers (and therefore having to learn Python). Well JRuby is the perfect solution – installing/building Ruby on production servers is often verboten, but most production servers already have a JVM so installing JRuby in ${HOME} is a non-brainer and doesn’t litter /usr. Good: Ruby – fun, Python – <snore>.


Sridhar Dhanapalan: Installing Skype and Java on XOs

http://www.dhanapalan.com/blog/2011/05/05/installing-skype-and-java-on-xos/

We had a query about running Skype and WebEx on XOs.

Getting Skype to work is quite straightforward.

WebEx requires Java, which is trickier.

Note that I only tested to the point of being able to run a test applet in Firefox. If anyone wants to try WebEx, please let me know of your experiences.


2011 Sridhar Dhanapalan.
This work is licensed under a Creative Commons Attribution-Share Alike 2.5 Australia Licence.
Creative Commons BY-SA Licence

.

Silvia Pfeiffer: HTML5 multi-track audio or video

http://blog.gingertech.net/2011/05/01/html5-multi-track-audio-or-video/

In the last months, we’ve been working hard at the WHATWG and W3C to spec out new HTML markup and a JavaScript interface for dealing with audio or video content that has more than just one audio and video track.

This is particularly relevant when a Web page author wants to add a sign language track to a video or audio resource for deaf people, or an audio description track (i.e. a sound track in which a speaker explains the key things that can be seen on screen) for blind people. It is also relevant when a Web page author wants to publish a video with multiple audio tracks that are each a different language dub for the video and can be used for less common cases such as a director’s comment track, or making available different camera angles for an event.

Just to be clear: this is not a means to introduce video editing functionality into the Web browser. If you want to do edits, you’re better off with an application that will eventually render a new piece of content and includes fancy transitions etc. Similarly, this is not a means to introduce mixing functionality (as in what DJs do when they play with multiple audio recordings). You’re better off with an actual audio mixing or DJ application that will provide you all sorts of amazing effects and filters.

So, multi-track is squarely focused on synchronizing alternative or additional tracks to a single resource with a single timeline to which all tracks are slaved.

Two means of publishing such multi-track media content are possible:

1. In-band multi-track

In in-band multi-track, there is a single file that has all all the tracks inside it. For this single file, there is now an API in HTML5 that allows addressing and controlling these tracks.

Of the video file formats that Web browsers support, WebM is currently not defined to contain more than one audio or video track. However, since WebM is using the Matroska container format, which supports multi-track, it is possible to extend WebM for multi-track resources. I have seen multitrack Ogg, MP4 and Matroska files in the wild and most media players support their display.

The specification that has gone into HTML5 to support in-band multi-track looks as follows:

interface HTMLMediaElement : HTMLElement {
  [...]
  // tracks
  readonly attribute MultipleTrackList audioTracks;
  readonly attribute ExclusiveTrackList videoTracks;
};

interface TrackList {
  readonly attribute unsigned long length;
  DOMString getID(in unsigned long index);
  DOMString getKind(in unsigned long index);
  DOMString getLabel(in unsigned long index);
  DOMString getLanguage(in unsigned long index);

           attribute Function onchange;
};

interface MultipleTrackList : TrackList {
  boolean isEnabled(in unsigned long index);
  void enable(in unsigned long index);
  void disable(in unsigned long index);
};

interface ExclusiveTrackList : TrackList {
  readonly attribute unsigned long selectedIndex;
  void select(in unsigned long index);
};

You will notice that every audio and video track gets an index to address them. You can enable() and disable() individual audio tracks and you can select() a single video track for display. This means that one or more audio tracks can be active at the same time (e.g. main audio and audio description), but only one video track will be active at a time (e.g. main video or sign language).

Through the getID(), getKind(), getLabel() and getLanguage() functions you can find out more about what actual content is available in the individual tracks so as to activate/deactivate them correctly and display the right information about them.

getKind() identifies the type of content that the track exposes such as “description” (for audio description), “sign” (for sign language), “main” (for the default displayed track), “translation” (for a dubbed audio track), and “alternative” (for an alternative to the default track).

getLabel() provides a human readable string that describes the content of the track aiming to be used in a menu.

getID() provides a short machine-readable string that can be used to construct a media fragment URI for the track. The use case for this will be discussed later.

getLanguage() provides a machine-readable language code to identify which language is spoken or signed in an audio or sign language video track.

Example 1:

The following uses a video file that has a main video track, a main audio track in English and French, and an audio description track in English and French. (It likely also has caption tracks, but we will ignore text tracks for now.) This code sample switches the French audio tracks on and all other audio tracks off.

<video id="v1" poster=“video.png” controls>
 <source src=“video.ogv” type=”video/ogg”>
 <source src=“video.mp4” type=”video/mp4”>
</video>

<script type="text/javascript">
video = document.getElementsByTagName("video")[0];

for (i=0; i video.audioTracks.length; i++) {
  if (video.audioTracks.getLanguage(i) == "fr") {
    video.audioTracks.enable(i);
  } else {
    video.audioTracks.disable(i);
  }
}
</script>

Example 2:

The following uses a audio file that has a main audio track in English, no main video track, but sign language video tracks in ASL (American Sign Language), BSL (British Sign Language), and ASF (Australian Sign Language). This code sample switches the Australian sign language track on and all other video tracks off.

<video id="a1" controls>
 <source src=“audio_sign.ogg” type=”video/ogg”>
 <source src=“audio_sign.mp4” type=”video/mp4”>
</video>

<script type="text/javascript">
video = document.getElementsByTagName("video")[0];

for (i=0; i video.videoTracks.length; i++) {
  if (video.videoTracks.getLanguage(i) == "asf") {
    video.videoTracks.select(i);
    break;
  }
}
</script>

If you have more tracks in both examples that conflict with your intentions, you may need to further filter your activation / deactivation code using the getKind() function.

2. Synchronized resources

Sometimes the production process of media creates not a single resource with multiple contained tracks, but multiple resources that all share the same timeline. This is particularly useful for the Web, because it means the user can download only the required resources, typically saving a substantial amount of bandwidth.

For this situation, an attribute called @mediagroup can be added in markup to slave multiple media elements together. This is administrated in the JavaScript API through a MediaController object, which provides events and attributes for the combined multi-track object.

The new IDL interfaces for HTMLMediaElement are as follows:

interface HTMLMediaElement : HTMLElement {
  [...]
  // media controller
           attribute DOMString mediaGroup;
           attribute MediaController controller;
};

interface MediaController {
  readonly attribute TimeRanges buffered;
  readonly attribute TimeRanges seekable;
  readonly attribute double duration;
           attribute double currentTime;

  readonly attribute boolean paused;
  readonly attribute TimeRanges played;
  void play();
  void pause();

           attribute double defaultPlaybackRate;
           attribute double playbackRate;

           attribute double volume;
           attribute boolean muted;

           attribute Function onemptied;
           attribute Function onloadedmetadata;
           attribute Function onloadeddata;
           attribute Function oncanplay;
           attribute Function oncanplaythrough;
           attribute Function onplaying;
           attribute Function onwaiting;
           attribute Function ondurationchange;
           attribute Function ontimeupdate;
           attribute Function onplay;
           attribute Function onpause;
           attribute Function onratechange;
           attribute Function onvolumechange;
};

You will notice that the MediaController replicates some of the states and events of the slave media elements. In general the approach is that the attributes represent the summary state from all the elements and the writable attributes when set are handed through to all the slave elements.

Importantly, if the individual media elements have @controls activated, then the displayed controls interact with the MediaController thus allowing synchronized playback and interaction with the combined multi-track object.

Example 3:

The following uses a video file that has a main video track, a main audio track in English. There is another video file with the ASL sign language for the video, and an audio file with the audio description in English. This code sample creates controls on the first file, which then also control the audio description and the sign language video, neither of which have controls. Since the audio description doesn’t have controls, it doesn’t get visually displayed. The sign language video will just sit next to the main video without controls.

<video id="v1" poster=“video.png” controls mediagroup="a11y_vid">
 <source src=“video.webm” type=”video/webm”>
 <source src=“video.mp4” type=”video/mp4”>
</video>

<video id="v2" poster=“sign.png” mediagroup="a11y_vid">
 <source src=“sign.webm” type=”video/webm”>
 <source src=“sign.mp4” type=”video/mp4”>
</video>

<audio id="a1" mediagroup="a11y_vid">
 <source src=“audio.ogg” type=”audio/ogg”>
 <source src=“audio.mp3” type=”audio/mp3”>
</video>

Example 4:

We now accompany a main video with three sign language video tracks in ASL, BSL and ASF. We could just do this in JavaScript and replace the currentSrc of a second video element with the links to BSL and ASF as required, but then we need to run our own media controls to list the available tracks. So, instead, we create a video element for each one of the tracks and use CSS to remove the inactive ones from the page layout. The code sample activates the ASF track and deactivates the other sign language tracks.

<style>
  video.inactive { display: none; }
</style>

<video id="v1" poster=“video.png” controls mediagroup="a11y_vid">
 <source src=“video.webm” type=”video/webm”>
 <source src=“video.mp4” type=”video/mp4”>
</video>

<video id="v2" poster=“sign_asl.png” mediagroup="a11y_vid" class="active">
 <source src=“sign_asl.webm” type=”video/webm”>
 <source src=“sign_asl.mp4” type=”video/mp4”>
</video>

<video id="v3" poster=“sign_bsl.png” mediagroup="a11y_vid" class="inactive">
 <source src=“sign_bsl.webm” type=”video/webm”>
 <source src=“sign_bsl.mp4” type=”video/mp4”>
</video>

<video id="v4" poster=“sign_asf.png” mediagroup="a11y_vid" class="inactive">
 <source src=“sign_asf.webm” type=”video/webm”>
 <source src=“sign_asf.mp4” type=”video/mp4”>
</video>

<script type="text/javascript">
videos = document.getElementsByTagName("video");

for (i=0; i videos.length; i++) {
  if (video[i].videoTracks.getLanguage(0) == "asf") {
    video[i].setAttribute("class", "active");
  } else {
    video[i].setAttribute("class", "inactive");
  }
}
</script>

Example 5:

In this final example we look at what to do when we have a in-band multi-track resource with multiple video tracks that should all be displayed on screen. This is not a simple problem to solve because a video element is only allowed to display a single video track at a time. Therefore for this problem you need to use both approaches: in-band and synchronized resources.

We take a in-band multitrack resource with a main video and audio track and three sign language tracks in ASL, BSL and ASF. The second resource will be made up from the URI of the first resource with a media fragment address of the sign language tracks. (If required, these can be discovered using the getID() function on the first resource.) The markup will look as follows:

<video id="v1" poster=“video.png” controls mediagroup="a11y_vid">
 <source src=“video.ogv#track=v_main&track=a_main” type=”video/ogv”>
 <source src=“video.mp4#track=v_main&track=a_main” type=”video/mp4”>
</video>

<video id="v2" poster=“sign.png” controls mediagroup="a11y_vid">
 <source src=“video.ogv#track=asl&track=bsl&track=asf” type=”video/ogv”>
 <source src=“video.mp4#track=asl&track=bsl&track=asf” type=”video/mp4”>
</video>

Note that with multiple video elements you can always style them in the way that you want them displayed on screen. E.g. if you want a picture-in-picture display, you scale the second video down and absolutely position it on top of the first one in the appropriate location. You can even grab the second video into a canvas, chroma-key your sign language speaker on a green or blue screen and remove that background through some canvas processing before popping it on top of the video.

The world is all yours!

HOWEVER: There is one big caveat on all these specs – while they have all found entry into the HTML5 specification, it would be expecting a bit much to have browser support already. :-)

Pia Waugh: Twitter Weekly Updates for 2011-05-01

http://pipka.org/blog/2011/05/01/twitter-weekly-updates-for-2011-05-01/

Silvia Pfeiffer: WordPress plugin for external videos updated

http://blog.gingertech.net/2011/04/29/wordpress-plugin-for-external-videos-updated/

Over the last weeks I’ve updated my “external videos” wordpress plugin. I’ve fixed bugs and added some new functionality.

List of changes:

Download the new plugin version 0.13

Note: there is something weird going on with the wordpress plugins site, which still shows version 0.7 as the current one, but when you download it, it gets the latest version 0.12. If somebody knows how to fix this, that would be awesome. I think it also stops people from auto-updating this plugin, which is sad with this many improvements.
(I think I fixed it by actually changing the version number in the external-videos.php file – how silly of me – and thanks to the WordPress Forum person who pointed it out to me! Download 0.13 now.)

Sridhar Dhanapalan: Documentary about OLPC Designer Yves Behar

http://www.dhanapalan.com/blog/2011/03/20/documentary-about-olpc-designer-yves-behar/

SBS television recently screened a documentary about Yves Behar, the person behind the distinctive industrial design of the OLPC XO laptop. It’s a fascinating insight into the mind and influences behind one of the most influential designers around. The documentary was originally aired in November 2008, so it is a little dated. For example, Yves talks about the “XOXO” XO-2, which has since been replaced with the XO-3. Nevertheless, it is well worth watching.

You can view the full version at the SBS Web site. The section on OLPC starts at 14:48.

UPDATE: if you are having trouble viewing the video, try this one instead. The attention to detail and quality is astounding. Yves rightly points out that products seen in lesser economically developed countries are normally second hand or second rate. The design is rugged and functional. It provides scope for personalisation. What was most interesting to me is Yves’ commentary on the keyboard. Its one-piece design means that the letters can be printed in one silkscreening process. This makes it feasible to translate into languages that would be uneconomical with a standard keyboard design.


©2011 Sridhar Dhanapalan.
This work is licensed under a Creative Commons Attribution-Share Alike 2.5 Australia Licence.
Creative Commons BY-SA Licence

.

Ian Wienand: Toolchains, libc and interesting interactions

http://www.technovelty.org/linux/libc-symbol-visibility.html

When building a user space binary, the -lc that gcc inserts into the final link seems pretty straight forward — link the C library. As with all things system-library related there is more to investigate.

If you look at /usr/lib/libc.so; the "library" that gets linked when you specify -lc, it is not a library as such, but a link script which specifies the libraries to link, which also includes the dynamic linker itself:

$ cat /usr/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) )

On very old glibc's the AS_NEEDED didn't appear; so every binary really did have a DT_NEEDED entry for the dynamic linker itself. This can be somewhat confusing if you're ever doing forensics on an old binary which seems to have these entries for not apparent reason. However, we can see that that /lib/libc.so itself does actually require symbols from the dynamic linker:

$ readelf --dynamic /lib/libc.so.6 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

Although you're unlikely to encounter it, a broken toolchain can cause havoc if it gets the link order wrong here, because the glibc dynamic linker defines minimal versions of malloc and friends -- you've got a chicken-and-egg problem using libc's malloc before you've loaded it! You can simulate this havoc with something like:

$ cat foo.c
#include <stdio.h>
#include <syslog.h>

int main(void)
{
   syslog(LOG_DEBUG, "hello, world!");
   return 0;
}

$ cat libbroken.so
GROUP ( /lib/ld-linux-x86-64.so.2 )

$ gcc -o -Wall -Wl,-rpath=. -L. -lbroken -g -o foo foo.c

$ ./foo
Inconsistency detected by ld.so: dl-minimal.c: 138: realloc: Assertion `ptr == alloc_last_block' failed!

Depending on various versions of things, you might see that assert or possibly just strange, corrupt output in your logs as syslog calls the wrong malloc. You could debug something like this by asking the dynamic linker to show you its bindings as it resolves them:

$ LD_DEBUG_OUTPUT=foo.txt LD_DEBUG=bindings ./foo
Inconsistency detected by ld.so: dl-minimal.c: 138: realloc: Assertion `ptr == alloc_last_block' failed!

$ cat foo.txt.11360 | grep "\`malloc'"
     11360:	binding file /lib/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `malloc' [GLIBC_2.2.5]
     11360:	binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `malloc' [GLIBC_2.2.5]
     11360:	binding file /lib/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `malloc' [GLIBC_2.2.5]

Above, because the dynamic loader comes first in the link order, libc.so.6's malloc has bound to the minimal implementation it provides, rather the full-featured one it provides internally.

As an aside, AFAICT, there is really only one reason why a normal library will link against the dynamic loader -- for the thread-local storage support function __tls_get_addr. You can try this yourself:

$ cat tls.c
char __thread *foo;

char* moo(void) {
	return foo;
}
$ gcc -fPIC -o libtls.so -shared tls.c
$ readelf -d ./libtls.so  | grep NEED
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux.so.2]
 0x6ffffffe (VERNEED)                    0x314
 0x6fffffff (VERNEEDNUM)                 2

Thread-local storage is worthy of a book of its own, but the gist is that this support function says "hey, give me an address of foo in libtls.so", the magic being that if the current thread has never accessed foo then it may not actually have any storage for foo yet, so the dynamic linker can allocate some memory for it lazily and then return the right thing. Otherwise, every thread that started would need to reserve room for foo "just in case", even if it never cares about moo.

But looking a little closer at the symbols of libc.so is also interesting. libc.so doesn't actually have many functions you can override. You can see what it is possible to override by checking the relocations against the procdure-lookup table (PLT).

Relocation section '.rela.plt' at offset 0x1e770 contains 8 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000035b000  084600000007 R_X86_64_JUMP_SLO 00000000000a2100 sysconf + 0
00000035b008  02e000000007 R_X86_64_JUMP_SLO 0000000000075e50 calloc + 0
00000035b010  01dd00000007 R_X86_64_JUMP_SLO 0000000000077910 realloc + 0
00000035b018  029300000007 R_X86_64_JUMP_SLO 00000000000661c0 feof + 0
00000035b020  046f00000007 R_X86_64_JUMP_SLO 00000000000768c0 malloc + 0
00000035b028  000400000007 R_X86_64_JUMP_SLO 0000000000000000 __tls_get_addr + 0
00000035b030  01b400000007 R_X86_64_JUMP_SLO 0000000000076dd0 memalign + 0
00000035b038  086000000007 R_X86_64_JUMP_SLO 00000000000767e0 free + 0

i.e. instead of jumping directly to the malloc defined in the libc code section, any internal calls will jump to this stub which, the first time, asks the dynamic linker to go out and find the address of malloc (it then saves it, so the second time the stub just jumps to the saved location).

This is an interesting list, seemingly without much order. feof, for example, stands out as worth checking out a bit closer — why would that be there when fopen isn't, say? We can track down where it comes from with a bit of detective work; knowing that the value of the symbol feof will be placed into 0x35b018 we can disassemble libc.so to see that this address is used by the feof PLT stub at 0x1e870 (luckily, objdump has done the math to offest from the rip for us; i.e. 0x1e876 + 0x33c7a2 = 0x35b018)

000000000001e870 <feof@plt>:
   1e870:       ff 25 a2 c7 33 00       jmpq   *0x33c7a2(%rip)        # 35b018 <_IO_file_jumps+0xb18>
   1e876:       68 03 00 00 00          pushq  $0x3
   1e87b:       e9 b0 ff ff ff          jmpq   1e830 <h_errno+0x1e7dc>

From there we can search for anyone jumping to that address, and find out the caller:

$ objdump --disassemble-all /lib/libc.so.6 | grep 1e870
000000000001e870 <feof@plt>:
   1e870:	ff 25 a2 c7 33 00    	jmpq   *0x33c7a2(%rip)        # 35b018 <_IO_file_jumps+0xb18>
   f19f7:	e8 74 ce f2 ff       	callq  1e870 <feof@plt>

$ addr2line -e /lib/libc.so.6 f19f7
/home/aurel32/eglibc/eglibc-2.11.2/sunrpc/bindrsvprt.c:70

Which turns out to be part of a local patch which probably gets things a little wrong, as described below. The sysconf relocation is from a similar add-on patch (being used to find the page size, it seems).

libc, like all sensible libraries, uses the hidden attribute on symbols to restrict what is exported by the library. The benefit of this is that when the linker knows you are referencing a hidden symbol it knows that the value can never be overridden, and thus does not need to emit extra code to do indirection just in case you ever wish to redirect the symbols. In the above, it appears that feof has never been marked as hidden, probably because no internal glibc functions used it until that add-on patch, and since it is not considered an internal function the linker must allow for the possibility that it will be overridden at run time and provide a PLT slot for it. There are consequences; if this was on a fast-path then the extra jumps required to go via the PLT may matter for performance and it may also cause strange behaviour if somebody had preloaded something that took over feof.

Note, this is different from saying that your library can override symbols provided by libc.so; such as when you LD_PRELOAD a library to wrap an open call. What you can not override is the open call that say, the internal libc.so function getusershell does to read /etc/shells.

Having the malloc related calls as preemptable seems intentional and sane; although I can not find a comment to the effect of "we deliberately leave these like this so that users may use alternative malloc implementations", it makes sense so that libc.so is internally using the same malloc as everything else if you choose to use something such as tc-malloc.

tl;dr? Digging into your system libraries is always interesting. Be careful with your link order when creating toolchains, and be careful about symbol visibility when you're working with libraries.

James Purser: It's called Freedom of Speech, not Freedom from criticism

http://jamespurser.com.au/blog/its-called-freedom-speech-not-freedom-criticism

You know what I love? When trolls make outrageous statements such as "Gay marriage will destroy the institution of marriage!" or "Women shouldn't be on the frontline because they can't open jam jars!" and when they get called on it, the first thing they do is accuse the person/people calling them out of infringing on the trolls right to freedom of speech.

This of course displays a pretty fundamental misunderstanding of what the right to freedom of speech actually entails. It doesn't actually mean that you can spout whatever crap you want and no one is allowed to call you a moron. Instead it pretty much ensures that if someone disagrees with you, they are perfectly entitled to use their own personal freedom of speech to ensure that you are made aware of their disagreement.

So the next time someone waves the old "right to speak" in your face during an argument, wave yours right back. 

James Purser: Twitter Liberation Front!

http://jamespurser.com.au/blog/twitter-liberation-front

Yesterday Jim Wallace of the Australian Christian Lobby tweeted the following:

"Just hope that as we remember Servicemen and women today we remember the Australia they fought for - wasn't gay marriage and islamic."

As to be expected, there was a firestorm of condemnation from the twittersphere with most calling him to task not just for his homophobic and anti-muslim views, but for daring to use ANZAC day (a day which is generally viewed as being above politics) to push his own narrow minded views.

After a couple of hours of being told to pull his head in he deleted the original post (of course this was useless as numerous screen caps had been taken of the offending tweet), and put out a media release in which he used the Scott Morrisson defence of claiming that the only thing wrong with the tweet was the timing.

Sigh.

Of course, given that there was a lot of interest around the tweet, Channel 7's Sunrise programme invited Jim onto this morning to be flayed with a scented boot lace:

http://www.youtube.com/watch?v=TeTvimoQoyo

This of couse inspired me to spend 5 seconds knocking up a logo for the twitter activists mentioned in the interview above:

 Twitter Liberation Front 

James Purser: A Rambling ANZAC Day post

http://jamespurser.com.au/blog/rambling-anzac-day-post

So today is ANZAC day, for many it commemorates the initial landing by the British, Australian, New Zealand and French forces at Gallipolli on the Turkish coast, the start of what can only be called a militarily disasterous campaign for the British. For many more it is a day to think about those who have served their country (whether you accept the validity of the conflict they served in or not).

Personally I've gone through a whole range of views on ANZAC day. When I was a kid, I was unpopular because I was taught by my father that it celebrated failure and I took his view as gospel. As I grew up I realised that this view somewhat missed the whole point of the exercise.

Yes, Gallopoli was a failure. Tens of thousands of men on both sides lost their lives over the course of nine months, with many more wounded or made ill by the horrendous conditions in the trenches. However, and this is important, it was on the shores of Gallipoli that the Australian forces really did start to define themselves as seperate and above the british. This carried on into the rest of the war, with Australia increasingly unwilling to be treated like some sort of pseudo-english force, and demanding more of a partnership role in the conflict.

What started in the failure of the English commanders, ended with the Australians becoming a respected fighting force in their own right, from the digger in the trenches (just ask the Turks or Germans who they would have preferred to face, Tommy's or Diggers), to the commanders (Sir John Monash was one of the leading proponents of the new air/armour/artillery/infantry combined arms operations). This is and should be a source of pride for the nation.

There are those who argue that we should never have gotten involved with World War 1. We weren't threatened in any way by the conflict (the Japanese were part of the Allies the first time round) and we were just starting out as a nation, why were we running around on mummy's apron tails, and to be honest I think they're right.

However we're looking at it in a time where the call of Empire and Home Country has largely become a joke, only taken seriously by the ACL and Consitutional Monarchists. In those days, the prevailing view was very much that Australia was still part of the greatest Empire the world had ever seen. Those who signed up for the entirely voluntary AEF (Australian Expeditionary Force) were going on a grand adventure, going to thwack fritz or johny turk on the nose and be back for Christmas.

Now we are much more aware of the cost of war. The advent of film, television, radio and now the internet has brought the true costs of conflict into our homes with an immediacy that Australians in 1915 could never have imagined. Today we are treated daily to news footage covering conflicts raging in Afghanistan, Libya and which ever african conflict is erupting at the moment. We don't have the luxury of romanticising war as some sort of boys own adventure. 

Hrmm I've rambled a bit I think. I personally have a lot of respect for those who are willing to put their lives on the line for their country, and while I think we should be a lot more careful about we as a nation use those lives, ANZAC day for me will always be a day when I think about those people who served and sacrificed for our country.

Pia Waugh: Twitter Weekly Updates for 2011-04-24

http://pipka.org/blog/2011/04/24/twitter-weekly-updates-for-2011-04-24/

Mark Greenaway: Guitar

http://certifiedwaif.livejournal.com/386684.html

Does anyone still blog? It seems nearly everyone has moved onto Twitter/Facebook. I miss being able to express thoughts in more than 160 characters.

I went to a picnic recently, and some people were passing a steel string guitar around. I'm not a good acoustic player, but it was fun so I had a bash. Someone played Under The Bridge, and took liberties with the chord voicings. So I was inspired to pick up my guitar and work through the official transcription, which I own. While the basic form of the song is pretty simple, as you can hear, the clever part is the choice of chord voicings and fills. I'll be practicing that one for a while.

I've also started over working through the Berklee method books, starting at Volume 2. I learned by playing by ear and memorising, so sight reading is still something I'm getting used to, and sometimes I'm not disciplined enough to do it properly. But I'm getting better at that. I'll be so happy when I start to get good at position playing.

Pia Waugh: Twitter Weekly Updates for 2011-04-17

http://pipka.org/blog/2011/04/17/twitter-weekly-updates-for-2011-04-17/

Scott Sinclair: Google Video Shutting Down

http://techcrunch.com/2011/04/15/google-video-prepares-to-enter-the-deadpool-for-good/

Jason Kincaid, Techcrunch:

Google just sent out an email to users who have previously uploaded content to the service informing them that on April 29 2011, the site will no longer host any more videos.

While this is probably something a long time coming, it does mean there is the potential for there to be a lot of videos that only exist on Google Video, and haven't been migrated to other services (Youtube, Vimeo, etc.). A thread on reddit is trying to list all the good ones that the userbase can find.

Jeremy Visser: Theora/Vorbis over RTSP over IPv6

https://jeremy.visser.name/2011/04/16/theora-vorbis-over-rtsp-over-ipv6/

I’ve always wondered if it were possible. Theora. Vorbis. Played with RTSP. All over IPv6.

VLC is truly the swiss army knife of streaming, and for many years has been my network streaming client/server of choice. Except when you have a red X on this page. And there are a lot of red X’s on that page.

VLC includes an RTSP server based on the live555 codebase. Unfortunately the codebase is very dated, and does not support IPv6. Given the fact that VLC has been the flagship IPv6 network streaming utility for just about every other protocol in existence, hacking live555 to support IPv6 is probably non-trivial. Not only that, but VLC does not support Theora or Vorbis payloads for RTP, which is required for RTSP support.

Do I want to be forced to stream MPEG or H.264 over RTSP over IPv4? No thanks.

Turns out the only two RTSP servers that support Theora and Vorbis are Feng, and…wait for it…GStreamer. And there are only half as many clients that support consuming Theora/Vorbis over RTSP — GStreamer (a.k.a. Totem, gst-launch, or similar) is your only choice.

Feng

The first thing I tried was Feng. Compiled version 2.1.0 from source. Using GStreamer (Totem) as a client, listening to MP3 audio over IPv6 worked great. However, Vorbis stuttered and was unusable. Not only that, but after playing a Vorbis file a second time, the Feng server would crash.

Depressingly, Feng does not support Theora (yet), which means that even if the stuttering Vorbis bug was fixed, it is unsuitable for Theora/Vorbis video streaming.

GStreamer

Now this is what I’m talkin’ about. GStreamer has an RTSP server library called gst-rtsp-server (README) that is able to use any RTP payload format that GStreamer supports.

For a list of what you can stream using GStreamer’s RTSP server, run:

$ gst-inspect-0.10 | grep 'rtp\w*pay'

(Forgive me if that \w usage kills kittens.)

To get started using gst-rtsp-server, just download the latest tarball from the site, and compile the programs in the examples/ directory. My favourite program was test-launch.c, which allows you to pass a GStreamer pipeline on the command line without having to recompile every time you want to try a new pipeline.

Using gst-rtsp-server, I’ve been able to stream Ogg Theora/Vorbis files and devices using Totem as a client. Everything works extremely well, although it is a bit tough to set up. Once you get test-launch.c compiled, you can add the following line of code after the call to gst_rtsp_server_new() in order to enable IPv6:

gst_rtsp_server_set_address (server, "::");

Then, you can, for example, stream your webcam and microphone by running the following:

./test-launch '(
	v4l2src
	! videorate
	! video/x-raw-yuv,width=320,height=240,framerate=7/1
	! theoraenc quality=10
	! queue
	! rtptheorapay name=pay0 pt=96

	autoaudiosrc
	! vorbisenc
	! queue
	! rtpvorbispay name=pay1 pt=97
)'

You can then connect to your new IPv6–enabled Theora/Vorbis RTSP server by running the following:

$ totem rtsp://localhost:8554/test

Please note that you need the very latest GStreamer for Totem to support IPv6 (for example, in Ubuntu 10.04 it does not work, but in 10.10 it does), and for the server end, gst_rtsp_server_set_address() is only supported in the very latest versions.

If anybody knows why lip sync is out for the first 30 seconds or so of the stream, please let me know. My knowledge of GStreamer is extremely limited (and the documentation is not very conducive to reciprocating that).

James Purser: Can Barry Sack Marrickville Council?

http://jamespurser.com.au/blog/can-barry-sack-marrickville-council

So Barry O'Farrel has decided that if Marrickville Council doesn't stop with the Anti-Israeli Boycott, he will step in and recommend that the Governor sack them.

Can he do it?

Yes. The legislation is pretty clear on this.

Governor may dismiss mayor and councillors

255 Governor may dismiss mayor and councillors

(1) The Governor may, by proclamation, declare all civic offices in relation to a council to be vacant if:
(a) a public inquiry concerning the council has been held, and
(b) after considering the results of the inquiry, the Minister has recommended that the Governor make such a declaration.

(2) The Governor may, by proclamation, declare all civic offices in relation to a council to be vacant if the Independent Commission Against Corruption, in a report referred to in section 74C of the Independent Commission Against Corruption Act 1988 , recommends that consideration be given to the making of such a declaration because of systemic corruption within the council.

(3) If the Independent Commission Against Corruption makes such a recommendation, all civic offices in relation to the council may be declared vacant under subsection (2) without the holding of a public or other inquiry concerning the council. However, the making of such a recommendation does not preclude the holding of a public or other inquiry.

It's the first clause that the Premier can use to sack the Council for not coming to heel on the BDS issue. At no point does it say that anything other than an inquiry has to be held (it certainly doesn't specify that if the council is not found to be neglegent, corrupt or incompetent it should be left alone). All the Premier has to do is consider the results and then make his recommendation to the Governer.

It's actually a very open ended clause when you think about it, and it gives the State a hell of a lot of power of the Local Government bodies.

Should Barry sack Marrickville Council?

That's a completely different question. Personally I don't think the BDS will do much to scare Israel back to the path of the right and good, we're talking about a country that has had a siege mentality essentially since it was formed (with some justification I might add). It also would be better balanced with similar action against say Iran (known backer of terrorist organisations and brutaliser of its own people).

However. There is a very fine line that O'Farrell needs to tread when dealing with this. While everyone is bang on the side of dealing with ineffective or corrupt councils, sacking an elected council because they take an idealogical stance different to your own is scary making. Barry might be better off by letting the people decide in six months time (Local Government elections are being held in September this year). If the move is unpopular then the people will express that at the ballot box.

Taking the decision to remove elected officials out of the hands of the people is always worrying, especially when there is no evidence of anything but a difference in ideologies.

Scott Sinclair: Bad advice on IPv6

http://www.techworld.com.au/article/383241/why_shouldn_t_worry_about_switching_ipv6_now/

Logan G. Harbaug of PCWorld:

A lot of coverage of IPv6 over the past few years gives the impression that you need to switch to IPv6 soon. That's not necessarily the case, though, especially for a consumer or a small business.

However, it may be at least three to five years before the average small business needs to work on this.

I think many people will disagree with his argument. Not taking action soon enough on IPv6, is just going to cause people to be unprepared when it is required.

Consumers and Small Businesses would only replace hardware (and software for that matter), after the previous one has had a very long lifespan. (Hell, I know people who are still using Pentium 3 Desktops!). So to say to them "don't do anything now, do it when everyone else is ready", will mean that when it is ready, there will be a mad rush to get it sorted, and there will be no "smooth" transition (and probably cause more issues!).

So please, plan your next Software and Hardware purchases in mind of them needing IPv6 support. Chances are half of what you have, won't have IPv6 support yet.

James Purser: MobileFOSS 2012

http://jamespurser.com.au/blog/mobilefoss-2012

Okay, so I thought I would get in early and see if people would be interested in me attempting to run another MobileFOSS miniconf at next years linux.conf.au.

Given the changes that have occured in the MobileFOSS space in the four months since MobileFOSS 2011 imagine what it's going to look like in another eight?

Leave a comment, contact me on twitter or send a carrier pigeon.

Sonia Hamilton: Python-iView

http://soniahamilton.wordpress.com/2011/04/11/python-iview/

A nice tool written by Jeremy Visser – Python-iView.

As Jeremy says on his blog: “Python-iView is an alternative frontend to ABC iView, which … is an awesome ABC TV programme catchup service that lets you watch most ABC programs from the last month in your browser… However, the iView website has some major problems… To address this, I wrote the open source (GPLv3) application Python-iView, which … allows you to download episodes to your hard drive in their original FLV format.”

Thanks Jeremy!


Pia Waugh: Twitter Weekly Updates for 2011-04-10

http://pipka.org/blog/2011/04/10/twitter-weekly-updates-for-2011-04-10/

Jeff Waugh: Trying to will a world into existence

http://bethesignal.org/blog/2011/04/09/trying-to-will-a-world-into-existence/

FromJames Cameron’sintroduction to an earlyscriptment of Strange Days:

At the beginning of any writing project is the agonizing period in which nebulous ideas dance before the mind’s eye like memories of a dream, and vaporous vague shapes take on human form and begin to answer to their names.

Trying to will a world into existence. I circle around it, nibbling at the edges, writing notes about the social infrastructure and expounding to no one in particular about the themes of the thing.

Then slowly a change happens. Without warning, it becomes easier to write a scene than to write notes about the scene. I start sticking words in the mouths of characters who are still mannequins, forcing them to move and to walk. Slowly their movements become more human.

The curve inflects upward, the pace increases. The characters begin to say things in their own words. By the end of this period I’m writing ten pages a day. The last day becomes endless, often stretching round the clock to the following noon. The curve becomes almost vertical as the thing seems to come alive. I become a witness only, a court reporter getting it down as fast as I can.

James Purser: VaxMeUp

http://jamespurser.com.au/blog/vaxmeup

As you may already know, I'm not overly fond of the Anti-Vaccination crowd, they play on ignorance and fear to push their own agendas.

Anyhoo, tonight I found through the magic of twitter that one of the people I follow - Julie Posetti - had been diagnosed with Whooping Cough. She was another victim of the falling numbers of pertussis vaccinations that have led to a marked increase in cases (including sadly, the death of an infant a few weeks ago).

Snap. I've had enough of letting the anti-vaxers get away with spreading misinformation and crippling an effective public health measure. So after discussing it with the twitterverse I've come up with VaxMeUp Day. On a day to be announced I want to organise a mass Pertussis Vaccination. Not only will this be a public show of support for the concept of vaccinations, but it will actually help to boost the herd immunity against the disease. Remember as an adult, there is every chance that your original immunisation has worn off.

So let's see what sort of support we can drum up. I'm going to be contacting my local members (State and Federal) as well as health bodies. I want to see this happen, and I want to see it happen soon.

Andre Pang: Immutability and Blocks, Lambdas and Closures [UPDATE x2]

http://algorithm.com.au/blog/files/immutability-blocks-lambdas-closures.php#unique-entry-id-613

I recently ran into some “interesting” behaviour when using lambda in Python. Perhaps it’s only interesting because I learnt lambdas from a functional language background, where I expect that they work a particular way, and the rest of the world that learnt lambdas through Python, Ruby or JavaScript disagree. (Shouts out to you Objective-C developers who are discovering the wonders of blocks, too.) Nevertheless, I thought this would be blog-worthy. Here’s some Python code that shows the behaviour that I found on Stack Overflow:

Since I succumb to reading source code in blog posts by interpreting them as “blah”1, a high-level overview of what that code does is:

  1. iterate over a list of strings,
  2. create a new list of functions that prints out the strings, and then
  3. call those functions, which prints the strings.

Simple, eh? Prints “do”, then “re”, then “mi”, eh? Wrong. It prints out “mi”, then “mi”, then “mi”. Ka-what?

(I’d like to stress that this isn’t a theoretical example. I hit this problem in production code, and boy, it was lots of fun to debug. I hit the solution right away thanks to the wonders of Google and Stack Overflow, but it took me a long time to figure out that something was going wrong at that particular point in the code, and not somewhere else in my logic.)

The second answer to the Stack Overflow question is the clearest exposition of the problem, with a rather clever solution too. I won’t repeat it here since you all know how to follow links. However, while that answer explains the problem, there’s a deeper issue. The inconceivable Manuel Chakravarty provides a far more insightful answer when I emailed him to express my surprise at Python’s lambda semantics:

This is a very awkward semantics for lambdas. It is also probably almost impossible to have a reasonable semantics for lambdas in a language, such as Python.

The behaviour that the person on SO, and I guess you, found surprising is that the contents of the free variables of the lambdas body could change between the point in time where the closure for the lambda was created and when that closure was finally executed. The obvious solution is to put a copy of the value of the variable (instead of a pointer to the original variable) into the closure.

But how about a lambda where a free variable refers to a 100MB object graph? Do you want that to be deep copied by default? If not, you can get the same problem again.

So, the real issue here is the interaction between mutable storage and closures. Sometimes you want the free variables to be copied (so you get their value at closure creation time) and sometimes you don’t want them copied (so you get their value at closure execution time or simply because the value is big and you don’t want to copy it).

And, indeed, since I love being categorised as a massive Apple fanboy, I found the same surprising behaviour with Apple’s blocks semantics in C, too:

You can see the Gist page for this sample code to see how to work around the problem in Objective-C (basically: copy the block), and also to see what it’d look like in Haskell (with the correct behaviour).

In Python, the Stack Overflow solution that I linked to has an extremely concise way of giving the programmer the option to either copy the value or simply maintain a reference to it, and the syntax is clear enough—once you understand what on Earth what the problem is, that is. I don’t understand Ruby or JavaScript well enough to comment on how they’d capture the immediate value for lambdas or whether they considered this design problem. C++0x will, unsurprisingly, give programmers full control over lambda behaviour that will no doubt confuse the hell out of people. (See the C++0x language draft, section 5.1.2 on page 91.)

In his usual incredibly didactic manner, Manuel then went on to explain something else insightful:

I believe there is a deeper issue here. Copying features of FP languages is the hip thing in language design these days. That’s fine, but many of the high-powered FP language features derive their convenience from being unspecific, or at least unconventional, about the execution time of a piece of code. Lambdas delay code execution, laziness provides demand-dependent code execution plus memoisation, continuations capture closures including their environment (ie, the stack), etc. Another instance of that problem was highlighted by Joe Duffy in his STM retrospective.

I would say, mutability and flexible control flow are fundamentally at odds in language design.

Indeed, I’ve been doing some language exploration again lately as the lack of static typing in Python is really beginning to bug me, and almost all the modern languages that attempt to pull functional programming concepts into object-oriented land seem like a complete Frankenstein, partially due to mutability. Language designers, please, this is 2011: multicore computing is the norm now, whether we like it or not. If you’re going to make an imperative language—and that includes all your OO languages—I’ll paraphrase Tim Sweeney: in a concurrent world, mutable is the wrong default! I’d love a C++ or Objective-C where all variables are const by default.

One take-away point from all this is to try to keep your language semantics simple. I love Dan Ingall’s quote from Design Principles Behind Smalltalk: “if a system is to serve the creative spirit, it must be entirely comprehensible to a single individual”. I love Objective-C partially because its message-passing semantics are straightforward, and its runtime has a amazingly compact API and implementation considering how powerful it is. I’ve been using Python for a while now, and I still don’t really know the truly nitty-gritty details about subtle object behaviours (e.g. class variables, multiple inheritance). And I mostly agree with Guido’s assertion that Python should not have included lambda nor reduce, given what Python’s goals are. After discovering this quirk about them, I’m still using the lambda in production code because the code savings does justify the complexity, but you bet your ass there’s a big comment there saying “warning, pretentous code trickery be here!”

1. See point 13 of Knuth et al.’s Mathematical Writing report.

UPDATE: There’s a lot more subtlety at play here than I first realised, and a couple of statements I’ve made above are incorrect. Please see the comments if you want to really figure out what’s going on: I’d summarise the issues, but the interaction between various language semantics are extremely subtle and I fear I’d simply get it wrong again. Thank you to all the commenters for both correcting me and adding a lot of value to this post. (I like this Internet thing! Other people do my work for me!)

Update #2

I’ve been overwhelmed by the comments, in both the workload sense and in the pleasantly-surprised-that-this-provoked-some-discussion sense. Boy, did I get skooled in a couple of areas. I’ve had a couple of requests to try to summarise the issues here, so I’ll do my best to do so.

Retrospective: Python

It’s clear that my misunderstanding of Python’s scoping/namespace rules is the underlying cause of the problem: in Python, variables declared in for/while/if statements will be declared in the compound block’s existing scope, and not create a new scope. So in my example above, using a lambda inside the for loop creates a closure that references the variable m, but m’s value has changed by the end of the for loop to “mi”, which is why it prints “mi, mi, mi”. I’d prefer to link to the official Python documentation about this here rather than writing my own words that may be incorrect, but I can’t actually find anything in the official documentation that authoritatively defines this. I can find a lot of blog posts warning about it—just Google for “Python for while if scoping” to see a few—and I’ve perused the entire chapter on Python’s compound statements, but I just can’t find it. Please let me know in the comments if you do find a link, in which case I’ll retract half this paragraph and stand corrected, and also a little shorter.

I stand by my assertion that Python’s for/while/if scoping is slightly surprising, and for some particular scenarios—like this—it can cause some problems that are very difficult to debug. You may call me a dumbass for bringing assumptions about one language to another, and I will accept my dumbassery award. I will happily admit that this semantics has advantages, such as being able to access the last value assigned in a for loop, or not requiring definitions of variables before executing an if statement that assigns to those variables and using it later in the same scope. All language design decisions have advantages and disadvantages, and I respect Python’s choice here. However, I’ve been using Python for a few years, consider myself to be at least a somewhat competent programmer, and only just found out about this behaviour. I’m surprised 90% of my code actually works as intended given these semantics. In my defence, this behaviour was not mentioned at all in the excellent Python tutorials, and, as mentioned above, I can’t a reference for it in the official Python documentation. I’d expect that this behaviour is enough of a difference vs other languages to at least be mentioned. You may disagree with me and regard this as a minor issue that only shows up when you do crazy foo like use lambda inside a for loop, in which case I’ll shrug my shoulders and go drink another beer.

I’d be interested to see if anyone can come up an equivalent for the “Closures and lexical closures” example at http://c2.com/cgi/wiki?ScopeAndClosures, given another Python scoping rule that assignment to a variable automatically makes it a local variable. (Thus, the necessity for Python’s global keyword.) I’m guessing that you can create the createAdder closure example there with Python’s lambdas, but my brain is pretty bugged out today so I can’t find an equivalent for it right now. You can simply write a callable class to do that and instantiate an object, of course, which I do think is about 1000x clearer. There’s no point using closures when the culture understands objects a ton better, and the resulting code is more maintainable.

Python summary: understand how scoping in for/while/if blocks work, otherwise you’ll run into problems that can cost you hours, and get skooled publicly on the Internet for all your comrades to laugh at. Even with all the language design decisions that I consider weird, I still respect and like Python, and I feel that Guido’s answer to the stuff I was attempting would be “don’t do that”. Writing a callable class in Python is far less tricky than using closures, because a billion times more people understand their semantics. It’s always a design question of whether the extra trickiness is more maintainable or not.

Retrospective: Blocks in C

My C code with blocks failed for a completely different reason unrelated to the Python version, and this was a total beginner’s error with blocks, for which I’m slightly embarrassed. The block was being stack-allocated, so upon exit of the for loop that assigns the function list, the pointers to the blocks are effectively invalid. I was a little unlucky that the program didn’t crash. The correct solution is to perform a Block_copy, in which case things work as expected.

Retrospective: Closures

Not all closures are the same; or, rather, closures are closures, but their semantics can differ from language to language due to many different language design decisions—such as how one chooses to define the lexical environment. Wikipedia’s article on closures has an excellent section on differences in closure semantics.

Retrospective: Mutability

I stand by all my assertions about mutability. This is where the Haskell tribe will nod their collective heads, and all the anti-Haskell tribes think I’m an idiot. Look, I use a lot of languages, and I love and hate many things about each of them, Haskell included. I fought against Haskell for years and hated it until I finally realised that one of its massive benefits is that things bloody well work an unbelievable amount of the time once your code compiles. Don’t underestimate how much of a revelation this is, because that’s the point where the language’s beauty, elegance and crazy type system fade into the background and, for the first time, you see one gigantic pragmatic advantage of Haskell.

One of the things that Haskell does to achieve this is the severe restriction on making things immutable. Apart from the lovely checkbox reason that you can write concurrent-safe algorithms with far less fear, I truly believe that this makes for generally more maintainable code. You can read code and think once about what value a variable holds, rather than keep it in the back of your mind all the time. The human mind is better at keeping track of multiple names, rather than a single name with different states.

The interaction of state and control flow is perhaps the most complex thing to reason about in programming—think concurrency, re-entrancy, disruptive control flow such as longjmp, exceptions, co-routines—and mutability complicates that by an order of magnitude. The subtle difference in behaviour between all the languages discussed in the comments is exemplar that “well-understood” concepts such as lexical scoping, for loops and closures can produce a result that many people still don’t expect; at least for this simple example, these issues would have been avoided altogether if mutability was disallowed. Of course mutability has its place. I’m just advocating that we should restrict it where possible, and at least a smattering of other languages—and hopefully everyone who has to deal with thread-safe code—agrees with me.

Closing

I’d truly like to thank everyone who added their voice and spent the time to comment on this post. It’s been highly enlightening, humbling, and has renewed my interest in discussing programming languages again after a long time away from it. And hey, I’m blogging again. (Though perhaps after this post, you may not think that two of those things are good things.) It’s always nice when you learn something new, which I wouldn’t have if not for the excellent peer review. Science: it works, bitches!

Sonia Hamilton: Python, Lambda, Closures

http://soniahamilton.wordpress.com/2011/04/05/python-lambda-closures/

(apologies to slug planeteers for the duplicate info).

A very interesting post from Andr Pang about some issues he came across with lamdas and closures in Python. Well, interesting to me because I’m playing with Lisp and Python at the moment…

Some links from his blog, in case the original post ever disappears:

A nice snippet of code to remind me of the issues:

def callback(msg):
    print msg
funcList=[]
for m in ('do', 're', 'mi'):
    # funcList.append(lambda: callback(m))
    funcList.append(lambda m=m: callback(m))
for f in funcList:
    f()

Rich Buggy: phpConf.au

http://buggy.id.au/2011/04/05/phpconf-au/

I don’t know much about phpConf.au or who is organizing it. If you’re in Sydney and interested in a PHP Conference then it’s probably worth registering your email address for more information.

Update: It’s amazing how easy it is to write config instead of conf when you’re writing while talking to a 3 year old.

Craige McWhirter: Honey Goat's Milk Soap Launched!

http://mcwhirter.com.au/craige/blog/2011/Honey_Goat_s_Milk_Soap_Launched

Honey Goat's Milk Soap

Kristina and I are proud to launch the latest addition to our goat's milk soap range:

Honey Goat's Milk Soap

With fresh goat's milk and honey, this makes a luxurious and soothing soap that is perfect for all skin types. 

This soap is available in the following forms:

Our honey goat's milk soap is also available from our wonderful stockists :)
 

read more

James Purser: This mornings Insiders and Bolts performance

http://jamespurser.com.au/blog/mornings-insiders-and-bolts-performance

Okay I have a splitting headache so this is going to be brief:

Andrew Bolt used this mornings Insiders appearance to reach out to one audience only, the nascent Australian Tea Party. He was talking directly to those people who held up the signs equating the UN with genocide and who view the political debate through the eyes of a cranky 4 year old.

Bolt's attack on Lenore was typical of the sort of tactics that the masters of this art in the US use. Straight from the book of Beck, he was overbearing, refused to let Lenore Taylor actually answer his question and when she did actually manage to answer it (effectively and fully by the way), he simply ignored that fact and inserted his own. It seems that monsuier Bolt may have ambitions for a leadership position in the new order.

The thing is, if he wants to be a "thought leader" for the Tea Party(keeping in mind that he's competing with such luminaries as Angry Anderson), then more luck to him. He's got plenty of opportunities to spout his own brand of "wisdom", he has a radio gig and a blog with the biggest newspaper in Victoria. Insiders is meant to be more than a personal soap box for a budding demagogue, it's meant to be an exploration of the nations politics by people who know what they are talking about. 

Gah, head is pounding, but seriously from now on if Bolt is on Insiders I will not be watching. The show just loses all value. Oh and that applies to Piers Akerman as well. It is possible to have a right wing commentator on the show who can debate policy. Gerard Henderson is one such person. I may never agree with what he says but at least he knows how to argue rationally and cogently.