rss.xml

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Store Half Byte-Reverse Indexed</title><link>https://sthbrx.github.io/</link><description>A Power Technical Blog</description><atom:link href="https://sthbrx.github.io/rss.xml" rel="self"></atom:link><lastBuildDate>Tue, 22 Mar 2016 18:00:00 +1100</lastBuildDate><item><title>Getting logs out of things</title><link>https://sthbrx.github.io/blog/2016/03/22/getting-logs-out-of-things/</link><description>&lt;p&gt;Here at OzLabs, we have an unfortunate habit of making our shiny Power computers very sad, which is a common problem in systems programming and kernel hacking. When this happens, we like having logs. In particular, we like to have the kernel log and the OPAL firmware log, which are, very surprisingly, rather helpful when debugging kernel and firmware issues.&lt;/p&gt;
&lt;p&gt;Here's how to get them.&lt;/p&gt;
&lt;h2&gt;From userspace&lt;/h2&gt;
&lt;p&gt;You're lucky enough that your machine is still up, yay! As every Linux sysadmin knows, you can just grab the kernel log using &lt;code&gt;dmesg&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;As for the OPAL log: we can simply ask OPAL to tell us where its log is located in memory, copy it from there, and hand it over to userspace. In Linux, as per standard Unix conventions, we do this by exposing the log as a file, which can be found in &lt;code&gt;/sys/firmware/opal/msglog&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Annoyingly, the &lt;code&gt;msglog&lt;/code&gt; file reports itself as size 0 (I'm not sure exactly why, but I &lt;em&gt;think&lt;/em&gt; it's due to limitations in sysfs), so if you try to copy the file with &lt;code&gt;cp&lt;/code&gt;, you end up with just a blank file. However, you can read it with &lt;code&gt;cat&lt;/code&gt; or &lt;code&gt;less&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;From &lt;code&gt;xmon&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;xmon&lt;/code&gt; is a really handy in-kernel debugger for PowerPC that allows you to do basic debugging over the console without hooking up a second machine to use with &lt;code&gt;kgdb&lt;/code&gt;. On our development systems, we often configure &lt;code&gt;xmon&lt;/code&gt; to automatically begin debugging whenever we hit an oops or panic (using &lt;code&gt;xmon=on&lt;/code&gt; on the kernel command line, or the &lt;code&gt;XMON_DEFAULT&lt;/code&gt; Kconfig option). It can also be manually triggered:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;root@p86:~# echo x &amp;gt; /proc/sysrq-trigger
sysrq: SysRq : Entering xmon
cpu 0x7: Vector: 0  at [c000000fcd717a80]
pc: c000000000085ad8: sysrq_handle_xmon+0x68/0x80
lr: c000000000085ad8: sysrq_handle_xmon+0x68/0x80
sp: c000000fcd717be0
msr: 9000000000009033
current = 0xc000000fcd689200
paca    = 0xc00000000fe01c00   softe: 0        irq_happened: 0x01
pid   = 7127, comm = bash
Linux version 4.5.0-ajd-11118-g968f3e3 (ajd@ka1) (gcc version 5.2.1 20150930 (GCC) ) #1 SMP Tue Mar 22 17:01:58 AEDT 2016
enter ? for help
7:mon&amp;gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;From &lt;code&gt;xmon&lt;/code&gt;, simply type &lt;code&gt;dl&lt;/code&gt; to dump out the kernel log. If you'd like to page through the log rather than dump the entire thing at once, use &lt;code&gt;#&amp;lt;n&amp;gt;&lt;/code&gt; to split it into groups of &lt;code&gt;n&lt;/code&gt; lines.&lt;/p&gt;
&lt;p&gt;Until recently, it wasn't as easy to extract the OPAL log without knowing magic offsets. A couple of months ago, I was debugging a nasty CAPI issue and got rather frustrated by this, so one day when I had a couple of hours free I &lt;a href="http://patchwork.ozlabs.org/patch/581775/"&gt;refactored&lt;/a&gt; the existing sysfs interface and &lt;a href="http://patchwork.ozlabs.org/patch/581774/"&gt;added&lt;/a&gt; the &lt;code&gt;do&lt;/code&gt; command to &lt;code&gt;xmon&lt;/code&gt;. These patches will be included from kernel 4.6-rc1 onwards.&lt;/p&gt;
&lt;p&gt;When you're done, &lt;code&gt;x&lt;/code&gt; will attempt to recover the machine and continue, &lt;code&gt;zr&lt;/code&gt; will reboot, and &lt;code&gt;zh&lt;/code&gt; will halt.&lt;/p&gt;
&lt;h2&gt;From the FSP&lt;/h2&gt;
&lt;p&gt;Sometimes, not even &lt;code&gt;xmon&lt;/code&gt; will help you. In production environments, you're not generally going to start a debugger every time you have an incident. Additionally, a serious hardware error can cause a 'checkstop', which completely halts the system. (Thankfully, end users don't see this very often, but kernel developers, on the other hand...)&lt;/p&gt;
&lt;p&gt;This is where the Flexible Service Processor, or FSP, comes in. The FSP is an IBM-developed baseboard management controller used on most IBM-branded Power Systems machines, and is responsible for a whole range of things, including monitoring system health. Among its many capabilities, the FSP can automatically take "system dumps" when fatal errors occur, capturing designated regions of memory for later debugging. System dumps can be configured and triggered via the FSP's web interface, which is beyond the scope of this post but is &lt;a href="https://www.ibm.com/support/knowledgecenter/POWER8/p8ha5/mainstoragedump.htm?cp=POWER8%2F1-3-14-2"&gt;documented&lt;/a&gt; in IBM Power Systems user manuals.&lt;/p&gt;
&lt;p&gt;How does the FSP know what to capture? As it turns out, skiboot (the firmware which implements OPAL) maintains a &lt;a href="https://github.com/open-power/skiboot/blob/master/hw/fsp/fsp-mdst-table.c"&gt;Memory Dump Source Table&lt;/a&gt; which tells the FSP which memory regions to dump. MDST updates are recorded in the OPAL log:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2690088026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Max&lt;/span&gt; &lt;span class="n"&gt;entries&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;MDST&lt;/span&gt; &lt;span class="nl"&gt;table&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2690090666&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Addr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mh"&gt;0x31000000&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nl"&gt;size&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mh"&gt;0x100000&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;added&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;MDST&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2690093767&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Addr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mh"&gt;0x31100000&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nl"&gt;size&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mh"&gt;0x100000&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;added&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;MDST&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2750378890&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Table&lt;/span&gt; &lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;11199672771&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Addr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mh"&gt;0x1fff772780&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nl"&gt;size&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mh"&gt;0x200000&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;added&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;MDST&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;11215193760&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Table&lt;/span&gt; &lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;28031311971&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Table&lt;/span&gt; &lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;28411709421&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Addr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mh"&gt;0x1fff830000&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nl"&gt;size&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mh"&gt;0x100000&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;added&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;MDST&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;28417251110&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nl"&gt;MDST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Table&lt;/span&gt; &lt;span class="n"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;In the above log, we see four entries: the skiboot/OPAL log, the &lt;a href="https://github.com/open-power/hostboot"&gt;hostboot&lt;/a&gt; runtime log, the petitboot Linux kernel log (which doesn't make it into the final dump) and the real Linux kernel log. skiboot obviously adds the OPAL and hostboot logs to the MDST early in boot, but it also exposes the &lt;a href="https://github.com/open-power/skiboot/blob/master/doc/opal-api/opal-register-dump-region-101.txt"&gt;&lt;code&gt;OPAL_REGISTER_DUMP_REGION&lt;/code&gt;&lt;/a&gt; call which can be used by the operating system to register additional regions. Linux uses this to &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/powernv/opal.c#L608"&gt;register the kernel log buffer&lt;/a&gt;. If you're a kernel developer, you could potentially use the OPAL call to register your own interesting bits of memory.&lt;/p&gt;
&lt;p&gt;So, the MDST is all set up, we go about doing our business, and suddenly we checkstop. The FSP does its sysdump magic and a few minutes later it reboots the system. What now?&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;After we come back up, the FSP notifies OPAL that a new dump is available. Linux exposes the dump to userspace under &lt;code&gt;/sys/firmware/opal/dump/&lt;/code&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;a href="https://sourceforge.net/projects/linux-diag/files/ppc64-diag/"&gt;ppc64-diag&lt;/a&gt; is a suite of utilities that assist in manipulating FSP dumps, including the &lt;code&gt;opal_errd&lt;/code&gt; daemon. &lt;code&gt;opal_errd&lt;/code&gt; monitors new dumps and saves them in &lt;code&gt;/var/log/dump/&lt;/code&gt; for later analysis.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;opal-dump-parse&lt;/code&gt; (also in the &lt;code&gt;ppc64-diag&lt;/code&gt; suite) can be used to extract the sections we care about from the dump:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;root@p86:/var/log/dump# opal-dump-parse -l SYSDUMP.842EA8A.00000001.20160322063051 
|---------------------------------------------------------|
|ID              SECTION                              SIZE|
|---------------------------------------------------------|
|1              Opal-log                           1048576|
|2              HostBoot-Runtime-log               1048576|
|128            printk                             1048576|
|---------------------------------------------------------|
List completed
root@p86:/var/log/dump# opal-dump-parse -s 1 SYSDUMP.842EA8A.00000001.20160322063051 
Captured log to file Opal-log.842EA8A.00000001.20160322063051
root@p86:/var/log/dump# opal-dump-parse -s 2 SYSDUMP.842EA8A.00000001.20160322063051 
Captured log to file HostBoot-Runtime-log.842EA8A.00000001.20160322063051
root@p86:/var/log/dump# opal-dump-parse -s 128 SYSDUMP.842EA8A.00000001.20160322063051 
Captured log to file printk.842EA8A.00000001.20160322063051
&lt;/pre&gt;&lt;/div&gt;


&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There's various other types of dumps and logs that I won't get into here. I'm probably obliged to say that if you're having problems out in the wild, you should probably contact your friendly local IBM Service Representative...&lt;/p&gt;
&lt;h2&gt;Acknowledgements&lt;/h2&gt;
&lt;p&gt;Thanks to &lt;a href="https://flamingspork.com"&gt;Stewart Smith&lt;/a&gt; for pointing me in the right direction regarding FSP sysdumps and related tools.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Andrew Donnellan</dc:creator><pubDate>Tue, 22 Mar 2016 18:00:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-03-22:blog/2016/03/22/getting-logs-out-of-things/</guid><category>debugging</category><category>skiboot</category><category>OPAL</category><category>FSP</category><category>kernel</category><category>development</category><category>OpenPOWER</category></item><item><title>The Elegance of the Plaintext Patch</title><link>https://sthbrx.github.io/blog/2016/03/22/the-elegance-of-the-plaintext-patch/</link><description>&lt;p&gt;I've only been working on the Linux kernel for a few months.  Before that, I worked with proprietary source control at work and common tools like GitHub at home.  The concept of the mailing list seemed obtuse to me.  If I noticed a problem with some program, I'd be willing to open an issue on GitHub but not to send an email to a mailing list.  Who still uses those, anyway?&lt;/p&gt;
&lt;p&gt;Starting out with the kernel meant I had to figure this email thing out.  &lt;code&gt;git format-patch&lt;/code&gt; and &lt;code&gt;git send-email&lt;/code&gt; take most of the pain out of formatting and submitting a patch, which is nice.  The patch files generated by &lt;code&gt;format-patch&lt;/code&gt; open nicely in Emacs by default, showing all whitespace and letting you pick up any irregularities.  &lt;code&gt;send-email&lt;/code&gt; means you can send it to yourself or a friend first, finding anything that looks stupid before being exposed to the public.&lt;/p&gt;
&lt;p&gt;And then what?  You've sent an email.  It gets sent to hundreds or thousands of people.  Nowhere near that many will read it.  Some might miss it due to their mail server going down, or the list flagging your post as spam, or requiring moderation.  Some recipients will be bots that archive mail on the list, or publish information about the patch.  If you haven't formatted it correctly, someone will let you know quickly.  If your patch is important or controversial, you'll have all sorts of responses.  If your patch is small or niche, you might not ever hear anything back.&lt;/p&gt;
&lt;p&gt;I remember when I sent my first patch.  I was talking to a former colleague who didn't understand the patch/mailing list workflow at all.  I sent him a link to my patch on a mail archive.  I explained it like a pull request - here's my code, you can find the responses.  What's missing from a GitHub-esque pull request?  We don't know what tests it passed.  We don't know if it's been merged yet, or if the maintainer has looked at it.  It takes a bit of digging around to find out who's commented on it.  If it's part of a series, that's awkward to find out as well.  What about revisions of a series?  That's another pain point.&lt;/p&gt;
&lt;p&gt;Luckily, these problems do have solutions.  &lt;a href="http://jk.ozlabs.org/projects/patchwork/"&gt;Patchwork&lt;/a&gt;, written by fellow OzLabs member &lt;a href="http://jk.ozlabs.org"&gt;Jeremy Kerr&lt;/a&gt;, changes the way we work with patches.  Project maintainers rely on Pathwork instances, such as &lt;a href="https://patchwork.ozlabs.org"&gt;https://patchwork.ozlabs.org&lt;/a&gt;, for their day-to-day workflow: tagging reviewers, marking the status of patches, keeping track of tests, acks, reviews and comments in one place.  Missing from this picture is support for series and revisions, which is a feature that's being developed by the &lt;a href="https://www.freedesktop.org/wiki/"&gt;freedesktop&lt;/a&gt; project.  You can check out their changes in action &lt;a href="https://patchwork.freedesktop.org"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;So, Patchwork helps patches and email catch up to what GitHub has in terms of ease of information.  We're still missing testing and other hooks.  What about review?  What can we do with email, compared to GitHub and the like?&lt;/p&gt;
&lt;p&gt;In my opinion, the biggest feature of email is the ease of review.  Just reply inline and you're done.  There's inline commenting on GitHub and GitLab, which works well but is a bit tacky, people commenting on the same thing overlap and conflict, each comment generates a notification (which can be an email until you turn that off).  Plus, since it's email, it's really easy to bring in additional people to the conversation as necessary.  If there's a super lengthy technical discussion in the kernel, it might just take Linus to resolve.&lt;/p&gt;
&lt;p&gt;There are alternatives to just replying to email, too, such as &lt;a href="https://www.gerritcodereview.com/"&gt;Gerrit&lt;/a&gt;.  Gerrit's pretty popular, and has a huge amount of features.  I understand why people use it, though I'm not much of a fan.  Reason being, it doesn't add to the email workflow, it replaces it.  Plaintext email is supported on pretty much any device, with a bunch of different programs.  From the goals of Patchwork: "patchwork should supplement mailing lists, not replace them".&lt;/p&gt;
&lt;p&gt;Linus Torvalds famously explained why he prefers email over GitHub pull requests &lt;a href="https://github.com/torvalds/linux/pull/17"&gt;here&lt;/a&gt;, using &lt;a href="https://groups.google.com/forum/#!topic/linux.kernel/w957vpu3PPU"&gt;this&lt;/a&gt; pull request from Ben Herrenschmidt as an example of why git's own pull request format is superior to that of GitHub.  Damien Lespiau, who is working on the freedesktop Patchwork fork, &lt;a href="http://damien.lespiau.name/2016/02/augmenting-mailing-lists-with-patchwork.html"&gt;outlines on his blog&lt;/a&gt; all the issues he has with mailing list workflows and why he thinks mailing lists are a relic of the past.  His work on Patchwork has gone a long way to help fix those problems, however I don't think mailing lists are outdated and superceded, I think they are timeless.  They are a technology-agnostic, simple and free system that will still be around if GitHub dies or alienates its community.&lt;/p&gt;
&lt;p&gt;That said, there's still the case of the missing features.  What about automated testing?  What about developer feedback?  What about making a maintainer's life easier?  We've been working on improving these issues, and I'll outline how we're approaching them in a future post.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Tue, 22 Mar 2016 13:53:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-03-22:blog/2016/03/22/the-elegance-of-the-plaintext-patch/</guid><category>development</category><category>education</category><category>kernel</category><category>patches</category></item><item><title>No Network For You</title><link>https://sthbrx.github.io/blog/2016/03/21/no-network-for-you/</link><description>&lt;p&gt;In POWER land &lt;a href="https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface"&gt;IPMI&lt;/a&gt; is mostly known as the method to access the machine's console and start interacting with Petitboot. However it also has a plethora of other features, handily described in the 600ish page &lt;a href="http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html"&gt;IPMI specification&lt;/a&gt; (which you can go read yourself).&lt;/p&gt;
&lt;p&gt;One especially relevant feature to Petitboot however is the 'chassis bootdev' command, which you can use to tell Petitboot to ignore any existing boot order, and only consider boot options of the type you specify (eg. 'network', 'disk', or 'setup' to not boot at all). Support for this has been in Petitboot for a while and should work on just about any machine you can get your hands on.&lt;/p&gt;
&lt;h2&gt;Network Overrides&lt;/h2&gt;
&lt;p&gt;Over in OpenPOWER&lt;sup&gt;1&lt;/sup&gt; land however, someone took this idea and pushed it further - why not allow the network configuration to be overwritten too? This isn't in the IPMI spec, but if you cast your gaze down to page 398 where the spec lays out the entire format of the IPMI request, there is a certain field named "OEM Parameters". This is an optional amount of space set aside for whatever you like, which in this case is going to be data describing an override of the network config.&lt;/p&gt;
&lt;p&gt;This allows a user to tell Petitboot over IPMI to either;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Disable the network completely,&lt;/li&gt;
&lt;li&gt;Set a particular interface to use DHCP, or&lt;/li&gt;
&lt;li&gt;Set a particular interface to use a specific static configuration.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Any of these options will cause any existing network configurations to be ignored.&lt;/p&gt;
&lt;h2&gt;Building the Request&lt;/h2&gt;
&lt;p&gt;Since this is an OEM-specific command, your average ipmitool package isn't going to have a nice way of making this request, such as 'chassis bootdev network'. Rather you need to do something like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span class="x"&gt;ipmitool -I lanplus -H &lt;/span&gt;&lt;span class="p"&gt;$&lt;/span&gt;&lt;span class="nv"&gt;yourbmc&lt;/span&gt;&lt;span class="x"&gt; -U &lt;/span&gt;&lt;span class="p"&gt;$&lt;/span&gt;&lt;span class="nv"&gt;user&lt;/span&gt;&lt;span class="x"&gt; -P &lt;/span&gt;&lt;span class="p"&gt;$&lt;/span&gt;&lt;span class="nv"&gt;pass&lt;/span&gt;&lt;span class="x"&gt; raw 0x00 0x08 0x61 0x80 0x21 0x70 0x62 0x21 0x00 0x01 0x06 0x04 0xf4 0x52 0x14 0xf3 0x01 0xdf 0x00 0x01 0x0a 0x3d 0xa1 0x42 0x10 0x0a 0x3d 0x2 0x1&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Horrific right? In the near future the Petitboot tree will include a helper program to format this request for you, but in the meantime (and for future reference), lets lay out how to put this together:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;Specify the &amp;quot;chassis bootdev&amp;quot; command, field 96, data field 1:
    0x00 0x08 0x61 0x80

Unique value that Petitboot recognises:
    0x21 0x70 0x62 0x21

Version field (1)
    0x00 0x01 ..   ..

Size of the hardware address (6):
    ..   ..   0x06 ..

Size of the IP address (IPv4/IPv6):
    ..   ..   ..   0x04

Hardware (MAC) address:
    0xf4 0x52 0x14 0xf3
    0x01 0xdf ..   ..

&amp;#39;Ignore flag&amp;#39; and DHCP/Static flag (DHCP is 0)
    ..   ..   0x00 0x01

(Below fields only required if setting a static IP)

IP Address:
    0x0a 0x3d 0xa1 0x42

Subnet Mask (eg, /16):
    0x10 ..   ..   ..
Gateway IP Address:
    ..   0x0a 0x3d 0x02
    0x01
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Clearing a network override is as simple as making a request empty aside from the header:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;0x00 0x08 0x61 0x80 0x21 0x70 0x62 0x21 0x00 0x01 0x00 0x00
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;You can also read back the request over IPMI with this request:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;0x00 0x09 0x61 0x00 0x00
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;That's it! Ideally this is something you would be scripting rather than bashing out on the keyboard - the main use case at the moment is as a way to force a machine to netboot against a known good source, rather than whatever may be available on its other interfaces.&lt;/p&gt;
&lt;p&gt;[1] The reason this is only available on OpenPOWER machines at the moment is that support for the IPMI command itself depends on the BMC firmware, and non-OpenPOWER machines use an FSP which is a different platform.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Mon, 21 Mar 2016 15:23:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-03-21:blog/2016/03/21/no-network-for-you/</guid><category>petitboot</category><category>power</category><category>p8</category><category>openpower</category><category>goodposts</category><category>realcontent</category><category>ipmi</category><category>bmc</category><category>based16</category></item><item><title>And now for something completely different: approximate computing</title><link>https://sthbrx.github.io/blog/2016/03/15/and-now-for-something-completely-different-approximate-computing/</link><description>&lt;p&gt;In early February I had the opportunity to go the the NICTA Systems Summer School, where Cyril and I were invited to represent IBM. There were a number of excellent talks across a huge range of systems related subjects, but the one that has stuck with me the most was a talk given by &lt;a href="http://homes.cs.washington.edu/~luisceze/"&gt;Luis Ceze&lt;/a&gt;  on a topic called approximate computing. So here, in hopes that you too find it interesting, is a brief run-down on what I learned.&lt;/p&gt;
&lt;p&gt;Approximate computing is fundamentally about trading off accuracy for something else - often speed or power consumption. Initially this sounded like a very weird proposition: computers do things like 'running your operating system' and 'reading from and writing to disks': things you need to always be absolutely correct if you want anything vaguely resembling reliability. It turns out that this is actually not as big a roadblock as I had assumed - you can work around it fairly easily.&lt;/p&gt;
&lt;p&gt;The model proposed for approximate computing is as follows. You divide your computation up into two classes: 'precise', and 'approximate'. You use 'precise' computations when you need to get exact answers: so for example if you are constructing a JPEG file, you want the JPEG header to be exact. Then you have approximate computations: so for example the contents of your image can be approximate.&lt;/p&gt;
&lt;p&gt;For correctness, you have to establish some boundaries: you say that precise data can be used in approximate calculations, but that approximate data isn't allowed to cross back over and pollute precise calculations. This, while intuitively correct, poses some problems in practise: when you want to write out your approximate JPEG data, you need an operation that allows you to 'bless' (or in their terms 'endorse') some approximate data so it can be used in the precise file system operations.&lt;/p&gt;
&lt;p&gt;In the talk we were shown an implementation of this model in Java, called &lt;a href="http://sampa.cs.washington.edu/research/approximation/enerj.html"&gt;EnerJ&lt;/a&gt;. EnerJ allows you to label variables with either &lt;code&gt;@Precise&lt;/code&gt; if you're dealing with precise data, or &lt;code&gt;@Approx&lt;/code&gt; if you're dealing with approximate data. The compiler was modified so that it would do all sorts of weird things when it knew it was dealing with approximate data: for example, drop loop iterations entirely, do things in entirely non-determistic ways - all sorts of fun stuff. It turns out this works surprisingly well.&lt;/p&gt;
&lt;p&gt;However, the approximate computing really shines when you can bring it all the way down to the hardware level. The first thing they tried was a CPU with both 'approximate' and precise execution engines, but this turned out not to have the power savings hoped for. What seemed to work really well was a model where some approximate calculations could be identified ahead of time, and then replaced with neural networks in hardware. These neural networks approximated the calculations, but did so at significantly lower power levels. This sounded like a really promising concept, and it will be interesting to see if this goes anywhere over the next few years.&lt;/p&gt;
&lt;p&gt;There's a lot of work evaluating the quality of the approximate result, for cases where the set of inputs is known, and when the inputs is not known. This is largely beyond my understanding, so I'll simply refer you to some of the papers &lt;a href="http://sampa.cs.washington.edu/research/approximation/enerj.html"&gt;listed on the website&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The final thing covered in the talk was bringing approximate computing into current paradigms by just being willing to accept higher user-visible error rates. For example, they hacked up a network stack to accept packets with invalid checksums. This has had mixed results so far. A question I had (but didn't get around to asking!) would be whether the mathematical properties of checksums (i.e. that they can correct a certain number of bit errors) could be used to correct some of the errors, rather than just accepting/rejecting them blindly. Perhaps by first attempting to correct errors using the checksums, we will be able to fix the simpler errors, reducing the error rate visible to the user.&lt;/p&gt;
&lt;p&gt;Overall, I found the NICTA Systems Summer School to be a really interesting experience (and I hope to blog more about it soon). If you're a university student in Australia, or an academic, see if you can make it in 2017!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Tue, 15 Mar 2016 11:30:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-03-15:blog/2016/03/15/and-now-for-something-completely-different-approximate-computing/</guid><category>nicta</category><category>conferences</category></item><item><title>linux.conf.au 2016: A set of brief thoughts</title><link>https://sthbrx.github.io/blog/2016/03/15/linuxconfau-2016-a-set-of-brief-thoughts/</link><description>&lt;p&gt;Recently most of us attended LCA2016. This is one set of reflections on what we heard and what we've thought since. (Hopefully not the only set of reflections that will be posted on this blog either!)&lt;/p&gt;
&lt;p&gt;LCA was 2 days of miniconferences plus 3 days of talks. Here, I've picked some of the more interesting talks I attended, and I've written down some thoughts. If you find the thoughts interesting, you can click through and watch the whole talk video, because LCA is awesome like that.&lt;/p&gt;
&lt;h4&gt;Life is better with Rust's community automation&lt;/h4&gt;
&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=dIageYT0Vgg"&gt;This talk&lt;/a&gt; is probably the one that's had the biggest impact on our team so far. We were really impressed by the community automation that Rust has: the way they can respond to pull requests from new community members in a way that lets them keep their code quality high and be nice to everyone at the same time.&lt;/p&gt;
&lt;p&gt;The system that they've developed is fascinating (and seems fantastic). However, their system uses pull requests, while we use mailing lists. Pull requests are easy, because github has good hook support, but how do we link mailing lists to an automatic test system?&lt;/p&gt;
&lt;p&gt;As it turns out, this is something we're working on: we already have &lt;a href="http://patchwork.ozlabs.org/"&gt;Patchwork&lt;/a&gt;, and &lt;a href="https://openpower.xyz/"&gt;Jenkins&lt;/a&gt;: how do we link them? We have something brewing, which we'll open source real soon now - stay tuned!&lt;/p&gt;
&lt;h4&gt;Usable formal methods - are we there yet?&lt;/h4&gt;
&lt;p&gt;I liked &lt;a href="https://www.youtube.com/watch?v=RxHjhBVOCSU"&gt;this talk&lt;/a&gt;, as I have a soft spot for formal methods (as I have a soft spot for maths). It covers applying a bunch of static analysis and some of the less intrusive formal methods (in particular &lt;a href="http://www.cprover.org/cbmc/"&gt;cbmc&lt;/a&gt;) to an operating system kernel. They were looking at eChronos rather than Linux, but it's still quite an interesting set of results.&lt;/p&gt;
&lt;p&gt;We've also tried to increase our use of static analysis, which has already found a &lt;a href="http://patchwork.ozlabs.org/patch/580629/"&gt;real bug&lt;/a&gt;. We're hoping to scale this up, especially the use of sparse and cppcheck, but we're a bit short on developer cycles for it at the moment.&lt;/p&gt;
&lt;h4&gt;Adventures in OpenPower Firmware&lt;/h4&gt;
&lt;p&gt;Stewart Smith - another OzLabber - gave &lt;a href="https://www.youtube.com/watch?v=a4XGvssR-ag"&gt;this talk&lt;/a&gt; about, well, OpenPOWER firmware. This is a large part of our lives in OzLabs, so it's a great way to get a picture of what we do each day. It's also a really good explanation of the open source stack we have: a POWER8 CPU runs open-source from the first cycle.&lt;/p&gt;
&lt;h4&gt;What Happens When 4096 Cores &lt;code&gt;All Do synchronize_rcu_expedited()&lt;/code&gt;?&lt;/h4&gt;
&lt;p&gt;Paul McKenney is a parallel programming genius - he literally &lt;a href="https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html"&gt;'wrote the book'&lt;/a&gt; (or at least, wrote &lt;em&gt;a&lt;/em&gt; book!) on it. &lt;a href="https://www.youtube.com/watch?v=1nfpjHTWaUc"&gt;His talk&lt;/a&gt; is - as always - a brain-stretching look at parallel programming within the RCU subsystem of the Linux kernel. In particular, the tree structure for locking that he presents is really interesting and quite a clever way of scaling what at first seems to be a necessarily global lock.&lt;/p&gt;
&lt;p&gt;I'd also really recommed &lt;a href="https://www.youtube.com/watch?v=tFmajPt0_hI"&gt;RCU Mutation Testing&lt;/a&gt;, from the kernel miniconf, also by Paul.&lt;/p&gt;
&lt;h4&gt;What I've learned as the kernel docs maintainer&lt;/h4&gt;
&lt;p&gt;As an extra bonus: I mention &lt;a href="https://www.youtube.com/watch?v=gsJXf6oSbAE"&gt;this talk&lt;/a&gt;, just to say "why on earth have we still not fixed the Linux kernel &lt;a href="https://www.kernel.org/doc/linux/README"&gt;README&lt;/a&gt;"?!!?&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Tue, 15 Mar 2016 11:30:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-03-15:blog/2016/03/15/linuxconfau-2016-a-set-of-brief-thoughts/</guid><category>lca2016</category><category>conferences</category></item><item><title>Learning From the Best</title><link>https://sthbrx.github.io/blog/2016/03/03/learning-from-the-best/</link><description>&lt;p&gt;When I first started at IBM I knew how to alter Javascript and compile it. This is because of my many years playing Minecraft (yes I am a nerd). Now I have leveled up! I can understand and use Bash, Assembly, Python, Ruby and C! Writing full programs in any of these languages is a very difficult prospect but none the less achievable with what I know now. Whereas two weeks ago it would have been impossible. Working here even for a short time has been an amazing Learning experience for me, plus it looks great on a resume! Learning how to write C has been one of the most useful things I have learnt. I have already written programs for use both in and out of IBM. The first program I wrote was the standard newbie 'hello world' exercise. I have now expanded on that program so that it now says, "Hello world! This is Callum Scarvell". This is done using strings that recognise my name as a set character. Then I used a header file called conio.h or curses.h to recognise 'cal' as the short form of my name. This is so now I can abbreviate my name easier. Heres what the code looks like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span class="cp"&gt;#include &amp;lt;stdio.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include &amp;lt;string.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include &amp;lt;curses.h&amp;gt;&lt;/span&gt;

&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Hello, World! This Is cal&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="n"&gt;first_name&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Callum&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Scarvell&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

  &lt;span class="cm"&gt;/* testing code */&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strncmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;first_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Callum&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strncmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Scarvell&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sc"&gt;&amp;#39;S&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;sprintf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;%s %s&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;first_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;last_name&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strncmp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Callum Scarvell&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;This is %s&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="cm"&gt;/*printf(&amp;quot;actual string is -%s-\n&amp;quot;,name);*/&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;Name_Rec&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
 &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
 &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

 &lt;span class="n"&gt;clrscr&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
 &lt;span class="n"&gt;puts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Callum Scarvell : &lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
 &lt;span class="n"&gt;gets&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

 &lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s"&gt;cal : &lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s"&gt;%c&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
 &lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="sc"&gt;&amp;#39;\0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The last two lines have been left out to make it a challenge to recreate. Feel free to test your own knowledge of C to finish the program! My ultimate goal for this program is to make it generate the text 'Hello World! This is Callum Scarvell's computer. Everybody else beware!'(which is easy) then import it into the Linux kernel to the profile login screen. Then I will have my own unique copy of the kernel. And I could call myself an LSD(Linux system developer). That's just a small pet project I have been working on in my time here. Another pet project of mine is my own very altered copy of the open source game NetHack. It's written in C as well and is very easy to tinker with. I have been able to do things like set my characters starting hit points to 40, give my character awesome starting gear and keep save files even after the death of a character. These are just a couple small projects that made learning C so much easier and a lot more fun. And the whole time I was learning C, Ruby, or Python I had some of the best system developers in the world showing me the ropes. This made things even easier, and much more comprehensive. So really its no surprise that in three short weeks I managed to learn almost four different languages and how to run a blog from the raw source code. The knowledge given to me by the OzLabs team is priceless and invaluable. I will forever remember all the new faces and what they taught me. And the &lt;em&gt;Linux Gods&lt;/em&gt; will answer your prayers whether e-mail or in person because they walk among us! So if you ever get an opportunity to do work experience, internship or a graduate placement take the chance to do it because you will learn many things that are not taught in school.&lt;/p&gt;
&lt;p&gt;If you would like to reveiw the source code for the blog or my work in general you can find me at &lt;a href="https://github.com/CallumScar"&gt;CallumScar.github.com&lt;/a&gt; or find me on facebook, &lt;a href="https://www.facebook.com/callum.scarvell/about"&gt;Callum Scarvell&lt;/a&gt;. &lt;/p&gt;
&lt;p&gt;And a huge thankyou to the OzLabs team for taking me on for the three weeks and for teaching me so much! I am forever indebted to everyone here.   &lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Callum Scarvell</dc:creator><pubDate>Thu, 03 Mar 2016 00:00:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-03-03:blog/2016/03/03/learning-from-the-best/</guid><category>education</category><category>work experience</category><category>Linux Gods</category></item><item><title>Work Experience At Ozlabs</title><link>https://sthbrx.github.io/blog/2016/02/25/work-experience-at-ozlabs/</link><description>&lt;p&gt;As a recent year twelve graduate my knowledge of computer science was very limited and my ability to write working programs was all but none. So you can imagine my excitement when I heard of an opening for work experience with IBM's internationally renowned Ozlabs team, or as I knew them the &lt;em&gt;Linux Gods&lt;/em&gt;. My first day of working at Ozlabs I learnt more about programing then in six years of secondary education. I met most of the Ozlabs team and made connections that will certainly help with my pursuit of a career in IT. Because in business its who you know more than what you know, and now I know the guys at Ozlabs I know how to write code and run it on my own Linux Distro. And on top of all the extremely valuable knowledge I am on a first name basis with the &lt;em&gt;Linux Gods&lt;/em&gt; at the LTC.&lt;/p&gt;
&lt;p&gt;After my first week at Ozlabs I cloned this blog from Octopress and reformatted it for pelican static site generator.For those who don't know Octopress is a ruby based static site generator so converting the embedded ruby gems to pelicans python code was no easy task for this newbie. Luckily I had a team of some of the best software developers in the world to help and teach me their ways. After we sorted the change from ruby to python and I was able to understand both languages, I presented my work to the team. They then decided to throw me a curve ball as they did not like any of pelicans default themes, instead they wanted the original Octopress theme on the new blog. This is how I learnt GitHub is my bestest friend, because some kind soul had already converted the ruby theme into python and it ran perfectly!&lt;/p&gt;
&lt;p&gt;Now it was a simple task of re-formatting the ruby-gem text files into markdown which is pelican compatible(which is why we chose pelican in the first place). So now we had a working pelican blog on the Octopress theme, one issue it was very annoying to navigate. Using my newly learned skills and understanding of python I inserted tags, categories, web-links, navigation bar and I started learning how to code C. And it all worked fine! That was what I a newbie could accomplish in one week. I still have two more weeks left here and I have plenty of really interesting work left to do. This has been one of the greatest learning experiences of my life and I would do it again if I could! So if you are looking for experience in it or software development look no further because you could be learning to code from the people who wrote the language itself. The &lt;em&gt;Linux Gods&lt;/em&gt;.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Callum Scarvell</dc:creator><pubDate>Thu, 25 Feb 2016 00:00:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-02-25:blog/2016/02/25/work-experience-at-ozlabs/</guid><category>Work Experience</category><category>Ozlabs</category></item><item><title>Panic, flushing and compromise</title><link>https://sthbrx.github.io/blog/2016/02/15/panic/</link><description>&lt;p&gt;This is a tale of a simple problem, with a relatively simple solution, that ended up being pretty complicated.&lt;/p&gt;
&lt;p&gt;The BMC of OpenPOWER machines expose a serial console.  It's pretty useful for getting information as the system is booting, or when it's having issues and the network is down.  OpenPOWER machines also have runtime firmware, namely &lt;a href="https://github.com/open-power/skiboot"&gt;skiboot&lt;/a&gt;, which the Linux kernel calls to make certain things happen.  One of those is writing to the serial console.  There's a function that &lt;a href="https://github.com/open-power/skiboot/blob/master/core/opal.c"&gt;skiboot exposes&lt;/a&gt;, &lt;code&gt;opal_poll_events()&lt;/code&gt; (which then calls &lt;code&gt;opal_run_pollers()&lt;/code&gt;), which the kernel calls frequently.  Among other things, it performs a partial flush of the serial console.  And that all works fine...until the kernel panics.&lt;/p&gt;
&lt;p&gt;Well, the kernel is in panic.  Who cares if it flushes the console?  It's dead.  It doesn't need to do anything else.&lt;/p&gt;
&lt;p&gt;Oh, right.  It prints the reason it panicked.  Turns out that's pretty useful.&lt;/p&gt;
&lt;p&gt;There's a pretty simple fix here that we can push into the firmware.  Most kernels are configured to reboot after panic, typically with some delay.  In OpenPOWER, the kernel reboots by calling into skiboot with the &lt;code&gt;opal_cec_reboot()&lt;/code&gt; function.  So all we need to do is flush out the console buffer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="n"&gt;int64&lt;/span&gt; &lt;span class="nf"&gt;opal_cec_reboot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;OPAL: Reboot request...&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="n"&gt;console_complete_flush&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="c1"&gt;// &amp;lt;-- what I added&lt;/span&gt;

    &lt;span class="c1"&gt;// rebooting stuff happens here...&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;OPAL_SUCCESS&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Writing a complete flushing function was pretty easy, then call it from the power down and reboot functions.  Easy, all nicely contained in firmware.&lt;/p&gt;
&lt;p&gt;Now, what if the kernel isn't configured to reboot after panic.  Or, what if the reboot timer is really long?  Do you want to wait 3 minutes to see your panic output?  Probably not.  We need to call the pollers after panic.&lt;/p&gt;
&lt;p&gt;First, I had to figure out what the kernel actually &lt;em&gt;does&lt;/em&gt; when it panics.  Let's have a look at the &lt;a href="https://github.com/torvalds/linux/blob/master/kernel/panic.c"&gt;panic function itself&lt;/a&gt; to figure out where we could work some code in.&lt;/p&gt;
&lt;p&gt;In the &lt;code&gt;panic()&lt;/code&gt; function, the easiest place I found to put in some code was &lt;code&gt;panic_blink()&lt;/code&gt;.  This is supposed to be a function to blink the LEDs on your keyboard when the kernel is panicking, but we could set it to &lt;code&gt;opal_poll_events()&lt;/code&gt; and it'd work fine.  There, problem solved!&lt;/p&gt;
&lt;p&gt;Oh, wait.  That will never get accepted upstream, ever.  Let's try again.&lt;/p&gt;
&lt;p&gt;Well, there are &lt;code&gt;#ifdef&lt;/code&gt;s in the code that are architecture specific, for s390 and SPARC.  I could add an &lt;code&gt;#ifdef&lt;/code&gt; to check if we're an OpenPOWER machine, and if so, run the pollers a bunch of times.  That would also involve including architecture specific code from &lt;code&gt;arch/powerpc&lt;/code&gt;, and that's somewhat gross.  Maybe I could upstream this, but it'd be difficult.  There must be a better way.&lt;/p&gt;
&lt;p&gt;As a kernel noob, I found myself digging into what every function called by &lt;code&gt;panic()&lt;/code&gt; actually did, to see if there's a way I could use it.  I looked over it at first, but eventually I started looking harder at this line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;    &lt;span class="n"&gt;kmsg_dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;KMSG_DUMP_PANIC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;It turns out &lt;code&gt;kmsg_dump()&lt;/code&gt; does what it says: dumps messages from the kernel.  Different parts of the kernel can register their own dumpers, so the kernel can have a variety of dumpers for different purposes.  One existing example in OpenPOWER is a kmsg dumper that stores messages in &lt;code&gt;nvram&lt;/code&gt; (non-volatile RAM), so you can find it after you reboot.&lt;/p&gt;
&lt;p&gt;Well, we don't really want to dump any output, it's already been sent to the output buffer.  We just need to flush it.  Pretty simple, just call &lt;code&gt;opal_poll_events()&lt;/code&gt; a whole bunch of times, right?  That &lt;em&gt;would&lt;/em&gt; work, though it'd be nice to have a better way than just calling the pollers.  Instead, we can add a new API call to skiboot specifically for console flushing, and call it from the kmsg dumper.&lt;/p&gt;
&lt;p&gt;Initially, I wired up the skiboot complete console flushing function to a new OPAL API call, and called that from the kernel.  After some feedback, this was refactored into a partial, incremental flush so it was more generic.  I also had to consider what happened if the machine was running a newer kernel and an older skiboot, so if the skiboot version didn't have my new flushing call it would fall back to calling the pollers an arbitrary amount of times.&lt;/p&gt;
&lt;p&gt;In the end, it looks like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt; * Console output is controlled by OPAL firmware.  The kernel regularly calls&lt;/span&gt;
&lt;span class="cm"&gt; * OPAL_POLL_EVENTS, which flushes some console output.  In a panic state,&lt;/span&gt;
&lt;span class="cm"&gt; * however, the kernel no longer calls OPAL_POLL_EVENTS and the panic message&lt;/span&gt;
&lt;span class="cm"&gt; * may not be completely printed.  This function does not actually dump the&lt;/span&gt;
&lt;span class="cm"&gt; * message, it just ensures that OPAL completely flushes the console buffer.&lt;/span&gt;
&lt;span class="cm"&gt; */&lt;/span&gt;
&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;force_opal_console_flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;kmsg_dumper&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;dumper&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                                     &lt;span class="k"&gt;enum&lt;/span&gt; &lt;span class="n"&gt;kmsg_dump_reason&lt;/span&gt; &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kt"&gt;int64_t&lt;/span&gt; &lt;span class="n"&gt;ret&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt;     * Outside of a panic context the pollers will continue to run,&lt;/span&gt;
&lt;span class="cm"&gt;     * so we don&amp;#39;t need to do any special flushing.&lt;/span&gt;
&lt;span class="cm"&gt;     */&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reason&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;KMSG_DUMP_PANIC&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;opal_check_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;OPAL_CONSOLE_FLUSH&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;ret&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;opal_console_flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ret&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;OPAL_UNSUPPORTED&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;ret&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;OPAL_PARAMETER&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="cm"&gt;/* Incrementally flush until there&amp;#39;s nothing left */&lt;/span&gt;
        &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;opal_console_flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;OPAL_SUCCESS&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="cm"&gt;/*&lt;/span&gt;
&lt;span class="cm"&gt;         * If OPAL_CONSOLE_FLUSH is not implemented in the firmware,&lt;/span&gt;
&lt;span class="cm"&gt;         * the console can still be flushed by calling the polling&lt;/span&gt;
&lt;span class="cm"&gt;         * function enough times to flush the buffer.  We don&amp;#39;t know&lt;/span&gt;
&lt;span class="cm"&gt;         * how much output still needs to be flushed, but we can be&lt;/span&gt;
&lt;span class="cm"&gt;         * generous since the kernel is in panic and doesn&amp;#39;t need&lt;/span&gt;
&lt;span class="cm"&gt;         * to do much else.&lt;/span&gt;
&lt;span class="cm"&gt;         */&lt;/span&gt;
        &lt;span class="n"&gt;printk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;KERN_NOTICE&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;opal: OPAL_CONSOLE_FLUSH missing.&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;opal_poll_events&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;You can find the full code in-tree &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/platforms/powernv/opal-kmsg.c"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And thus, panic messages now roam free 'cross the countryside, causing developer frustration around the world.  At least now they know why they're frustrated.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Russell Currey</dc:creator><pubDate>Mon, 15 Feb 2016 14:23:00 +1100</pubDate><guid>tag:sthbrx.github.io,2016-02-15:blog/2016/02/15/panic/</guid><category>openpower</category></item><item><title>Evolving into a systems programmer</title><link>https://sthbrx.github.io/blog/2015/11/06/evolving-into-a-systems-programmer/</link><description>&lt;p&gt;In a previous life I tutored first year computing. The university I
attended had a policy of using C to introduce first years to programming.
One of the most rewarding aspects of teaching is opening doors of
possibility to people by sharing my knowledge.&lt;/p&gt;
&lt;p&gt;Over the years I had a mixture of computer science or computer engineering
students as well as other disciplines of engineering who were required to
learn the basics (notably electrical and mechanical). Each class was
different and the initial knowledge always varied greatly. The beauty of
teaching C meant that there was never someone who truly knew it all, heck,
I didn't and still don't. The other advantage of teaching C is that I could
very quickly spot the hackers, the shy person at the back of the room who's
eyes light up when you know you've correctly explained pointers (to them
anyway) or when asked "What happens if you use a negative index into an
array" and the smile they would make upon hearing "What do you think happens".&lt;/p&gt;
&lt;p&gt;Right there I would see the makings of a hacker, and this post is dedicated
to you or to anyone who wants to be a hacker. I've been asked "What did you
do to get where you are?", "How do I get into Linux?" (vague much) at
careers fairs. I never quite know what to say, here goes a braindump.&lt;/p&gt;
&lt;p&gt;Start with the basics, one of the easiest way we tested the first years was
to tell them they can't use parts of libc. That was a great exam, taking
aside those who didn't read the question and used &lt;code&gt;strlen()&lt;/code&gt; when they were
explicitly told they couldn't &lt;code&gt;#include &amp;lt;string.h&amp;gt;&lt;/code&gt; a true hacker doesn't
need libc, understand it won't always be there. I thought of this example
because only two weeks ago I was writing code in an environment where I
didn't have libc. Ok sure, if you've got it, use it, just don't crumble
when you don't. Oh how I wish I could have told those students who argued
that it was a pointless question that they were objectively wrong.&lt;/p&gt;
&lt;p&gt;Be a fan of assembly, don't be afraid of it, it doesn't bite and it can be
a lot of fun. I wouldn't encourage you to dive right into the PowerISA,
it's intense but perhaps understand the beauty of GCC, know what it's doing
for you. There is a variety of little 8 bit processors you can play with
these days.&lt;/p&gt;
&lt;p&gt;At all levels of my teaching I saw almost everyone get something which
'worked', and that's fine, it probably does but I'm here to tell you that
it doesn't work until you know why it works. I'm all for the 'try it and
see' approach but once you've tried it you have to explain why the
behaviour changed otherwise you didn't fix it. As an extension to that,
know how your tools work, I don't think anyone would expect you to be able
to write tools to the level of complexity of GCC or GDB or Valgrind but
have a rough idea as to how they achieve their goals.&lt;/p&gt;
&lt;p&gt;A hacker is paranoid, yes, &lt;code&gt;malloc()&lt;/code&gt; fails. Linux might just decide now
isn't a good time for you to &lt;code&gt;open()&lt;/code&gt; and your &lt;code&gt;fopen()&lt;/code&gt; calling function had
better be cool with that. A hacker also doesn't rely on the kindness of the
operating system theres an &lt;code&gt;munmap()&lt;/code&gt; for a reason. Nor should you even
completely trust it, what are you leaving around in memory?&lt;/p&gt;
&lt;p&gt;Above all do a it for the fun of it, so many of my students asked how I
knew everything I knew (I was only a year ahead of them in my first year of
teaching) and put simply, write code on a Saturday night.&lt;/p&gt;
&lt;p&gt;None of these things do or don't make you a hacker, being a hacker is a
frame of mind and a way of thinking but all of the above helps.&lt;/p&gt;
&lt;p&gt;Unfortunately there isn't a single path, I might even say it is a path that
chooses you. Odds are you're here because you approached me at some point
and asked me one of those questions I never quite know how to answer.
Perhaps this is the path, at the very least you're asking questions and
approaching people. I'm hope I did on the day, but once again, all the very
best with your endeavours into the future&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Fri, 06 Nov 2015 11:13:00 +1100</pubDate><guid>tag:sthbrx.github.io,2015-11-06:blog/2015/11/06/evolving-into-a-systems-programmer/</guid><category>education</category><category>offtopic</category></item><item><title>What the HILE is this?</title><link>https://sthbrx.github.io/blog/2015/11/03/what-the-hile-is-this/</link><description>&lt;p&gt;One of the cool features of POWER8 processors is the ability to run in either big- or little-endian mode. Several distros are already available in little-endian, but up until recently Petitboot has remained big-endian. While it has no effect on the OS, building Petitboot little-endian has its advantages, such as making support for vendor tools easier.
So it should just be a matter of compiling Petitboot LE right? Well...&lt;/p&gt;
&lt;h3&gt;Switching Endianess&lt;/h3&gt;
&lt;p&gt;Endianess, and several other things besides, are controlled by the Machine State Register (MSR). Each processor in a machine has an MSR, and each bit of the MSR controls some aspect of the processor such as 64-bit mode or enabling interrupts. To switch endianess we set the LE bit (63) to 1.&lt;/p&gt;
&lt;p&gt;When a processor first starts up it defaults to big-endian (bit 63 = 0). However the processor doesn't actually know the endianess of the kernel code it is about to execute - either it is big-endian and everything is fine, or it isn't and the processor will very quickly try to execute an illegal instruction.&lt;/p&gt;
&lt;p&gt;The solution to this is an amazing little snippet of code in &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/boot/ppc_asm.h#L65"&gt;arch/powerpc/boot/ppc_asm.h&lt;/a&gt; (follow the link to see some helpful commenting):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span class="cp"&gt;#define FIXUP_ENDIAN&lt;/span&gt;
    &lt;span class="n"&gt;tdi&lt;/span&gt;   &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mh"&gt;0x48&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;b&lt;/span&gt;     &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;36&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0x05009f42&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0xa602487d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0x1c004a39&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0xa600607d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0x01006b69&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0xa6035a7d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0xa6037b7d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="mh"&gt;0x2400004c&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;By some amazing coincidence if you take the opcode for &lt;code&gt;tdi 0, 0, 0x48&lt;/code&gt; and flip the order of the bytes it forms the opcode for &lt;code&gt;b . + 8&lt;/code&gt;. So if the kernel is big-endian, the processor will jump to the next instruction after this snippet. However if the kernel is little-endian we execute the next 8 instructions. These are written in reverse so that if the processor isn't in the right endian it interprets them backwards, executing the instructions shown in the linked comments above, resulting in MSR&lt;sub&gt;LE&lt;/sub&gt; being set to 1.&lt;/p&gt;
&lt;p&gt;When booting a little-endian kernel all of the above works fine - but there is a problem for Petitboot that will become apparent a little further down...&lt;/p&gt;
&lt;h3&gt;Petitboot's Secret Sauce&lt;/h3&gt;
&lt;p&gt;The main feature of Petitboot is that it is a full (but small!) Linux kernel and userspace which scans all available devices and presents possible boot options. To boot an available operating system Petitboot needs to start executing the OS's kernel, which it accomplishes via &lt;a href="https://en.wikipedia.org/wiki/Kexec"&gt;kexec&lt;/a&gt;. Simply speaking kexec loads the target kernel into memory, shuts the current system down most of the way, and at the last moment sets the instruction pointer to the start of the target kernel. From there it's like booting any other kernel, including the FIXUP_ENDIAN section above.&lt;/p&gt;
&lt;h3&gt;We've Booted! Wait...&lt;/h3&gt;
&lt;p&gt;So our LE Petitboot kernel boots fine thanks to FIXUP_ENDIAN, we kexec into some other kernel.. and everything falls to pieces.&lt;br /&gt;
The problem is we've unwittingly changed one of the assumptions of booting a kernel; namely that MSR&lt;sub&gt;LE&lt;/sub&gt; defaults to zero. When kexec-ing from an LE kernel we start executing the next kernel in LE mode. This itself is ok, the FIXUP_ENDIAN macro will handle the switch if needed. The problem is that the FIXUP_ENDIAN macro is relatively recent, first entering the kernel in early 2014. So if we're booting, say, an old Fedora 19 install with a v3.9 kernel - things go very bad, very quickly.&lt;/p&gt;
&lt;h3&gt;Fix #1&lt;/h3&gt;
&lt;p&gt;The solution seems pretty straightforward: find where we jump into the next kernel, and just before that make sure we reset the LE bit in the MSR. That's exactly what &lt;a href="https://github.com/antonblanchard/kexec-lite/commit/150b14e76a4b51f865b929ad9a9bf4133e2d3af7"&gt;this patch&lt;/a&gt; to kexec-lite does.&lt;br /&gt;
That worked up until I tested on a machine with more than one CPU. Remembering that the MSR is processor-specific, we also have to &lt;a href="https://github.com/torvalds/linux/commit/ffebf5f391dfa9da3e086abad3eef7d3e5300249"&gt;reset the endianess of each secondary CPU&lt;/a&gt;&lt;br /&gt;
Now things are looking good! All the CPUs are reset to big-endian, the target kernel boots fine, and then... 'recursive interrupts?!'&lt;/p&gt;
&lt;h3&gt;HILE&lt;/h3&gt;
&lt;p&gt;Skipping the debugging process that led to this (hint: &lt;a href="https://www.flamingspork.com/blog/2014/12/03/running-skiboot-opal-on-the-power8-simulator/"&gt;mambo&lt;/a&gt; is actually a pretty cool tool), these were the sequence of steps leading up to the problem:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Little-endian Petitboot kexecs into a big-endian kernel&lt;/li&gt;
&lt;li&gt;All CPUs are reset to big-endian&lt;/li&gt;
&lt;li&gt;The big-endian kernel begins to boot successfully&lt;/li&gt;
&lt;li&gt;Somewhere in the device-tree parsing code we take an exception&lt;/li&gt;
&lt;li&gt;Execution jumps to the exception handler at &lt;a href="https://github.com/torvalds/linux/blob/master/arch/powerpc/kernel/exceptions-64s.S#L199"&gt;0x300&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;I notice that MSR&lt;sub&gt;LE&lt;/sub&gt; is set to 1&lt;/li&gt;
&lt;li&gt;WHAT WHY IS THE LE BIT IN THE MSR SET TO 1&lt;/li&gt;
&lt;li&gt;We fail to read the first instruction at 0x300 because it's written in big-endian, so we jump to the exception handler at 0x300... oh no.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And then we very busily execute nothing until the machine is killed. I spend some time staring incredulously at my screen, then appeal to a &lt;a href="https://github.com/torvalds/linux/blob/master/MAINTAINERS"&gt;higher authority&lt;/a&gt; who replies with "What is the HILE set to?"  &lt;/p&gt;
&lt;p&gt;..the WHAT?&lt;br /&gt;
Cracking open the &lt;a href="https://www.power.org/documentation/power-isa-v-2-07b/"&gt;PowerISA&lt;/a&gt; reveals this tidbit:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The Hypervisor Interrupt Little-Endian (HILE) bit is a bit
in an implementation-dependent register or similar
mechanism. The contents of the HILE bit are copied
into MSR&lt;sub&gt;LE&lt;/sub&gt; by interrupts that set MSR&lt;sub&gt;HV&lt;/sub&gt; to 1 (see Section
6.5), to establish the Endian mode for the interrupt
handler. The HILE bit is set, by an implementation-dependent
method, during system initialization,
and cannot be modified after system initialization.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;To be fair, there are use cases for taking exceptions in a different endianess. The problem is that while HILE gets switched on when setting MSR&lt;sub&gt;LE&lt;/sub&gt; to 1, it &lt;em&gt;doesn't&lt;/em&gt; get turned off when MSR&lt;sub&gt;LE&lt;/sub&gt; is set to zero. In particular the line "...cannot be modified after system initialization." led to a fair amount of hand wringing from myself and whoever would listen; if we can't reset the HILE bit, we simply can't use little-endian kernels for Petitboot.  &lt;/p&gt;
&lt;p&gt;Luckily while on some other systems the machinations of the firmware might be a complete black box, Petitboot runs on OPAL systems - which means the firmware source is &lt;a href="https://github.com/open-power/skiboot"&gt;right here&lt;/a&gt;. In particular we can see here the OPAL call to &lt;a href="https://github.com/open-power/skiboot/blob/master/core/cpu.c#L702"&gt;opal_reinit_cpus&lt;/a&gt; which among other things resets the HILE bit.&lt;br /&gt;
This is actually what turns on the HILE bit in the first place, and is meant to be called early on in boot since it also clobbers a large amount of state. Luckily for us we don't need to hold onto any state since we're about to jump into a new kernel. We just need to choose an appropriate place where we can be sure we won't take an exception before we get into the next kernel: thus the &lt;a href="https://github.com/torvalds/linux/commit/e72bb8a5a884d022231149d407653923a1d79e53"&gt;final patch to support PowerNV machines.&lt;/a&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 03 Nov 2015 15:02:00 +1100</pubDate><guid>tag:sthbrx.github.io,2015-11-03:blog/2015/11/03/what-the-hile-is-this/</guid><category>petitboot</category><category>power</category><category>p8</category><category>openpower</category><category>goodposts</category><category>autoboot</category><category>realcontent</category><category>kexec</category><category>kernel</category></item><item><title>Docker: Just Stop Using AUFS</title><link>https://sthbrx.github.io/blog/2015/10/30/docker-just-stop-using-aufs/</link><description>&lt;p&gt;Docker's default storage driver on most Ubuntu installs is AUFS.&lt;/p&gt;
&lt;p&gt;Don't use it. Use Overlay instead. Here's why.&lt;/p&gt;
&lt;p&gt;First, some background. I'm testing the performance of the basic LAMP
stack on POWER. (LAMP is Linux + Apache + MySQL/MariaDB + PHP, by the
way.) To do more reliable and repeatable tests, I do my builds and
tests in Docker containers. (See &lt;a href="/blog/2015/10/12/a-tale-of-two-dockers/"&gt;my previous post&lt;/a&gt; for more info.)&lt;/p&gt;
&lt;p&gt;Each test downloads the source of Apache, MariaDB and PHP, and builds
them. This should be quick: the POWER8 system I'm building on has 160
hardware threads and 128 GB of memory. But I was finding that it was
only just keeping pace with a 2 core Intel VM on BlueMix.&lt;/p&gt;
&lt;p&gt;Why? Well, my first point of call was to observe a compilation under
&lt;code&gt;top&lt;/code&gt;. The header is below.&lt;/p&gt;
&lt;p&gt;&lt;img alt="top header, showing over 70 percent of CPU time spent in the kernel" src="/images/dja/aufs/top-bad.png" /&gt;&lt;/p&gt;
&lt;p&gt;Over 70% of CPU time is spent in the kernel?! That's weird. Let's dig
deeper.&lt;/p&gt;
&lt;p&gt;My next port of call for analysis of CPU-bound workloads is
&lt;code&gt;perf&lt;/code&gt;. &lt;code&gt;perf top&lt;/code&gt; reports astounding quantities of time in
spin-locks:&lt;/p&gt;
&lt;p&gt;&lt;img alt="display from perf top, showing 80 percent of time in a spinlock" src="/images/dja/aufs/perf-top-spinlock.png" /&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;perf top -g&lt;/code&gt; gives us some more information: the time is in system
calls. &lt;code&gt;open()&lt;/code&gt; and &lt;code&gt;stat()&lt;/code&gt; are the key culprits, and we can see a
number of file system functions are in play in the call-chains of the
spinlocks.&lt;/p&gt;
&lt;p&gt;&lt;img alt="display from perf top -g, showing syscalls and file ops" src="/images/dja/aufs/perf-top-syscalls.png" /&gt;&lt;/p&gt;
&lt;p&gt;Why are open and stat slow? Well, I know that the files are on an AUFS
mount. (&lt;code&gt;docker info&lt;/code&gt; will tell you what you're using if you're not
sure.) So, being something of a kernel hacker, I set out to find out
why. This did not go well. AUFS isn't upstream, it's a separate patch
set. Distros have been trying to deprecate it for years. Indeed, RHEL
doesn't ship it. (To it's credit, Docker seems to be trying to move
away from it.)&lt;/p&gt;
&lt;p&gt;Wanting to avoid the minor nightmare that is an out-of-tree patchset,
I looked at other storage drivers for Docker. &lt;a href="https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html"&gt;This presentation is particularly good.&lt;/a&gt;
My choices are pretty simple: AUFS, btrfs, device-mapper or
Overlay. Overlay was an obvious choice: it doesn't need me to set up
device mapper on a cloud VM, or reformat things as btrfs.&lt;/p&gt;
&lt;p&gt;It's also easy to set up on Ubuntu:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;export/save any docker containers you care about.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;add &lt;code&gt;--storage-driver=overlay&lt;/code&gt; option to &lt;code&gt;DOCKER_OPTS&lt;/code&gt; in &lt;code&gt;/etc/default/docker&lt;/code&gt;, and restart docker (&lt;code&gt;service docker restart&lt;/code&gt;)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;import/load the containters you exported&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;verify that things work, then clear away your old storage directory (&lt;code&gt;/var/lib/docker/aufs&lt;/code&gt;). &lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Having moved my base container across, I set off another build.&lt;/p&gt;
&lt;p&gt;The first thing I noticed is that images are much slower to create with Overlay. But once that finishes, and a compile starts, things run much better:&lt;/p&gt;
&lt;p&gt;&lt;img alt="top, showing close to zero system time, and around 90 percent user time" src="/images/dja/aufs/top-good.png" /&gt;&lt;/p&gt;
&lt;p&gt;The compiles went from taking painfully long to astonishingly fast. Winning.&lt;/p&gt;
&lt;p&gt;So in conclusion:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;If you use Docker for something that involves open()ing or stat()ing files&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If you want your machine to do real work, rather than spin in spinlocks&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If you want to use code that's upstream and thus much better supported&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;If you want something less disruptive than the btrfs or dm storage drivers&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;...then drop AUFS and switch to Overlay today.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Fri, 30 Oct 2015 13:30:00 +1100</pubDate><guid>tag:sthbrx.github.io,2015-10-30:blog/2015/10/30/docker-just-stop-using-aufs/</guid><category>docker</category><category>aufs</category><category>overlay</category><category>performance</category><category>power</category></item><item><title>A tale of two Dockers</title><link>https://sthbrx.github.io/blog/2015/10/12/a-tale-of-two-dockers/</link><description>&lt;p&gt;(This was published in an internal technical journal last week, and is now being published here. If you already know what Docker is, feel free to skim the first half.)&lt;/p&gt;
&lt;p&gt;Docker seems to be the flavour of the month in IT. Most attention is focussed on using Docker for the deployment of production services. But that's not all Docker is good for. Let's explore Docker, and two ways I use it as a software developer.&lt;/p&gt;
&lt;p&gt;Docker: what is it?&lt;/p&gt;
&lt;p&gt;Docker is essentially a set of tools to deal with &lt;em&gt;containers&lt;/em&gt; and &lt;em&gt;images&lt;/em&gt;. &lt;/p&gt;
&lt;p&gt;To make up an artificial example, say you are developing a web app. You first build an &lt;em&gt;image&lt;/em&gt;: a file system which contains the app, and some associated metadata. The app has to run on something, so you also install things like Python or Ruby and all the necessary libraries, usually by installing a minimal Ubuntu and any necessary packages.&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1" rel="footnote"&gt;1&lt;/a&gt;&lt;/sup&gt; You then run the image inside an isolated environment called a &lt;em&gt;container&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;You can have multiple containers running the same image, (for example, your web app running across a fleet of servers) and the containers don't affect each other.  Why? Because Docker is designed around the concept of &lt;em&gt;immutability&lt;/em&gt;. Containers can write to the image they are running, but the changes are specific to that container, and aren't preserved beyond the life of the container.&lt;sup id="fnref:2"&gt;&lt;a class="footnote-ref" href="#fn:2" rel="footnote"&gt;2&lt;/a&gt;&lt;/sup&gt; Indeed, once built, images can't be changed at all, only rebuilt from scratch.&lt;/p&gt;
&lt;p&gt;However, as well as enabling you to easily run multiple copies, another upshot of immutability is that if your web app allows you to upload photos, and you restart the container, your photos will be gone. Your web app needs to be designed to store all of the data outside of the container, sending it to a dedicated database or object store of some sort.&lt;/p&gt;
&lt;p&gt;Making your application Docker friendly is significantly more work than just spinning up a virtual machine and installing stuff. So what does all this extra work get you? Three main things: isolation, control and, as mentioned, immutability. &lt;/p&gt;
&lt;p&gt;&lt;em&gt;Isolation&lt;/em&gt; makes containers easy to migrate and deploy, and easy to update. Once an image is built, it can be copied to another system and launched. Isolation also makes it easy to update software your app depends on: you rebuild the image with software updates, and then just deploy it. You don't have to worry about service A relying on version X of a library while service B depends on version Y; it's all self contained. &lt;/p&gt;
&lt;p&gt;&lt;em&gt;Immutability&lt;/em&gt; also helps with upgrades, especially when deploying them across multiple servers. Normally, you would upgrade your app on each server, and have to make sure that every server gets all the same sets of updates. With Docker, you don't upgrade a running container. Instead, you rebuild your Docker image and re-deploy it, and you then know that the same version of everything is running everywhere. This immutability also guards against the situation where you have a number of different servers that are all special snowflakes with their own little tweaks, and you end up with a fractal of complexity.&lt;/p&gt;
&lt;p&gt;Finally, Docker offers a lot of &lt;em&gt;control&lt;/em&gt; over containers, and for a low performance penalty. Docker containers can have their CPU, memory and network controlled easily, without the overhead of a full virtual machine. This makes it an attractive solution for running untrusted executables.&lt;sup id="fnref:3"&gt;&lt;a class="footnote-ref" href="#fn:3" rel="footnote"&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;As an aside: despite the hype, very little of this is actually particularly new. Isolation and control are not new problems. All Unixes, including Linux, support 'chroots'. The name comes from “change root”: the system call changes the processes idea of what the file system root is, making it impossible for it to access things outside of the new designated root directory.  FreeBSD has jails, which are more powerful, Solaris has Zones, and AIX has WPARs. Chroots are fast and low overhead. However, they offer much lower ability to control the use of system resources. At the other end of the scale, virtual machines (which have been around since ancient IBM mainframes) offer isolation much better than Docker, but with a greater performance hit.&lt;/p&gt;
&lt;p&gt;Similarly, immutability isn't really new: Heroku and AWS Spot Instances are both built around the model that you get resources in a known, consistent state when you start, but in both cases your changes won't persist. In the development world, modern CI systems like Travis CI also have this immutable or disposable model – and this was originally built on VMs. Indeed, with a little bit of extra work, both chroots and VMs can give the same immutability properties that Docker gives.&lt;/p&gt;
&lt;p&gt;The control properties that Docker provides are largely as a result of leveraging some Linux kernel concepts, most notably something called namespaces.&lt;/p&gt;
&lt;p&gt;What Docker does well is not something novel, but the engineering feat of bringing together fine-grained control, isolation and immutability, and – importantly – a tool-chain that is easier to use than any of the alternatives. Docker's tool-chain eases a lot of pain points with regards to building containers: it's vastly simpler than chroots, and easier to customise than most VM setups. Docker also has a number of engineering tricks to reduce the disk space overhead of isolation.&lt;/p&gt;
&lt;p&gt;So, to summarise: Docker provides a toolkit for isolated, immutable, finely controlled containers to run executables and services.&lt;/p&gt;
&lt;h2&gt;Docker in development: why?&lt;/h2&gt;
&lt;p&gt;I don't run network services at work; I do performance work. So how do I use Docker?&lt;/p&gt;
&lt;p&gt;There are two things I do with Docker: I build PHP 5, and do performance regression testing on PHP 7. They're good case studies of how isolation and immutability provide real benefits in development and testing, and how the Docker tool chain makes life a lot nicer that previous solutions.&lt;/p&gt;
&lt;h3&gt;PHP 5 builds&lt;/h3&gt;
&lt;p&gt;I use the &lt;em&gt;isolation&lt;/em&gt; that Docker provides to make building PHP 5 easier. PHP 5 depends on an old version of Bison, version 2. Ubuntu and Debian long since moved to version 3. There are a few ways I could have solved this:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;I could just install the old version directly on my system in &lt;code&gt;/usr/local/&lt;/code&gt;, and hope everything still works and nothing else picks up Bison 2 when it needs Bison 3. Or I could install it somewhere else and remember to change my path correctly before I build PHP 5.&lt;/li&gt;
&lt;li&gt;I could roll a chroot by hand. Even with tools like debootstrap and schroot, working in chroots is a painful process.&lt;/li&gt;
&lt;li&gt;I could spin up a virtual machine on one of our development boxes and install the old version on that. That feels like overkill: why should I need to run an entire operating system? Why should I need to copy my source tree over the network to build it?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Docker makes it easy to have a self-contained environment that has Bison 2 built from source, and to build my latest source tree in that environment. Why is Docker so much easier?&lt;/p&gt;
&lt;p&gt;Firstly, Docker allows me to base my container on an existing container, and there's an online library of containers to build from.&lt;sup id="fnref:4"&gt;&lt;a class="footnote-ref" href="#fn:4" rel="footnote"&gt;4&lt;/a&gt;&lt;/sup&gt; This means I don't have to roll a base image with &lt;code&gt;debootstrap&lt;/code&gt; or the RHEL/CentOS/Fedora equivalent.&lt;/p&gt;
&lt;p&gt;Secondly, unlike a chroot build process, which ultimately is just copying files around, a docker build process includes the ability to both copy files from the host and &lt;em&gt;run commands&lt;/em&gt; in the context of the image. This is defined in a file called a &lt;code&gt;Dockerfile&lt;/code&gt;, and is kicked off by a single command: &lt;code&gt;docker build&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;So, my PHP 5 build container loads an Ubuntu Vivid base container, uses apt-get to install the compiler, tool-chain and headers required to build PHP 5, then installs old bison from source, copies in the PHP source tree, and builds it. The vast majority of this process – the installation of the compiler, headers and bison, can be cached, so they don't have to be downloaded each time. And once the container finishes building, I have a fully built PHP interpreter ready for me to interact with.&lt;/p&gt;
&lt;p&gt;I do, at the moment, rebuild PHP 5 from scratch each time. This is a bit sub-optimal from a performance point of view. I could alleviate this with a Docker volume, which is a way of sharing data persistently between a host and a guest, but I haven't been sufficiently bothered by the speed yet. However, Docker volumes are also quite fiddly, leading to the development of tools like &lt;code&gt;docker compose&lt;/code&gt; to deal with them. They also are prone to subtle and difficult to debug permission issues.&lt;/p&gt;
&lt;h3&gt;PHP 7 performance regression testing&lt;/h3&gt;
&lt;p&gt;The second thing I use docker for takes advantage of the throwaway nature of docker environments to prevent cross-contamination.&lt;/p&gt;
&lt;p&gt;PHP 7 is the next big version of PHP, slated to be released quite soon. I care about how that runs on POWER, and I preferably want to know if it suddenly deteriorates (or improves!). I use Docker to build a container with a daily build of PHP 7, and then I run a benchmark in it. This doesn't give me a particularly meaningful absolute number, but it allows me to track progress over time. Building it inside of Docker means that I can be sure that nothing from old runs persists into new runs, thus giving me more reliable data. However, because I do want the timing data I collect to persist, I send it out of the container over the network.&lt;/p&gt;
&lt;p&gt;I've now been collecting this data for almost 4 months, and it's plotted below, along with a 5-point moving average. The most notable feature of the graph is a the drop in benchmark time at about the middle. Sure enough, if you look at the PHP repository, you will see that a set of changes to improve PHP performance were merged on July 29: changes submitted by our very own Anton Blanchard.&lt;sup id="fnref:5"&gt;&lt;a class="footnote-ref" href="#fn:5" rel="footnote"&gt;5&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;p&gt;&lt;img alt="Graph of PHP 7 performance over time" src="/images/dja/php7-perf.png" /&gt;&lt;/p&gt;
&lt;h2&gt;Docker pain points&lt;/h2&gt;
&lt;p&gt;Docker provides a vastly improved experience over previous solutions, but there are still a few pain points. For example:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;Docker was apparently written by people who had no concept that platforms other than x86 exist. This leads to major issues for cross-architectural setups. For instance, Docker identifies images by a name and a revision. For example, &lt;code&gt;ubuntu&lt;/code&gt; is the name of an image, and &lt;code&gt;15.04&lt;/code&gt; is a revision. There's no ability to specify an architecture. So, how you do specify that you want, say, a 64-bit, little-endian PowerPC build of an image versus an x86 build? There have been a couple of approaches, both of which are pretty bad. You could name the image differently: say &lt;code&gt;ubuntu_ppc64le&lt;/code&gt;. You can also just cheat and override the &lt;code&gt;ubuntu&lt;/code&gt; name with an architecture specific version. Both of these break some assumptions in the Docker ecosystem and are a pain to work with.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Image building is incredibly inflexible. If you have one system that requires a proxy, and one that does not, you need different Dockerfiles. As far as I can tell, there are no simple ways to hook in any changes between systems into a generic Dockerfile. This is largely by design, but it's still really annoying when you have one system behind a firewall and one system out on the public cloud (as I do in the PHP 7 setup).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Visibility into a Docker server is poor. You end up with lots of different, anonymous images and dead containers, and you end up needing scripts to clean them up. It's not clear what Docker puts on your file system, or where, or how to interact with it.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Docker is still using reasonably new technologies. This leads to occasional weird, obscure and difficult to debug issues.&lt;sup id="fnref:6"&gt;&lt;a class="footnote-ref" href="#fn:6" rel="footnote"&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;Final words&lt;/h2&gt;
&lt;p&gt;Docker provides me with a lot of useful tools in software development: both in terms of building and testing. Making use of it requires a certain amount of careful design thought, but when applied thoughtfully it can make life significantly easier.&lt;/p&gt;
&lt;div class="footnote"&gt;
&lt;hr /&gt;
&lt;ol&gt;
&lt;li id="fn:1"&gt;
&lt;p&gt;There's some debate about how much stuff from the OS installation you should be using. You need to have key dynamic libraries available, but I would argue that you shouldn't be running long running processes other than your application. You shouldn't, for example, be running a SSH daemon in your container. (The one exception is that you must handle orphaned child processes appropriately: see &lt;a href="https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/"&gt;https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/&lt;/a&gt;) Considerations like debugging and monitoring the health of docker containers mean that this point of view is not universally shared.&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:2"&gt;
&lt;p&gt;Why not simply make them read only? You may be surprised at how many things break when running on a read-only file system. Things like logs and temporary files are common issues.&amp;#160;&lt;a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:3"&gt;
&lt;p&gt;It is, however, easier to escape a Docker container than a VM. In Docker, an untrusted executable only needs a kernel exploit to get to root on the host, whereas in a VM you need a guest-to-host vulnerability, which are much rarer.&amp;#160;&lt;a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:4"&gt;
&lt;p&gt;Anyone can upload an image, so this does require running untrusted code from the Internet. Sadly, this is a distinctly retrograde step when compared to the process of installing binary packages in distros, which are all signed by a distro's private key.&amp;#160;&lt;a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:5"&gt;
&lt;p&gt;See &lt;a href="https://github.com/php/php-src/pull/1326"&gt;https://github.com/php/php-src/pull/1326&lt;/a&gt;&amp;#160;&lt;a class="footnote-backref" href="#fnref:5" rev="footnote" title="Jump back to footnote 5 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id="fn:6"&gt;
&lt;p&gt;I hit this last week: &lt;a href="https://github.com/docker/docker/issues/16256"&gt;https://github.com/docker/docker/issues/16256&lt;/a&gt;, although maybe that's my fault for running systemd on my laptop.&amp;#160;&lt;a class="footnote-backref" href="#fnref:6" rev="footnote" title="Jump back to footnote 6 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Mon, 12 Oct 2015 14:14:00 +1100</pubDate><guid>tag:sthbrx.github.io,2015-10-12:blog/2015/10/12/a-tale-of-two-dockers/</guid><category>docker</category><category>php</category><category>peformance</category></item><item><title>Running ppc64le_hello on real hardware</title><link>https://sthbrx.github.io/blog/2015/06/03/ppc64le-hello-on-real-hardware/</link><description>&lt;p&gt;So today I saw &lt;a href="https://github.com/andreiw/ppc64le_hello"&gt;Freestanding “Hello World” for OpenPower&lt;/a&gt; on &lt;a href="https://news.ycombinator.com/item?id=9649490"&gt;Hacker News&lt;/a&gt;. Sadly Andrei hadn't been able to test it on real hardware, so I set out to get it running on a real OpenPOWER box. Here's what I did.&lt;/p&gt;
&lt;p&gt;Firstly, clone the repo, and, as mentioned in the README, comment out &lt;code&gt;mambo_write&lt;/code&gt;. Build it.&lt;/p&gt;
&lt;p&gt;Grab &lt;a href="https://github.com/open-power/op-build"&gt;op-build&lt;/a&gt;, and build a Habanero defconfig. To save yourself a fair bit of time, first edit &lt;code&gt;openpower/configs/habanero_defconfig&lt;/code&gt; to answer &lt;code&gt;n&lt;/code&gt; about a custom kernel source. That'll save you hours of waiting for git.&lt;/p&gt;
&lt;p&gt;This will build you a PNOR that will boot a linux kernel with Petitboot. This is almost what you want: you need Skiboot, Hostboot and a bunch of the POWER specific bits and bobs, but you don't actually want the Linux boot kernel.&lt;/p&gt;
&lt;p&gt;Then, based on &lt;code&gt;op-build/openpower/package/openpower-pnor/openpower-pnor.mk&lt;/code&gt;, we look through the output of &lt;code&gt;op-build&lt;/code&gt; for a  &lt;code&gt;create_pnor_image.pl&lt;/code&gt; command, something like this monstrosity:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;PATH="/scratch/dja/public/op-build/output/host/bin:/scratch/dja/public/op-build/output/host/sbin:/scratch/dja/public/op-build/output/host/usr/bin:/scratch/dja/public/op-build/output/host/usr/sbin:/home/dja/bin:/home/dja/bin:/home/dja/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/openpower/common/x86_64/bin" /scratch/dja/public/op-build/output/build/openpower-pnor-ed1682e10526ebd85825427fbf397361bb0e34aa/create_pnor_image.pl -xml_layout_file /scratch/dja/public/op-build/output/build/openpower-pnor-ed1682e10526ebd85825427fbf397361bb0e34aa/"defaultPnorLayoutWithGoldenSide.xml" -pnor_filename /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/pnor/"habanero.pnor" -hb_image_dir /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/hostboot_build_images/ -scratch_dir /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/openpower_pnor_scratch/ -outdir /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/pnor/ -payload /scratch/dja/public/op-build/output/images/"skiboot.lid" -bootkernel /scratch/dja/public/op-build/output/images/zImage.epapr -sbe_binary_filename "venice_sbe.img.ecc" -sbec_binary_filename "centaur_sbec_pad.img.ecc" -wink_binary_filename "p8.ref_image.hdr.bin.ecc" -occ_binary_filename /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/occ/"occ.bin" -targeting_binary_filename "HABANERO_HB.targeting.bin.ecc" -openpower_version_filename /scratch/dja/public/op-build/output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/openpower_version/openpower-pnor.version.txt&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Replace the &lt;code&gt;-bootkernel&lt;/code&gt; arguement with the path to ppc64le_hello, e.g.: &lt;code&gt;-bootkernel /scratch/dja/public/ppc64le_hello/ppc64le_hello&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;Don't forget to move it into place! &lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;mv output/host/usr/powerpc64-buildroot-linux-gnu/sysroot/pnor/habanero.pnor output/images/habanero.pnor
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Then we can use skiboot's boot test script (written by Cyril and me, coincidentally!) to flash it.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;ppc64le_hello/skiboot/external/boot-tests/boot_test.sh -vp -t hab2-bmc -P &amp;lt;path to&amp;gt;/habanero.pnor
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;It's not going to get into Petitboot, so just interrupt it after it powers up the box and connect with IPMI. It boots, kinda:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;[11012941323,5] INIT: Starting kernel at 0x20010000, fdt at 0x3044db68 (size 0x11cc3)
Hello OPAL!
           _start = 0x20010000
                              _bss   = 0x20017E28
                                                 _stack = 0x20018000
                                                                    _end   = 0x2001A000
                                                                                       KPCR   = 0x20017E50
                                                                                                          OPAL   = 0x30000000
                                                                                                                             FDT    = 0x3044DB68
                                                                                                                                                CPU0 not found?

                                                                                                                                                               Pick your poison:
                                                                                                                                                                                Choices: (MMU = disabled):
                                                                                                                                                                                                             (d) 5s delay
                                                                                                                                                                                                                            (e) test exception
    (n) test nested exception
                                (f) dump FDT
                                               (M) enable MMU
                                                                (m) disable MMU
                                                                                  (t) test MMU
                                                                                                 (u) test non-priviledged code
                                                                                                                                 (I) enable ints
                                                                                                                                                   (i) disable ints
                                                                                                                                                                      (H) enable HV dec
                                                                                                                                                                                          (h) disable HV dec
                                                                                                                                                                                                               (q) poweroff
                                                                                                                                                                                                                             1.42486|ERRL|Dumping errors reported prior to registration
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;Yes, it does wrap horribly. However, the big issue here (which you'll have to scroll to see!) is the "CPU0 not found?". Fortunately, we can fix this with a little patch to &lt;code&gt;cpu_init&lt;/code&gt; in main.c to test for a PowerPC POWER8:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;    &lt;span class="n"&gt;cpu0_node&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fdt_path_offset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fdt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;/cpus/cpu@0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cpu0_node&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;cpu0_node&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fdt_path_offset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fdt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;/cpus/PowerPC,POWER8@20&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cpu0_node&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;printk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;CPU0 not found?&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;This is definitely the &lt;em&gt;wrong&lt;/em&gt; way to do this, but it works for now.&lt;/p&gt;
&lt;p&gt;Now, correcting for weird wrapping, we get:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;Hello OPAL!
_start = 0x20010000
_bss   = 0x20017E28
_stack = 0x20018000
_end   = 0x2001A000
KPCR   = 0x20017E50
OPAL   = 0x30000000
FDT    = 0x3044DB68
Assuming default SLB size
SLB size = 0x20
TB freq = 512000000
[13205442015,3] OPAL: Trying a CPU re-init with flags: 0x2
Unrecoverable exception stack top @ 0x20019EC8
HTAB (2048 ptegs, mask 0x7FF, size 0x40000) @ 0x20040000
SLB entries:
1: E 0x8000000 V 0x4000000000000400
EA 0x20040000 -&amp;gt; hash 0x20040 -&amp;gt; pteg 0x200 = RA 0x20040000
EA 0x20041000 -&amp;gt; hash 0x20041 -&amp;gt; pteg 0x208 = RA 0x20041000
EA 0x20042000 -&amp;gt; hash 0x20042 -&amp;gt; pteg 0x210 = RA 0x20042000
EA 0x20043000 -&amp;gt; hash 0x20043 -&amp;gt; pteg 0x218 = RA 0x20043000
EA 0x20044000 -&amp;gt; hash 0x20044 -&amp;gt; pteg 0x220 = RA 0x20044000
EA 0x20045000 -&amp;gt; hash 0x20045 -&amp;gt; pteg 0x228 = RA 0x20045000
EA 0x20046000 -&amp;gt; hash 0x20046 -&amp;gt; pteg 0x230 = RA 0x20046000
EA 0x20047000 -&amp;gt; hash 0x20047 -&amp;gt; pteg 0x238 = RA 0x20047000
EA 0x20048000 -&amp;gt; hash 0x20048 -&amp;gt; pteg 0x240 = RA 0x20048000
...
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;The weird wrapping seems to be caused by NULLs getting printed to OPAL, but I haven't traced what causes that.&lt;/p&gt;
&lt;p&gt;Anyway, now it largely works! Here's a transcript of some things it can do on real hardware.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;Choices: (MMU = disabled):
   (d) 5s delay
   (e) test exception
   (n) test nested exception
   (f) dump FDT
   (M) enable MMU
   (m) disable MMU
   (t) test MMU
   (u) test non-priviledged code
   (I) enable ints
   (i) disable ints
   (H) enable HV dec
   (h) disable HV dec
   (q) poweroff
&amp;lt;press e&amp;gt;
Testing exception handling...
sc(feed) =&amp;gt; 0xFEEDFACE
Choices: (MMU = disabled):
   (d) 5s delay
   (e) test exception
   (n) test nested exception
   (f) dump FDT
   (M) enable MMU
   (m) disable MMU
   (t) test MMU
   (u) test non-priviledged code
   (I) enable ints
   (i) disable ints
   (H) enable HV dec
   (h) disable HV dec
   (q) poweroff
&amp;lt;press t&amp;gt;
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = RA 0x20010000
mapped 0xFFFFFFF000 to 0x20010000 correctly
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = unmap
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = RA 0x20011000
mapped 0xFFFFFFF000 to 0x20011000 incorrectly
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = unmap
Choices: (MMU = disabled):
   (d) 5s delay
   (e) test exception
   (n) test nested exception
   (f) dump FDT
   (M) enable MMU
   (m) disable MMU
   (t) test MMU
   (u) test non-priviledged code
   (I) enable ints
   (i) disable ints
   (H) enable HV dec
   (h) disable HV dec
   (q) poweroff
&amp;lt;press u&amp;gt;
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = RA 0x20080000
returning to user code
returning to kernel code
EA 0xFFFFFFF000 -&amp;gt; hash 0xFFFFFFF -&amp;gt; pteg 0x3FF8 = unmap
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;I also tested the other functions and they all seem to work. Running non-priviledged code with the MMU on works. Dumping the FDT and the 5s delay both worked, although they tend to stress IPMI a &lt;em&gt;lot&lt;/em&gt;. The delay seems to correspond well with real time as well.&lt;/p&gt;
&lt;p&gt;It does tend to error out and reboot quite often, usually on the menu screen, for reasons that are not clear to me. It usually starts with something entirely uninformative from Hostboot, like this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;1.41801|ERRL|Dumping errors reported prior to registration
  2.89873|Ignoring boot flags, incorrect version 0x0
&lt;/pre&gt;&lt;/div&gt;


&lt;p&gt;That may be easy to fix, but again I haven't had time to trace it.&lt;/p&gt;
&lt;p&gt;All in all, it's very exciting to see something come out of the simulator and in to real hardware. Hopefully with the proliferation of OpenPOWER hardware, prices will fall and these sorts of systems will become increasingly accessible to people with cool low level projects like this!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Wed, 03 Jun 2015 12:16:00 +1000</pubDate><guid>tag:sthbrx.github.io,2015-06-03:blog/2015/06/03/ppc64le-hello-on-real-hardware/</guid><category>openpower</category><category>power</category><category>p8</category></item><item><title>Petitboot Autoboot Changes</title><link>https://sthbrx.github.io/blog/2015/06/02/autoboot/</link><description>&lt;p&gt;The way autoboot behaves in Petitboot has undergone some significant changes recently, so in order to ward off any angry emails lets take a quick tour of how the new system works.&lt;/p&gt;
&lt;h2&gt;Old &amp;amp; Busted&lt;/h2&gt;
&lt;p&gt;For some context, here is the old (or current depending on what you're running) section of the configuration screen.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Old Autoboot" src="/images/sammj/oldstyle.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;This gives you three main options: don't autoboot, autoboot from anything, or autoboot only from a specific device. For the majority of installations this is fine, such as when you have only one default option, or know exactly which device you'll be booting from.&lt;/p&gt;
&lt;p&gt;A side note about default options: it is important to note that not all boot options are valid &lt;em&gt;autoboot&lt;/em&gt; options. A boot option is only considered for auto-booting if it is marked default, eg. 'set default' in GRUB and 'default' in PXE options.&lt;/p&gt;
&lt;h2&gt;New Hotness&lt;/h2&gt;
&lt;p&gt;Below is the new autoboot configuration.&lt;/p&gt;
&lt;p&gt;&lt;img alt="New Autoboot" src="/images/sammj/newstyle.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;The new design allows you to specify an ordered list of autoboot options.
The last two of the three buttons are self explanatory - clear the list and autoboot any device, or clear the list completely (no autoboot).&lt;/p&gt;
&lt;p&gt;Selecting the first button, 'Add Device' brings up the following screen:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Device Selection" src="/images/sammj/devices.jpg" /&gt;&lt;/p&gt;
&lt;p&gt;From here you can select any device or &lt;em&gt;class&lt;/em&gt; of device to add to the boot order. Once added to the boot order, the order of boot options can be changed with the left and right arrow keys, and removed from the list with the minus key ('-').&lt;/p&gt;
&lt;p&gt;This allows you to create additional autoboot configurations such as "Try to boot from sda2, otherwise boot from the network", or "Give priority to PXE options from eth0, otherwise try any other netboot option".
You can retain the original behaviour by only putting one option into the list (either 'Any Device' or a specific device).&lt;/p&gt;
&lt;p&gt;Presently you can add any option into the list and order them how you like - which means you can do silly things like this:&lt;/p&gt;
&lt;p&gt;&lt;img alt="If you send me a bug report with this in it I may laugh at you" src="/images/sammj/redundant.jpg" /&gt;&lt;/p&gt;
&lt;h2&gt;IPMI&lt;/h2&gt;
&lt;p&gt;Slightly prior to the boot order changes Petitboot also received an update to its IPMI handling. IPMI 'bootdev' commands allow you to override the current autoboot configuration remotely, either by specifying a device type to boot (eg. PXE), or by forcing Petitboot to boot into the 'setup' or 'safe' modes. IPMI overrides are either persistent or non-persistent. A non-persistent override will disappear after a successful boot - that is, a successful boot of a boot option, not booting to Petitboot itself - whereas a persistent override will, well, persist!&lt;/p&gt;
&lt;p&gt;If there is an IPMI override currently active, it will appear in the configuration screen with an option to manually clear it:&lt;/p&gt;
&lt;p&gt;&lt;img alt="IPMI Overrides" src="/images/sammj/ipmi.jpg" /&gt;&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;That sums up the recent changes to autoboot; a bit more flexibility in assigning priority, and options for more detailed autoboot order if you need it. New versions of Petitboot are backwards compatible and will recognise older saved settings, so updating your firmware won't cause your machines to start booting things at random.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Samuel Mendoza-Jonas</dc:creator><pubDate>Tue, 02 Jun 2015 08:11:00 +1000</pubDate><guid>tag:sthbrx.github.io,2015-06-02:blog/2015/06/02/autoboot/</guid><category>petitboot</category><category>power</category><category>p8</category><category>openpower</category><category>goodposts</category><category>autoboot</category><category>realcontent</category></item><item><title>Joining the CAPI project</title><link>https://sthbrx.github.io/blog/2015/05/27/joining-the-capi-project/</link><description>&lt;p&gt;(I wrote this blog post a couple of months ago, but it's still quite relevant.)&lt;/p&gt;
&lt;p&gt;Hi, I'm Daniel! I work in OzLabs, part of IBM's Australian Development Labs. Recently, I've been assigned to the CAPI project, and I've been given the opportunity to give you an idea of what this is, and what I'll be up to in the future!&lt;/p&gt;
&lt;h2&gt;What even is CAPI?&lt;/h2&gt;
&lt;p&gt;To help you understand CAPI, think back to the time before computers. We had a variety of machines: machines to build things, to check things, to count things, but they were all specialised --- good at one and only one thing.&lt;/p&gt;
&lt;p&gt;Specialised machines, while great at their intended task, are really expensive to develop. Not only that, it's often impossible to change how they operate, even in very small ways.&lt;/p&gt;
&lt;p&gt;Computer processors, on the other hand, are generalists. They are cheap. They can do a lot of things. If you can break a task down into simple steps, it's easy to get them to do it. The trade-off is that computer processors are incredibly inefficient at everything.&lt;/p&gt;
&lt;p&gt;Now imagine, if you will, that a specialised machine is a highly trained and experienced professional, a computer processor is a hungover university student.&lt;/p&gt;
&lt;p&gt;Over the years, we've tried lots of things to make student faster. Firstly, we gave the student lots of caffeine to make them go as fast as they can. That worked for a while, but you can only give someone so much caffeine before they become unreliable. Then we tried teaming the student up with another student, so they can do two things at once. That worked, so we added more and more students. Unfortunately, lots of tasks can only be done by one person at a time, and team-work is complicated to co-ordinate. We've also recently noticed that some tasks come up often, so we've given them some tools for those specific tasks. Sadly, the tools are only useful for those specific situations.&lt;/p&gt;
&lt;p&gt;Sometimes, what you really need is a professional.&lt;/p&gt;
&lt;p&gt;However, there are a few difficulties in getting a professional to work with uni students. They don't speak the same way; they don't think the same way, and they don't work the same way. You need to teach the uni students how to work with the professional, and vice versa.&lt;/p&gt;
&lt;p&gt;Previously, developing this interface – this connection between a generalist processor and a specialist machine – has been particularly difficult. The interface between processors and these specialised machines – known as &lt;em&gt;accelerators&lt;/em&gt; – has also tended to suffer from bottlenecks and inefficiencies.&lt;/p&gt;
&lt;p&gt;This is the problem CAPI solves. CAPI provides a simpler and more optimised way to interface specialised hardware accelerators with IBM's most recent line of processors, POWER8. It's a common 'language' that the processor and the accelerator talk, that makes it much easier to build the hardware side and easier to program the software side. In our Canberra lab, we're working primarily on the operating system side of this. We are working with some external companies who are building CAPI devices and the optimised software products which use them.&lt;/p&gt;
&lt;p&gt;From a technical point of view, CAPI provides &lt;em&gt;coherent&lt;/em&gt; access to system memory and processor caches, eliminating a major bottleneck in using external devices as accelerators. This is illustrated really well by the following graphic from &lt;a href="https://www.youtube.com/watch?v=4ZyXc12J6FA"&gt;an IBM promotional video&lt;/a&gt;. In the non-CAPI case, you can see there's a lot of data (the little boxes) stalled in the PCIe subsystem, whereas with CAPI, the accelerator has direct access to the memory subsystem, which makes everything go faster.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Slide showing CAPI's memory access" src="/images/dja/capi-memory.png" /&gt;&lt;/p&gt;
&lt;h2&gt;Uses of CAPI&lt;/h2&gt;
&lt;p&gt;CAPI technology is already powering a few really cool products.&lt;/p&gt;
&lt;p&gt;Firstly, we have an implementation of Redis that sits on top of flash storage connected over CAPI. Or, to take out the buzzwords, CAPI lets us do really, really fast NoSQL databases. There's &lt;a href="https://www.youtube.com/watch?v=cCmFc_0xsvA"&gt;a video online&lt;/a&gt; giving more details.&lt;/p&gt;
&lt;p&gt;Secondly, our partner &lt;a href="http://www.mellanox.com/page/products_dyn?product_family=201&amp;amp;mtag=connectx_4_vpi_card"&gt;Mellanox&lt;/a&gt; is using CAPI to make network cards that run at speeds of up to 100Gb/s.&lt;/p&gt;
&lt;p&gt;CAPI is also part of IBM's OpenPOWER initiative, where we're trying to grow a community of companies around our POWER system designs. So in many ways, CAPI is both a really cool technology, and a brand new ecosystem that we're growing here in the Canberra labs. It's very cool to be a part of!&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Daniel Axtens</dc:creator><pubDate>Wed, 27 May 2015 15:08:00 +1000</pubDate><guid>tag:sthbrx.github.io,2015-05-27:blog/2015/05/27/joining-the-capi-project/</guid><category>capi</category><category>open-power</category></item><item><title>OpenPOWER Powers Forward</title><link>https://sthbrx.github.io/blog/2015/05/21/openpower-powers-forward/</link><description>&lt;p&gt;I wrote this blog post late last year, it is very relevant for this blog though so I'll repost it here.&lt;/p&gt;
&lt;p&gt;With the launch of &lt;a href="http://www.tyan.com/campaign/openpower/"&gt;TYAN's OpenPOWER reference system&lt;/a&gt; now is a good time to reflect on the team responsible for so much of the research, design and development behind this very first ground breaking step of &lt;a href="http://openpowerfoundation.org/"&gt;OpenPOWER&lt;/a&gt; with their start to finish involvement of this new Power platform.&lt;/p&gt;
&lt;p&gt;ADL Canberra have been integral to the success of this launch providing the Open Power Abstraction Layer (OPAL) firmware. OPAL breathes new life into Linux on Power finally allowing Linux to run on directly on the hardware.
While OPAL harnesses the hardware, ADL Canberra significantly improved Linux to sit on top and take direct control of IBMs new Power8 processor without needing to negotiate with a hypervisor. With all the Linux expertise present at ADL Canberra it's no wonder that a Linux based bootloader was developed to make this system work. Petitboot leverage's all the resources of the Linux kernel to create a light, fast and yet extremely versatile bootloader. Petitboot provides a massive amount of tools for debugging and system configuration without the need to load an operating system.&lt;/p&gt;
&lt;p&gt;TYAN have developed great and highly customisable hardware. ADL Canberra have been there since day 1 performing vital platform enablement (bringup) of this new hardware. ADL Canberra have put all the work into the entire software stack, low level work to get OPAL and Linux to talk to the new BMC chip as well as the higher level, enabling to run Linux in either endian and Linux is even now capable of virtualising KVM guests in either endian irrespective of host endian. Furthermore a subset of ADL Canberra have been key to getting the Coherent Accelerator Processor Interface (CAPI) off the ground, enabling more almost endless customisation and greater diversity within the OpenPOWER ecosystem.&lt;/p&gt;
&lt;p&gt;ADL Canberra is the home for Linux on Power and the beginning of the OpenPOWER hardware sees much of the hard work by ADL Canberra come to fruition.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Cyril Bur</dc:creator><pubDate>Thu, 21 May 2015 11:29:00 +1000</pubDate><guid>tag:sthbrx.github.io,2015-05-21:blog/2015/05/21/openpower-powers-forward/</guid><category>open-power</category></item></channel></rss>