Linux Engineer's random thoughtshttps://anisse.astier.eu/2020-02-02T20:22:02+01:00A beginner hacker's guide to IPv62020-02-02T20:22:02+01:002020-02-02T20:22:02+01:00Anisse Astiertag:anisse.astier.eu,2020-02-02:/ipv6.html<p>I noticed recently how little I knew about IPv6. For someone working on broadband gateways, that's not something I'm most proud of. But I've learned a little in the past few months, and I thought I'd share it here.</p> <h1>How to read an IPv6 address: the zeroes are hidden</h1> <p>An …</p><p>I noticed recently how little I knew about IPv6. For someone working on broadband gateways, that's not something I'm most proud of. But I've learned a little in the past few months, and I thought I'd share it here.</p> <h1>How to read an IPv6 address: the zeroes are hidden</h1> <p>An IPv6 address is 128 bits wide, and written in eight groups of 4 hex-digits, separated by colons, like this:</p> <p><code>2001:41d0:0001:c38f:0000:0000:0000:0001</code></p> <p>Leading zeroes are ignored, so the previous address can be written like this:</p> <p><code>2001:41d0:1:c38f:0:0:0:1</code></p> <p>And a sequence of one ore more groups that are all zeroes can be replaced with <code>::</code>, so the previous address is canonically written like this:</p> <p><code>2001:41d0:1:c38f::1</code></p> <p>This is the address of this blog at the time of this writing.</p> <h2>Put the address inside square brackets for URLs</h2> <p>To reduce the confusion with the IP:PORT notation, the IPv6 in URLs is enclosed in square brackets for URLs:</p> <p><code>curl -v http://[2001:41d0:1:c38f::1]</code></p> <p><code>curl -v https://[2001:41d0:1:c38f::1]:443</code></p> <p>But not all tools would need that:</p> <p><code>ping 2001:41d0:1:c38f::1</code></p> <p><code>ssh 2001:41d0:1:c38f::1</code></p> <h2>Types of IPv6 addresses</h2> <p>There are three types of addresses to know about:</p> <ul> <li>the <strong>loopback</strong> address <strong><code>::1</code></strong>; that's your <code>localhost</code> or <code>127.0.0.1</code> in IPv4 (or the whole <code>127.0.0.0/8</code> subnet)</li> <li><strong>link-local</strong> addresses in the <code>fe80::/10</code> range : their scope is local, and they shouldn't be routed/forwarded. Equivalent to <code>169.254.0.0/16</code> in IPv4.</li> <li><strong>global</strong> addresses; those are globally routed addresses; they're any other address.</li> </ul> <p>Other interesting address types not covered here:</p> <ul> <li>multicast addresses <code>ff00::/8</code></li> <li>unique local addresses (previously named site-local) <code>fc00::/7</code> (can start with <code>fd00:</code>); they are used to build a private network.</li> <li>the unspecified address <code>::</code> (all zeroes) used for broadcast.</li> </ul> <h1>Client IPv6 addresses are autoconfigured by default</h1> <p>In IPv6, client hosts don't use DHCP by default (except in a few cases). They use autoconfiguration, which means the host decided of its address by itself, inside a pool of available addresses: this is <a href="https://tools.ietf.org/html/rfc4862">Stateless Address Auto Configuration (SLAAC)</a>. Instead of asking your router to give you an address, in SLAAC, the client machine sends a Router Solicitation, which responds with a Router Advertisement that contains the range in which the client can configure an address; then chooses an address at random (or based on its MAC address), and runs a collision detection algorithm (Duplicate Address Detection) to prevent having the same address as a peer.</p> <p>For link-local addresses, there does not even need to be router advertisements, since the range is known by default, a host can pick an address and then run it's duplicate detection algorithm.</p> <h2>DHCPv6 has lost and shouldn't be used</h2> <p>There exist a spec to attribute addresses via a DHCP mechanism: DHCPv6. But it isn't supported in Android, by choice. Admins that want to match IPv6 addresses to MAC addresses (say, for compliance purpose) should watch ICMPv6 SLAAC advertisements instead. But in a world of random MAC addresses by default on consumer devices, it doesn't really make sense. It's better to enforce a <a href="https://en.wikipedia.org/wiki/Zero_Trust">zero-trust network</a> with a higher level authentication and a VPN.</p> <h1>Use AAAA DNS records</h1> <p>You store IPv6 addresses in an AAAA record instead of an A record:</p> <div class="highlight"><pre><span></span>$ dig anisse.astier.eu AAAA +short <span class="m">2001</span>:41d0:1:c38f::1 </pre></div> <h1>Your ISP gives you more IPv6 addresses than you could ever use</h1> <p>Your ISP probably gives you a <code>/56</code> or <code>/48</code> (mine gives a <code>/61</code>): it means that you have 2<sup>35</sup> to 2<sup>48</sup> <em>more</em> addresses than available in the whole IPv4 range (ignoring reserved ranges). I could address 2<sup>67</sup> devices in my network; that's 147,573,952,589,676,412,928 IPs.</p> <p><a href="https://xkcd.com/865/" title="I think the IETF hit the right balance with the 128 bits thing. We can fit MAC addresses in a /64 subnet, and the nanobots will only be able to devour half the planet."><img alt="xkcd #865 Nanobots IPv6" src="images/nanobots-ipv6.png"></a></p> <p>Some cloud providers might give you single IPv6 address (<code>/128</code>). I think this is wrong and short-sighted; mostly used for market segmentation (the simplicity of configuration does not hold up a cursory look).</p> <h1>All your devices are globally reachable</h1> <p>That's the thing that surprises most people used to an IPv4 NAT-ed world mindset. Since your ISP gives you so many addresses, all your local devices can have a globally routed address. And it's a <strong>good</strong> thing, <em>mostly</em>. What this implies:</p> <ul> <li>you can access your dev web server from anywhere in the world if you listen on global addresses instead of local or loopback one.</li> <li>you can send your DNS requests to your home pi-hole wherever you are. Ditto for any service hosted on a random machine in your network.</li> <li>you can directly connect to a peer for video chat, exchange data, etc. No need for UPnP, UDP hole punching, STUN gateways, and many other types of NAT-traversal technologies.</li> </ul> <h2>Security implications: you should make sure privacy extensions are enabled</h2> <p>Unfortunately, by default many OSes used their MAC address to choose a global address with SLAAC. This means that whole-plage scanning with masscan-like tools are possible on given <a href="https://en.wikipedia.org/wiki/Organizationally_unique_identifier">OUI</a>s. It takes about <a href="https://zmap.io">5 minutes to scan the whole IPv4 range</a> nowadays. That's what companies like Shodan or CybelAngel do continously.</p> <p>The <a href="https://www.si6networks.com/tools/ipv6toolkit/">scan6 tool</a> of the <a href="https://github.com/fgont/ipv6toolkit">SI6 IPv6 toolkit</a> (<a href="https://www.bortzmeyer.org/7707.html">covered by Stéphane Bortzmeyer in french</a>) can be used to do this type of scanning on IPv6.</p> <p>This has the following implications for IPv6:</p> <ul> <li>if badly-configured software starts some VMs with open services and a static network MAC address, you could instantaneously scan a given IPv6 prefix (provided you know its size) to for the presence of such a VM; because the last bits of the IPv6 would always be same same.</li> <li>If you know a particular (say, IoT) device has open services and is vulnerable; you could scan its <a href="https://en.wikipedia.org/wiki/Organizationally_unique_identifier">OUI</a> for a given IPv6 prefix. That's still 2<sup>24</sup> IPs to scan; much less if you know the MAC sequencing pattern. This could be used for targeted attacks, or botnet/worms.</li> <li>If a device connects to a malicious site, it can be scanned; for example that dev web server on your laptop; and the type of devices leaks because the <a href="https://en.wikipedia.org/wiki/Organizationally_unique_identifier">OUI</a> is in the IPv6. This breaks expectation of IPv4 NAT-ed world where you need to manually forward port for them to be publicly available</li> </ul> <p>Luckily, all of this was understood a long time ago, and there are privacy extensions in IPv6: a way to randomize your SLAAC address, just like we now randomize Wifi MACs on untrusted networks. It is now implemented in most modern OSes. Unfortunately older ones, and some Linux distributions don't enable privacy extensions by default. In Linux, those are represented as <code>temporary</code> instead of <code>global</code> in <code>ip -6 addr</code>. Those temporary addresses are used when connecting to a service over IPv6, so that your MAC doesn't leak. The address should change regularly.</p> <p>As far as I understand, privacy extensions address the issue of data leakage, but not the fact that you can then be scanned for mis-configured software (which to be fair is already the case in a NAT-ed IPv4 + websocket world). That's because the initial expectation was to have every device with its own firewall; and consider that it's up to the device to properly handle what's exposed to the world.</p> <p>Therefore, a small issue I see, is that it's hard to have a server application to listen only on global IPs. It should enumerate the interfaces and IPs, and only listen on those that have "scope global" but not "temporary" or "secondary" addresses.</p> <h1>Random Trivia</h1> <ul> <li>It has been mandatory to support IPv6 on new devices in Brazil for three years.</li> <li>It will soon be mandatory in France for 5G carriers.</li> <li>When using link-local addresses, you should do <em>scoping</em>; which is add a % after the address to specify through which interface your connection should go through. But wget does not support link-local scoping; use curl instead. To connect to a link-local IP on port 8080 : <code>curl -v http://[fe80::6e60:6ddd:d354:2234%wlp2s0]:8080</code></li> <li>It's 2020 and Fedora still puts <code>ip</code> and <code>ifconfig</code> in <code>/sbin</code> ; without adding it to non-root users' <code>PATH</code>. I don't know why.</li> <li>At FOSDEM, the default network is IPv6 only, and it works really well. But from time to time you might discover that something is not working. It's because it's not connecting over IPv6 (like the Steam client for instance).</li> </ul> <p>There are many RFCs on IPv6; I couldn't cover here anywhere near everything that a professional network engineer should know. I hope I've covered the basics so that you can search for the rest yourself.</p> <p><em>Thanks to Stéphane Bortzmeyer and Neil Armstrong for feedback on this article.</em></p>36th Chaos Communication Congress2019-12-28T13:00:00+01:002019-12-28T13:00:00+01:00Anisse Astiertag:anisse.astier.eu,2019-12-28:/36c3.html<p>I’m at CCC for the first time this year ! Here my notes for a few of the talks.</p> <h1>Tamago</h1> <p>In an ideal world, one could pick the programming language of their choice and always generate the optimal machine code. But we live in the real world, and it isn't …</p><p>I’m at CCC for the first time this year ! Here my notes for a few of the talks.</p> <h1>Tamago</h1> <p>In an ideal world, one could pick the programming language of their choice and always generate the optimal machine code. But we live in the real world, and it isn't yet possible. Running code in a constrained environment, like baremetal SoCs means that your choice of languages is reduced, and towards low-level languages like C, which come with their own issues. The motivation for Tamago is to be able to run a higher-level language like Go on baremetal.</p> <p>The goal of the Tamago project is to run Go on baremetal ARMv7, for example to be able to write security-sensitive bootloaders (firmware) on an NXP i.MX6ULL based <a href="https://inversepath.com/usbarmory.html">usbarmory</a>.</p> <p>This isn't really a new idea, as Unikernels or library OSes often match the same description. But <a href="https://github.com/cetic/unikernels">unikernels</a> didn't match the requirements of depending on no C code (not importing another OS or library), or running on small embedded systems, not the cloud.</p> <p>Go was chosen here because it's relatively easier to learn than other proven-on-embedded languages like Rust. Currently, Tamago is a patched Go compiler that adds another OS support <code>GOOS=tamago</code>, to be able to run <code>GOARCH=arm</code> on baremetal. The scope is different from <a href="https://tinygo.org/">TinyGo</a>, that targets micro-controllers instead, and uses a completely different compiler implementation, as well as not supporting the full runtime and Go language yet. It's also different from the recently announced <a href="https://embeddedgo.github.io/">Embedded Go</a> that targets Thumb and bigger microcontrollers.</p> <p>The goal of Tamago is to be upstream-able in Go. The current patchset is about 3000 lines of code. That's about ~300 lines of glue code, ~2700 lines of re-used code, like the <code>plan9</code> memory allocator or locking from <code>js,wasm</code>. Then there's ~600 lines of new code to provide ARMv7 and heap init functions.</p> <p>There are few functions that should be provided in a board-support package: how to get random data, init hw, print a byte on the console, what is the ram size and offset, and how to get high-res time.</p> <p>Writing drivers in go can be done easily either using Go's assembly support or using unsafe; it also means that the sensitive parts are identified and can be written with care and audited if needed. In the runtime, the only syscall that is implemented is write(), and it is used for debugging on the serial port.</p> <p>In order to write a baremetal Go program, one should import the board package library to get its side-effects.</p> <p>The Tamago authors wrote i.MX6 drivers to test their model: for the DCP co-processor, the HW random generator, a USB driver and a USB networking card (CDC-ACM).</p> <h1>Boot2root: Auditing bootloaders by example</h1> <p>Bootloaders are a critical part of any secure boot chain. The authors of this talk had a look at common open source bootloaders to look for any issue they could fine.</p> <p>They looked at u-boot, a very commonly used bootloader in embedded systems, Coreboot is targeted at modern OSes and used in Chromebooks, Grub used in most Linux distros, Broadcom CFE, iPXE for network boot, Tianocore etc.</p> <p>In most opensource bootloaders, there is no privilege separation: all codes runs with maximum privileges. The attack surfaces are NVRAM, files and file systems, busses (I2C/SPI…), etc.</p> <p>For NVRAM, where environment variables are stored, one should look at where the variables are read, and if there's any kind of sanitization. For u-boot, many places were found where env_get() is called without any sort of size check of the data read. The attack scenario is a device for which NVRAM is modifiable (through hardware or host OS); this is then open for exploitation, for which a demo was shown with u-boot.</p> <p>For the filesystems attack surface, often the images are read on a filesystem that is not signed or integrity checked. An example was given with grub's ext2 symlink handling.</p> <p>When looking at TCP/IP, many issues can be found: TLV parsing is tricky for example. DNS can be poisoned, DHCP leases can be stolen. In u-boot, the DNS TID is hardcoded to 1, enabling easy mitm. In broadcom CFE, many memory corruptions were found in DHCP, ICMP handling, IP header length etc.</p> <p>In iPXE for 802.11, a memory corruption bug was found SSID handling. Bluetooth was attacked, and issues were found in proprietary bootloaders. The most interesting attack vectors were in large frame handling and fragmentation, but no example was shown because of NDAs.</p> <p>USB is also a big attack surface when booting from storage or ethernet dongles. Often, descriptor parsing is wrong, with overflows or double fetches causing TOCTOU issues. In Grub, Tianocore and Seabios, memory corruptions and double fetches issues were found. Recent similar issues in proprietary devices include the Nintendo Switch bootloader issue and iPhone checkm8.</p> <p>Another attack surface are SMM, for which many problems were found with UEFI Tianocore in the last 15 years. A recent issue was found in Coreboot, for which the range checks were simply not implemented yet.</p> <p>If DMA is an attack surface (rogue device), IOMMUs are an absolute minimum to defend against the attacks. But then you should make sure that any data you get from a device is properly checked, so old driver code needs to be rewritten and audited. There might also be HW bugs in IOMMUs, sidechannels, data leaks, etc. In edkII (Tianocore), the spec mandates shutting down the IOMMU before chainloading to an OS, so it opens a huge window during which a rogue device can do attacks.</p> <p>The next attack surfaces are glitching, which means injecting faults in the hardware clocks, voltage sources, or via lasers or EMIs. These hardware side channels are used to extract secrets. Some companies (Chipsec…) also provide optical ROM code extraction by decapping chips.</p> <p>In conclusion, there's a surprising amount of low quality code in bootloaders (closed and open); and one often encounters NDA walls when looking at proprietary code. The advice is to minimize the image, and turn most extra features off. Enable basic exploit mitigations when compiling. In all of this, Tianocore is well ahead of the game.</p> <p>The authors called the audience for action in reviewing, fuzzing, and analysing bootloaders.</p> <p><em>That's it for 36C3 ! Most talks were recorded with <a href="https://media.ccc.de/c/36c3">videos already available</a>.</em></p>Linux Security Summit Europe 20192019-10-31T16:00:00+01:002019-10-31T16:00:00+01:00Anisse Astiertag:anisse.astier.eu,2019-10-31:/lsseu-2019.html<p>Following my <a href="elce-2019.html">ELCE notes</a>, you'll find below my notes for some of the talks the Linux Security Summit Europe 2019.</p> <h1>Exploiting race conditions using the scheduler — Jann Horn</h1> <p>The first bug Jann talked about is a race between mremap and fallocate. mremap allows moving a memory mapping from an address …</p><p>Following my <a href="elce-2019.html">ELCE notes</a>, you'll find below my notes for some of the talks the Linux Security Summit Europe 2019.</p> <h1>Exploiting race conditions using the scheduler — Jann Horn</h1> <p>The first bug Jann talked about is a race between mremap and fallocate. mremap allows moving a memory mapping from an address to another. The associated page table entries are moved as well, and then the TLB is flushed.</p> <p>fallocate allocates and de-allocates space for a file. It interactes with the page cache so it's possible to exploit a race between the page table modification and the TLB flushing.</p> <p>To widen the race-exploitation window, the scheduler is used. The approach is to pin two own tasks to the same CPU, and set on of the task to idle (low) priority; it's then possible to interrupt the idle task, even when it's inside the kernel, simply by having the other task being active. This is used to have a syscall take much longer that needed, and make the race window also last longer.</p> <p>The second example is a refcount decrement on struct file.</p> <p>Both userfaultd and FUSE allow handling page faults in userspace.</p> <p>The kcmp syscall, made for checkpoint/restore in userspace is useful for reliable Linux user-after-free exploitation. It compares two processes' kernel resources (file descriptors) to determine if they are sharing pointers.</p> <p>Combined exploitation then creates a FUSE mapping, open a writable file, make a write that then blocks to FUSE write, then open another RO destination file and verify that we reuse the same file structure with kcmp() before then resolving the FUSE page fault and writing to the destination (forbidden) file.</p> <p>The third bug uses getpidcon(); it exploits an issue in Android binder that used the caller PID to get the context, before then making a decision based on this.</p> <p>Here the race window is widened by creating an artificial priority inversion inside the kernel, which mutexes are vulnerable to. This is done by interacting with the VFS's mutexes with userfaultd.</p> <h1>Kernel Runtime Security Instrumentation — KP Singh</h1> <p>Signals from Audit or perf can correlate with malicious activity, but not necessarily. LSM errors and denies are Mitigations that deny certain actions. You need both to have security.</p> <p>To detect a new attack, you need both an Audit update (to log new events), and a mitigation update (e.g new LD_PRELOAD signature).</p> <p>KP gave a few examples of Signals: for example a process that runs and deletes its binary. For Mitigations, the first example was to have a dynamic whitelist of known Kernel modules. The point is that both go hand in hand, and you need both.</p> <p>KP says they want to implement KRSI with eBPF; LSM hooks should be implemented as eBPF programs. The LSMs are a good match because they map to security behaviours, are associated with kernel data structures with "blobs", and benefit from years of research and verification of the LSM system. It also benefits the LSMs because the framework can then get extended from security analysts feedback.</p> <p>With this eBPF infrastructures, both actions are possible on an LSM action: returning an error and/or logging an audit event.</p> <p>The new BTF format helps with eBPF programs because now the addresses of symbols are distilled in a compact format, allowing to read inside kernel data structures in a version and platform-independent way.</p> <p>KP showed an example BPF program to do the denial of unlinking its own executable (or a parent, or another process's), or to logging a write /proc/self/mem.</p> <p>Compared to the landlock patchset, KP says that KRSI has different roles: the goal is to do system-wide MAC, not unprivileged DAC.</p> <p>Performance-wise, the latency impact of KRSI is three times lower than audit, and with less variation.</p> <h1>Address space for namespaces — Mike Rappoport</h1> <p>Address space isolation is on of the best protection methods since the invention of virtual memory, Mike says.</p> <p>Page Table Isolation (PTI) is the first example of this type of isolation. There are other work in progress, like KVM address space isolation or process local memory.</p> <p>Mike's group at IBM wants to improve container isolation; a way to do that can be to assign dedicated page tables to namespaces.</p> <p>Previously, they tried something called System Call Isolation (SCI), to run system calls with very limited page tables, but it didn't provide enough guarantees, and had a heavy performance impact. Another thing they worked on was <code>mmap(MAP_EXCLUSIVE)</code> to create private memory mappings that can't be accessed by the rest of the system. A similar thing was done with a char device mapping : can transfer the rights with SCM_RIGHTS, and doesn't need a new page flag.</p> <p>The Address spaces for namespaces approach would have processes within a namespace sharing a page table, but their mappings wouldn't be available outside of that namespace. A first version was built for netns in order to have an isolated network stack for network namespaces. Even the internal kernel pages and objects aren't visible outside of the namespace. It required extending <code>alloc_page()</code> and <code>kmalloc()</code>.</p> <p>The proof of concept implementation still does not work, but there is good hope that it can be improved.</p> <h1>Using a different LSM from the Host in a Container — John Johansen</h1> <p>The main use case to have a different LSM in a container is cross-distro container running: e.g, Fedora (selinux) inside a container on Ubuntu(apparmor), or an Android (selinux) inside something else.</p> <p>LSM Stacking is very similar to what needs to be done here, so one can leverage it for this use case.</p> <p>But an issue John found is that LSM order matters. His first tries with apparmor (main LSM), then selinux (stacked) failed to boot, because in Ubuntu dbus is built with both apparmor and selinux. It finds that selinux is enabled, and then when initializing apparmor, it thinks it already has an (empty) policy. That was the first gotcha.</p> <p>Then, another gotcha was selinux was blocking apparmor code.</p> <p>There also needs to be a way to namespace LSMs: inside a chroot (or container), you don't an applied policy to impact the whole system.</p> <p>In order to do that, Apparmor implements virtualization of parameters, policy, etc. That's the way to be namespace aware, and it will need to be done in every LSM.</p> <p>Another issue that was encountered was securityfs, which isn't multi-mount capable, so a non-container aware image can't boot because it would fail to mountit.</p> <p>It's also not possible to use seccomp and no_new_privs. So to work around that, AppArmor support for no_new_privs with stacking was added: at lockdown, the apparmor confinement is saved, and then checked again whenever necessary to make sure it's enforced properly.</p> <p>Container nesting is also another issue. There are specifically issues around user namespaces: for example there's no way to know when a user namespace is created; new LSM hooks are probably missing to fix these issues.</p> <p>It's now possible to run an Ubuntu container on Fedora, with the help of LXD (handling mappings, etc.).</p> <h1>Keylime: Open Source software for remote trust — Luke Hinds</h1> <p>Traditionally, software trust used to reside in memory or disks. TPMs changed this by providing hardware-backed software trust sources. They are now ubiquituous, and used pervasively accross various industries.</p> <p>The basic model relies on an RSA keypair, with the private key being inaccessible to software. A TPM can hash critical sections of firmware, a boot process, and then those hashes can be made public and verified with the public key.</p> <p>Keylime is a remote trust framework using TPMs. TPM 2.0 and 1.2 are supported, but the latter will be deprecated. It provides Measured Boot (for firmware, shim, grub, kernel, etc.), it can measure secure boot (EFI DBX, MokListX, etc.), and an IMA Runtime for attestation. It' possible to have encrypted payload execution, depending on the previous measurements, which would then unwrap appropriate keys. Keylime also has a revocation framework.</p> <p>In the target node (IoT / Edge platform / Provider Cloud), the Keylime Agent runs, with the TPM software stack. On the trusted infrastructure, the Keylime server runs, with these components: Verifier, Registrar, Revocation Service, and Keylime CA.</p> <p>It's possible to have many nodes, and multiple verifiers combinations.</p> <p>Enrollment relies on the TPM-backed hardware keys. Once it is done, the keylime infrastructure (Verifier and Registrar) will provide a way for a tenant to access an enrolled node, and send it a cryptographically secure payload.</p> <p>With IMA, it's possible to have continous remote attestations (with polling). When an attacker triggers an IMA error, the Keylime Verifier will notice, and the revocation mechanisms can kick in. The Keylime Verifier will send a signed revocation event to all nodes, for example to revoke trust to the compromised node.</p> <p>The <a href="https://github.com/keylime/keylime">Keylime project</a> is active, has documentation and working CI. Luke would of course like to have even more contributors.</p> <p>The agent is being ported from Python to Rust for performance and security reasons. Support for virtual TPM (vTPM) is being worked on as well, while still binding it to the hardware TPM's cryptographic trust.</p> <h1>Securing TPM Secrets with Intel TXT and Kernel Signatures — Paul Moore</h1> <p>Paul's goal is to store secrets in the TPM, and only allow access to these secrets to authorized kernels. He also wants this to work on both UEFI Secure boot systems and legacy BIOS systems.</p> <p>Sealing TPM secrets against a set of PCR values. These values are measured against system state with secure boot, in the firmware, bootloader, kernel, etc.</p> <p>For UEFI Secure boot, PCR 7 is a hash of the kernel signing authority; it's stable across kernel updates with the same signer.</p> <p>With Intel TXT, the hardware and firmware create a dynamic root of trust; it uses the tboot bootloader, which makes a hash of the kernel image. But that hash isn't stable across updates, so the PCRs are unstable.</p> <p>So Paul's solution to this is to extend tboot to support the UEFI PECOFF signature format. Its verification is rooted in the TPM; and it then updates the PCRs.</p> <p>The <a href="https://github.com/pcmoore/misc-tboot/tree/working-txtsig">prototype code</a> has been released.</p> <p>There's still need to work on command line and initrd verification, but the solutions that were applied with UEFI secure boot could work.</p> <p><em>That's it for this edition of LSS EU!</em></p>ELCE, OS Summit and KVM Forum 2019 notes2019-10-31T00:00:00+01:002019-10-31T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2019-10-31:/elce-2019.html<p>After a two years hiatus, I couldn't miss this year's ELCE happening in Lyon, the first in France since 10 years ago in Grenoble, which coincidentally was one of my first conferences. OS Summit and KVM forum events are co-located and with the same ticket, which is nice, and why …</p><p>After a two years hiatus, I couldn't miss this year's ELCE happening in Lyon, the first in France since 10 years ago in Grenoble, which coincidentally was one of my first conferences. OS Summit and KVM forum events are co-located and with the same ticket, which is nice, and why I attempted a few talks there as well.</p> <h1>Making device identity trustworthy with TPMs — Matthew Garett and Brandon Weeks</h1> <p>For access to any internal Google Service (BeyondCorp), both the user and the device need to be authenticated. The devices that are allowed need to be well-known and inventoried.</p> <p>A device identity needs to be unique (serial number), bound to hardware and stable for the lifetime of the device. It also shouldn't be unforgeable and resistant to tampering.</p> <p>Existing solutions are inadequate: self-identification can be forged, keys on disk can be duplicated, and trust bootstrapping is hard, especially remotely.</p> <p>TPMs are specific chips that provide a store and generation for keys that never leave the hardware: they can't be extracted and duplicated to other hardware. Modern TPMs allow tracking the hardware manufacturer, thanks to endorsement keys (EK): they provide proof that a TPM comes from a certain manufacturer. Attestation Keys (AK) are then used to prove whether a key has been generated on a specific TPM.</p> <p>An issue is that TPMs aren't directly related to devices. A solution to that is runtime binding: run a script that extracts the TPM EK and place in an internal database. But an issue with this is that you don't know if the device state is trustable at this time, or if the TPM is indeed internal to the device.</p> <p>Binding at provisioning helps with this by reducing the window where a system could be compromised by having an IT officer do this operation before it's given to a user. But it's still dependent on having a well-functioning inventory system.</p> <p>Binding at manufacture goes even further: it maps a given device with a TPM with the help of Platform Certificates. Those are sent out-of-band at device ordering.</p> <p>The lifecycle then looks like this: - first, the device is provisioned: at this point an Attestation Key is generated by the TPM, and both the EK and AK are registered through the Attestation CA - then, a Client certificate is issued: it is signed by the AK - the client certificate is then provided to the access gateway as part of mutual TLS auth to access services.</p> <p>Another possibility once you know how to bootstrap device trust, is to use TPM-backed trust to authentify services for example.</p> <p>The low level code, including Platform Certificates parsing has been <a href="https://github.com/google/go-attestation">released here</a>.</p> <h1>What's new in Buildroot — Thomas Petazzoni</h1> <p>Two years ago, LTS support was added: the february release is maintained for a year for security updates and bug fixes.</p> <p>Internal toochain support has been updated, with new gcc, binutils, and various libc versions. They are now tested automatically for architectures supported by qemu. External toolchains have been updated as well. It's also now possible to declare external toochains from <code>BR2_EXTERNAL</code>.</p> <p>Two new common package infrastructures were added for go and meson packages.</p> <p>Git caching has been improved, for git-fetched packages; as well as the whole package download infrastructure which has been rewritten.</p> <p>Many packages were updated, and added; a few obsolete ones have been removed.</p> <p>New global options have been added to force building with security-related options (relro, stack protection, etc.)</p> <p>A new <code>make show-info</code> was added to dump the state of enabled packages as a json to be used by external tools.</p> <p>Work has been done to improve the testability of reproducible builds, as part as a GSoC: if <code>BR2_REPRODUCIBLE=y</code>, the build is done twice, and the outputs of the two are compared with diffoscope.</p> <p>Work has also continued to improve parallel builds; one of the last series on the subject is on the per-package directories for HOST.</p> <p>The runtime test infrastructure has been improved to add more tests. The tooling around buildroot has been augmented with the support of release-monitoring.org to track packages that are outdated.</p> <h1>Boot time optimization with systemd — Chris Simmonds</h1> <p>systemd runs as init, so it's PID 1. It launched and monitors daemons, configures stuff, etc.</p> <p>For embedded systems, systemd is a much bigger init system, with 50 binaries and a 34MB footprint. It supports many features: event logging with journald, user login with logind, device management with udevd, etc.</p> <p>systemd has many features for resource control, free parallel boot, can have a system boot without a shell etc. It has unit (a generic type), services (a given job), and targets (a group of services, e.g a runlevel).</p> <p>systemd searches for units first in /etc/systemd/system for local configuration, then in /run/systemd/system for runtime config, then in /usr/lib/systemd/system .</p> <p>Units can depend on each other, with three types of deps: <code>Requires:</code> describes a hard dependency, <code>Wants:</code> is a weaker one meaning it won't be stopped if the dep fails, and <code>Conflicts:</code>.</p> <p>systemd also provides an other concept: ordering. <code>Before:</code> and <code>After:</code> determine when a unit is started. It's used for example when starting a unit web server after <code>network.target</code>. Without ordering, units are started in no particular order.</p> <p>At boot, systemd starts the <code>default.target</code>. On most systems, this is by default a symbolic link to the <code>multi-user.target</code>.</p> <p>It's also possible to describe a reverse dependency with <code>WantedBy:</code>, which is used to add services to be started by a target for example: <code>WantedBy: multi-user.target</code>. This is implemented by creating a symbolic link in the <code>multi-user.target.wants</code> directory.</p> <p><code>systemctl</code> is the cli tool used to interface with systemd at runtime.</p> <h2>How to reduce boot time then ?</h2> <p>Boot time is defined by the time to power on to running the critical app.</p> <p>When using a generic system image (yocto, debian), those are designed conservatively to cater to all common cases. So to reduce boot time, one should make it less generic, either by disabling services, or reducing their dependencies.</p> <p>The main tool to optimize boot time is <code>systemd-analyze</code>, that can give you a summary of the boot time; <code>systemd-analyze blame</code> list units by order of start-up time. The most important is <code>systemd-analyze critcal-chain</code> that shows the time for the units in the critical path.</p> <p>In an example, Chris showed that the critical-chain depends on a timeout because of a non-existant ttyGS0, removing the associated getty unit saved a lot of time. Changing the default target and disabling unused daemons also helped a lot.</p> <h2>Other useful features in embedded systems</h2> <p>The watchdog is a very useful feature of systemd: if a service does not reply to watchdog, it can be restarted automatically. It's even possible to force a reboot if the watchdog has been triggered a certain amount of time above a given threshold.</p> <p>Resource limits like CPU and memory limiting can also be very useful; this is implemented through cgroups.</p> <h1>Crypto API: using hardware protected keys — Gilad Ben Yossef</h1> <p>In the Linux crypto API, there are transformation providers, that can either use dedicated hardware, specialized instructions or a software implementation. There are used by the crypto user API, dm-crypt or ipsec for example.</p> <p>The crypto API is used in multiple steps - crypto_alloc_skcipher, for example to get xts(aes) transformation handle - set key to tfm - get a request handle - set request callback - set input, output, IVs, - etc.</p> <p>Tranformation providers have a generic name (the algorithm), a driver name (the implementation), and a priority, to know which is most important. There are other properties describing the synchronicity, min/max keysize, etc.</p> <p>The key is usally just stored in RAM, like everything else. It makes it vulnerable to various key-extraction attacks. It should be possible to have a transformation provider that support a hardware-backed key.</p> <p>It was implemented a few years ago for IBM mainframes, which means that the infrastructure could be reused for embedded devices.</p> <p>In the implementation, it means the user of the API would pass a tag instead of the key bytes. The tag describes a storage and key from inside a secure domain. The tag can be an index, or an encrypted key in case of key ladder.</p> <p>In practice the security of this key depends on the security of the secure domain (hardware or software, e.g tee), its provisioning, etc.</p> <p>The cipher's name is prefixed with 'p', for example "paes", for protected key. Because the tag value is specific to hardware implementation, when requesting a cipher, the specific name of the driver is used instead of just the algorithm name.</p> <p>When instantiating it with dm-crypt, one should use the crypto-api algo driver name and instead of the key, a tag describing the key (e.g key slot).</p> <p>A future challenge is that TPMs act very much in the same way, yet aren't using the same API.</p> <h1>iwd - state of the union — Marcel Holtmann</h1> <p>iwd 1.0 has been released on this day (October 30th 2019)</p> <p>Marcel says Wi-Fi on Linux sucks. It's because the roles are split between many projects (kernel, wpa_supplicant, dhcpcd, network manager, etc.), and there's still a lot of code to write on top of this to ship a consumer product.</p> <p>iwd's goal is consolidate the wifi information in one place, which is then used by network-manager. The goal is to only have one entity interacting with nl80211 for better performance.</p> <p>For example, when you wakeup your laptop, you don't want to rescan the ever-growing list of channels before re-joining a network.</p> <p>In addition to being the central known-network database, iwd has many features: - it has optimized scanning since it's the only daemon to do scanning in a system - it can do enterprise provisioning - supports fast roaming and transitions - it supports WPA3 and OWE (Opportunistic Wireless Encryption), and no UI change was needed to add this support - there's an integrated EAP engine that uses the kernel keyring system - it support the hotspot 2.0 spec - push-buttons method work (WPS, etc.) - address randomization is supported - AP mode is supported to do tethering</p> <p>Enterprise provisioning can be very complex. Most OSes have a lot of settings that are hard to manage, etc. With Windows 10 and iOS there's now a downloadable configuration file, like for OpenVPN for example.</p> <p>iwd has now support for configuration files with embedded certificates so that everything can be in a single file. An enterprise admin can now provide this configuration, the user installs it, and connects to the network. This format in documented in the manpage <code>man iwd.network.5</code>. Unfortunately, there's still no standard for Wi-Fi provisioning, and Marcel wants to address that.</p> <p>Marcel says that in some cases, just the overhead of communicating with other daemons (systemd-networkd, connman or network-manager) in order to trigger dhcp, is too big. Some systems also don't necessarily have those daemons. That's why iwd added support for an experimental DHCPv4 daemon. This is documented in <code>iwd.config.5</code>.</p> <p>The goal with iwd is to complete a connection in 100ms or less (with an IP address). Right now, it's not there yet. PAE in the kernel nl80211 interface helps reducing this. Address randomization adds 300ms on top of this. In Android, it can add up to a 3s penalty, because one needs to power down the phy and power-it up again with Linux. There's work in the kernel to reduce this time as well, but it's not there yet Marcel says.</p> <p>iwd does not depend on wpa_supplicant, and has improved a lot.</p> <p>Marcel says they have reached the limit of what is possible to improve inside iwd. There needs to be other features in nl80211 to continue doing optimizations.</p> <p>iwd has 40k SLOC, which might be a lot, but only a tenth of wpa_supplicant.</p> <p>There are other daemons in the work: ead for ethernet authentication; its code is in the repo, and still being worked on.</p> <p>apd, the access point daemon is still private and being prototyped and should land next year in the repo.</p> <p>rsd is a resolving service daemon is pretty much a replacement for systemd-resolved; the DNS part is quite tricky according to Marcel; the goal is to be able to chose the correct path (e.g through a proxy or not) for a given URL. It's not planned to be released anytime soon though.</p> <h1>VirtIO without the Virt: towards implementations in hardware — Michael Tsirkin</h1> <p>virtio enables re-utilisation of drivers that are already in the OS. There are already many types of devices that are supported.</p> <p>Hardware helps implement userspace drivers, that can also be simpler. Another motivation for hardware virtio would be passthrough for performance, while retaining the advantages of software implementations.</p> <p>If there is a precise virtio spec, when a bug happens it's possible to find if it's the fault of the driver or the card. If you have hardware, you can switch to a different card or software implementation to find out.</p> <p>Virtio feature negociation allows implementing only certain features in the driver or hardware, and then use only the intersection.</p> <p>For virtio-net, the virtualized hardware uses PCI, so it's possible to forward guest access to real hardware by giving it access directly to hardware memory range for example.</p> <p>Virtio ring has a standard lockless access model that looks a lot like DMA systems that hardware vendors are used to implement.</p> <p>Depending on the hardware, there might be cache coherency issues, which means that hardware has a different feature flags, in particular VIRTIO_F_ORDER_PLATFORM and VIRTIO_F_ACCESS_PLATFORM. Version 1 needs to be implemented as well, without the legacy interface.</p> <p>The simplest way to implement hardware virtio is to just use passthrough; for example, if a network card implements the spec properly, just pass-through everything. Another possibility is to only have data path offloading: the control path is intercepted in an mdev driver.</p> <p>It would also be possible to do partitioning in the last case, by tagging requests for a given virtqueue depending on each VM if we want to share a device between VMs. Another use of the mdev driver is for migration: force it to a matching subset of features between two machines, and then do the migration between the two transpararently.</p> <p>If there are device quirks, the best way to address that is to use feature bits instead.</p> <p>Virtio 1.2 spec plans to be frozen by end of November 2019. When a adding a device to the spec, and ID should be reserved, and a flag for new features.</p> <h1>Authenticated encryption storage — Jan Lübbe</h1> <p>It's possible to integrate authentication and encryption at various layer of the storage stack, from userspace, filesystems&amp;VFS to device-mapper.</p> <p>With dm-verity, a tree of hashes is built, and the root hash is provided out-of-band (kernel command line), or via signature in super block since Linux 5.4. It's the best choice for read-only data.</p> <p>dm-integrity arrived in Linux 4.12, and provides integrity for writing as well. There's one metadata block for n data blocks, and they are interleaved. It needs additional space, and has a performance overhead, because a write happens twice because of journalling (for both data and metadata) to prevent power issues.</p> <p>dm-crypt handles sector-based encryption with multiple algorithms. It's length preserving, which means that data cannot be authenticated. It's a good choice for RW block devices without authentication.</p> <p>Recently, dm-crypt added support for authentication as well with AEAD cipher modes. But it authenticates individual sectors, so replay is possible (is it the last version ?). The recommended algorithm is AEGIS-128-random.</p> <p>fsverity is now "dm-verity for files", and has been integrated into ext4. A single (large)file has root hash (provided out-of-band), and once written, is then immutable. Biggest user is likely Android for .apk files.</p> <p>fscrypt has the same idea of encryption at the file level. It's interesting for a multi-user system where each user has its own keys. It's possible to mount a filesystem and remove files without having the keys. It also has no authentication.</p> <p>Since Linux v4.20, UBIFS can provide authentication. The root hash is authenticated via HMAC or signature since Linux 5.3. It's the only FS that authenticates both data and metadata. It's the best choice for raw NAND/MTD devices.</p> <p>ecryptfs is a stacked filesystem (mounts on top of another fs), and was used by Ubuntu at some point for per-user home directory encryption, but has now been superseded by fscrypt.</p> <p>IMA/EVM was initially developed for remote attestation wit TPMs, and uses extended attributes. It protects from file data modification, but it is vulnerable to directory modifications (file move/cp).</p> <p>Master key storage is also problematic, and platform dependent. Many SoCs provide key wrapping to encrypt secrets per-device, but it needs a secure boot chain. Other possibilities include using a TPM or (OP-)TEE.</p> <p>Authenticated writable storage can only detect offline attacks, not runtime ones, so there's the need to have RO part of the system (recovery) in order to be able to restore the system in a good state.</p> <p>How to analyze device problems from devices that are returned from the field ? There might be the need for a mode to erase the keys (protects the data) and disable authenticated boot (for HW analysis).</p> <h1>That's it ?</h1> <p>If you want more notes, I invite you to read Arnout Vandecappelle's at <a href="https://www.mind.be/2019/10/28/OSS-EU-2019.html">Mind Embedded Development's blog</a>.</p> <p>I'm also attending <a href="lsseu-2019.html">Linux Security Summit, so stay tuned</a>!</p>What are you working on ?2019-10-20T00:00:00+02:002019-10-20T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-10-20:/edge-computing.html<p><em>This was <a href="https://www.linkedin.com/pulse/what-you-working-anisse-astier/">previously published on LinkedIn</a></em></p> <p>Until very recently I couldn’t really answer this question. But it has shipped, so now I can tell you: I've been working on Edge Computing.</p> <p>To be more precise, this is the first compute service available to ISP subscribers directly on their ISP …</p><p><em>This was <a href="https://www.linkedin.com/pulse/what-you-working-anisse-astier/">previously published on LinkedIn</a></em></p> <p>Until very recently I couldn’t really answer this question. But it has shipped, so now I can tell you: I've been working on Edge Computing.</p> <p>To be more precise, this is the first compute service available to ISP subscribers directly on their ISP gateway.</p> <p>Those of you that read my LinkedIn profile know that I work on the consumer-facing boxes of the french ISP called Free, at the Iliad subsidiary called Freebox. Free is often credited as the inventor of ADSL triple-play, and has been making its own hardware with Freebox for almost 20 years now. The latest gateway/set-top-box triple play offer is called Freebox Delta and is relatively high-end; non-french readers can find a <a href="https://www.engadget.com/2019/02/23/freebox-delta-set-top-box-devialet-speakers-impressions/">good overview</a> of the Freebox Delta at Engadget. The gateway, Freebox Server, ships with RAID, 10Gbps support, wifi with 160Mhz bands, lots of NAS features, etc. And now it's first ISP gateway to also provide compute, via installable VMs, available to subscribers.</p> <p>You can use it to host your personal cloud, home automation, multimedia server, etc. It's truly generic compute, with an arm64 OS, so your imagination is the limit. I'm already using it at home to run a few services that were on a Raspberry Pi: it's much faster thanks to the 64-bit OS, and more reliable than microSD cards.</p> <p><img alt="UI in french showing a few installed VMs and how to install a new one" src="/images/vm-preins-choix.png"></p> <p>It's built on top of QEMU and supports cloud-init, USB passthrough, remote display, and lot of other small goodies. In the process of building it, I found two QEMU bugs (one of which danpb fixed upstream in QEMU 4.1.0), two ubuntu bugs and <a href="https://github.com/anisse/websocktty">released a debug tool for websocket ttys</a>.</p> <p>You'll find some of the coverage of this <a href="https://www.inpact-hardware.com/article/1126/freebox-os-4-1-dessous-machines-virtuelles-dans-freebox-delta">release in french here, with quotes from yours truly</a>, as well as the <a href="https://dev.freebox.fr/blog/?p=5450">article that was published on the Freebox developers blog (french)</a>.</p> <p><video controls width="590" heighgt="331"><source src="/images/vm-install.webm" type="video/webm" >Install of a VM</video></p>r2wars 20192019-09-08T00:00:00+02:002019-09-08T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-09-08:/r2wars-2019.html<p>You might have noticed I was at <a href="r2con-2019.html">r2con 2019</a>. This is my writeup for the r2wars 2019 challenge.</p> <h1>r2wars primer: a Core Wars-like game</h1> <p>In radare2, there's an intermediate language and VM called ESIL. It is used to emulate code, and supports many architectures. r2wars is built on top of …</p><p>You might have noticed I was at <a href="r2con-2019.html">r2con 2019</a>. This is my writeup for the r2wars 2019 challenge.</p> <h1>r2wars primer: a Core Wars-like game</h1> <p>In radare2, there's an intermediate language and VM called ESIL. It is used to emulate code, and supports many architectures. r2wars is built on top of r2's ESIL VM, so it supports any ESIL instruction sets, for example x86-32, mips-32 or arm-64.</p> <p>Since it's based on <a href="https://en.wikipedia.org/wiki/Core_War">Core Wars</a>, in r2wars two opponents create "bots", short assembly programs, which are executed one after the other, in the same memory space, thanks to ESIL emulation. The goal is be the latest to survive; usually by wiping your opponent so that it executes and invalid instruction.</p> <p>The ESIL vm is initialized with a 1024 bytes memory space plus a stack (more on that later), and the two opponents are placed randomly in this space. They are executed in round-robin: one after the other, one instruction each. A particularity is that you can have an opponent using a different architecture, since multi-architectures combats are supported: you only share the memory space, not the CPU state (registers). The server makes sure your memory does not overlap your opponents' at launch; afterwards, all bets are off. The CPU state is dumped after each instruction, to let the opponent run, and then restored before running the next instruction.</p> <p>A contestant's bot looses when it attempts to:</p> <ul> <li>execute and invalid instruction</li> <li>write or read outside of the arena or stack</li> <li>execute instructions outside of the arena</li> <li>trigger an interrupt, trap, io error or exception.</li> </ul> <p>The most reliable way to win is therefore to trigger one of the previous condition for the opponent by overwriting where it executes. But an other strategy could also be to survive until the other one suicides.</p> <p>You send the source of your bot to the organizers ; the <a href="https://github.com/radare/r2wars">r2wars software</a> is built in mono with a web interface, and automates the following tasks: it uses rasm2, the radare2 assembler to build your bot, it launches radare2, initializes ESIL, then launches 1v1 matches in a tournament, to determine the global ranking.</p> <p>The challenge is ran over the two days of r2 con, with tournament runs at 10am, 2pm, and 5-6pm. So one has 5 runs to perfect their bot (with small prizes after each), and one final run for the final ranking.</p> <h1>Initial approach</h1> <p>I'm not a beginner to this challenge, since I <a href="r2con2018.html">participated last year</a>; so my first idea was to simply to take my last bot, which ranked second last year, and submit it. Here is the commented arm64 assembly source:</p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="no">start</span> <span class="c">; get address of start label with pc-relative adr instruction</span> <span class="nf">mov</span> <span class="no">x3</span><span class="p">,</span> <span class="mi">1008</span> <span class="c">; put relocation address in x3 (just before the end of arena)</span> <span class="nf">neg</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x0</span> <span class="c">; put some value looking like ffffff in x5 for writing</span> <span class="nf">mov</span> <span class="no">x4</span><span class="p">,</span> <span class="no">x5</span> <span class="c">; copy x5 to x4</span> <span class="nf">ldp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">]</span> <span class="c">; load code at start label into x1 and x2 (2 4-bytes instructions in each)</span> <span class="nf">stp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">]</span> <span class="c">; store code to relocation address</span> <span class="nf">br</span> <span class="no">x3</span> <span class="c">; jump to the core loop at the relocation address</span> <span class="nl">start:</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="c">; decrement x3 by 16, then store 16 bytes of data (x5 and x4) in address pointed by x3</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="c">; same write 16 bytes + pre-decrement</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="c">; same write 16 bytes + pre-decrement</span> <span class="nf">b</span> <span class="no">start</span> <span class="c">; loop with relative backward jump</span> </pre></div> <p>The core of this bot is 4 instructions, with a small setup to relocate the bot at the end of memory. Once the whole memory is overwritten, this bot makes a write to invalid address (x3 underflows) and dies.</p> <p>I reused my <a href="https://github.com/anisse/r2warsbots/">workspace from last year</a> meaning I could immediately do debug runs in radare2, but I had to reinstall the official mono builds (the only ones that work with r2wars) to run the game simulations.</p> <p>I mildly tested my bot in radare2 (it worked, like last year), and submitted it immediately once my plane landed so that I could do the first round. I didn't want to spend to much time on it this year and wanted to follow more talks.</p> <h1>First roadblock</h1> <p>I got a message from the organizer asking me if I tested my bot. Hmmm, my spidey sense told me this wasn't a good sign. Of course, I tested it in radare2, it did overwrite the whole memory.</p> <p>But as the first round came, something weird happened. It seemed like my bot committed suicide after only a few cycles. Did I miss something or trigger an exception ? I tried it again, and found I was able to reproduce the problem in r2wars, so I went to see the organizer to ask for help. It seemed the problem was only reproduced when running the bot in r2wars; the only difference with my testing script, is that r2wars needs to save and restore the CPU state after each instruction to execute each contestant in round-robin.</p> <p>Very soon, skuater had a minimal reproducer of the issue, which helped pancake <a href="https://github.com/radare/radare2/commit/7bcac956b181ac43960516dc31ca5234cd781df5">issue a fix</a>. It was an underflow on restoration of the zero-register (x31), which instead of being ignored, triggered writes side-effects in x0, corrupting it.</p> <p>I took some time for the fix to propagate to the machine used to run the tournament, so I missed the next two rounds of the day as well: not such a big issue since it gave me more time to follow the talks and socialize.</p> <h1>Reminescence</h1> <p>I took some time to try out an idea I had last year and for which I had <a href="https://github.com/radare/radare2/commit/1b36c2485e55f28f7d41fbc8ce5fe8c303de1b5c">submitted an ESIL patch</a>: the single instruction bot, that consisted of only one instruction, that writes itself at the next address with a post-increment:</p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="no">start</span> <span class="nf">ldr</span> <span class="no">x1</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">]</span> <span class="nf">str</span> <span class="no">x1</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">]</span> <span class="nf">br</span> <span class="no">x3</span> <span class="nl">start:</span> <span class="nf">str</span> <span class="no">x1</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">,</span> <span class="mi">4</span><span class="p">]!</span> <span class="nf">str</span> <span class="no">x1</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">,</span> <span class="mi">4</span><span class="p">]!</span> </pre></div> <p>It still needs to write 8 bytes (two identical instructions), but it's effectively only using one at a time. It takes advantage of the fact that all registers (here x2 and x3) are initialized to zero.</p> <p>While the bot did work, it wasn't very effective in my simulations, since the smaller footprint (4 bytes at a given time) wasn't worth much lower write throughput, which seems to be a core metric (but not sole, as we'll see later) in bot performance.</p> <h1>Insomnia</h1> <p>I didn't mean to, but it was bound to happen. I couldn't sleep, despite waking up at very early to catch my morning flight.</p> <p>It occurred to me that I could simply workaround the radare2 issue by changing the register allocation in last year's bot: the bot was, after all, only using six out of thirty-one available registers on aarch64.</p> <p>So I changed <a href="https://github.com/anisse/r2warsbots/blob/master/bot13.asm">the bot</a> to use different registers. It did work with a buggy radare2. But it wasn't enough.</p> <p>I couldn't just keep testing against my simple bots, so I decided to do simulations against some of the best contestants from last year. Some <a href="https://github.com/Ik4ru5/r2wars-bots/blob/master/r2con.x86-32.asm">had been</a> <a href="https://github.com/pageflt/r2wars-2018/blob/master/t_pageflt-v1.0.x86-32.asm">published</a> on github, so I downloaded them for simulation.</p> <p>And they were still better. So I changed strategies... Along the way, I tried changing the loop to have one restore between each instruction, as well as delaying the relocation as much as possible, in order to beat other bots that would relocate at the end, like ik4ru5. In retrospect, the former (continuous restoration) was a mistake: it divided write throughput by 3, which is a hard blow to the bot performance.</p> <p>Here, you can see how I duplicated the relocation code store instruction in order to delay the main loop as much as possible:</p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x10</span><span class="p">,</span> <span class="no">start</span> <span class="nf">mov</span> <span class="no">x13</span><span class="p">,</span> <span class="mi">1008</span> <span class="nf">mov</span> <span class="no">x14</span><span class="p">,</span> <span class="no">x13</span> <span class="nf">sub</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x5</span><span class="p">,</span> <span class="mi">1</span> <span class="nf">mov</span> <span class="no">x4</span><span class="p">,</span> <span class="no">x5</span> <span class="nf">ldp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x10</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">br</span> <span class="no">x13</span> <span class="nl">start:</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x14</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x13</span><span class="p">]</span> <span class="nf">b</span> <span class="no">start</span> </pre></div> <p>It did provide better performance against ik4ru5 and t_pageflt, but neither of them were participating in this year's r2 wars… So the result the next day on the first round wasn't really good:</p> <p><img alt="Round 4/6: I ranked 6/12" src="/images/r2con2019/round4.jpg"></p> <p>I ranked 6 out 12.</p> <h1>Intelligence</h1> <p>Luckily, I had decided to fully record this round (or as much as possible) in order to gather intel on how the other bots <em>exactly</em> worked. I just couldn't continue going blind since there was only two tournament runs, which wouldn't provide a good feedback loop for bot improvement.</p> <p>I recorded videos and took a few photos in order to have a view on every participant, then decided to <strong>rewrite the top-5 bots from scratch</strong>. I first thought of using the assembly source, but since the r2 disassembler converts relative addresses (jumps, etc.) to absolute addresses, it would be hard, so I "simply" copied the hex code of each instruction into <code>.hex</code> directives for the assembler. It wasn't that simple though, as the quality was sometimes less than optimal:</p> <p><img alt="Hexadecimal copy-writing" src="/images/r2con2019/hex-copywriting.jpg"></p> <p>The challenge for next year would be to automate this task with OCR or Google Lens.</p> <p>After a few typos, I finally had <a href="https://github.com/anisse/r2warsbots/commit/0e478bec9531139e1bbcbcb1e8d3c152fb5c5db9">the code of the other top-5 contestants</a> <em>for this tournament iteration</em>. I just had to hope that they wouldn't modify their bots too much before the next run.</p> <p>Without surprise, they were very good, and were beating all the bots I had ever written (mostly). It might have stemmed from the <code>pushal</code> strategy that all the x86-32 employed: this instruction enables a throughput of 32 bytes per ESIL cycle, twice the 16 bytes per second of arm64 <code>stp</code>.</p> <p>So I ran more simulations and iterated over the bot from last year.</p> <h1>Improvements</h1> <p>After a few iterations (not <a href="https://github.com/anisse/r2warsbots/blob/master/bot19.asm">shown</a> here), here is the bot that had a good win-rate in simulations (5 out of 5 adversaries). You can see it has many small changes:</p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x10</span><span class="p">,</span> <span class="no">start</span> <span class="c">; 1</span> <span class="nf">mov</span> <span class="no">x2</span><span class="p">,</span> <span class="mi">512</span><span class="err">+</span><span class="mi">32</span> <span class="c">; 2 - heuristic</span> <span class="nf">mov</span> <span class="no">x8</span><span class="p">,</span> <span class="mi">0x180</span> <span class="c">; 3 - heuristic</span> <span class="nf">mov</span> <span class="no">x9</span><span class="p">,</span> <span class="mi">0x180</span> <span class="c">; 4 - heuristic</span> <span class="nf">ldp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x10</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 5</span> <span class="nf">ldp</span> <span class="no">x14</span><span class="p">,</span> <span class="no">x15</span><span class="p">,</span> <span class="p">[</span><span class="no">x10</span><span class="p">]</span> <span class="c">; 6</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x9</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 7</span> <span class="nf">stp</span> <span class="no">x14</span><span class="p">,</span> <span class="no">x15</span><span class="p">,</span> <span class="p">[</span><span class="no">x9</span><span class="p">]</span> <span class="c">; 8</span> <span class="nf">br</span> <span class="no">x8</span> <span class="c">; 9</span> <span class="nl">start:</span> <span class="c">; 10</span> <span class="nf">and</span> <span class="no">x2</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="c">#0x3f0 ; 11 - looping with a modulo</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 12</span> <span class="nf">stp</span> <span class="no">x14</span><span class="p">,</span> <span class="no">x15</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 13</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 14</span> <span class="nf">stp</span> <span class="no">x14</span><span class="p">,</span> <span class="no">x15</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 15</span> <span class="nf">stp</span> <span class="no">x11</span><span class="p">,</span> <span class="no">x12</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 16</span> <span class="nf">stp</span> <span class="no">x14</span><span class="p">,</span> <span class="no">x15</span><span class="p">,</span> <span class="p">[</span><span class="no">x2</span><span class="p">],</span> <span class="mi">16</span> <span class="c">; 17</span> <span class="nf">b</span> <span class="no">start</span> <span class="c">; 18</span> </pre></div> <p>From line 2 you can see that something changed: I've split the main heuristic of relocating to the end of memory and writing from there into two: now the relocation address is at 384 bytes, and the start of writing address is 544.</p> <p>The main loop (line 10-18) is now 8 instructions instead of 4, so it needs to be loaded with two <code>ldp</code> (line 5-6), and stored with two <code>stp</code> (line 7-8). Both used post-increment instead of pre-increment addressing.</p> <p>In the loop, you see that we repeat the two <code>stp</code> in line (12-17): that's because we now write the bot itself all over memory, instead of 0xFF. I found this was just as efficient as 0xFF against x86 bots, and luckily, there was no arm64 participant this year !</p> <p>Finally, this bot never dies by itself: as you can see the first instruction is an <code>and</code> in order to only keep the lower bits of the write address register <code>x2</code>. We need to make sure at the end of the of the loop that we finish with 0x400 in x2; this has a consequence of limiting the choices of the write-start heuristic (must be in the form n*96+64), and we absolutely need to use a post-increment instead of a pre-increment to detect the overflow before writing out of the arena.</p> <p>Since this bot was unbeatable in my simulations (except by some arm64 bots with different heuristics) against "real" bots, I was very confident, and submitted my new iteration to the organizer, full of hubris:</p> <p><img alt="Hubris: this is a winner" src="/images/r2con2019/hubris.png"></p> <p>But unfortunately, it wasn't the winner of the next round either, although it came close:</p> <p><img alt="Round 5/6: I ranked 2/15" src="/images/r2con2019/round5.jpg"></p> <p>This was mostly because the CAP had improved his bot again, and we were now very close in terms of performance. I missed the beginning of this tournament round, so I don't know which which combat round was lost or why.</p> <h1>Interlude: funny bots spotted</h1> <p>During the first tournament round of the second day, we noticed something weird: an x86 bot was quite big, and took a lot of cycles of doing almost nothing:</p> <p><img alt="Bier" src="/images/r2con2019/bier.jpg"></p> <p>Turns out the bot was writing a message on screen, which took lots of cycles to write, because it was encoded in a bitmap that needed to be decoded. This bot had little chance of survival, in addition to triggering the timeout (4000 cycles), before finishing the full message, which I was told needed ~6000 cycles to complete. But a colleague of the author of PROSTmahlzeit was kind enough to submit the surviv0r bot so that it would get a chance to write its message. It was noticed by the organizers, and he got the prize he asked for :-)</p> <p>Then, on the next tournament run, someone copied the idea, but implemented it in mips:</p> <p><img alt="Help" src="/images/r2con2019/help.jpg"></p> <p>No bitmap though here since every address is hardcoded, making for a huge bot :-)</p> <h1>ARMv7 explorations</h1> <p>Over lunch, I was reminded of an ARMv7 instruction a colleague told me about: <code>stmia</code> and its <code>ldmia</code> counter part. These are used to store/load a set of registers with a post-increment of the address register. They're usually used for stack push/pop, fast memcpys, or context switching.</p> <p>More importantly, it means you can reach a write throughput of 64 bytes per cycle (32 bits * 16 registers). So I modified my workspace to <a href="https://github.com/anisse/r2warsbots/commit/b21d3658b5337c75aa3038b9488b7031ee7dfea3">add support for arm32 bots</a> and wrote a simple bot. Unfortunately, the post-increment in stmia did not work in ESIL.</p> <p>Since I had decided to leave my bot as-is for the last round, I took instead some time to fix the <code>stmia</code> behaviour in ESIL, and <a href="https://github.com/radare/radare2/pull/14983">submitted a pull request</a> even <a href="https://github.com/radareorg/radare2-regressions/pull/1946">with tests</a> this time !</p> <p>It's quite simple, and if you want to contribute to radare2, you can look at these patches, and do the same thing for <code>stmib</code>, <code>stmdb</code>, <code>stmdb</code> or <code>ldmia</code> variants. You can look at one of the <a href="https://github.com/radareorg/r2con2019/blob/master/talks/journey-through-esil/slides.pdf">ESIL</a> <a href="https://www.youtube.com/watch?v=qiuLdZ9kXLY">introductory</a> <a href="https://www.youtube.com/watch?v=R2jBSNrfAaI&amp;list=PLjIhlLNy_Y9Po69BDCTEnrKvwLneSkG26&amp;index=11&amp;t=0s">talks</a> to understand how it works.</p> <h1>Final results and notes</h1> <h2>Tips</h2> <p>Here's what I learned from the r2wars competition: Measure the average throughput per cycles (in my case, it was 12, just like last year). Measure the cycles to first write (7 cycles), and the length of your bot's main loop (32 bytes). Test against real bots; you can look at <a href="https://github.com/anisse/r2warsbots/tree/master/competitors">my repo</a> for an overview, but for best results, try the ones you competed with in a previous round ! Once you know your competitors, you can tune the bot style (static or mobile), your start position, what type of data you write (are you writing valid x86 opcodes?), where you start writing it, etc.</p> <h2>Results</h2> <p>So the last round came and ran. I hadn't modified the bot since last time, as I wasn't sure what to improve, and I was thinking it best to leave well enough alone. Was it a good decision? Lets find out:</p> <p><img alt="Final round: I arrived 1/16 !" src="/images/r2con2019/finalround.jpg"></p> <p>It turns out it was: I arrived first ! Of course it was luck since neither me nor <a href="https://twitter.com/CaptnBanana">the cap</a> decided to modify their bot.</p> <p>Note that the cap arrived in the <a href="https://bananamafia.dev/post/r2ctf-2019/">top 3 of the CTF</a> <em>and</em> also co-won the r2 <a href="https://bananamafia.dev/post/r2-pwndebian/">PwnDebian challenge</a> !</p> <p>I also learned that I was the only one to have copied and run simulation with the competitors' bots, so the "intelligence" I described above gave me a bit of an advantage :-)</p> <h2>r2wars future</h2> <p>As we saw before, the write throughput is an important metric. And since ESIL isn't cycle accurate compared to the architectures it emulates, anyone adding instructions with higher throughput to the ESIL machine would have a huge advantage. </p> <p>So the only logical conclusions is to move to SIMD instruction sets. Once NEON and AVX-512 are implemented, it will make for much faster bots. NEON <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0406b/index.html">VSTM</a> can write 128 bytes in one instruction for example. We'll see if people plan r2wars by adding instructions to ESIL in advance :-) If it happens, the organizers might need to change the rules: have a bigger arena, ban some instructions, or make them take more cycles (i.e put a write throughput cycle cap in ESIL).</p> <p>I want to thank the organizers for the incredible turn-around after I reported an issue. The second day, r2wars runs moved from Windows to MAC, they added color legends in the corners, and it ran and timeout-ed faster !</p>r2con 20192019-09-06T00:00:00+02:002019-09-06T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-09-06:/r2con-2019.html<p>I'm back in Barcelona for this year's edition of <a href="https://rada.re/con/2019">r2con</a>. You can read <a href="r2con-2018.html">my r2con 2018 report</a>.</p> <p>radare2 is debugging and reverse engineering toolkit. It's mostly used from the command line or through a programming interface (r2pipe) that is identical to the command line one.</p> <h1>Cutter</h1> <p>Cutter is the official …</p><p>I'm back in Barcelona for this year's edition of <a href="https://rada.re/con/2019">r2con</a>. You can read <a href="r2con-2018.html">my r2con 2018 report</a>.</p> <p>radare2 is debugging and reverse engineering toolkit. It's mostly used from the command line or through a programming interface (r2pipe) that is identical to the command line one.</p> <h1>Cutter</h1> <p>Cutter is the official graphical user interface for radare2. It's cross-platform (written in C++ with Qt), and built on top of radare2.</p> <p>It has a dynamic graph view, a linear disassembly view, and an hexdump view for data, and other various widgets.</p> <p>During the last year, plugin support was added as well as a graph overview, a theme editor, and many new translations and bug fixes.</p> <p>A new entrant in the reversing landscape this year was Ghidra, a new tool with a particularly powerful decompiler. The decompiler part is now integrated directly into radare, with the r2ghidra-dec plugin. This plugin also works with cutter.</p> <p>It allows exploring the decompiled C-code side-by-side with the disassembly, as well as import headers to decode struct accesses, etc.</p> <h1>Who you gonna' syscall ?</h1> <p>Grant's goal with this talk is to share how he improved with frida and r2 on iOS, to automate analysis of arm64 protected iOS apps that include anti-debug.</p> <h1>down the business with r2dwarf</h1> <p>Dwarf in this talk, is a Frida frontend, and a framework to allow debugging a target process with a GUI. r2dwarf is a pipe between Dwarn and r2. It wraps common Frida operations to make dynamic debugging and reversing easier.</p> <h1>Understanding ESIL emulation</h1> <p>ESIL in an emulator inside radare2. It's built on top of an intermediate language based on reverse polish notation.</p> <p>The ESIL machine is based on infinite memory and registers. There are then bindings/aliases to map ESIL registers to the architecture-specific ones.</p> <p>The ESIL machine is based on a set of instructions with a stack machine (using the polish notation). Every native instruction is converted into a "transformation", which is an ESIL string in polish notation.</p> <h1>Overview of the Linux threat landscape</h1> <p>An issue with Linux is the low visiblity of the threat landscape. At Intezer, the team discovered many new threats, from crypto-mining to trojans and botnets, some of them coming from nation-state actors.</p> <p>As the landscape evolves, threat detection will improve, as well as the malware methods. The goal of the talk is to present a few of the techniques for defender awareness, in particular ELF tricks.</p> <p>ELF parsing can be complex. Sections in binaries get loaded into memory segments depending on their types. A common obfuscation technique is to remove or scramble some sections. To workaround that is to simply, the best is to ignore or scrub scrambled sections when analyzing.</p> <p>It's often possible to break a parser with just one byte modification, for example, by modifying the endianness of the file.</p> <p>Another presented technique is to hide dynamic entries. It has been found is a lot of native objects packed in android malware. The technique uses a mismatch between the section offset, and the address that is used to map the segment in order to have a fake dynamic section, and a real one that will be mapped in memory.</p> <p>Relocation hijacking uses the relocation features of ELF: there are usually well known and easy to detect, but it's possible to use a few tricks to avoid detection.</p>Pass the SALT 2019 live report (part 3/3)2019-07-03T00:00:00+02:002019-07-03T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-07-03:/pass-the-salt-2019-3.html<p>This year I'm at the <a href="https://2019.pass-the-salt.org/">Pass the Salt 2019</a> conferences. You'll find my <a href="pass-the-salt-2019-1.html">part 1</a> and <a href="pass-the-salt-2019-2.html">part 2 of my notes here</a>.</p> <h1>Configurations, Do you prove yours ?</h1> <p>by Alexandre Brianceau (<a href="https://2019.pass-the-salt.org/files/slides/17-Configurations_do_you_prove_yours.pdf">slides</a>)</p> <p>Alexandre went a bit more into details on the core concepts behind DevSecOps, and why continous configuration and observability …</p><p>This year I'm at the <a href="https://2019.pass-the-salt.org/">Pass the Salt 2019</a> conferences. You'll find my <a href="pass-the-salt-2019-1.html">part 1</a> and <a href="pass-the-salt-2019-2.html">part 2 of my notes here</a>.</p> <h1>Configurations, Do you prove yours ?</h1> <p>by Alexandre Brianceau (<a href="https://2019.pass-the-salt.org/files/slides/17-Configurations_do_you_prove_yours.pdf">slides</a>)</p> <p>Alexandre went a bit more into details on the core concepts behind DevSecOps, and why continous configuration and observability are important to understand what's happening in an IT system and match compliance targets.</p> <p><a href="https://rudder.io">Rudder</a>, while it started 10 years ago as an ops and reliability tool, now takes an approach to configuration management and observability focused on compliance, and is often used by SecOps teams.</p> <h1>What you most likely did not know about sudo…</h1> <p>by Peter Czanik (<a href="https://2019.pass-the-salt.org/files/slides/18-sudo.pdf">slides</a>)</p> <p>Most people do not know what sudo is (a prefix?) or what it can be used for, Peter says.</p> <p>A basic rule set describes who can do what, where and as which user. But sudo also allows defining aliases and groups for each of these, making configuration less error prone.</p> <p>sudo can modify or filter environment variables, or even spew insults on when someone types a wrong password, although it isn't enable by default. It's possible to add rules to verify integrity of binary before running it, or even record terminal sessions of sudo commands; although those sessions logs are easy to delete. Peter did a demo of this feature with the <code>sudoreplay</code> command that replayed a recorded session.</p> <p>It has a plugin-based architecture, and there are many open source and commercial plugins for sudo. An interesting one is <a href="https://github.com/square/sudo_pair">sudo_pair</a>, which allows real-time approval of sudo commands by an admin user, coupled with live session viewing and control by the admin.</p> <p>The configuration of sudo is interpreted from top to bottom, so one should put the most generic rules first, and exceptions at the end. It's possible to configure sudo through LDAP, allowing to have a remote-only configuration. Peter showed a sample sudoers file, where we can see the importance of the order of rules interpretation.</p> <h2>Logging</h2> <p>By default all logs are sent to syslog. Peter advises using central logging with sudo for analysis. It's possible to use syslog-ng for that, with a minimal configuration.</p> <p>Since he works on syslog-ng, Peter showed an example of building a pipeline for sudo logs and alerting, and sending alerts to slack.</p> <p>In conclusion, sudo is not just a prefix, but a very powerful and versatile tool.</p> <h1>Be secret like a ninja with Hashicorp Vault</h1> <p>by Mehdi Laruelle</p> <p>Credentials sharing between persons or programs is often an issue in an enterprise environment. Hashicorp Vault attempts to solve this issue by providing controlled-access and encrypting "secrets".</p> <p>There are different kind of secrets in Vault: static, key/value secrets; dynamic secrets (cloud…), and the ones that are encrypted on-demand.</p> <p>Vault works by giving access to secrets to an application; the simplest way to use it, is to store static secrets in Vault, and giving access to apps that have the proper role. Then, one should make the secrets dynamic. Finally, sensitive data should also be encrypted, and vault provides a service to do that.</p> <p>Mehdi did a demo showing how a simple service can access credentials to a database with Vault. The Vault app role id and secrets are passed through the environment, then the app uses those connects to Vault's API and get the DB credentials. This app also encrypted sensitive data with the Vault API before storing it in the DB. In this case, the DB username and passwords were generated dynamically by Vault. Then, using Vault web UI, Mehdi decrypted the data that was encrypted.</p> <p>In conclusion, Mehdi says one should always attempt to implement the principle of least privilege, and Vault helps to do that.</p> <h1>Scale Your Auditing Events</h1> <p>by Philipp Krenn</p> <p>Auditd is component that works with the linux kernel auditing system. It can be used to monitor network access, system calls, commands, etc. The raw logs are hard to understand, Philipp says. There are tools to show statistics, or search through the event with auditd, and it is well documented. Namespace support is still a work in progress though.</p> <p>To centralize all of this, the Elastic stack can be used. There's a Filebeat module for Logstash for that, but it relies on regular expressions to parse the raw logs. That's why the Auditbeat module was built in order to get the structured information directly. It's implemented on top of <a href="https://github.com/elastic/go-libaudit">go-libaudit</a>. Philipp showed a few examples of Auditbeat configurations and what can be done with it.</p> <p>Elastic SIEM is a new software that builds on top of the Elastic Common Schema(ECS) and Auditbeat in order to provide a high-level view with search capability on the events that are put into the Elastic stack.</p> <h1>Programming research: a missed opportunity for secure and libre software?</h1> <p>by Gabriel Scherer</p> <p>Public research often picks a hard problem at attempts to solve that. But the produced software are most of the time unmaintained proofs of concept. The free software community also has hard problems to solve, and there a few collaborations between the academic community and free software community, like Coccinellle with the Linux kernel, but not enough, Gabriel says.</p> <p>He showed a demo of a programming environment (Why3) made for writing correct programs and proving them.</p> <p>There are static analyzers that can be very useful to rule out entire classes of failure. Annotations are good, Gabriel says, because they help both humans and tools alike. There a few success stories in that domain, like Astrée, that proved that there were no errors in Airbus flight control software.</p> <p>Verified programming is the next step, were the annotations are used to prove the correctness, like it is done in Spark/Ada.</p> <p>With proof assistants, the users write a full mathematical proof that is verified by the checker. The micro-kernel seL4 was proven this way for example.</p> <p>Unfortunately, free software lacks adoption of those tools, and this is also the research community's fault Gabriel says. An easy way to fix that, is to use safer languages for new projects, citing modern C++ Rust as examples, and stop using C or PHP. A bit more work is required to try to adopt static-analysis tools. And finally, keeping up-to-date on programming research is also important, and funding programmers to go to academic conferences, or even collaborating with academia.</p> <h1>D4 Project - Design and Implementation of an Open Source Distributed and Collaborative Security Monitoring</h1> <p>by Alexandre Dulaunoy, Jean-Louis Huynen and Aurelien Thirion</p> <p>An issue between organizations that want to share sensor information easily and automatically. One of the initial goal of the D4 project was to have flexibility on the type of sensor and the type of information that could be shared. The goal isn't to reinvent existing tools, but to build on top of them and providing sharing capability in the platform.</p> <p>The <a href="https://d4-project.org/">D4 project</a> was started in late 2018, so it's still very young, but it has been fully open from the start.</p> <p>The monitoring protocol is very simple, and can be extended very easily to plug new data sources. The D4 server provides a web interface to browse the monitored data. The team showed a demo of the project, with various types of sensors (DNS, TLS) to show the powerful capabilities of the tool.</p> <h1>No IT security without Free Software</h1> <p>by Max Mehl</p> <p>Free software provides four different freedoms: to use, study, share and modify a piece of software. Security in itself is a process, Max cites Bruce Schneier.</p> <p>Security benefits from free software, through transparency: independent audits increase trust, Max says. Releasing code can be scary, but this is for the best: it pushes one to look closely at what is released when the code is available. Sharing synergies with the community and giving independence to users is also helps security.</p> <p>Max went through a few of the arguments against making software free. He then cited the example of the Huawei 5G controversies, and how they could be solved by moving through free software, whether or not it's realistic in the short term.</p> <h1>Managing a growing fleet of WiFi routers combining OpenWRT, WireGuard, Salt and Zabbix</h1> <p>by Kenan Ibrović</p> <p>Kenan's organization provides secure routers to journalists around the world. They want to manage their fleet of routers around the world, with no on-site technical support.</p> <p>The routers are based on OpenWRT and use Wireguard to provide VPN access to the devices. SaltStack is used to manage the devices, which allows running commands on all the devices securely and remotely.</p> <p>Salt node groups are used to organize the inventory. Pillars makes the states reusable, by storing per-device credentials, variables, etc. Zabbix is used to monitor the routers.</p> <p>All the devices use a public VPN based on OpenVPN for the public connection, with a shared account. Since they don't have enough space on the devices, they use external USB flash drive with <a href="https://openwrt.org/docs/guide-user/additional-software/extroot_configuration">ExtRoot</a> to store more data and install more apps.</p> <p>When the device is updated, by default the apps are deleted, so they have a configuration script to do the reinstallation after updates or even after USB-unplug.</p> <h1>Better curl !</h1> <p>by Yoann Lamouroux</p> <p>The curl project started in 1996 by Daniel Stenberg. It's composed of libcurl which has bindings in most languages, and the curl binary that installed on most OSes.</p> <p>curl is stable and widely deployed. The most basic use is to fetch a URL and show the response body or headers with <code>-v</code>.</p> <p>You can also use <code>--trace-ascii</code> to see the detailed transferred bytes. <code>--trace-time</code> will show detailed timing information when used with <code>-v</code>.</p> <p>When you need to change the IP resolution it's possible to use the <code>--resolve</code> parameter. Yoann says there's no need to remember the parameters, you can put the options in your <code>~/.curlrc</code> if you always use them. If you use browser dev tools, it has a "Copy as curl" feature which gives you a command line you can reuse.</p> <p>It's also possible to generate C source code with the <code>--libcurl</code> option.</p> <h1>PatrOwl - Orchestrating SecOps with an open-source SOAR platform</h1> <p>by Nicolas Mattiocco</p> <p><a href="https://patrowl.io/">PatrOwl</a> is an open source platform to automate security scans, for use in SecOps teams. It has pluggable connectors for data sources, with many already provided by default.</p> <p>According to Nicolas, there's a growing set of challenges in cybersecurity. Automation and orchestration can help address them, but only if you do it properly, and at better scale than attackers. That's why PatrOwl was built. It integrates the best-of-breed scan tools to analyze a network.</p> <p>Written in python3, it integrates multiple engines, by domain; these are applications or web services, like nmap or VirusTotal, that are used in various use cases. PatrOwl is currently looking for contributors and user feedback.</p> <p><em>That's it for Pass the SALT 2019. Thanks to the team for organizing this event !</em></p>Pass the SALT 2019 live report (part 2/3)2019-07-02T00:00:00+02:002019-07-02T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-07-02:/pass-the-salt-2019-2.html<p>This year I'm at the <a href="https://2019.pass-the-salt.org/">Pass the Salt 2019</a> conferences. You'll find my <a href="pass-the-salt-2019-1.html">part 1 of my notes</a> and part 2 here.</p> <h1>Time-efficient assessment of open-source projects for Red Teamers</h1> <p>by Thomas Chauchefoin and Julien Szlamowicz (<a href="https://2019.pass-the-salt.org/files/slides/11-Time_efficient_%20assessment_of_%20open-source_projects_for_Red_Teamers.pdf">slides</a>)</p> <p>In their pentest team, Julien often does red team assessments with big scopes …</p><p>This year I'm at the <a href="https://2019.pass-the-salt.org/">Pass the Salt 2019</a> conferences. You'll find my <a href="pass-the-salt-2019-1.html">part 1 of my notes</a> and part 2 here.</p> <h1>Time-efficient assessment of open-source projects for Red Teamers</h1> <p>by Thomas Chauchefoin and Julien Szlamowicz (<a href="https://2019.pass-the-salt.org/files/slides/11-Time_efficient_%20assessment_of_%20open-source_projects_for_Red_Teamers.pdf">slides</a>)</p> <p>In their pentest team, Julien often does red team assessments with big scopes, often facing a blue team. The talk is a case study on the work they did assessing GPLI, and the methodology they used.</p> <p>GLPI is a GPLv2 inventory tool often used by sysadmins, widely deployed in France and Brazil, which made it an interesting target.</p> <p>In Red Team assessments, discretion is key, as opposed to traditional pentests, where noise does not matter as much. Thomas says the forensing footprint should be as low as possible. Therefore, a good Red Team vulnerability should be silent.</p> <p>An aspect of assessing Open Source Software, is that you don't work with a blackbox, and it's easier to replicate an accurate environment in a lab. In this case, the attack surface that was analyzed is only comprised of non-authenticated code paths. PHP apps often have scripts that are directly accessible. They used an internal tool to help finding public-facing paths, as well as looked as previous GLPI vulnerabilities.</p> <p>They didn't have semantic tooling, but used various hooks for DB queries, or low-level PHP functions, as well as profilers to do the analysis. They also wrapped <code>$_GET</code> and <code>$_POST</code> objects in order to have automatic analysis of bad usage.</p> <p>The first issue they found was an infoleak, that exposed the various versions of GPLI, PHP, the OS. Then, they found an SQL injection, which wasn't immediately usable because the queries parameter were sanitized. This sanitization was still bypassable in a few cases, which they were able to do with another injection.</p> <p>They then looked at the way the "Remember me" feature was implemented with json in a cookie. This allowed controlling the algorithm of password verification, therefore enabling a denial of service on the server. It also used PHP loose comparisons, which allows abusing string compare when a string starts with '0e' (see <a href="https://anisse.astier.eu/qual-sigsegv1-rtfm.html">my writeup of 'La simplicité' in SIGSEGv1 CTF</a>).</p> <p>They also found a Local File Inclusion (LFI) issue in a query, which then allowed calling arbitrary functions.</p> <p>The communication with GPLI maintainers was very smooth, and they were quick to react and apply patches, even if it took some time to arrive in released versions.</p> <h1>Hacking Jenkins!</h1> <p>by Orange Tsai (<a href="https://2019.pass-the-salt.org/files/slides/12-Orange_Tsai%20-%20Hacking_Jenkins_for_public.pdf">slides</a>)</p> <p>Jenkins is the most used CI/CD software in the world. It's a very interesting target, because it has access to source code, might have access to credentials, or compute nodes. It has been exploited in the wild.</p> <p>The most common attack vector is a dictionary attack on the login page. Then, the previous known vulnerabilities, like previous deserialization bugs, of which there were many instances, because the initial fixes were blacklist-based. The serialization has since been rewritten to replace Java serialization with an HTTP API.</p> <p>Orange then decided to review core Jenkins code, starting with the router. He found an issue with crafting URLs, that were mapped to class names and methods. But since access paths were whitelisted for non-authenticated users, he had to find a given path through whitelist objects in order to reach a dangerous invocation.</p> <p>The next step, was to find another vulnerability to chain with in order to reach code-execution on the server. For this, he looked at Pipeline, a DSL built on top of groovy that allows doing reproducible, trackable in VCS Jenkins scripts.</p> <p>Pipeline scripts must have a valid syntax in order to be interpreted: this is simple to do, but only path he found only did parsing, and no execution. To bypass that, Orange used Groovy meta-programming with the <code>@Grab</code> and <code>@Asttest</code> decorators that allowed to execute code. Finally, he found an issue in the <code>@Grab</code> implementation that allowed injecting a jar file by URL with <code>@GrabResolver</code>.</p> <p>After the vulnerabilities were reported and fixed, new vulnerabilities were found by other researchers, to have a more generic entry points and ease exploitation. Unfortunately, public exploitation of these issues were common, including the infamous hack of the Matrix infrastructure, because many people were slow to update their Jenkins instances.</p> <h1>VLC Security</h1> <p>by Jean-Baptiste Kempf (<a href="https://2019.pass-the-salt.org/files/slides/13-VLC_Security.pdf">slides</a>)</p> <p>VLC is the most popular video player, and its popularity comes from the fact that it can understand most video formats, even incomplete files. Jean-Baptiste estimates that it has more than 450 millions users, even there is no telemetry to have an exact count, because that would be "spying on users".</p> <p>VLC has about 1 million line of codes, but lots(100+) of dependencies, which brings the total to more than 15 millions lines of code, include C, C++ and handcrafted ASM, of varying quality.</p> <p>VLC development happens on mailing list, with relatively long review processes. Static and dynamic analysis is done by most developers. Fuzzing has been added recently. Hardening has started in 3.0, from PIE code to fixing most warnings of modern compilers, enabling ASRL, DEP etc.</p> <p>The release process is very strict, with offline signing and very well defined steps.</p> <p>Despite Jean-Baptiste's hate for bug bounties, VLC participated in the EU-FOSSAv2 program, with a twist: they decided to add bonuses for researchers that provided fixes. The result of this bug bounty were 31 security issues, with one classified as high. The program was successful in the end.</p> <p>The best researchers were very good, but the worst were very bad going as far as insults or death threats. In general, half of the security reports are "total crap", Jean-Baptiste says. There's also a tendency to overblowing security issues: from bad CVSS scores to click-baiting articles. The evaluation of the impact is also very bad, since even very-hard to exploit issues are given up to 9.8 CVSS scores without PoCs. Jean-Baptiste followed with more examples of bad behaviour coming from parts of the infosec community.</p> <p>A research project inside VLC is to put a sandbox inside VLC to segment the different parts of VLC to have different permissions; hopefully, this should improve the general security, but this is a complex endeavour.</p> <h1>OSS in the quest for GDPR compliance</h1> <p>by Aaron Macsween (<a href="https://2019.pass-the-salt.org/files/slides/14-OSS-in-the-quest-for-GDPR-compliance.pdf">slides</a>)</p> <p>Aaron started by saying he was filling in for Critina Delisle, the original author of the talk that couldn't make it.</p> <p>Privacy and security are often "added at the end" of a project which doesn't work and has terrible consequences. And there's no single fix to this, since these domains are complicated, and often dependents. For both, one must evaluate what the threat model : what you're protecting, for how long, from whom, etc. In some cases, you need to chose between Privacy and Security.</p> <p>An example, Aaron says, is that you might optimize for security by reducing privacy via surveillance: for example what you're bank does with financial transactions.</p> <p>At the other end of the spectrum, you can optimize for privacy with less security, by having web services that have no authentication, like privacy pastebins, or mega's ciphered uploads for example.</p> <p>Cryptpad, which Aaron is the lead developer of, is a real-time collaboration tool like Etherpad, but with encryption. The browser-based "thick" client is doing the most work, with an append-only log data structure on the server. It has many extensions, from read/write/delete features, to file-server capability, etc. It's used by various users in hackerspace or activist groups; it was funded with a grant from BPI France, the NLNet Foundation, and donations.</p> <h2>GDPR</h2> <p>The European privacy regulation has been in effect since May 2018, and has made Aaron's job much easier by raising awareness on the privacy issues, he says.</p> <p>The strategy in Cryptpad is of data minimization, by reducing what's needed at a given moment, like the way cryptpad does peer-to-peer conflict resolution instead of server-based one.</p> <p>The Data Protection Officer (DPO) role (Cristina's) can be adversarial, but always useful: it forces auditable traces around the handled data for example. The data controllers are the DPO's employers, the ones handling the data. And the data processors can be any third parties handling the data, like the hosting or payment processing companies.</p> <p>Aaron says there still a few areas of uncertainty, like when a self-host becomes a data controller, or how to challenge the "legitimate use" that has a fuzzy definition in the law.</p> <h1>TLS 1.3: Solving new challenges for next generation firewalls</h1> <p>by Nicolas Pamart</p> <p>Nicolas is presenting joint work with Damien Deville and Thomas Malherbe on how they adapted their firewall and IPS product to work with TLS 1.3.</p> <p>The Intrusion Prevention System inside the proprietary Stormshield product does TLS analysis: it looks at the data in the Client Hello Server Hello handshake packets to get the client and server certificates. With TLS 1.3, it's no longer possible to get the Server Certificate just by looking at network traffic.</p> <p>Since they didn't want to decrypt in order to stay passive, they elected to buffer the ClientHello and replay it once the connection was approved. In order to get the Server certificate, the IPS contacts the destination server with the same SNI and cipher list, but with its own KeyShare. It can then make a decision and replay the original ClientHello so that its connection can be established. A cache was added on top of that to have only one request per domain per time period.</p> <p>In order to handle TLS 1.3 session resumption, in the case SNI isn't provided; there is also an SNI coherence layer, which is a cache of SNI presence.</p> <p>In a response to a question, Nicolas said that encrypted SNI with DNSSEC might completely break this feature of the IPS.</p> <h1>Lookyloo: A complete solution to investigate complex websites - with a decent UI</h1> <p>by Quinn Norton and Raphaël Vinot (<a href="https://2019.pass-the-salt.org/files/slides/16-Lookyloo.pdf">slides</a>)</p> <p><a href="https://github.com/CIRCL/lookyloo">Lookyloo</a> is an UI and visualize requests done in complex websites. It allows visualizing exactly which URLs are loaded when contacting a website. It's built on top Splash and the ETE toolkit.</p> <p>When looking at a tree, you can see when a requests switches to insecure mode, or how many ads toolkit are loaded.</p> <p>It can be used to detect when websites use a technique to bypass TLS mixed-content warnings, or when there are transparent HTTP meta redirects.</p> <p>It can help popular sites analyze what resources are pulled by the single ad network code they put on their frontpage. Everytime you load a page, it might change, and lookyloo allows looking at the requests, saving them and analyzing it offline.</p> <p>Quinn and Raphaël showed an example where a very popular website showed a GDPR warning, but still loaded dozens of resources before user consent was given.</p> <h1>Rump sessions</h1> <p>The rumps are five minutes talks on various subjects.</p> <h2>BPFCTRL</h2> <p>by Eloïse Brocas and Eric Leblond</p> <p><a href="https://github.com/StamusNetworks/bpfctrl">bpfctrl</a> is a new tool to analyze and manipulate eBPF maps loaded in the Linux kernel. Eloïse built it as wrapper on top of bpftool. It's higher-level and written in a mix of C and rust. It was missing in Suricata, to debug the traffic that was filtered.</p> <h2>$0.02 DNS Firewall with MISP</h2> <p>by Xavier Mertens (<a href="https://2019.pass-the-salt.org/files/slides/RUMP-XME.pdf">slides</a>)</p> <p>Xavier recommends using your own resolver, and log all queries, because everything goes through the DNS. With RPZ, it's possible to filter malicious domains by returning fake addresses.</p> <p>There are plenty of malicious domain sources, but Xavier chose to use MISP, an incident response and sharing platform, to handle this list. A script does the extraction of malicious domains, which is then used by the bind configuration. Xavier <a href="https://isc.sans.edu/forums/diary/DNS+Firewalling+with+MISP/24556/">posted about his configuration here</a></p> <h2>Gamebuino as a keyboard</h2> <p>by Antoine Cervoise (<a href="https://2019.pass-the-salt.org/files/slides/RUMP-ANTOINE-Gamebuino%20as%20a%20keyboard.pdf">slides</a>)</p> <p>Antoine presented how he added <a href="https://github.com/cervoise/so-i-became-a-keyboard">keyboard functionality</a>, to his Gamebuino, an open console based on Arduino, and used it to run automated commands, with different keyboard layouts supported.</p> <h2>RUDDER</h2> <p>by Alexandre Brianceau (<a href="https://2019.pass-the-salt.org/files/slides/RUMP-ALEXANDRE-BRIANCEAU-Rudder.pdf">slides</a>)</p> <p><a href="https://www.rudder.io/">Rudder</a> is an open-source continuous compliance auditing et configuration platform.</p> <h2>Why fuzz Rust code ?</h2> <p>by Pierre Chifflier (<a href="https://2019.pass-the-salt.org/files/slides/RUMP-PIERRE-rump-why-fuzz-rust.pdf">slides</a>)</p> <p>Is the Rust memory safety and a test suite enough to ensure correctness ? Pierre says no, and every error should be handled properly without crashing. That's why rust code should be fuzzed. The <code>cargo-fuzz</code> crate can help for that, especially if you combine it with code coverage analysis.</p> <p>Pierre says fuzzing is necessary, but neither sufficient, nor a starting point. You should also share the fuzzing corpus, because it has a lot of value.</p> <h2>Total recall</h2> <p>by Alexandre Dulaunoy (<a href="https://2019.pass-the-salt.org/files/slides/RUMP-ALEXANDRE-rumps-total-recall.pdf">slides</a>)</p> <p>CIRCL does a lot of crawling of websites, including on Tor, where they take a lot of screenshots of webpages. They have a lot of data, but need to analyze it. Total recall is a tool to do large-scale image comparison and classification, to find phishing sites that look like popular websites.</p> <p>A tool named <a href="https://github.com/CIRCL/douglas-quaid">Douglas Quaid</a> has been released, as well as the <a href="http://www.circl.lu/opendata/circl-phishing-dataset-01/">CIRCL phishing dataset</a>.</p> <h2>Psychological manipulation</h2> <p>by Simon Heilles (<a href="https://2019.pass-the-salt.org/files/slides/RUMP-SIMON.pdf">slides</a>)</p> <p>Simon goes over a few ways to manipulate humans. Understanding the different techniques is very useful to protects oneself.</p> <p><em>That's it for part 2. <a href="pass-the-salt-2019-3.html">Part 3 is continued here</a>.</em></p>Pass the SALT 2019 live report (part 1/3)2019-07-01T00:00:00+02:002019-07-01T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-07-01:/pass-the-salt-2019-1.html<p>This year I'm at the <a href="https://2019.pass-the-salt.org/">Pass the Salt 2019</a> conferences. You'll find my notes updated in real time here.</p> <h1>Kill MD5</h1> <p>by Ange Albertini (<a href="https://2019.pass-the-salt.org/files/slides/01-KILL_MD5.pdf">slides</a>)</p> <p>Ange starts by saying that he does not know much about crypto, and he had help from Marc Stevens, a hash-collision cryptographer. This talk should …</p><p>This year I'm at the <a href="https://2019.pass-the-salt.org/">Pass the Salt 2019</a> conferences. You'll find my notes updated in real time here.</p> <h1>Kill MD5</h1> <p>by Ange Albertini (<a href="https://2019.pass-the-salt.org/files/slides/01-KILL_MD5.pdf">slides</a>)</p> <p>Ange starts by saying that he does not know much about crypto, and he had help from Marc Stevens, a hash-collision cryptographer. This talk should provide a high-level overview.</p> <p>A property of hashes is that you shouldn't be able to guess a hashed content from the hash value. This is still true, even for MD2, Ange says.</p> <p>Hash collisions are separated in two types: identical-prefix and chosen prefix.</p> <p>A collision is separated in 4 parts: the prefix, its padding, the collision block, and the suffix. Only the collision block changes between two collided files.</p> <p>For a chosen prefix collision, the two prefixes can be completely different. It means the state of the hash function will be different in the two files when computing the collision blocks.</p> <h2>MD5</h2> <p>MD5 is mostly dead since 2008 when a practical attack was demonstrated.</p> <p>To further prove this point, Ange computed a few demos colliding JPGs, executables, MP4s, GIFs, etc. He even went as far as doing a collision of four different file type for PoC||GTFO: PDF, PNG, PE and MP4.</p> <p>There a few different algorithms to exploit MD5 collisions, with varying difficulty.</p> <p>To craft a file that is collidable, the common approach is to have a comment in the file; in the collision block, the comment will mask a chunk that is isn't in the original file. In order to have a file format that isn't collidable, don't put comment in it, or remove comments before computing hashes.</p> <p>To conclude, Ange said that MD5 isn't a cryptographic hash, but a toy function. Do not use it to compute hashes. His work was meant to prove the real-world feasibility of hash attacks.</p> <h1>Dexcalibur - automate your android app reverse</h1> <p>by Georges-B. Michel (<a href="https://2019.pass-the-salt.org/files/slides/02-Dexcalibur.pdf">slides</a>)</p> <p>An obfuscated Android app will usually be split in multiple .dex stages, first a packer, then a loader and then the payload. Only the first stage is usually available for static analysis, the rest is usually done dynamically, and can be very cumbersome.</p> <p>That's why Georges started developing a toolkit in order to discover automate this process: <a href="https://github.com/FrenchYeti/dexcalibur">Dexcalibur</a>.</p> <p>Dexcalibur integrates many tools: baksmali for disassembling, as well as file identifiers, parsers, static and dynamic code analyzers, and instrumentatioon with Frida. He added a modular heuristic search engine on top of that, as well a web UI.</p> <p>The current limitations is that it can't hook native code on do symbolic execution of bytecode. To fix that, Georges intends to integrate radare2 in Dexcalibur in the future.</p> <p>In a demo, Georges showed a crackme running in an emulator, and being analyzed in real-time in the Dexcalibur web UI. It works by extracting all the code present in an APK, then instrumenting it before running it. There are default hooks, for filesystem access for example. The code is then run, and at hook points, the analysis engine will evaluate a few rules in order to provide the most complete view of the reversed app possible.</p> <p>As an example, if code is called dynamically (reflection), Dexcalibur will still update the cross-references (xref) in order to have a complete call graph.</p> <h1>Reversing a firmware uploader &amp; other NFC stories</h1> <p>by Aurélien Pocheville (<a href="https://2019.pass-the-salt.org/files/slides/03-Reversing_a_Firmware_uploader_others_NFC_stories.pdf">slides</a>)</p> <p>Aurélien started with a warning to always be careful when manipulating LF NFC tags. He once erased one by mistake when attempting to use a brand new proxmark.</p> <p>The Chameleon Mini: Rev E is an NFC writer tool, that is widely available online. Unfortunately, it only worked with Windows, so Aurélien decided to reverse its loader in order to create a multiplatform free and open source solution. Since it was based on a reversed iceman tool, he thought it was just a case of finding an AES key. When launching the tool, it looked like a simple DFU programmer, but he didn't know that at the time.</p> <p>In order to reverse the firmware builder, he started with Ghidra, a relatively recent RE and decompiling toolkit. Very quickly by looking at the code, he found an AES key, and attempted to decrypt the code with it. It wasn't successful. Afterwards, he looked again at the data-preprocessing, and after a two attempts, found that both the data was run through a rolling xor, and the key was used in decryption mode for encryption (and vice-versa).</p> <p>Looking the flasher, he found that the AES was decrypted (in encryption mode), but the rolling-XOR was kept. On-device bootloader undid the rolling XOR.</p> <p>In the end, he had a working solution, but it can still be improved.</p> <p>He then presented a small puzzle with NFC tags controlling access to an apartment complex, which he cloned for a friend, but then ran into an anti-clone protection that triggered erratic behaviour.</p> <h1>Improving your firmware security analysis process with FACT</h1> <p>by Johannes vom Dorp (<a href="https://2019.pass-the-salt.org/files/slides/04-FACT.pdf">slides</a>)</p> <p>IoT devices are growing, and more and more devices are used as part of network attacks. <a href="https://github.com/fkie-cad/FACT_core">FACT</a> is a tool to help doing firmware analysis in order to improve security.</p> <p>A typical firmware analysis, Johannes says, starts with unpacking. Then you'll gather information with the help of a tool in order to attempt identifying obvious weaknesses. Then, if there is nothing obvious, one should start doing reverse engineering. The goal of FACT is to automate the first three parts, leaving RE as a manual phase.</p> <p>The initial goal in 2015 was to improve on binwalk, which mostly does extraction and discovery of files. The goal was to make a new tool with even more automated phases, and being as extendable as possible. FACT is still using binwalk for a lot of use cases, but its internal tools have also replaced it for many operations.</p> <p>In addition to combining unpacking with various tools, FACt does parallel analysis, provides result visualization, a plugin system, and stores its results in a query-able database, which can be used for comparison or statistical analysis. Many statistics are already provided by default.</p> <p>In a demonstration of the tool, Johannes showed how a binary can be found vulnerable to heartbleed in a router firmware, as well as the many generated statistics. In another, he showed the crypto material that was found in the image, in an executable; you can then search for other occurrences of the same key for example. There also various ways to do a search in all the firmwares: from a quick search, to an advanced one, to a raw mongodb query; it's also possible to search by yara rules.</p> <p>The compare feature allows comparing two different versions of the same firmware for example, to find only what changed, or the common software. This can be used for example to find what bug was fixed in a given version. Or, in Johannes example, if a fix actually fixes the core issue.</p> <h1>cwe_checker: Hunting Binary Code Vulnerabilities Across CPU Architectures</h1> <p>by Thomas Barabosch and Nils-Edvin Enkelmann (<a href="https://2019.pass-the-salt.org/files/slides/05-cwe_checker.pdf">slides</a>)</p> <p>Another tool is presented here, to automate a part of the vulnerability research tasks, with an accent on supporting multiple CPU architectures. In order to do that, cwe_checker works with an intermediate representation (IR), generated by BAP, an open source reverse engineering tool.</p> <p>Nils says cwe_checker is inspired by clang-analyzer: it find potential bugs by looking for heuristics of varying complexity. It has a modular structure, and is written in OCaml, like BAP which it builds atop. CWE stands for Common Weakness Enumeration, and is named like that because each specific heuristic analyzer has a given number, and they check for various issues: integer overflow, null dereference, TOCTOU race, etc.</p> <p>cwe_checker does symbolic execution BAP's Primus engine, although it can be time consuming, especially with path explosion. That's why it's only optional by default.</p> <p>cwe_checker uses a variety of techniques; for integer overflow, it currently checks for instructions run before a malloc-family function, but other functions can be added. For NULL dereference, it has a list of function that can return NULL, and verifies if NULL is checked for in the return value with varied flow analysis in the IR.</p> <p>FACT, presented in the previous talk, integrates cwe_checker. The project also provides integration with IDA Pro with python scripts.</p> <p>In conclusion, Thomas says there are few false positives or negatives, so they want to improve the checks, but also add more checks and different type of analysis.</p> <h1>Unlocking secrets of the proxmark3 RDV4</h1> <p>by Christian Herrmann</p> <p>Proxmark3 is the third iteration of a board designed for RFID research. It works at 125kHz, 13.56MHz, as well as contact tags. It's a versatile tool.</p> <p>Recent high profile uses include an attack on the Tesla model S key fob that could be cloned with a Proxmark3; another one was the VingCard vulnerability, that affected 140 millions door locks.</p> <p>The previous revision had a few issues, so a kickstarter was made to launch a newer, smaller, more powerful version. The new design allows for antenna customisation which is critical depending what tag you want to address, has onboard memory, etc.</p> <p>The new version now supports chip-and-pin cards (ISO-7816), with an extension for snooping what happens in an exchange and run commands. It also has a bluetooth and battery add-on, allowing to use it wirelessly with a smartphone (if it had an Android/iOS app).</p> <p>There's an ARM processor on board, and an FPGA on-board for RF processing. The command line tool has many different commands, as well as high-level lua scripts. It can work in standalone mode, or with client that does the computation or crypto attacks.</p> <h1>Workshop: Let’s play ColTris - understand and exploit hash collisions</h1> <p>by Ange Albertini</p> <p><a href="https://speakerdeck.com/ange/colltris">slides</a> This workshop is an intro to hash collisions, and how to exploit each one with relation to file formats.</p> <p>At first we used <a href="https://github.com/cr-marcstevens/hashclash">hashclash</a> to compute md5 collisions with an empty prefix with the <code>md5_fastcoll</code> tool. Then we used a prefix we generated. These are identical prefix collisions.</p> <p>Fastcoll is easy to do, but hard to use, because you need a file format where you can skip the identical random data in the collided blocks, but also use the modified bytes to have an effect on the files being different. For example, if the different bytes control the length of a comment block, then you can have variable comment blocks length, different on each of the two files, meaning they will be interpreted differently.</p> <p>When you want to craft a reusable collision, you need to plan a header that will contain two files, depending on where the comment blocks are, and this way, with your two headers, you can generate two colliding files at will.</p> <p>Next, we started using unicoll, with the <code>poc_no.sh</code> script in the hashclash tool set. This attack works by modifying only a single byte in the collision block; there is no padding, and only two bytes are modified, by being incremented. It takes longer to compute (a few minutes instead of seconds), but since the modified byte is only incremented, it gives more control when creating collisions with some file formats.</p> <p>Using unicoll, it's possible to exploit the PNG file structure, by varying the byte size of a block in big endian: this gives you a jump of +0x100, more that enough to have varying data in the skipped chunk.</p> <p>Next, Ange showed how to use fastcool with the GIF format in particular, which has a structure that allows using it. GIF is quite old, and its sublock and comment structure allows using fastcoll, which is very useful for computing fast collisions.</p> <p>In the end, Ange explained the constraints of the chosen prefix attack, and how to implement it using the hashclash <code>cpc.sh</code> script. The length of the prefixes don't matter, but having a file format that tolerates appended data is important. It's this attack that Ange used to generate a reusable PE format collision. If block alignment is properly handled, it's possible to chain collisions, which can be used for mixing file types, or modifying content in a per-block fashion with unicoll.</p> <p><em>That's it for part 1. <a href="pass-the-salt-2019-2.html">Part 2 is continued here</a>, and <a href="pass-the-salt-2019-3.html">part 3 here</a>.</em></p>How I traded my first software project 15 years ago2019-06-26T00:00:00+02:002019-06-26T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2019-06-26:/gmail-binary-clock-rust.html<h1>2004</h1> <p>Gmail recently celebrated its 15 years anniversary, which led me down the path of how I got my first gmail invite. Back then, the private Gmail beta was all the rage. In the first few months, gmail invites <a href="https://en.wikipedia.org/wiki/History_of_Gmail#Extended_beta_phase">were scarce</a>. I got one by doing the most popular thing …</p><h1>2004</h1> <p>Gmail recently celebrated its 15 years anniversary, which led me down the path of how I got my first gmail invite. Back then, the private Gmail beta was all the rage. In the first few months, gmail invites <a href="https://en.wikipedia.org/wiki/History_of_Gmail#Extended_beta_phase">were scarce</a>. I got one by doing the most popular thing at the time: sending a postcard to a stranger. He was nice enough to send it right away, but the postcard took so long to arrive (2 weeks+) that he initially thought I had scammed him :-) That's how I was initially part of that name rush.</p> <p>To get my second invite, I decided walking to a post office and spending ~2€ was a bit too much effort, so instead of a postcard, I went back to the gmail-invite exchanges and proposed something only I had: a small software project I had been writing on my free time: a <a href="https://en.wikipedia.org/wiki/Binary_clock">binary clock</a>.</p> <p>I don't remember how I learned about binary clocks, but at the time, various existed in watch, alarm clock, or software form. As a student, I decided to put to use my freshly-learned C and SDL skills to rewrite one for my desktop.</p> <p>I had been using it for a few weeks when I decided to trade this software in a gmail invite marketplace. I sent the windows build to a random stranger for a gmail invite. Fun fact: I remember it not working on the first try because it depended on the standard C runtime DLL (msvcrt.dll) which I didn't ship with it. I didn't know what a runtime or C library was back then, I just knew I had to ship SDL.dll and that it would work. I got my gmail invite and completely forgot about this binary clock after a few months.</p> <h1>Present day</h1> <p>I rediscovered this bit of code in my backups. It survived despite my backup strategy not being as safe as I'd like: there was only one copy for a long time. It's now multi-site and multi-copy, but I wouldn't mind one more of each.</p> <p>Needless to say, the code was ugly, despite being a very simple project (less than 200 lines of C). But it still built and worked (mostly) as expected. SDL is truly a work of art, and with <a href="https://hg.libsdl.org/sdl12-compat">sdl12-compat</a>, the projects based on it should continue to live on for a long time.</p> <p>I fixed all the warnings given by modern gcc, passed it through automatic indenting, rewrote the makefiles, changed the newlines to unix ones, and am <a href="https://github.com/anisse/binclock">releasing it today</a>. The stranger who received the windows build had a long enough exclusivity period :-).</p> <h2>Rewriting</h2> <p>As I am learning the Rust programming language with the <a href="https://doc.rust-lang.org/book/">Rust book</a>, I thought this small project would be good idea for a rewrite, since it's simple enough. You can have a look at the code <a href="https://github.com/anisse/binclock-rs">here as well</a>.</p> <p>There is an <a href="https://github.com/Rust-SDL2/rust-sdl2">sdl2</a> crate, which allows calling the SDL2 library in safe Rust. The crate doc is good enough, and the crate itself has enough examples to start just by modifying some code. It took me some time to work out the image loading. The examples load pngs, which requires SDL_Image in C, or the "image" feature of the crate in Rust, which means I had to add that. This is clearly documented in the README, which I unfortunately managed to skip. But since I was loading bmps in the original code, it took me a while to realize that the load_bmp method from sdl2::surface::Surface would do the job, from which I could create a Texture for use in a Canvas.</p> <p>While I expected this after reading so many examples from the Rust book, it's still surprising how much Rust ownership constraints forces you to structure your code in a different way in order to match the safety constraints. Luckily, I've found the rustc errors to be (mostly) explanatory, although the suggested solution wasn't always what I needed. Maybe this little program lacks depth, but it seems to me that the Rust team has taken to heart the initial criticisms that rustc errors were hard to grok.</p> <p>Since this is a clock, I was surprised to find date/time package in the rust standard library. But the <a href="https://rust-lang-nursery.github.io/rust-cookbook/datetime/parse.html#examine-the-date-and-time">rust cookbook</a> recommends using <a href="https://github.com/chronotope/chrono">chrono</a>, which looks like the de-facto crate for this job. It looks quite good and is well documented, but it's hard to discover such a crucial missing part when you're doing offline work and can't go search for crates.</p> <p>The rewrite differs from the original a bit since it uses SDL2 instead of SDL 1.2. It also uses an event pump instead to build an event loop, instead of a manually built one. Unlike games, this program does not need to update at 60 frames per second, but at best only once per second. The original program had two updates per second as an heuristic not to miss updates. This time I wanted to be a bit more clever with the event loop, and only update on window events (window just reappeared) or every second. Unfortunately, there's no timer event in SDL, so my solution is a bit hackish: the program just checks if the second changed to do the refresh.</p> <p>In the end, the new program seems to consume more CPU resources than the original, but I haven't looked in depth at the cause (rust-c ffi? SDL2 event loop?); this will be an area of improvement for future work.</p>Bash : so long, and thanks for all the fish !2019-03-31T00:00:00+01:002019-03-31T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2019-03-31:/bash-to-fish.html<p>I recently took the plunge and decided to move to <a href="https://fishshell.com/">fish</a>, a friendly modern shell.</p> <h1>Background</h1> <p>I'm heavily invested in bash. I wrote countless scripts, for my personal use, and for various jobs or projects, so I didn't think it would be an easy thing to move away from it …</p><p>I recently took the plunge and decided to move to <a href="https://fishshell.com/">fish</a>, a friendly modern shell.</p> <h1>Background</h1> <p>I'm heavily invested in bash. I wrote countless scripts, for my personal use, and for various jobs or projects, so I didn't think it would be an easy thing to move away from it. But I tried anyway, because if you always stay in your comfort zone, you never learn anything, and stop growing. And I didn't regret it. I've moved to fish (3.0.2 at the time of this writing) on all my machines, including Termux on my mobile phone.</p> <p>The trigger was probably, in retrospect, the fact that the bash development process makes AOSP feel like a model of open source development, see for example <a href="https://git.savannah.gnu.org/cgit/bash.git/commit/?id=d233b485e83c3a784b803fb894280773f16f2deb">this git commit</a>.</p> <p>Fish is in written in C++ (but it looks mostly like modern C), has a github, CI, pull requests, and <a href="https://github.com/fish-shell/fish-shell/graphs/contributors">more than one contributor</a>. It's 14 years old (vs 30 for bash or 29 for zsh), so it's pretty young compared to other shells. But the project is still very mature.</p> <h1>Differences from bash</h1> <p>One important aspect, is that fish is designed around a simpler language, and is not POSIX-compatible. It makes the command lines and the scripts <strong>much</strong> easier to read. This also means that you shouldn't uninstall bash just yet, it might be useful to run those legacy scripts :-)</p> <p>Although you can write scripts pretty easily, fish is mostly designed around the command line use. It works very well by default, and requires very little configuration. For example, while there's no full-featured linter like shellcheck (which you should really use with bash), fish has live-command line syntax check: if it knows it won't be able to run your command because of a syntax error or non-existing command, it will highlight it in red, allowing you to fix it before even running.</p> <p>A core fish feature is the auto-suggestions: they mostly replace and remove the need to use reverse history search (Ctrl+R); they are enabled by default and take some time to get used to, but make you very productive in the end. You can still search in history by typing part of a previous command, and then pressing UP. This is useful since auto-suggestions work only search history (and completions, file paths) the beginning of a command.</p> <p>While there's an <a href="https://github.com/junegunn/fzf">fzf</a> <a href="https://github.com/jethrokuan/fzf">integration</a>, I didn't really feel the need to use it.</p> <p>Since the language is different, you cannot simply add environment to a command like this: <code>VAR=x cmd</code>, you need to use <code>env</code> to run the command: <code>env VAR=x cmd</code>. It's a bit longer to type. You also need to be careful if you have such command in your config files, for example I had to change this vim fugitive configuration:</p> <p><code>let g:fugitive_git_executable="LANG=C LC_ALL=C git"</code></p> <p>into this:</p> <p><code>let g:fugitive_git_executable="env LANG=C LC_ALL=C git"</code> (that's because fugitive parses git output in english). </p> <p>This construct is portable to other shells as well, so that's fine.</p> <p>Another big difference is command substitution: <code>$(cmd)</code> becomes <code>(cmd)</code> (and <code>`cmd`</code> isn't supported at all, but you shouldn't be using it anyway).</p> <h1>Pitfalls and limitations</h1> <p>When attempting to port some bash functions over to fish, I noticed other missing features:</p> <ul> <li>there is no short <code>&amp;&gt;</code> or <code>|&amp;</code> combined stderr/stdout redirection (<a href="https://github.com/fish-shell/fish-shell/issues/110">issue</a>)</li> <li>there is no parameter expansion of variables (<a href="https://github.com/fish-shell/fish-shell/issues/156">out of scope</a>)</li> <li>Process substitution only works for input, not output, with <code>psub</code>. For example <code>diff &lt;(sort file1) &lt;sort file2)</code> in bash, becomes <code>diff (sort file1 | psub) (sort file2 | psub)</code> in fish. (<a href="https://github.com/fish-shell/fish-shell/issues/1786">issue for output</a>)</li> <li>there is no <code>fc</code> to edit the last command in your $EDITOR, but you can do that with UP, then Alt+E to edit the current line.</li> <li>there is no history substitution (<code>!!</code> for example), <a href="http://fishshell.com/docs/current/faq.html#faq-history">use arrow keys</a>, like you'd do in bash.</li> </ul> <h1>Good surprises</h1> <p>Pasting in fish works as it always should have: it does not execute commands, and wraps multiple lines properly. This invalidates <a href="http://thejh.net/misc/website-terminal-copy-paste">pastejacking</a> <a href="https://nakedsecurity.sophos.com/2016/05/26/why-you-cant-trust-things-you-cut-and-paste-from-web-pages/">attacks</a> for example.</p> <p>History is managed transparently by fish: the <code>~/.local/share/fish/fish_history</code> text file is using an internal fish format. It means searching through it is more efficient when history goes bigger. And fish supports merging the history from other fish sessions with a single <code>history --merge</code> command ! This means I'll never have to run <code>exec bash</code> in all my open sessions to do a history sync again !</p> <p>Fish also imports bash history automatically on first run, but it might take a while if you have a big history (a few minutes for 50k lines on one of the machines).</p> <p>Python3 venvs work as they should if you include activate.fish.</p> <h1>Getting started</h1> <p>Just <a href="https://rootnroll.com/d/fish-shell/">try fish in browser</a>; it follows the <a href="https://fishshell.com/docs/current/tutorial.html">tutorial</a>, and explains all the basic features !</p>Running latest kernels on ARM Scaleway servers2018-12-21T00:00:00+01:002018-12-21T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2018-12-21:/distro-kernel-scaleway-arm.html<p>Scaleway, a french cloud provider, has been renting baremetal ARM servers for a few years now, and virtual ARM64 servers more recently. They ship with a scaleway-provided kernel and initird, which isn't updated as often as I'd like. The latest ARMv7 (32 bits) kernel, is 4.9.93, while the …</p><p>Scaleway, a french cloud provider, has been renting baremetal ARM servers for a few years now, and virtual ARM64 servers more recently. They ship with a scaleway-provided kernel and initird, which isn't updated as often as I'd like. The latest ARMv7 (32 bits) kernel, is 4.9.93, while the latest 4.9 LTS at the time of this writing is 4.9.146. 53 versions behind is a lot, so I've been looking at how to work around this.</p> <h1>A bad surprise</h1> <p>At the latest Golang Paris meetup, I did a livecoding introduction to <a href="https://godoc.org/golang.org/x/crypto/acme/autocert">autocert</a>. Unfortunately,the <a href="https://twitter.com/Aissn/status/1072839595115585537">demo at the end failed</a>, despite the code being correct (it was still my fault, though.). After digging through, it all pointed out to something wrong on the server.</p> <p>This server, was a C1 ARM server from Scaleway, was one of the first ever (baremetal) ARM servers available at cloud provider. Based on custom hardware with a Marvell ARMv7 SoC, it was also very cheap at launch, and still one of the cheapest baremetal server to rent out there. Since then, Packet has launched ARM64 servers based on the Cavium ThunderX (much more expensive, with 96 cores and 2 SoCs on board), and Scaleway followed suite with virtual servers based on the same platform (with 4 to 64 cores), and much more affordable.</p> <p>The C1 server was updated regularly, in addition to <a href="https://wiki.debian.org/UnattendedUpgrades">unattended-upgrades</a> being <a href="https://github.com/jnv/ansible-role-unattended-upgrades">enabled</a>. But what seemed odd was the old kernel version (4.5.7). Since I had provisioned it (more or less), it had been running the same kernel version, despite having been rebooted a few times. Which isn't really a good idea, at least for security reasons.</p> <p>And it turned out, for at least one other reason as well: golang binaries starting with Go 1.9, failed to initialize the crypto-rng using the getrandom syscall, blocking forever. Updating to a more recent kernel (4.9.93) fixed the issue. But the update process required using the Scaleway web interface or <a href="https://developer.scaleway.com/#servers-server-put">the API</a>, the <a href="https://github.com/scaleway/scaleway-cli">cli tool</a> does not (AFAIK) support this operation. <em>Sidenote</em>: I know that in a cloud world I should just spin up a new server and redeploy to it. I'll get there once I'm comfortable enough that it can work with my apps :-)</p> <p>While this fixed this particular issue, it got me thinking about the general process for managing these servers. Should I setup a script or an ansible role to update the bootscript regularly ? Isn't there a better way, in order to use the distro kernels ? That led me to contemplate using kexec.</p> <h1>ARMv7 kexec attempts</h1> <p>Fortunately, I was not the first to have this idea, since Scaleway's initramfs scripts directly <a href="https://github.com/scaleway/initrd/blob/master/Linux/README.md#boot-options">support using kexec to a new kernel</a> ! You can find a <a href="https://blog.simos.info/how-to-run-the-stock-ubuntu-linux-kernel-on-scaleway-using-kexec-and-server-tags/">tutorial here</a>, but unfortunately, it only covers x86 servers.</p> <p>I quickly learned that the serial console on the web interface is inferior to the one provided by the cli tool: <code>./scw attach &lt;server-name&gt;</code>. All the boot logs from this post are captured with it.</p> <p>My first attempts were therefore to use the <code>KEXEC_KERNEL=/vmlinuz</code> and <code>KEXEC_INIRTD=/initrd.img</code> server tags, but it failed to work. Here is the boot log output with <code>INITRD_VERBOSE=1</code></p> <div class="highlight"><pre><span></span>** Message: /dev/nbd6 is not used[ 30.528536] kexec_core: Starting new kernel ** Message: cm[ 30.583224] Disabling non-boot CPUs ... d check mode ** Message: /dev/n[ 30.672735] CPU1: shutdown bd7 is not used ** Message: cmd check mode ** Message: /dev/nbd8 is not used [ 30.791469] CPU2: shutdown ** Message: cmd check mode ** Message: /dev/nbd9 is not used *[ 30.891720] CPU3: shutdown * Message: cmd check mode ** Message: /dev/nbd1[ 30.960773] Bye! 0 is not used </pre></div> <p>The output is a bit mangled, and I lack visibility into how it's being done. So I wanted to add more kernel debug options: I tried the using <code>KEXEC_APPEND="debug initcall_debug"</code>. But then I discovered that the server tags did not support having spaces inside, since the tokenisation was space-based.</p> <p>I then decided to use <code>INITRD_DROPBEAR=1</code> to start a shell in the initrd, giving me control into how the kexec is run. Initially, I was wondering if the fact that I didn't boot with a device tree was causing an issue. So I dumped the device-tree from the running process and re-built it with <code>dtc</code>. I made sure to re-use the command line from the current boot, and to properly detach the nbd block device. I attempted to use a more recent kexec userspace tool, and add a debugging option. After many attempts, I had a script to run inside the initramfs that looked like this:</p> <div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span> <span class="nb">export</span> <span class="nv">PATH</span><span class="o">=</span>/sbin/:/usr/sbin:<span class="nv">$PATH</span> cp /newroot/initrd.img / cp /newroot/vmlinuz / cp /newroot/sbin/kexec / /newroot/usr/bin/dtc -I fs -O dtb -o /generated-dtb /proc/device-tree/ umount /newroot xnbd-client -c /dev/nbd0 xnbd-client -d /dev/nbd0 /kexec -d -l --append<span class="o">=</span><span class="s2">&quot;verbose debug </span><span class="k">$(</span>cat /proc/cmdline<span class="k">)</span><span class="s2"> is_in_kexec=yes root=/dev/nbd0 nbdroot=10.1.52.66,4448,nbd0&quot;</span> --dtb<span class="o">=</span>/generated-dtb --ramdisk<span class="o">=</span>/initrd.img --type<span class="o">=</span>zImage /vmlinuz /kexec -d -e </pre></div> <p>Since the dropbear in initramfs lacks the scp server part, and kept generating new host keys on boot, I pushed it like this:</p> <div class="highlight"><pre><span></span>cat kexec-initramfs-script.sh <span class="p">|</span> ssh -o <span class="nv">UserKnownHostsFile</span><span class="o">=</span>/dev/null -o <span class="nv">StrictHostKeyChecking</span><span class="o">=</span>no root@myc1server <span class="s2">&quot;tee -a kexec.sh &amp;&amp; chmod +x kexec.sh&quot;</span> </pre></div> <p>Then, it ran without any obvious error:</p> <div class="highlight"><pre><span></span><span class="n">kernel</span><span class="o">:</span> <span class="mh">0xb66ef008</span> <span class="n">kernel_size</span><span class="o">:</span> <span class="mh">0x7d8200</span> <span class="n">MEMORY</span> <span class="n">RANGES</span> <span class="mi">0000000000000000</span><span class="o">-</span><span class="mi">000000007</span><span class="n">fffefff</span> <span class="o">(</span><span class="mi">0</span><span class="o">)</span> <span class="n">zImage</span> <span class="n">header</span><span class="o">:</span> <span class="mh">0x016f2818</span> <span class="mh">0x00000000</span> <span class="mh">0x007d8200</span> <span class="n">zImage</span> <span class="n">size</span> <span class="mh">0x7d8200</span><span class="o">,</span> <span class="n">file</span> <span class="n">size</span> <span class="mh">0x7d8200</span> <span class="n">zImage</span> <span class="n">has</span> <span class="n">tags</span> <span class="n">offset</span> <span class="mh">0x0000ae48</span> <span class="n">tag</span> <span class="mh">0x5a534c4b</span> <span class="n">size</span> <span class="mi">8</span> <span class="n">kernel</span> <span class="n">image</span> <span class="n">size</span><span class="o">:</span> <span class="mh">0x015c5d14</span> <span class="n">kexec_load</span><span class="o">:</span> <span class="n">entry</span> <span class="o">=</span> <span class="mh">0x8000</span> <span class="n">flags</span> <span class="o">=</span> <span class="mh">0x280000</span> <span class="n">nr_segments</span> <span class="o">=</span> <span class="mi">3</span> <span class="n">segment</span><span class="o">[</span><span class="mi">0</span><span class="o">].</span><span class="n">buf</span> <span class="o">=</span> <span class="mh">0xb66ef008</span> <span class="n">segment</span><span class="o">[</span><span class="mi">0</span><span class="o">].</span><span class="n">bufsz</span> <span class="o">=</span> <span class="mh">0x7d8200</span> <span class="n">segment</span><span class="o">[</span><span class="mi">0</span><span class="o">].</span><span class="n">mem</span> <span class="o">=</span> <span class="mh">0x8000</span> <span class="n">segment</span><span class="o">[</span><span class="mi">0</span><span class="o">].</span><span class="n">memsz</span> <span class="o">=</span> <span class="mh">0x7d9000</span> <span class="n">segment</span><span class="o">[</span><span class="mi">1</span><span class="o">].</span><span class="n">buf</span> <span class="o">=</span> <span class="mh">0xb3603008</span> <span class="n">segment</span><span class="o">[</span><span class="mi">1</span><span class="o">].</span><span class="n">bufsz</span> <span class="o">=</span> <span class="mh">0x30eba91</span> <span class="n">segment</span><span class="o">[</span><span class="mi">1</span><span class="o">].</span><span class="n">mem</span> <span class="o">=</span> <span class="mh">0x15ce000</span> <span class="n">segment</span><span class="o">[</span><span class="mi">1</span><span class="o">].</span><span class="n">memsz</span> <span class="o">=</span> <span class="mh">0x30ec000</span> <span class="n">segment</span><span class="o">[</span><span class="mi">2</span><span class="o">].</span><span class="n">buf</span> <span class="o">=</span> <span class="mh">0x4f45a8</span> <span class="n">segment</span><span class="o">[</span><span class="mi">2</span><span class="o">].</span><span class="n">bufsz</span> <span class="o">=</span> <span class="mh">0x45bc</span> <span class="n">segment</span><span class="o">[</span><span class="mi">2</span><span class="o">].</span><span class="n">mem</span> <span class="o">=</span> <span class="mh">0x46ba000</span> <span class="n">segment</span><span class="o">[</span><span class="mi">2</span><span class="o">].</span><span class="n">memsz</span> <span class="o">=</span> <span class="mh">0x5000</span> </pre></div> <p>But the serial console output was always the same:</p> <div class="highlight"><pre><span></span>[ 129.248360] kexec_core: Starting new kernel [ 129.298586] Disabling non-boot CPUs ... [ 129.393572] CPU1: shutdown [ 129.532515] CPU2: shutdown [ 129.632399] CPU3: shutdown [ 129.700758] Bye! </pre></div> <p>And no new kernel seemed to boot… That's when I gave up, and decided to try something new. While writing this post, I also <a href="https://github.com/scaleway/initrd/issues/199">opened an issue</a> to inform Scaleway of this status.</p> <h1>ARMv8 servers</h1> <p>I decided to check the ARMv8 virtual servers I had heard about. I already have <a href="r2con-2018.html">arm64 experience</a>, and I noticed that the pricing was similar (3€ per months for 4 cores + 2GB). So I instantiated one and tried to see if kexec could work on it. I first used the <code>KEXEC_KERNEL</code> and <code>KEXEC_INITRD</code> parameters, but it failed since there is <a href="https://github.com/scaleway/initrd/issues/200">no kexec in the arm64 initramfs</a>:</p> <div class="highlight"><pre><span></span>&gt;&gt;&gt; kexec: kernel=/vmlinuz initrd=/initrd.img append=&#39;&#39; /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: xnbd-client: not found /init: line 337: kexec: not found </pre></div> <p>It wasn't really an issue, since I had already resolved to use the rootfs' kexec tool previously (to have a more recent version), so I just enabled the <code>INITRD_DROPBEAR</code> ssh server, and ran a script on it. And it worked. Well, mostly: the kernel booted, but it couldn't mount the rootfs because it was looking for in <code>/dev/vda</code> ; which is the full block device, not the root partition: <code>/dev/vda1</code>. This is due to a <a href="https://github.com/scaleway/initrd/issues/201">bad parameter on the kernel command line</a>; it doesn't affect <a href="https://github.com/scaleway/initrd/blob/master/Linux/tree-common/boot-local#L13-L41">Scaleway's initramfs because they do clever things</a>.</p> <p>After passing <code>root=/dev/vda1</code>, I finally had a working distro, with an up-to-date kernel. </p> <h2>Tutorial</h2> <p>After instaling <code>kexec-tools</code>, I added the following <code>/boot.sh</code> script:</p> <div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span> <span class="k">if</span> grep -q <span class="nv">is_in_kexec</span><span class="o">=</span>yes /proc/cmdline<span class="p">;</span> <span class="k">then</span> <span class="nb">exit</span> <span class="m">0</span> <span class="k">fi</span> kexec -f --ramdisk<span class="o">=</span>/initrd.img --append<span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span>cat /proc/cmdline<span class="k">)</span><span class="s2"> is_in_kexec=yes root=/dev/vda1&quot;</span> /vmlinuz </pre></div> <p>I don't use <code>systemctl kexec</code>, because it goes back to the initramfs, and kexec does not exist there…</p> <p>And this systemd unit (to be improved, it starts very late, doesn't umount or stop services) <code>kexec.service</code>:</p> <div class="highlight"><pre><span></span><span class="k">[Unit]</span> <span class="na">Description</span><span class="o">=</span><span class="s">Boot to kexec kernel if needed</span> <span class="k">[Service]</span> <span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span> <span class="na">ExecStart</span><span class="o">=</span><span class="s">/boot.sh</span> <span class="k">[Install]</span> <span class="na">WantedBy</span><span class="o">=</span><span class="s">network.target basic.target</span> </pre></div> <p>And then enabled it with <code>systemctl enable kexec.service</code>. That's all that's needed to always boot to the distribution's shipped kernel!</p> <h2>Bug notes</h2> <p>During my tests, I encountered many times <a href="https://github.com/scaleway/initrd/issues/202">IRQ exceptions on reboot; the VM is then broken, and needs api reboot</a>; during the last tests to write this blog post, a reboot caused a permanent crash: even after using the API restart, the server was blocked in a transient state("rebooting server"), forbidding any other action. I hope a simple reboot in a VM can't crash the orchestrator or worse (hypervisor), affecting other clients. <strong>Update:</strong> after I contacted Scaleway support, they gave me be back access to the server: it was still rebooting endlessly and I was able to restart it with the API; I'm guessing the hypervisor didn't crash, and probably no other customers were affected.</p> <p>Also during my explorations, I accidentally accessed the boot menu on the server (using a keyboard shortcut on the serial console). I don't think that's an issue since this is due to the fact that the full EFI stack is emulated as well. It might be possible to configure the bootloader to boot directly the kernel I want, but I haven't explored this possibility. It might require the EFI bootloader to understand virtio block devices, <a href="https://github.com/tianocore/edk2/blob/master/OvmfPkg/VirtioBlkDxe/VirtioBlk.c">which might be possible</a>.</p> <h2>Conclusion</h2> <p>The boot time is quite slow with this solution, since I have to boot the system twice (56 seconds before kexec, about 31 seconds after). Once the root= and kexec in initramfs bugs are fixed, I can use the server tags and have a faster boot; otherwise I might publish an ansible role to automate this process.</p> <p>I also decided to migrate my services on the ARMv8 server, since it performs much better : +50% to +1300% on sysbench; only the threads and hackbench message passing tests were slower, I'm guessing due to virtualization. It also has IPv6 available, if enabled.</p> <p>Be careful though: these servers are often out of stock, and I didn't notice, but I was lucky it was in stock when I provisioned it, since it isn't anymore in the Paris (par1) region, but is available in the Amsterdam ams1 datacenter (with low stock though). There might be a trick to bypass the "out of stock" status, but I doubt this works reliably.</p>SIGSEGV1 qualification CTF2018-10-12T00:00:00+02:002018-10-12T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2018-10-12:/qual-sigsegv1-rtfm.html<p>After my <a href="r2con-2018.html">r2con r2wars writeup</a>, here's another writeup of a "challenge". This challenge is the Capture-The-Flag (CTF) pre-qualifications for the <a href="https://sigsegv1.rtfm.re/">SIGSEGV1 conference in Paris</a>. It felt a bit weird to have a conference registration limited to those who pass a certain challenge, but I was curious about what it would …</p><p>After my <a href="r2con-2018.html">r2con r2wars writeup</a>, here's another writeup of a "challenge". This challenge is the Capture-The-Flag (CTF) pre-qualifications for the <a href="https://sigsegv1.rtfm.re/">SIGSEGV1 conference in Paris</a>. It felt a bit weird to have a conference registration limited to those who pass a certain challenge, but I was curious about what it would be like, so I thought: why not ?</p> <h1>Fun avec python</h1> <p>The first challenge was around python, one had to connect to a server, and try to capture the "flag", a file which you don't have access to, by exploiting a vulnerability in code on the server. This is what it looks like:</p> <div class="highlight"><pre><span></span>chall@ae805fd9fe99:~$ ls -l total <span class="m">16</span> -r--r----- <span class="m">1</span> root chall-pwned <span class="m">42</span> Oct <span class="m">5</span> <span class="m">17</span>:00 flag -rwxr-xr-x <span class="m">1</span> root root <span class="m">307</span> Sep <span class="m">20</span> <span class="m">00</span>:21 hello-world.py -rwxr-sr-x <span class="m">1</span> root chall-pwned <span class="m">6304</span> Oct <span class="m">5</span> <span class="m">17</span>:05 wrapper </pre></div> <p>The <code>flag</code> file is the goal, but we can't read it with our permission level (<code>chall</code> user. ). The <code>wrapper</code> is a suid binary that just calls the hello-world.py script. This is the content of the script:</p> <div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/python2.7</span> <span class="kn">from</span> <span class="nn">colors</span> <span class="kn">import</span> <span class="n">colors</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span> <span class="k">print</span><span class="p">(</span><span class="s1">&#39;This is an advanced hello-world&#39;</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">&#39;The world is more joyful with colors&#39;</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">&#39;So, here we are:&#39;</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s1">&#39;{}Hello-World !{}&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">colors</span><span class="o">.</span><span class="n">bcolors</span><span class="o">.</span><span class="n">OKBLUE</span><span class="p">,</span> <span class="n">colors</span><span class="o">.</span><span class="n">bcolors</span><span class="o">.</span><span class="n">ENDC</span><span class="p">))</span> <span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span> <span class="n">main</span><span class="p">()</span> </pre></div> <p>One of the issue, is that when calling suid-binaries, you control the environment, and if it isn't cleared, you can control how the executables behave. Here, we are attacking the python script (it's the name of the challenge). The goal will be to use the <code>import</code> clause of the script to run our one code.</p> <p>I reproduced the environment locally to do some tests, here is my <code>test.py</code> script:</p> <div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python2</span> <span class="kn">import</span> <span class="nn">colors</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;coucou&quot;</span><span class="p">)</span> </pre></div> <p>And here is the <code>colors.py</code> I used:</p> <div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span> <span class="k">print</span> <span class="nb">open</span><span class="p">(</span><span class="s2">&quot;flag&quot;</span><span class="p">,</span> <span class="s2">&quot;r&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span> </pre></div> <p>Running <code>test.py</code> on my machine shows that it's executing code in <code>colors.py</code> before doing anything, so it works. Now, I need to upload <code>colors.py</code> on the server, somewhere I can write files. Home isn't writable, so I just used <code>/tmp</code>. I put <code>colors.py</code> in <code>/tmp/colors/</code>. Then, I used the <code>PYTHONPATH</code> variable to run the script with the search path for modules modified:</p> <div class="highlight"><pre><span></span>chall@ae805fd9fe99:~$ <span class="nv">PYTHONPATH</span><span class="o">=</span>/tmp ./wrapper sigsegv<span class="o">{</span>518012356c8a2ed93b8d3e2416bb2274<span class="o">}</span> Traceback <span class="o">(</span>most recent call last<span class="o">)</span>: File <span class="s2">&quot;/home/chall/hello-world.py&quot;</span>, line <span class="m">3</span>, in &lt;module&gt; from colors import colors ImportError: cannot import name colors </pre></div> <p>The rest of the script fails, but, we can clearly see the flag: <code>sigsegv{518012356c8a2ed93b8d3e2416bb2274}</code></p> <h1>antistrings</h1> <p>This is a simple reverse engineering challenge, but with a few traps. I fell into all of them. The <a href="/static/linux_x64_chall_v1.bin">binary is backed up here</a> if you want to see for yourself.</p> <p>Despite the title being, "antistrings", I still went ahead and looked at the strings:</p> <div class="highlight"><pre><span></span>$ r2 linux_x64_chall_v1.bin -- This is just an existentialist experiment. <span class="o">[</span>0x00400650<span class="o">]</span>&gt; iz <span class="o">[</span>Strings<span class="o">]</span> Num Vaddr Paddr Len Size Section Type String <span class="m">000</span> 0x00000b48 0x00400b48 <span class="m">147</span> <span class="m">148</span> <span class="o">(</span>.rodata<span class="o">)</span> ascii Strings won<span class="s1">&#39;t help you that much.\n\n[+] Activating obfuscation layer 1...\n[+] Act]</span> <span class="s1">001 0x00000bdc 0x00400bdc 12 13 (.rodata) ascii KIS\bJED@\rL]_</span> <span class="s1">002 0x00000bed 0x00400bed 4 5 (.rodata) ascii \nBB^</span> <span class="s1">003 0x00000bf2 0x00400bf2 8 9 (.rodata) ascii _YMCJ@];</span> <span class="s1">004 0x00000c00 0x00400c00 15 16 (.rodata) ascii fIIO[K_YAO[Y^\@</span> <span class="s1">005 0x00000c15 0x00400c15 4 5 (.rodata) ascii RZJX</span> <span class="s1">006 0x00000c1a 0x00400c1a 13 14 (.rodata) ascii K($b%($!grdcA</span> <span class="s1">007 0x00000c28 0x00400c28 9 10 (.rodata) ascii lHDG[XNOY</span> <span class="s1">008 0x00000c32 0x00400c32 5 6 (.rodata) ascii F^AGG</span> <span class="s1">009 0x00000c3d 0x00400c3d 5 6 (.rodata) ascii [\]TP</span> <span class="s1">010 0x00000c48 0x00400c48 11 12 (.rodata) ascii ~\rz\b~OGOBCJ</span> <span class="s1">011 0x00000c5b 0x00400c5b 4 5 (.rodata) ascii jm|v</span> <span class="s1">012 0x00000c60 0x00400c60 10 11 (.rodata) ascii ^V^,-&#39;</span>-# L <span class="m">013</span> 0x00000c6b 0x00400c6b <span class="m">10</span> <span class="m">11</span> <span class="o">(</span>.rodata<span class="o">)</span> ascii ~<span class="se">\r</span>z<span class="se">\b</span>yFNM^K <span class="m">014</span> 0x00000c76 0x00400c76 <span class="m">5</span> <span class="m">6</span> <span class="o">(</span>.rodata<span class="o">)</span> ascii U_FVF <span class="m">015</span> 0x00000c80 0x00400c80 <span class="m">4</span> <span class="m">5</span> <span class="o">(</span>.rodata<span class="o">)</span> ascii <span class="se">\W</span><span class="o">]</span>Z </pre></div> <p>So, the strings are encrypted. No big deal, I'll find later how.</p> <div class="highlight"><pre><span></span>[0x00400650]&gt; aaa ...snip...8&lt;... [0x00400650]&gt; s main [0x00400aa2]&gt; pdf ┌ (fcn) main 19 │ main (int argc, char **argv, char **envp); │ ; DATA XREF from entry0 (0x40066d) │ 0x00400aa2 4883ec08 sub rsp, 8 │ 0x00400aa6 b800000000 mov eax, 0 │ 0x00400aab e830ffffff call fcn.004009e0 │ 0x00400ab0 4883c408 add rsp, 8 └ 0x00400ab4 c3 ret </pre></div> <p>A short <code>main()</code> (I skipped the libc entry point here), going to directly to another function.</p> <div class="highlight"><pre><span></span>[0x00400aa2]&gt; s fcn.004009e0 [0x004009e0]&gt; pdf 10 ┌ (fcn) fcn.004009e0 16 │ fcn.004009e0 (); │ ⁝ ; CALL XREF from main (0x400aab) │ ⁝ 0x004009e0 4883ec28 sub rsp, 0x28 ; &#39;(&#39; │ ⁝ 0x004009e4 50 push rax │ ⁝ 0x004009e5 31c0 xor eax, eax │ ⁝ 0x004009e7 85c0 test eax, eax │ ⁝ 0x004009e9 58 pop rax │ ┌──&lt; 0x004009ea 7502 jne 0x4009ee │ ┌───&lt; 0x004009ec 7401 je 0x4009ef │ │││ ; CODE XREF from fcn.004009e0 (0x4009ea) └ │└└─&lt; 0x004009ee ebb9 jmp 0x4009a9 ; sub.BB_7c2+0x1e7 </pre></div> <p>Here we can see the first trick: a jump to an unaligned address, after testing a very simple (always true) condition.</p> <div class="highlight"><pre><span></span>[0x004009e0]&gt; s 0x4009ef [0x004009ef]&gt; pd 10 │ ; CODE XREF from fcn.004009e0 (0x4009ec) │ 0x004009ef b900000000 mov ecx, 0 0x004009f4 ba01000000 mov edx, 1 0x004009f9 be00000000 mov esi, 0 0x004009fe bf00000000 mov edi, 0 0x00400a03 b800000000 mov eax, 0 0x00400a08 e823fcffff call sym.imp.ptrace 0x00400a0d 4885c0 test rax, rax ┌─&lt; 0x00400a10 791e jns 0x400a30 </pre></div> <p>And here we have the first anti-debug: <code>ptrace(PTRACE_TRACEME, 0, 0, 0)</code> is called. The <a href="http://man7.org/linux/man-pages/man2/ptrace.2.html">manpage</a> says this is to <em>Indicate that this process is to be traced by its parent.</em>. In other words, this is used to detect if the process is currently being <code>ptrace</code>d, which is the basic building block of all debuggers on Linux. On success, the program will print "not cool bro", and then exit.</p> <p>My first idea, was to modify the binary (it doesn't seem to have any integrity verification built-in). We want to simulate <code>ptrace()</code> returning an error; we'll just put -1 into <code>eax</code>, the return register in this x86 call convention. This is done with:</p> <div class="highlight"><pre><span></span>[0x004009ef]&gt; s 0x00400a08 [0x00400a08]&gt; &quot;wa sub eax, 1;nop;nop&quot; </pre></div> <p>Don't forget to open the binary with <code>r2</code> in write mode (<code>-w</code> command line switch). Afterwards, the code looks like this:</p> <div class="highlight"><pre><span></span> 0x004009ef b900000000 mov ecx, 0 0x004009f4 ba01000000 mov edx, 1 0x004009f9 be00000000 mov esi, 0 0x004009fe bf00000000 mov edi, 0 0x00400a03 b800000000 mov eax, 0 0x00400a08 83e801 sub eax, 1 0x00400a0b 90 nop 0x00400a0c 90 nop 0x00400a0d 4885c0 test rax, rax ┌─&lt; 0x00400a10 791e jns 0x400a30 </pre></div> <p>No more <code>ptrace()</code>! This might be useful if I want to run the binary in a VM with <code>gdb</code> or <code>strace</code>. Let's continue on the execution path.</p> <div class="highlight"><pre><span></span>[0x004009ef]&gt; s 0x400a30 [0x00400a30]&gt; pd 10 ;-- rip: ; CODE XREF from fcn.004009e0 (+0x30) 0x00400a30 bf480c4000 mov edi, str.z___OGOBCJ ; 0x400c48 ; &quot;~\rz\b~OGOBCJ\x10E]\x13@]S\x17jm|v\x1c^V^,-&#39;-# L&quot; 0x00400a35 e80cfdffff call sub.strlen_746 0x00400a3a 4889c7 mov rdi, rax 0x00400a3d b800000000 mov eax, 0 0x00400a42 e879fbffff call sym.imp.printf ; int printf(const char *format) </pre></div> <p>The first obfuscated string appears here. It is then passed to a function (that calls <code>strlen()</code>) that will transform it (decrypt?) before printing it. Let's see what this function looks like.</p> <div class="highlight"><pre><span></span><span class="cp">[</span><span class="mh">0x00400a30</span><span class="cp">]</span><span class="o">&gt;</span> <span class="nt">pdf</span> <span class="p">@</span><span class="k">sub</span><span class="p">.</span><span class="nc">strlen_746</span> <span class="err">┌</span> <span class="o">(</span><span class="nt">fcn</span><span class="o">)</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">strlen_746</span> <span class="nt">124</span> <span class="err">│</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">strlen_746</span> <span class="o">(</span><span class="nt">char</span> <span class="o">*</span><span class="nt">arg1</span><span class="o">)</span><span class="p">;</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">char</span> <span class="o">*</span><span class="nt">s</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x8</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">void</span> <span class="o">*</span><span class="nt">local_10h</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x10</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">size_t</span> <span class="nt">size</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x18</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">int</span> <span class="nt">local_1ch</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x1c</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">arg</span> <span class="nt">char</span> <span class="o">*</span><span class="nt">arg1</span> <span class="o">@</span> <span class="nt">rdi</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CALL</span> <span class="nt">XREFS</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="o">(</span><span class="nt">0x4009a6</span><span class="o">,</span> <span class="nt">0x4009c4</span><span class="o">)</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CALL</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">fcn</span><span class="p">.</span><span class="nc">004009e0</span> <span class="o">(+</span><span class="nt">0x3c</span><span class="o">)</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CALL</span> <span class="nt">XREFS</span> <span class="nt">from</span> <span class="nt">rip</span> <span class="o">(+</span><span class="nt">0x5</span><span class="o">,</span> <span class="o">+</span><span class="nt">0x1c</span><span class="o">)</span> <span class="err">│</span> <span class="nt">0x00400746</span> <span class="nt">4883ec28</span> <span class="nt">sub</span> <span class="nt">rsp</span><span class="o">,</span> <span class="nt">0x28</span> <span class="o">;</span> <span class="s1">&#39;(&#39;</span> <span class="err">│</span> <span class="nt">0x0040074a</span> <span class="nt">48897c2408</span> <span class="nt">mov</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">s</span><span class="cp">]</span><span class="o">,</span> <span class="nt">rdi</span> <span class="o">;</span> <span class="nt">arg1</span> <span class="err">│</span> <span class="nt">0x0040074f</span> <span class="nt">488b442408</span> <span class="nt">mov</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">s</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x8</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">8</span> <span class="err">│</span> <span class="nt">0x00400754</span> <span class="nt">4889c7</span> <span class="nt">mov</span> <span class="nt">rdi</span><span class="o">,</span> <span class="nt">rax</span> <span class="o">;</span> <span class="nt">const</span> <span class="nt">char</span> <span class="o">*</span><span class="nt">s</span> <span class="err">│</span> <span class="nt">0x00400757</span> <span class="nt">e854feffff</span> <span class="nt">call</span> <span class="nt">sym</span><span class="p">.</span><span class="nc">imp</span><span class="p">.</span><span class="nc">strlen</span> <span class="o">;</span> <span class="nt">size_t</span> <span class="nt">strlen</span><span class="o">(</span><span class="nt">const</span> <span class="nt">char</span> <span class="o">*</span><span class="nt">s</span><span class="o">)</span> <span class="err">│</span> <span class="nt">0x0040075c</span> <span class="nt">89442418</span> <span class="nt">mov</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">size</span><span class="cp">]</span><span class="o">,</span> <span class="nt">eax</span> <span class="err">│</span> <span class="nt">0x00400760</span> <span class="nt">8b442418</span> <span class="nt">mov</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">size</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x18</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">24</span> <span class="err">│</span> <span class="nt">0x00400764</span> <span class="nt">4898</span> <span class="nt">cdqe</span> <span class="err">│</span> <span class="nt">0x00400766</span> <span class="nt">4889c7</span> <span class="nt">mov</span> <span class="nt">rdi</span><span class="o">,</span> <span class="nt">rax</span> <span class="o">;</span> <span class="nt">size_t</span> <span class="nt">size</span> <span class="err">│</span> <span class="nt">0x00400769</span> <span class="nt">e8a2feffff</span> <span class="nt">call</span> <span class="nt">sym</span><span class="p">.</span><span class="nc">imp</span><span class="p">.</span><span class="nc">malloc</span> <span class="o">;</span> <span class="nt">void</span> <span class="o">*</span><span class="nt">malloc</span><span class="o">(</span><span class="nt">size_t</span> <span class="nt">size</span><span class="o">)</span> <span class="err">│</span> <span class="nt">0x0040076e</span> <span class="nt">4889442410</span> <span class="nt">mov</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">local_10h</span><span class="cp">]</span><span class="o">,</span> <span class="nt">rax</span> <span class="err">│</span> <span class="nt">0x00400773</span> <span class="nt">c744241c0000</span><span class="o">.</span> <span class="nt">mov</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_1ch</span><span class="cp">]</span><span class="o">,</span> <span class="nt">0</span> <span class="err">│</span> <span class="err">┌─</span><span class="o">&lt;</span> <span class="nt">0x0040077b</span> <span class="nt">eb31</span> <span class="nt">jmp</span> <span class="nt">0x4007ae</span> <span class="err">│</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CODE</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">strlen_746</span> <span class="o">(</span><span class="nt">0x4007b6</span><span class="o">)</span> <span class="err">│</span> <span class="err">┌──</span><span class="o">&gt;</span> <span class="nt">0x0040077d</span> <span class="nt">8b44241c</span> <span class="nt">mov</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_1ch</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x1c</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">28</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400781</span> <span class="nt">4863d0</span> <span class="nt">movsxd</span> <span class="nt">rdx</span><span class="o">,</span> <span class="nt">eax</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400784</span> <span class="nt">488b442410</span> <span class="nt">mov</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">local_10h</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x10</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">16</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400789</span> <span class="nt">4801d0</span> <span class="nt">add</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">rdx</span> <span class="o">;</span> <span class="s1">&#39;(&#39;</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x0040078c</span> <span class="nt">8b54241c</span> <span class="nt">mov</span> <span class="nt">edx</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_1ch</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x1c</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">28</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400790</span> <span class="nt">4863ca</span> <span class="nt">movsxd</span> <span class="nt">rcx</span><span class="o">,</span> <span class="nt">edx</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400793</span> <span class="nt">488b542408</span> <span class="nt">mov</span> <span class="nt">rdx</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">s</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x8</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">8</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400798</span> <span class="nt">4801ca</span> <span class="nt">add</span> <span class="nt">rdx</span><span class="o">,</span> <span class="nt">rcx</span> <span class="o">;</span> <span class="s1">&#39;&amp;&#39;</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x0040079b</span> <span class="nt">0fb612</span> <span class="nt">movzx</span> <span class="nt">edx</span><span class="o">,</span> <span class="nt">byte</span> <span class="cp">[</span><span class="nx">rdx</span><span class="cp">]</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x0040079e</span> <span class="nt">8b4c241c</span> <span class="nt">mov</span> <span class="nt">ecx</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_1ch</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x1c</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">28</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007a2</span> <span class="nt">83c125</span> <span class="nt">add</span> <span class="nt">ecx</span><span class="o">,</span> <span class="nt">0x25</span> <span class="o">;</span> <span class="s1">&#39;%&#39;</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007a5</span> <span class="nt">31ca</span> <span class="nt">xor</span> <span class="nt">edx</span><span class="o">,</span> <span class="nt">ecx</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007a7</span> <span class="nt">8810</span> <span class="nt">mov</span> <span class="nt">byte</span> <span class="cp">[</span><span class="nx">rax</span><span class="cp">]</span><span class="o">,</span> <span class="nt">dl</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007a9</span> <span class="nt">8344241c01</span> <span class="nt">add</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_1ch</span><span class="cp">]</span><span class="o">,</span> <span class="nt">1</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="o">;</span> <span class="nt">CODE</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">strlen_746</span> <span class="o">(</span><span class="nt">0x40077b</span><span class="o">)</span> <span class="err">│</span> <span class="err">⁝└─</span><span class="o">&gt;</span> <span class="nt">0x004007ae</span> <span class="nt">8b44241c</span> <span class="nt">mov</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_1ch</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x1c</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">28</span> <span class="err">│</span> <span class="err">⁝</span> <span class="nt">0x004007b2</span> <span class="nt">3b442418</span> <span class="nt">cmp</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">size</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x18</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">24</span> <span class="err">│</span> <span class="err">└──</span><span class="o">&lt;</span> <span class="nt">0x004007b6</span> <span class="nt">7cc5</span> <span class="nt">jl</span> <span class="nt">0x40077d</span> <span class="err">│</span> <span class="nt">0x004007b8</span> <span class="nt">488b442410</span> <span class="nt">mov</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">local_10h</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x10</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">16</span> <span class="err">│</span> <span class="nt">0x004007bd</span> <span class="nt">4883c428</span> <span class="nt">add</span> <span class="nt">rsp</span><span class="o">,</span> <span class="nt">0x28</span> <span class="o">;</span> <span class="s1">&#39;(&#39;</span> <span class="err">└</span> <span class="nt">0x004007c1</span> <span class="nt">c3</span> <span class="nt">ret</span> </pre></div> <p>Wow, that's a lot of code. Let's take some time to process this. The first part saves the size of the argument in local variable size, then allocates a second buffer of the same size.</p> <p>The second part will loop over both buffers, and put in the decoded buffer, each character, like this:</p> <div class="highlight"><pre><span></span><span class="n">decoded</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">^</span> <span class="mh">0x25</span> </pre></div> <p>It then returns the decoded string. I wrote a small python program to reproduce this:</p> <div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python3</span> <span class="kn">import</span> <span class="nn">sys</span> <span class="n">a</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="nb">chr</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">^</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mh">0x25</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">))</span> <span class="p">])</span> <span class="p">)</span> </pre></div> <p>We can then use this from r2 to decode a random string:</p> <div class="highlight"><pre><span></span><span class="p">[</span><span class="mh">0x00400a30</span><span class="p">]</span><span class="o">&gt;</span> <span class="s s-Atom">pr</span> <span class="mi">48</span> <span class="s s-Atom">@</span> <span class="s s-Atom">str</span><span class="p">.</span><span class="s s-Atom">fIIO_K_YAO_Y</span> <span class="p">|</span> <span class="p">.</span><span class="o">/</span><span class="s s-Atom">decode</span><span class="p">.</span><span class="s s-Atom">py</span> <span class="nv">Congratulations</span><span class="p">!</span> <span class="nv">You</span> <span class="s s-Atom">have</span> <span class="s s-Atom">the</span> <span class="nf">flag</span> <span class="o">:-</span><span class="p">)</span> </pre></div> <p>Hum, this looks interesting. Let's rename the r2 string flag to remember this, it will be useful later to locate where this is printed:</p> <div class="highlight"><pre><span></span><span class="p">[</span><span class="mh">0x00400a30</span><span class="p">]</span><span class="o">&gt;</span> <span class="s s-Atom">fr</span> <span class="s s-Atom">str</span><span class="p">.</span><span class="s s-Atom">fIIO_K_YAO_Y</span> <span class="s2">&quot;str.Congratulations! You have the flag :-)&quot;</span> </pre></div> <p>Let's continue where we left off: we wanted to decode, then print a string. Let's see what it was:</p> <div class="highlight"><pre><span></span>[0x00400a30]&gt; pr 48 @ str.z___OGOBCJ |./decode.py [+] Welcome to the RTFM challenge </pre></div> <p>Nice ! This is the programs's opening prompt. Let's continue with the next printed string:</p> <div class="highlight"><pre><span></span>[0x00400a42]&gt; pr 48 @ str.z__yFNM_K |./decode.py [+] Please enter the flag: @ACXG~ GHIBKLMV����ST </pre></div> <p>Once this is shown, we have the <code>read()</code> on stdin that asks for the flag:</p> <div class="highlight"><pre><span></span> 0x00400a6d 4889e0 mov rax, rsp 0x00400a70 ba14000000 mov edx, 0x14 ; 20 0x00400a75 4889c6 mov rsi, rax 0x00400a78 bf00000000 mov edi, 0 0x00400a7d e85efbffff call sym.imp.read ; ssize_t read(int fildes, void *buf, size_t nbyte) </pre></div> <p>And then another unaligned jump trick to fool the reader, but reversed (eax != 0):</p> <div class="highlight"><pre><span></span> 0x00400a82 50 push rax 0x00400a83 31c0 xor eax, eax 0x00400a85 85c0 test eax, eax 0x00400a87 58 pop rax ┌─&lt; 0x00400a88 7502 jne 0x400a8c ┌──&lt; 0x00400a8a 7401 je 0x400a8d ││ ; CODE XREF from rip (+0x58) ┌─└─&gt; 0x00400a8c eb48 jmp 0x400ad6 </pre></div> <p>Followed by a function call:</p> <div class="highlight"><pre><span></span>[0x00400a42]&gt; pd 10 @ 0x400a8c ; CODE XREF from rip (+0x58) ┌─&lt; 0x00400a8c eb48 jmp 0x400ad6 │ 0x00400a8e 89e0 mov eax, esp │ 0x00400a90 4889c7 mov rdi, rax │ 0x00400a93 e82afdffff call sub.BB_7c2 </pre></div> <p>In this function we'll see something curious:</p> <div class="highlight"><pre><span></span><span class="cp">[</span><span class="mh">0x00400650</span><span class="cp">]</span><span class="o">&gt;</span> <span class="nt">s</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="cp">[</span><span class="mh">0x004007c2</span><span class="cp">]</span><span class="o">&gt;</span> <span class="nt">pd</span> <span class="nt">30</span> <span class="err">┌</span> <span class="o">(</span><span class="nt">fcn</span><span class="o">)</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="nt">396</span> <span class="err">│</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="o">(</span><span class="nt">int</span> <span class="nt">arg1</span><span class="o">);</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">int</span> <span class="nt">local_8h</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x8</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">void</span> <span class="o">*</span><span class="nt">buf</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x10</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">unsigned</span> <span class="nt">int</span> <span class="nt">fildes</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x28</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">var</span> <span class="nt">signed</span> <span class="nt">int</span> <span class="nt">local_2ch</span> <span class="o">@</span> <span class="nt">rsp</span><span class="o">+</span><span class="nt">0x2c</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">arg</span> <span class="nt">int</span> <span class="nt">arg1</span> <span class="o">@</span> <span class="nt">rdi</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CALL</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">fcn</span><span class="p">.</span><span class="nc">004009e0</span> <span class="o">(+</span><span class="nt">0xb3</span><span class="o">)</span> <span class="err">│</span> <span class="nt">0x004007c2</span> <span class="nt">4883ec38</span> <span class="nt">sub</span> <span class="nt">rsp</span><span class="o">,</span> <span class="nt">0x38</span> <span class="o">;</span> <span class="s1">&#39;8&#39;</span> <span class="err">│</span> <span class="nt">0x004007c6</span> <span class="nt">48897c2408</span> <span class="nt">mov</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">local_8h</span><span class="cp">]</span><span class="o">,</span> <span class="nt">rdi</span> <span class="o">;</span> <span class="nt">arg1</span> <span class="err">│</span> <span class="nt">0x004007cb</span> <span class="nt">c744242c20a1</span><span class="o">.</span> <span class="nt">mov</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_2ch</span><span class="cp">]</span><span class="o">,</span> <span class="nt">0x7a120</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x7a120</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="err">│</span> <span class="err">┌─</span><span class="o">&lt;</span> <span class="nt">0x004007d3</span> <span class="nt">eb47</span> <span class="nt">jmp</span> <span class="nt">0x40081c</span> <span class="err">│</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CODE</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="o">(</span><span class="nt">0x400821</span><span class="o">)</span> <span class="err">│</span> <span class="err">┌──</span><span class="o">&gt;</span> <span class="nt">0x004007d5</span> <span class="nt">836c242c01</span> <span class="nt">sub</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_2ch</span><span class="cp">]</span><span class="o">,</span> <span class="nt">1</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007da</span> <span class="nt">be00000000</span> <span class="nt">mov</span> <span class="nt">esi</span><span class="o">,</span> <span class="nt">0</span> <span class="o">;</span> <span class="nt">int</span> <span class="nt">oflag</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007df</span> <span class="nt">bfed0b4000</span> <span class="nt">mov</span> <span class="nt">edi</span><span class="o">,</span> <span class="nt">str</span><span class="p">.</span><span class="nc">BB</span> <span class="o">;</span> <span class="nt">0x400bed</span> <span class="o">;</span> <span class="s2">&quot;\nBB^\x06_YMCJ@];&quot;</span> <span class="o">;</span> <span class="nt">const</span> <span class="nt">char</span> <span class="o">*</span><span class="nt">path</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007e4</span> <span class="nt">b800000000</span> <span class="nt">mov</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">0</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007e9</span> <span class="nt">e852feffff</span> <span class="nt">call</span> <span class="nt">sym</span><span class="p">.</span><span class="nc">imp</span><span class="p">.</span><span class="nc">open</span> <span class="o">;</span> <span class="nt">int</span> <span class="nt">open</span><span class="o">(</span><span class="nt">const</span> <span class="nt">char</span> <span class="o">*</span><span class="nt">path</span><span class="o">,</span> <span class="nt">int</span> <span class="nt">oflag</span><span class="o">)</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007ee</span> <span class="nt">89442428</span> <span class="nt">mov</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">fildes</span><span class="cp">]</span><span class="o">,</span> <span class="nt">eax</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x004007f2</span> <span class="nt">837c242800</span> <span class="nt">cmp</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">fildes</span><span class="cp">]</span><span class="o">,</span> <span class="nt">0</span> <span class="err">│</span> <span class="err">┌───</span><span class="o">&lt;</span> <span class="nt">0x004007f7</span> <span class="nt">7918</span> <span class="nt">jns</span> <span class="nt">0x400811</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="nt">0x004007f9</span> <span class="nt">488d4c2410</span> <span class="nt">lea</span> <span class="nt">rcx</span><span class="o">,</span> <span class="cp">[</span><span class="nx">buf</span><span class="cp">]</span> <span class="o">;</span> <span class="nt">0x10</span> <span class="o">;</span> <span class="nt">16</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="nt">0x004007fe</span> <span class="nt">8b442428</span> <span class="nt">mov</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">fildes</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x28</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="s1">&#39;(&#39;</span> <span class="o">;</span> <span class="nt">40</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="nt">0x00400802</span> <span class="nt">ba0a000000</span> <span class="nt">mov</span> <span class="nt">edx</span><span class="o">,</span> <span class="nt">0xa</span> <span class="o">;</span> <span class="nt">size_t</span> <span class="nt">nbyte</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="nt">0x00400807</span> <span class="nt">4889ce</span> <span class="nt">mov</span> <span class="nt">rsi</span><span class="o">,</span> <span class="nt">rcx</span> <span class="o">;</span> <span class="nt">void</span> <span class="o">*</span><span class="nt">buf</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="nt">0x0040080a</span> <span class="nt">89c7</span> <span class="nt">mov</span> <span class="nt">edi</span><span class="o">,</span> <span class="nt">eax</span> <span class="o">;</span> <span class="nt">int</span> <span class="nt">fildes</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="nt">0x0040080c</span> <span class="nt">e8cffdffff</span> <span class="nt">call</span> <span class="nt">sym</span><span class="p">.</span><span class="nc">imp</span><span class="p">.</span><span class="nc">read</span> <span class="o">;</span> <span class="nt">ssize_t</span> <span class="nt">read</span><span class="o">(</span><span class="nt">int</span> <span class="nt">fildes</span><span class="o">,</span> <span class="nt">void</span> <span class="o">*</span><span class="nt">buf</span><span class="o">,</span> <span class="nt">size_t</span> <span class="nt">nbyte</span><span class="o">)</span> <span class="err">│</span> <span class="err">│⁝│</span> <span class="o">;</span> <span class="nt">CODE</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="o">(</span><span class="nt">0x4007f7</span><span class="o">)</span> <span class="err">│</span> <span class="err">└───</span><span class="o">&gt;</span> <span class="nt">0x00400811</span> <span class="nt">8b442428</span> <span class="nt">mov</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">fildes</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x28</span><span class="p">:</span><span class="mi">4</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="s1">&#39;(&#39;</span> <span class="o">;</span> <span class="nt">40</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400815</span> <span class="nt">89c7</span> <span class="nt">mov</span> <span class="nt">edi</span><span class="o">,</span> <span class="nt">eax</span> <span class="o">;</span> <span class="nt">int</span> <span class="nt">fildes</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="nt">0x00400817</span> <span class="nt">e8b4fdffff</span> <span class="nt">call</span> <span class="nt">sym</span><span class="p">.</span><span class="nc">imp</span><span class="p">.</span><span class="nc">close</span> <span class="o">;</span> <span class="nt">int</span> <span class="nt">close</span><span class="o">(</span><span class="nt">int</span> <span class="nt">fildes</span><span class="o">)</span> <span class="err">│</span> <span class="err">⁝│</span> <span class="o">;</span> <span class="nt">CODE</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="o">(</span><span class="nt">0x4007d3</span><span class="o">)</span> <span class="err">│</span> <span class="err">⁝└─</span><span class="o">&gt;</span> <span class="nt">0x0040081c</span> <span class="nt">837c242c00</span> <span class="nt">cmp</span> <span class="nt">dword</span> <span class="cp">[</span><span class="nx">local_2ch</span><span class="cp">]</span><span class="o">,</span> <span class="nt">0</span> <span class="err">│</span> <span class="err">└──</span><span class="o">&lt;</span> <span class="nt">0x00400821</span> <span class="nt">7fb2</span> <span class="nt">jg</span> <span class="nt">0x4007d5</span> </pre></div> <p><code>open()</code> is called on a file, which once decoded the name is <code>/dev/urandom</code>. But the filename is never decoded. Is this a bug ? Or a last minute modification ? Then, 10 bytes are <code>read()</code> from this file. The file is closed. And this is done again. <code>0x7a120</code> times ! This is another anti-debug, probably designed to slow down <code>strace</code>. I tried running the binary (in a disposable VM!); <code>strace</code> is indeed <em>very</em> slow. Running the binary directly takes less than 5 seconds to pass this code. I'm guessing that it was probably decided that actually opening and reading the blocks in <code>/dev/urandom</code> would be too slow, or less portable. Or it's just a bug :smile:</p> <p>Since I had already disabled an anti-debug, I disable this one as well:</p> <div class="highlight"><pre><span></span>[0x004007c2]&gt; s 0x004007cb [0x004007cb]&gt; &quot;wa mov dword [rsp+0x2c], 0&quot; </pre></div> <p>By setting the loop counter to 0 instead of 0x7a120, this anti-debug code is never run.</p> <p>After this, there's another unaligned jump trick, and then the value that was <code>read()</code> previously is finally analyzed:</p> <div class="highlight"><pre><span></span><span class="cp">[</span><span class="mh">0x0040082e</span><span class="cp">]</span><span class="o">&gt;</span> <span class="nt">pd</span> <span class="nt">20</span> <span class="o">@</span> <span class="nt">0x40082e</span> <span class="err">│</span> <span class="o">;</span> <span class="nt">CODE</span> <span class="nt">XREF</span> <span class="nt">from</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span> <span class="o">(</span><span class="nt">0x40082b</span><span class="o">)</span> <span class="err">│</span> <span class="nt">0x0040082e</span> <span class="nt">488b442408</span> <span class="nt">mov</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">local_8h</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x8</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">8</span> <span class="nt">0x00400833</span> <span class="nt">0fb600</span> <span class="nt">movzx</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">byte</span> <span class="cp">[</span><span class="nx">rax</span><span class="cp">]</span> <span class="nt">0x00400836</span> <span class="nt">3c73</span> <span class="nt">cmp</span> <span class="nt">al</span><span class="o">,</span> <span class="nt">0x73</span> <span class="o">;</span> <span class="s1">&#39;s&#39;</span> <span class="o">;</span> <span class="nt">115</span> <span class="err">┌─</span><span class="o">&lt;</span> <span class="nt">0x00400838</span> <span class="nt">0f8581010000</span> <span class="nt">jne</span> <span class="nt">0x4009bf</span> <span class="o">;</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span><span class="o">+</span><span class="nt">0x1fd</span> <span class="err">│</span> <span class="nt">0x0040083e</span> <span class="nt">488b442408</span> <span class="nt">mov</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">rsp</span> <span class="o">+</span> <span class="mi">8</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x8</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">8</span> <span class="err">│</span> <span class="nt">0x00400843</span> <span class="nt">4883c001</span> <span class="nt">add</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">1</span> <span class="err">│</span> <span class="nt">0x00400847</span> <span class="nt">0fb600</span> <span class="nt">movzx</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">byte</span> <span class="cp">[</span><span class="nx">rax</span><span class="cp">]</span> <span class="err">│</span> <span class="nt">0x0040084a</span> <span class="nt">3c69</span> <span class="nt">cmp</span> <span class="nt">al</span><span class="o">,</span> <span class="nt">0x69</span> <span class="o">;</span> <span class="s1">&#39;i&#39;</span> <span class="o">;</span> <span class="nt">105</span> <span class="err">┌──</span><span class="o">&lt;</span> <span class="nt">0x0040084c</span> <span class="nt">0f856d010000</span> <span class="nt">jne</span> <span class="nt">0x4009bf</span> <span class="o">;</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span><span class="o">+</span><span class="nt">0x1fd</span> <span class="err">││</span> <span class="nt">0x00400852</span> <span class="nt">488b442408</span> <span class="nt">mov</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">qword</span> <span class="cp">[</span><span class="nx">rsp</span> <span class="o">+</span> <span class="mi">8</span><span class="cp">]</span> <span class="o">;</span> <span class="cp">[</span><span class="mh">0x8</span><span class="p">:</span><span class="mi">8</span><span class="cp">]</span><span class="o">=</span><span class="nt">-1</span> <span class="o">;</span> <span class="nt">8</span> <span class="err">││</span> <span class="nt">0x00400857</span> <span class="nt">4883c002</span> <span class="nt">add</span> <span class="nt">rax</span><span class="o">,</span> <span class="nt">2</span> <span class="err">││</span> <span class="nt">0x0040085b</span> <span class="nt">0fb600</span> <span class="nt">movzx</span> <span class="nt">eax</span><span class="o">,</span> <span class="nt">byte</span> <span class="cp">[</span><span class="nx">rax</span><span class="cp">]</span> <span class="err">││</span> <span class="nt">0x0040085e</span> <span class="nt">3c67</span> <span class="nt">cmp</span> <span class="nt">al</span><span class="o">,</span> <span class="nt">0x67</span> <span class="o">;</span> <span class="s1">&#39;g&#39;</span> <span class="o">;</span> <span class="nt">103</span> <span class="err">┌───</span><span class="o">&lt;</span> <span class="nt">0x00400860</span> <span class="nt">0f8559010000</span> <span class="nt">jne</span> <span class="nt">0x4009bf</span> <span class="o">;</span> <span class="nt">sub</span><span class="p">.</span><span class="nc">BB_7c2</span><span class="o">+</span><span class="nt">0x1fd</span> </pre></div> <p>It looks like a character-by-character comparison of the buffer read, starting with 's', then 'i', then 'g'. Is this the flag ? We know (it's in the rules) that the flags are in the format sigsegv{FLAG}, so this looks like it ! Two "unaligned jumps" later, we can gather all the characters for the flag. This is left as an exercise for the reader.</p> <p>This was a quite tedious debug. In fact, I could have ignored most of this, and jumped directly to the interesting part: the analysis of the <code>read()</code> result. Instead, I spent a lot of time disabling anti-debugs, analyzing the decryption function, and I even played a bit with ESIL emulation (not shown here). It was fun, but it could have been solved much more quickly. The top challenger did it ~4 minutes, while it took me a few hours, but I learned a lot along the way !</p> <h1>Javascript obfusqué</h1> <p>This challenge starts quite simply. You can find the <a href="/static/challenge.txt">backed-up source here</a>. Just opening the developer console in the browser allows you to quickly see the whole unpacked code (formatted a bit here):</p> <div class="highlight"><pre><span></span><span class="kd">function</span> <span class="nx">Kod</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">pass</span><span class="p">)</span> <span class="p">{</span> <span class="kd">var</span> <span class="nx">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="kd">var</span> <span class="nx">BlaBla</span><span class="o">=</span><span class="s2">&quot;&quot;</span><span class="p">;</span> <span class="k">for</span><span class="p">(</span><span class="nx">j</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="nx">j</span><span class="o">&lt;</span><span class="nx">s</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">j</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span> <span class="nx">BlaBla</span><span class="o">+=</span><span class="nb">String</span><span class="p">.</span><span class="nx">fromCharCode</span><span class="p">((</span><span class="nx">pass</span><span class="p">.</span><span class="nx">charCodeAt</span><span class="p">(</span><span class="nx">i</span><span class="o">++</span><span class="p">))</span><span class="o">^</span><span class="p">(</span><span class="nx">s</span><span class="p">.</span><span class="nx">charCodeAt</span><span class="p">(</span><span class="nx">j</span><span class="p">)));</span> <span class="k">if</span> <span class="p">(</span><span class="nx">i</span><span class="o">&gt;=</span><span class="nx">pass</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span> <span class="nx">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="p">}</span> <span class="k">return</span><span class="p">(</span><span class="nx">BlaBla</span><span class="p">);</span> <span class="p">}</span> <span class="kd">function</span> <span class="nx">f</span><span class="p">(</span><span class="nx">form</span><span class="p">){</span> <span class="kd">var</span> <span class="nx">pass</span><span class="o">=</span><span class="nb">document</span><span class="p">.</span><span class="nx">form</span><span class="p">.</span><span class="nx">pass</span><span class="p">.</span><span class="nx">value</span><span class="p">;</span> <span class="kd">var</span> <span class="nx">hash</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="k">for</span><span class="p">(</span><span class="nx">j</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="nx">j</span><span class="o">&lt;</span><span class="nx">pass</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">j</span><span class="o">++</span><span class="p">){</span> <span class="kd">var</span> <span class="nx">n</span><span class="o">=</span> <span class="nx">pass</span><span class="p">.</span><span class="nx">charCodeAt</span><span class="p">(</span><span class="nx">j</span><span class="p">);</span> <span class="nx">hash</span> <span class="o">+=</span> <span class="p">((</span><span class="nx">n</span><span class="o">-</span><span class="nx">j</span><span class="o">+</span><span class="mi">33</span><span class="p">)</span><span class="o">^</span><span class="mi">31025</span><span class="p">);</span> <span class="p">}</span> <span class="k">if</span> <span class="p">(</span><span class="nx">hash</span> <span class="o">==</span> <span class="mi">529387</span><span class="p">)</span> <span class="p">{</span> <span class="kd">var</span> <span class="nx">Secret</span> <span class="o">=</span><span class="s2">&quot;&quot;</span><span class="o">+</span><span class="s2">&quot;\x4f\x01\x13\x1e\x09\x59\x34\x09\x0b\x05\x26\x53\x31\x41\x5a\x18\x0e\x53\x1d\x15\x1c\x10\x11\x13\x5b\x06\x16\x69\x15\x29\x55\x1d\x55\x5d\x06\x1d\x0e\x1f\x0c\x14\x13\x5b\x06\x16\x69\x1e\x2a\x40\x5a\x1d\x18\x53\x19\x06\x00\x16\x02\x56\x0a\x1f\x16\x69\x07\x30\x14\x1b\x0a\x5d\x07\x1b\x08\x06\x13\x02\x56\x0b\x05\x06\x3b\x53\x33\x55\x16\x10\x19\x16\x1b\x47\x1f\x00\x47\x15\x13\x0b\x1f\x25\x16\x2b\x53\x1f\x45\x52\x1b\x1d\x0a\x1f\x5b&quot;</span><span class="o">+</span><span class="s2">&quot;&quot;</span><span class="p">;</span> <span class="kd">var</span> <span class="nx">s</span><span class="o">=</span><span class="nx">Kod</span><span class="p">(</span><span class="nx">Secret</span><span class="p">,</span> <span class="nx">pass</span><span class="p">);</span> <span class="nb">document</span><span class="p">.</span><span class="nx">write</span> <span class="p">(</span><span class="nx">s</span><span class="p">);</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="nx">alert</span> <span class="p">(</span><span class="s1">&#39;Wrong password!&#39;</span><span class="p">);</span> <span class="p">}</span> <span class="p">}</span> </pre></div> <p>The function <code>f()</code> is called on form submit. It first computes a custom checksum of the password, and if it matches, it tries to use to decrypt a secret with <code>Kod()</code>. This custom checksum contains a core XOR, that is run on very character, before adding to the total sum. We know that there's a good chance that each character's charCode will be &lt; 127 (in the ASCII space), so we can actually use the checksum to deduce the size of the password (the upper bits being more significant):</p> <div class="highlight"><pre><span></span>529387/31025 = 17.06323932312651 </pre></div> <p>It's 17 characters !</p> <p>The decode function is <code>Kod()</code>; it is run as a fixed-key XOR with the password as key, and the <code>Secret</code> variable as message. This should be crypto 101 (well, almost), and if not, you can follow the <a href="https://cryptopals.com/sets/1/challenges/6">cryptopals challenges</a> to learn how to do that. Since I wasn't so certain that I could do it on my own in a short-enough time, I just reused <a href="https://github.com/JesseEmond/matasano-cryptopals/">someone else's solution</a> with the key size I already found. It didn't give a perfect decrypt, but it was close enough to help me find the flag.</p> <p>In retrospect, I could have also used the fact that the start of the flags is always the same (<code>'sigsegv{'</code>), and then decode it by hand, but this was good enough.</p> <p>In the JS console of your browser, you can see how to decrypt the secret:</p> <div class="highlight"><pre><span></span><span class="nx">Kod</span><span class="p">(</span><span class="nx">Secret</span><span class="p">,</span> <span class="nx">flag</span><span class="p">)</span> <span class="s2">&quot;&lt;html&gt;Bravo tu as trouve le flag, utilise le mot de passe que tu as trouve pour valider le challenge&lt;/html&gt;&quot;</span> </pre></div> <h1>Un nouveau dialecte</h1> <p>This challenge, was in the "Crypto" section. Wait, didn't we have crypto in the two last challenges as well ?</p> <p>The goal was to decode this new "dialect":</p> <div class="highlight"><pre><span></span>ȃǹǷȃǵǷȆȋǜǑǣǤǕǗǑǓǕǣǤǠǑǣǣǙǖǑǓǙǜǕȍ </pre></div> <p>As I do in most challenges, I put the content in a file (named "file", I also lack imagination), loaded it in <code>ipython3</code>, and started visualizing and massaging the data</p> <div class="highlight"><pre><span></span><span class="n">Python</span> <span class="mf">3.6</span><span class="o">.</span><span class="mi">6</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">Jul</span> <span class="mi">19</span> <span class="mi">2018</span><span class="p">,</span> <span class="mi">14</span><span class="p">:</span><span class="mi">25</span><span class="p">:</span><span class="mi">17</span><span class="p">)</span> <span class="n">Type</span> <span class="s1">&#39;copyright&#39;</span><span class="p">,</span> <span class="s1">&#39;credits&#39;</span> <span class="ow">or</span> <span class="s1">&#39;license&#39;</span> <span class="k">for</span> <span class="n">more</span> <span class="n">information</span> <span class="n">IPython</span> <span class="mf">6.4</span><span class="o">.</span><span class="mi">0</span> <span class="o">--</span> <span class="n">An</span> <span class="n">enhanced</span> <span class="n">Interactive</span> <span class="n">Python</span><span class="o">.</span> <span class="n">Type</span> <span class="s1">&#39;?&#39;</span> <span class="k">for</span> <span class="n">help</span><span class="o">.</span> <span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="n">a</span><span class="o">=</span><span class="nb">open</span><span class="p">(</span><span class="s2">&quot;file&quot;</span><span class="p">,</span> <span class="s2">&quot;rb&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="k">print</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc8\x83\xc7\xb9\xc7\xb7\xc8\x83\xc7\xb5\xc7\xb7\xc8\x86\xc8\x8b\xc7\x9c\xc7\x91\xc7\xa3\xc7\xa4\xc7\x95\xc7\x97\xc7\x91\xc7\x93\xc7\x95\xc7\xa3\xc7\xa4\xc7\xa0\xc7\x91\xc7\xa3\xc7\xa3\xc7\x99\xc7\x96\xc7\x91\xc7\x93\xc7\x99\xc7\x9c\xc7\x95\xc8\x8d</span><span class="s1">&#39;</span> <span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="k">print</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&quot;utf-8&quot;</span><span class="p">))</span> <span class="err">ȃǹǷȃǵǷȆȋǜǑǣǤǕǗǑǓǕǣǤǠǑǣǣǙǖǑǓǙǜǕȍ</span> <span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">b</span> <span class="o">=</span> <span class="p">[</span> <span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">i</span><span class="o">+</span><span class="mi">2</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="mi">2</span><span class="p">)</span> <span class="p">]</span> <span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">b</span> <span class="n">Out</span><span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="p">[</span><span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc8\x83</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xb9</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xb7</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc8\x83</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xb5</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xb7</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc8\x86</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc8\x8b</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x9c</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x91</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa3</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa4</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x95</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x97</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x91</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x93</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x95</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa3</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa4</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa0</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x91</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa3</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\xa3</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x99</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x96</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x91</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x93</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x99</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x9c</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc7\x95</span><span class="s1">&#39;</span><span class="p">,</span> <span class="sa">b</span><span class="s1">&#39;</span><span class="se">\xc8\x8d</span><span class="s1">&#39;</span><span class="p">]</span> </pre></div> <p>I first analyzed the UTF-8 encoded codepoints (as binary data), but was soon lucky, and found that the Unicode codepoints (not their UTF-8 encoded version) were in fact all in the same range, hinting to a simple Caesar cipher. Here is the final visualizing and decoding program:</p> <div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python3</span> <span class="c1"># coding: utf-8</span> <span class="n">s</span><span class="o">=</span><span class="nb">open</span><span class="p">(</span><span class="s2">&quot;file&quot;</span><span class="p">,</span> <span class="s2">&quot;r&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">s</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;{} {:04x}: {:16b}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="nb">ord</span><span class="p">(</span><span class="n">i</span><span class="p">),</span> <span class="nb">ord</span><span class="p">(</span><span class="n">i</span><span class="p">)),</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39;</span><span class="se">\t</span><span class="s1">&#39;</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="nb">ord</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">-</span> <span class="mh">0x100</span> <span class="o">-</span> <span class="mi">144</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;{:09b} {} {}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="nb">chr</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">-</span> <span class="mi">400</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">s</span><span class="p">]))</span> </pre></div> <p>Note that this was very easy because I'm used to launch <code>python3</code> by default, which is unicode-native; <code>ord()</code> behaves as it should. Anyone used to python2, would be in a bit more pain, since the characters would have been interpreted byte-by-byte.</p> <h1>La simplicité</h1> <p>Under the "simplicity" title, this one might be the longest to solve. To start, you're given a "simple" website, that looks like this:</p> <div class="highlight"><pre><span></span>$ curl http://51.158.73.218:8880/ <span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">head</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">title</span><span class="p">&gt;</span>Un site simple<span class="p">&lt;/</span><span class="nt">title</span><span class="p">&gt;&lt;/</span><span class="nt">title</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">head</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">center</span><span class="p">&gt;&lt;</span><span class="nt">iframe</span> <span class="na">width</span><span class="o">=</span><span class="s">&quot;560&quot;</span> <span class="na">height</span><span class="o">=</span><span class="s">&quot;315&quot;</span> <span class="na">src</span><span class="o">=</span><span class="s">&quot;https://www.youtube.com/embed/2bjk26RwjyU?rel=0&amp;amp;controls=0&amp;amp;showinfo=0&quot;</span> <span class="na">frameborder</span><span class="o">=</span><span class="s">&quot;0&quot;</span> <span class="na">allow</span><span class="o">=</span><span class="s">&quot;autoplay; encrypted-media&quot;</span> <span class="na">allowfullscreen</span><span class="p">&gt;&lt;/</span><span class="nt">iframe</span><span class="p">&gt;&lt;/</span><span class="nt">center</span><span class="p">&gt;</span> <span class="c">&lt;!-- Si une méthode ne fonctionne pas il faut en utiliser une autre --&gt;</span> <span class="c">&lt;!-- Un formulaire c&#39;était pas assez simple donc on en a pas mis --&gt;</span> <span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span> </pre></div> <p>I did quite a lot of exploration on this: I tried different http methods (to no avail), I discovered that the name of the file was "index.php". I tried a few standard "admin" pages. Then I moved on to the other challenges, keeping this one last.</p> <p>When I came back to it, I had an epiphany. The solution to unlock the first step was simple, in retrospect (as announced):</p> <div class="highlight"><pre><span></span>$ curl http://51.158.73.218:8880/robots.txt backup.zip </pre></div> <p>Oh. So I download this <a href="/static/backup.zip">zip file, (backup here)</a>, and it's a password-protected zip containing an index.php:</p> <div class="highlight"><pre><span></span>$ unzip backup.zip Archive: backup.zip <span class="o">[</span>backup.zip<span class="o">]</span> index.php password: </pre></div> <p>No solution here, one has to use bruteforce to crack this zip. I download john the ripper, and extract the crackable hash:</p> <div class="highlight"><pre><span></span>$ ./JohnTheRipper/run/zip2john backup.zip &gt; john.hash ver <span class="m">2</span>.0 efh <span class="m">5455</span> efh <span class="m">7875</span> backup.zip-&gt;index.php PKZIP Encr: 2b chk, TS_chk, <span class="nv">cmplen</span><span class="o">=</span><span class="m">453</span>, <span class="nv">decmplen</span><span class="o">=</span><span class="m">680</span>, <span class="nv">crc</span><span class="o">=</span>70C7CB88 $ cat john.hash backup.zip:<span class="nv">$pkzip2$1</span>*2*2*0*1c5*2a8*70c7cb88*0*43*8*1c5*70c7*bce4*01f43e1d0eb0118661d22e480e38736de7c321d3ac1cf086601594c4ab54ebc7af0ad5ea01c8b64bda21aee19533a09808c0e7892fdb08f8df9644eeefc9aabe92b3c1cb10fb981090365d55229da292afba120f388d25a56e52c91b42af567d2ee897c5bd979b673a99fe187e4064f438165815d29fad2d1a7edbdf46ee2ff99afb546e1626cbb57897b6a108a3fb108495ec508243bffe3d050efe1b9aadf700695f8aca72e4e1977f827702ec5840fbe1559e0ac1e646323ea051ee69257030c3b33d305d9ab6f70dc600a2d4cc07482df8d95e4dd8741082540e3b2ec988eab2c99a595927eb31cc589d8bd28068ddd375588c668f52f5896d45e42de0d1933dc390a5c2a5ee3b8d30b91b763bb77892651dd9241bf03dde65ad8b6acee2bcb3942dc800aa3350d2f894c32fc0dcba5164d9db59dd09044d28b44181a19398d27c64b65bd1c8e4cdce21eeac513172d340ca4b54baf5570921dc182e3b02b0ff8d0b0ac4070a0715f6300f8fb99ffdc665270cc98fae8d28f3727742b79e2bd9392f35e4564a243234e9cf502beb0e3572c2c83a33b68c56cc317aece233f99a02838c9c562ebb3271d58aa6bb653b43803c9188b1c737cfa827c533ff301e453fb111*$/pkzip2$:::::backup.zip </pre></div> <p>Then I run john on it:</p> <div class="highlight"><pre><span></span>$ ./JohnTheRipper/run/john --show john.hash backup.zip:passw0rd:::::backup.zip <span class="m">1</span> password <span class="nb">hash</span> cracked, <span class="m">0</span> left </pre></div> <p>This runs almost instantly; so the password is "passw0rd". We can unzip the file, and get the source code of <code>index.php</code>. This is the same page as before, with the more interesting parts shown here:</p> <div class="highlight"><pre><span></span><span class="cp">&lt;?php</span> <span class="k">include</span> <span class="s2">&quot;auth.php&quot;</span><span class="p">;</span> <span class="cp">?&gt;</span><span class="x"></span> <span class="x">[…]</span> <span class="cp">&lt;?php</span> <span class="k">if</span><span class="p">(</span><span class="nb">isset</span><span class="p">(</span><span class="nv">$_POST</span><span class="p">[</span><span class="s2">&quot;h1&quot;</span><span class="p">]))</span> <span class="p">{</span> <span class="nv">$h1</span> <span class="o">=</span> <span class="nb">md5</span><span class="p">(</span><span class="nv">$_POST</span><span class="p">[</span><span class="s2">&quot;h1&quot;</span><span class="p">]</span> <span class="o">.</span> <span class="s2">&quot;Shrewk&quot;</span><span class="p">);</span> <span class="k">echo</span> <span class="s2">&quot;h1 vaut: &quot;</span><span class="o">.</span><span class="nv">$h1</span><span class="o">.</span><span class="s2">&quot;&lt;/br&gt;&quot;</span><span class="p">;</span> <span class="k">if</span><span class="p">(</span><span class="nv">$h1</span> <span class="o">==</span> <span class="s2">&quot;0&quot;</span><span class="p">)</span> <span class="p">{</span> <span class="k">echo</span> <span class="s2">&quot;&lt;!--Bien joué le flag est &quot;</span><span class="o">.</span><span class="nv">$flag</span><span class="o">.</span><span class="s2">&quot;--&gt;&quot;</span><span class="p">;</span> <span class="p">}</span> <span class="p">}</span> <span class="cp">?&gt;</span><span class="x"></span> <span class="x">[…]</span> </pre></div> <p>So, there's a POST parameter h1, which is hashed with a "Shrewk" salt, and then compared to the string "0". Wait, what ? How can md5() return "0" ?</p> <p>The php documentation doesn't mention anything of the sort. But you can quickly find that the <a href="http://php.net/manual/en/language.operators.comparison.php">comparison operation "=="</a> isn't recommended to compare strings, because of type juggling. Indeed, a string like "1e3", will be resolved to the integer 1000, for example. A secure string comparison should use "===".</p> <p>So, this might be the core of the challenge. Maybe what we need is an md5 hash string that starts with many "0", then "e" or "E", and then is followed by only digits ? It looks like we need to crack some md5 as well.</p> <p>I installed hashcat, and started looking at the documentation. I found a lot of ways to parametrize the input to be hashed, but no way to filter the hashes in output to look like a given format. I attempted to generate a lot of hashes in hope of finding a random password that would match one of them, with the given program:</p> <div class="highlight"><pre><span></span><span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp"></span> <span class="cp">#include</span> <span class="cpf">&lt;stdlib.h&gt;</span><span class="cp"></span> <span class="cp">#include</span> <span class="cpf">&lt;stdint.h&gt;</span><span class="cp"></span> <span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> <span class="kt">uint64_t</span> <span class="n">i</span><span class="p">;</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">seed</span><span class="p">;</span> <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="p">(</span><span class="mi">1</span><span class="o">&lt;&lt;</span> <span class="mi">29</span><span class="p">);</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="n">printf</span><span class="p">(</span><span class="s">&quot;0e%010d%010d%010d:Shrewk</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span> <span class="n">rand_r</span><span class="p">(</span><span class="o">&amp;</span><span class="n">seed</span><span class="p">),</span> <span class="n">rand_r</span><span class="p">(</span><span class="o">&amp;</span><span class="n">seed</span><span class="p">),</span> <span class="n">rand_r</span><span class="p">(</span><span class="o">&amp;</span><span class="n">seed</span><span class="p">));</span> <span class="p">}</span> </pre></div> <p>This writes data in the hashcat format hash:salt. I used it to generate a lot of potential hashes, but hashcat needs to load all of them into memory, and whether you have 1GB or 1TB of RAM, this is still is many order of magnitudes smaller than the space I want to explore. So while hashcat was running, I started working on a more exhaustive exploration.</p> <p>I decided to use a go program, because this is an embarrassingly parallel problem, and will be much easier to crack with go routines. First, here is the match function, to test if a hex-encoded hash has the format we want:</p> <div class="highlight"><pre><span></span><span class="kd">func</span> <span class="nx">match</span><span class="p">(</span><span class="nx">x</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span> <span class="k">for</span> <span class="nx">x</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">==</span> <span class="sc">&#39;0&#39;</span> <span class="p">{</span> <span class="nx">i</span><span class="o">++</span> <span class="p">}</span> <span class="k">if</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="mi">1</span> <span class="p">{</span> <span class="k">return</span> <span class="kc">false</span> <span class="p">}</span> <span class="k">if</span> <span class="nx">x</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">!=</span> <span class="sc">&#39;e&#39;</span> <span class="o">&amp;&amp;</span> <span class="nx">x</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">!=</span> <span class="sc">&#39;E&#39;</span> <span class="p">{</span> <span class="k">return</span> <span class="kc">false</span> <span class="p">}</span> <span class="k">for</span> <span class="nx">i</span><span class="o">++</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="nx">x</span><span class="p">)</span> <span class="o">&amp;&amp;</span> <span class="p">(</span><span class="nx">x</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">&gt;=</span> <span class="sc">&#39;0&#39;</span> <span class="o">&amp;&amp;</span> <span class="nx">x</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">&lt;=</span> <span class="sc">&#39;9&#39;</span><span class="p">);</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span> <span class="p">}</span> <span class="k">return</span> <span class="nx">i</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="nx">x</span><span class="p">)</span> <span class="p">}</span> </pre></div> <p>And its corresponding test:</p> <div class="highlight"><pre><span></span><span class="kd">func</span> <span class="nx">TestMatch</span><span class="p">(</span><span class="nx">t</span> <span class="o">*</span><span class="nx">testing</span><span class="p">.</span><span class="nx">T</span><span class="p">)</span> <span class="p">{</span> <span class="nx">samples</span> <span class="o">:=</span> <span class="p">[]</span><span class="kd">struct</span> <span class="p">{</span> <span class="nx">v</span> <span class="kt">string</span> <span class="nx">ret</span> <span class="kt">bool</span> <span class="p">}{</span> <span class="p">{</span><span class="s">&quot;000e1234123456781234567812345678&quot;</span><span class="p">,</span> <span class="kc">true</span><span class="p">},</span> <span class="p">{</span><span class="s">&quot;0a0e1234123456781234567812345678&quot;</span><span class="p">,</span> <span class="kc">false</span><span class="p">},</span> <span class="p">{</span><span class="s">&quot;00E01234123456781234567812345678&quot;</span><span class="p">,</span> <span class="kc">true</span><span class="p">},</span> <span class="p">{</span><span class="s">&quot;00EE1234123456781234567812345678&quot;</span><span class="p">,</span> <span class="kc">false</span><span class="p">},</span> <span class="p">{</span><span class="s">&quot;00E01234123456781234567812345ab8&quot;</span><span class="p">,</span> <span class="kc">false</span><span class="p">},</span> <span class="p">{</span><span class="s">&quot;0e000000000000000000000000000000&quot;</span><span class="p">,</span> <span class="kc">true</span><span class="p">},</span> <span class="p">}</span> <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">samples</span> <span class="p">{</span> <span class="k">if</span> <span class="nx">match</span><span class="p">(</span><span class="nx">samples</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">v</span><span class="p">)</span> <span class="o">!=</span> <span class="nx">samples</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">ret</span> <span class="p">{</span> <span class="nx">t</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">&quot;Error: &quot;</span><span class="p">,</span> <span class="nx">samples</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">v</span><span class="p">,</span> <span class="nx">samples</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">ret</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </pre></div> <p>This is the only function that was tested because of how core it was to finding a solution. Note that it might have been faster to work directly on byte data instead of converting the md5 to a hex-encoded string, but I found it an acceptable compromise to keep the code readable and correct.</p> <p>The rest of the code is just an exhaustive exploration of password space (with the salt), with a recursive core.</p> <div class="highlight"><pre><span></span><span class="kd">func</span> <span class="nx">core2</span><span class="p">(</span><span class="nx">share</span><span class="p">,</span> <span class="nx">max</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">alphabet</span> <span class="p">=</span> <span class="s">&quot;0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwyxz.;/-{}&quot;</span> <span class="kd">const</span> <span class="nx">salt</span> <span class="p">=</span> <span class="s">&quot;Shrewk&quot;</span> <span class="k">for</span> <span class="nx">size</span> <span class="o">:=</span> <span class="nx">share</span><span class="p">;</span> <span class="p">;</span> <span class="nx">size</span> <span class="o">+=</span> <span class="nx">max</span> <span class="p">{</span> <span class="k">if</span> <span class="nx">size</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> <span class="k">continue</span> <span class="p">}</span> <span class="nx">b</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span> <span class="nx">size</span><span class="o">+</span><span class="nb">len</span><span class="p">(</span><span class="nx">salt</span><span class="p">))</span> <span class="k">for</span> <span class="nx">c</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">c</span> <span class="p">&lt;</span> <span class="nx">size</span><span class="p">;</span> <span class="nx">c</span><span class="o">++</span> <span class="p">{</span> <span class="nx">b</span><span class="p">[</span><span class="nx">c</span><span class="p">]</span> <span class="p">=</span> <span class="nx">alphabet</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="p">}</span> <span class="k">for</span> <span class="nx">c</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">c</span> <span class="p">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="nx">salt</span><span class="p">);</span> <span class="nx">c</span><span class="o">++</span> <span class="p">{</span> <span class="nx">b</span><span class="p">[</span><span class="nx">c</span><span class="o">+</span><span class="nb">len</span><span class="p">(</span><span class="nx">b</span><span class="p">)</span><span class="o">-</span><span class="nb">len</span><span class="p">(</span><span class="nx">salt</span><span class="p">)]</span> <span class="p">=</span> <span class="nx">salt</span><span class="p">[</span><span class="nx">c</span><span class="p">]</span> <span class="p">}</span> <span class="kd">var</span> <span class="nx">charloop</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="kd">func</span><span class="p">())</span> <span class="nx">charloop</span> <span class="p">=</span> <span class="kd">func</span><span class="p">(</span><span class="nx">i</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">f</span> <span class="kd">func</span><span class="p">())</span> <span class="p">{</span> <span class="k">for</span> <span class="nx">c</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">c</span> <span class="p">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="nx">alphabet</span><span class="p">);</span> <span class="nx">c</span><span class="o">++</span> <span class="p">{</span> <span class="nx">b</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="p">=</span> <span class="nx">alphabet</span><span class="p">[</span><span class="nx">c</span><span class="p">]</span> <span class="k">if</span> <span class="nx">i</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span> <span class="nx">charloop</span><span class="p">(</span><span class="nx">i</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="nx">f</span><span class="p">)</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="nx">f</span><span class="p">()</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="c1">//recursion is much easier</span> <span class="nx">charloop</span><span class="p">(</span><span class="nx">size</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span> <span class="nx">h</span> <span class="o">:=</span> <span class="nx">md5</span><span class="p">.</span><span class="nx">Sum</span><span class="p">(</span><span class="nx">b</span><span class="p">)</span> <span class="nx">he</span> <span class="o">:=</span> <span class="nx">hex</span><span class="p">.</span><span class="nx">EncodeToString</span><span class="p">(</span><span class="nx">h</span><span class="p">[:])</span> <span class="k">if</span> <span class="nx">match</span><span class="p">(</span><span class="nx">he</span><span class="p">)</span> <span class="p">{</span> <span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;Found match: &quot;</span><span class="p">,</span> <span class="nb">string</span><span class="p">(</span><span class="nx">b</span><span class="p">),</span> <span class="nx">he</span><span class="p">)</span> <span class="nx">os</span><span class="p">.</span><span class="nx">Exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="p">}</span> <span class="p">})</span> <span class="p">}</span> <span class="p">}</span> </pre></div> <p>Particular attention was given on minimizing the allocations, to reduce the performance impact. This is why the core function works on a single <code>[]byte</code> slice. The hex.EncodeToString does many allocations though.</p> <p>The main function does the work sharing, in a naive way: the size of the password is used to slice the work between goroutines:</p> <div class="highlight"><pre><span></span><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span> <span class="nx">c</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kd">struct</span><span class="p">{})</span> <span class="k">for</span> <span class="nx">i</span> <span class="o">:=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">runtime</span><span class="p">.</span><span class="nx">NumCPU</span><span class="p">();</span> <span class="nx">i</span><span class="o">++</span> <span class="p">{</span> <span class="k">go</span> <span class="kd">func</span><span class="p">()</span> <span class="p">{</span> <span class="nx">core2</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="nx">runtime</span><span class="p">.</span><span class="nx">NumCPU</span><span class="p">())</span> <span class="nx">c</span> <span class="o">&lt;-</span> <span class="kd">struct</span><span class="p">{}{}</span> <span class="nx">os</span><span class="p">.</span><span class="nx">Exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="p">}()</span> <span class="p">}</span> <span class="o">&lt;-</span><span class="nx">c</span> <span class="p">}</span> </pre></div> <p>This means, that the program will be non-deterministic, depending on the number of cores we have, and the particular scheduling of goroutines.</p> <p>This code finds a password on my 2012 laptop in about 3 minutes.</p> <div class="highlight"><pre><span></span>$ <span class="nb">time</span> ./crack Found match: KgeM5000Shrewk 0e957579856924481004771378652894 real 2m45.885s user 10m28.607s sys 0m2.147s </pre></div> <p>And let's test this password we found:</p> <div class="highlight"><pre><span></span>$ curl -s -d <span class="nv">h1</span><span class="o">=</span>KgeM5000 http://51.158.73.218:8880/index.php <span class="p">|</span>grep flag h1 vaut: 0e957579856924481004771378652894&lt;/br&gt;&lt;!--Bien joué le flag est sigsegv<span class="o">{</span>a1a29afa647a20758e64b49d8eb453f4<span class="o">}</span>--&gt;&lt;!-- Si une méthode ne fonctionne pas il faut en utiliser une autre --&gt; </pre></div> <p>Unsurprisingly, it was much faster on a modern 8 core Xeon; and it found another password first because of the work sharing structure :</p> <div class="highlight"><pre><span></span>$ curl -s -d <span class="nv">h1</span><span class="o">=</span>QM8.B0 http://51.158.73.218:8880/index.php <span class="p">|</span>grep flag h1 vaut: 0e893977776066512259427456189998&lt;/br&gt;&lt;!--Bien joué le flag est sigsegv<span class="o">{</span>a1a29afa647a20758e64b49d8eb453f4<span class="o">}</span>--&gt;&lt;!-- Si une méthode ne fonctionne pas il faut en utiliser une autre --&gt; </pre></div> <p>And that's it for the challenges !</p>Kernel Recipes 2018 liveblog2018-09-26T00:00:00+02:002018-09-26T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2018-09-26:/recipes-conferences-2018.html<p>I had a lot of good feedback from the <a href="kernel-recipes-2017-day-1">previous</a> <a href="kernel-recipes-2016-notes.html">years</a> live-blogs of the Embedded and Kernel Recipes conferences. So much in fact, that this year I'm now doing the liveblog <a href="https://kernel-recipes.org/en/2018/live-blog/">directly on the official website of kernel recipes</a>.</p> <p>I hope you enjoy it !</p> <p>I also live-tweeted my impressions of …</p><p>I had a lot of good feedback from the <a href="kernel-recipes-2017-day-1">previous</a> <a href="kernel-recipes-2016-notes.html">years</a> live-blogs of the Embedded and Kernel Recipes conferences. So much in fact, that this year I'm now doing the liveblog <a href="https://kernel-recipes.org/en/2018/live-blog/">directly on the official website of kernel recipes</a>.</p> <p>I hope you enjoy it !</p> <p>I also live-tweeted my impressions of Embedded Recipes <a href="https://twitter.com/Aissn/status/1044137236621660160">day 1</a> and <a href="https://twitter.com/Aissn/status/1044499736571629568">day 2</a>.</p>r2con 2018 and r2wars2018-09-08T00:00:00+02:002018-09-08T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2018-09-08:/r2con-2018.html<p>I tried something new this year, by going to <a href="https://rada.re/con/2018">r2con</a>, a conference dedicated to <a href="https://radare.org">radare2</a>, a reverse engineering toolkit.</p> <h1>r2con</h1> <p>This conference is one of the most affordable security conference out there. I've used radare2 (r2) in the past, but I don't think I fully understood its philosophy until going …</p><p>I tried something new this year, by going to <a href="https://rada.re/con/2018">r2con</a>, a conference dedicated to <a href="https://radare.org">radare2</a>, a reverse engineering toolkit.</p> <h1>r2con</h1> <p>This conference is one of the most affordable security conference out there. I've used radare2 (r2) in the past, but I don't think I fully understood its philosophy until going there. The radare2 community is really focused on creating the best toolkit for analyzing, disassembling, debugging, reverse engineering software; all while keeping it fully free software.</p> <p>The training was very interesting, with demos and hands-on experience.</p> <p>The organization was great, and it was a sleek conference. Many thanks to the organizers.</p> <p>I've had to install Telegram for the first time to participate in the local discussions, and apparently I wasn't the only one new to Telegram. I've mostly interacted with the radare2 community through IRC before (they have a bridge).</p> <p>I've met many great people from all over the world; I was honestly surprised how welcoming is the community.</p> <p>The conference had a single track of talks (all recorded), a CTF (capture-the-flag) game focused on reverse engineering binaries: to crack them, exploit them, etc. with radare2, as well as a little game called "r2wars", on which I spent way more time than I care to admit.</p> <h1>r2wars</h1> <p>In radare2, there's an intermediate language and VM called ESIL. It is used to emulate code, and supports many architectures. r2wars is built on top of r2's ESIL VM, but the rules limit to the following architectures: x86-32, arm-32, arm-64 and mips-32.</p> <p>In r2wars, two opponents create "bots", short shellcodes, which are executed one after the other, in the same memory space, thanks to ESIL emulation. The goal is be the latest to survive; you must wipe, crash your opponent, or simply wait for your opponent to die by himself by writing or executing at invalid memory.</p> <p>The ESIL vm is initialized with a 1024 bytes memory space plus a stack (more on that later), and the two opponents are placed randomly in this space. They are executed in round-robin: one after the other, one instruction each. A particularity is that you can have an opponent using a different architecture, since multi-architectures combats are supported: you only share the memory space, not the CPU state (registers). The server makes sure your memory does not overlap your opponents' at launch; afterwards, all bets are off.</p> <p>You upload the source code of your bot to the <a href="https://github.com/radare/r2wars">r2wars server</a> through a <a href="http://radare.org/up">web interface</a> ; the server software uses rasm2, the radare2 assembler to build your bot, it launches radare2, initializes ESIL, then launches 1v1 matches in a tournament, to determine the winner.</p> <h2>My bots</h2> <p>I've dumped <a href="https://github.com/anisse/r2warsbots">the source code for all my bots</a> on github (spoilers!).</p> <h3>Naive approach</h3> <p>My very first idea was to create a self-replicating bot that would survive forever by copying itself in a loop. It took me long time to build as I was getting used to the aarch64 assembly (I mostly have experience with aarch32 and thumb).</p> <p>One of the first hurdles would be that the rasm2 assembler is still quite incomplete. It does not support all arm64 addressing modes for the branch instructions, it does not support load/store pre/post-indexing; it does labels in a naive way. For example, if you give a very short name to your label (say, 'x'), it will replace all occurrences of "x" in the program; which might be an issue in arm64, since the registers are named x0 to x30.</p> <p>One of the goal of the r2wars competition was to improve rasm2 and ESIL emulation in radare2. I looked at instruction encoding, finding a <a href="http://kitoslab-eng.blogspot.com/2012/10/armv8-aarch64-instruction-encoding.html">few references</a> in addition to the official manual, but I couldn't figure out the post-index address encoding for <code>ldp</code> in a short-enough time to be useful for the competition, so I moved to the gnu assembler included in the fedora package binutils-aarch64-linux-gnu. The binary code is then converted to <code>.hex</code> directives for rasm2.</p> <p>Once I did that, I discovered that ESIL emulation in radare2 was much more complete, and that the code I wrote behaved as expected (well, minus my bugs).</p> <p>At first I attempted to make use of "combined" instructions like <code>csel</code>, <code>cbz</code>, or <code>ldp</code>/<code>stp</code>; I've mostly kept <code>ldp</code>/<code>stp</code> afterwards, since they are the instructions which can read and write the most data that at once: two 8 bytes registers, with the option to modify/increment the addressing register at the same time with the so-called pre-index (modify the register before the load/store) and post-index addressing mode (modify it after the load/store). This allowed, when repeated, to do the most damage. This is my first bot:</p> <div class="highlight"><pre><span></span> <span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span> <span class="nf">add</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x0</span><span class="p">,</span> <span class="no">endprog</span> <span class="nf">add</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x1</span><span class="p">,</span> <span class="no">endprog</span> <span class="nf">cmp</span> <span class="no">x5</span><span class="p">,</span> <span class="mi">1024</span> <span class="p">-</span> <span class="mi">16</span> <span class="nf">csel</span> <span class="no">x1</span><span class="p">,</span> <span class="no">xzr</span><span class="p">,</span> <span class="no">x1</span><span class="p">,</span> <span class="no">gt</span> <span class="nf">add</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x1</span><span class="p">,</span> <span class="no">endprog</span> <span class="nf">add</span> <span class="no">x4</span><span class="p">,</span> <span class="no">x1</span><span class="p">,</span> <span class="mi">16</span> <span class="nl">looprog:</span> <span class="nf">ldp</span> <span class="no">x2</span><span class="p">,</span> <span class="no">x3</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">,</span> <span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x2</span><span class="p">,</span> <span class="no">x3</span><span class="p">,</span> <span class="p">[</span><span class="no">x1</span><span class="p">,</span> <span class="mi">16</span><span class="p">]!</span> <span class="nf">cmp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x5</span> <span class="nf">b.lt</span> <span class="no">looprog</span> <span class="nf">br</span> <span class="no">x4</span> <span class="nl">endprog:</span> <span class="nf">nop</span> </pre></div> <p>It's very naive, but it works. It copies itself in a loop, and wraps at 1024 bytes. It should never die when left a alone. The issues are numerous, but the biggest ones are that it's simply too slow, and too big, clocking at 48 bytes of useful instructions. I submitted it once, but never used it in a competition, replacing it with a better one.</p> <h3>Simplicity</h3> <p>Once I managed to build and run the official r2wars server (you need an official version of Mono for that, not the distro versions which are too old), I started writing other bots to have them compete with each other, before the official matches. I wrote a few variants of this bot:</p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="mi">68</span> <span class="nf">adr</span> <span class="no">x1</span><span class="p">,</span> <span class="p">-</span><span class="mi">4</span> <span class="nl">loop:</span> <span class="nf">stp</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x1</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">,</span> <span class="mi">16</span><span class="p">]!</span> <span class="nf">b</span> <span class="no">loop</span> </pre></div> <p>This bot will first get its own address + 68 (to be at the end of the reserved space by r2wars), and then will start writing groups of bytes in the up and down directions, until it died by reaching invalid memory.</p> <p>And most of those simple bots weren't statistically worse than the original bot. As it was already 3am before the competition started, I decided to call it a night, and submit my initial naive approach.</p> <h3>Insomnia</h3> <p>But. I just couldn't sleep. I had so many ideas, so I got up and wrote this one quickly: </p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="no">start</span> <span class="nf">mov</span> <span class="no">x3</span><span class="p">,</span> <span class="mi">1008</span> <span class="nf">ldp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">]</span> <span class="nf">br</span> <span class="no">x3</span> <span class="nl">start:</span> <span class="nf">stp</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">b</span> <span class="no">start</span> </pre></div> <p>This one copies it payload at the end of the arena in only 4 instructions, jumps to it, and then the payload will just overwrite all bytes at a significantly higher rate than the other ones. I wasn't sure it would be more efficient, but I went to sleep for real.</p> <p>The following morning, I ran the simulations again, and this new one was completely thrashing the others. I decided to submit it, and it was the one used in the first tournament.</p> <h2>The fights</h2> <p>It didn't fare so well during the first iteration of the tournament. It was the only arm-64 bot, mostly against x86 bots, and I didn't understand why, but it arrived in <a href="https://twitter.com/T_PAGEFLT/status/1038033365352570880">5th of 8 position</a>. Not too bad.</p> <p>I went on to work on my next idea: a single-instruction bot, providing a much slimmer target against non-linear writes, and always writing its next instruction. It needed to be slow, to not die quickly, so I had to write only 4 bytes at once. So I changed from the wide-store <code>stp</code>, to the smaller <code>str</code>. It supported the pre-indexing addressing mode, so it could work.</p> <p>But that's when I found my first ESIL bug. It turned out that ESIL arm64 emulation only implemented the pre-indexing addressing for <code>str</code>. I looked more into the code, and found that the ESIL VM used polish-notation instructions. It looked complicated, but I tried to understand how <code>stp</code> pre-indexing was implemented. I got derailed, so I missed the next r2wars tournament iteration, and didn't upload any update !</p> <p>But, magically, this time, my bot had it much better. Either the other opponents evolved to algorithms that were weaker against this strategy, or the new opponents gave me more points, or simply the randomness of the initial position was more favorable; but anyway, I arrived in <a href="https://twitter.com/T_PAGEFLT/status/1038084656724410369">3rd position</a> this time. I got a prize for being in the top 3, and was very happy. But this was only the beginning.</p> <h2>Stack escape</h2> <p>I noticed that some opponents would continue on living after I had overwritten them. How ? They were running x86-32 code. And it turns out, I was mostly writing zeroes; which are interpreted as valid x86 instructions. This architectures has simply too many valid opcodes.</p> <p>I also noticed that one of the x86-32 opponents, zutle, had a sudden weird bug: it was executing code at 0x01780xxx address. What sorcery was this ? Isn't the arena between 0 and 1024 ? It turns out, that in ESIL, that is the default initialized stack address. And it's executable! Time for a new bot.</p> <p>The idea of this bot, was to copy its 4-instructions payload somewhere on the stack, live there, and wipe the whole arena from there, in a loop: </p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="no">start</span> <span class="nf">mov</span> <span class="no">x3</span><span class="p">,</span> <span class="mi">0x7ff0</span> <span class="nf">movk</span> <span class="no">x3</span><span class="p">,</span> <span class="mi">0x0018</span><span class="p">,</span> <span class="no">lsl</span> <span class="mi">16</span> <span class="nf">ldp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">]</span> <span class="nf">mov</span> <span class="no">x5</span><span class="p">,</span> <span class="mi">0</span> <span class="nf">neg</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x0</span> <span class="nf">mov</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x0</span> <span class="nf">br</span> <span class="no">x3</span> <span class="nl">start:</span> <span class="nf">stp</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x1</span><span class="p">,</span> <span class="p">[</span><span class="no">x5</span><span class="p">],</span> <span class="mi">16</span> <span class="nf">stp</span> <span class="no">x0</span><span class="p">,</span> <span class="no">x1</span><span class="p">,</span> <span class="p">[</span><span class="no">x5</span><span class="p">],</span> <span class="mi">16</span> <span class="nf">and</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x5</span><span class="p">,</span> <span class="mi">0x3ff</span> <span class="nf">b</span> <span class="no">start</span> </pre></div> <p>And this time, it would only write 0xff bytes (well, mostly), to prevent the x86 valid 0x00 opcodes. The strategy is otherwise the same. It starts at 0, and overwrites the whole arena with a post-indexed <code>stp</code>, and the <code>and</code> here checks when the end is reached (a much better idea than my initial <code>csel</code>-based naive approach). And it worked well, in my simulations (with r2 and r2wars from git).</p> <p>But not in the tournament. In the tournament, it turns out that the post-index in the <code>stp</code>s were simply ignored. So the bot escaped to the stack, and then basically waited at the beginning of the address space for the opponent to come being overwritten. Sometimes it would work, sometimes the opponent would kill himself, and sometimes, it would run forever. Well, not forever, since the game has a timeout of 4000 cycles (already reduced from 8000 cycles in the first round). After the timeout, a draw would happen.</p> <p>The timeout shouldn't be an issue, since the timeout usually happens in less than 30 seconds... for an x86 bot. It turns out, ESIL arm64 emulation was much slower than x86, and the timeout felt like it was 10 times longer (I didn't measure). My bot was still the only arm-64 fighter at this stage (there was one arm-32, and one mips, the rest was x86-32), so I was responsible (well, with the ESIL bug in the r2 version of the organizer) for a VERY long tournament. It took ~1h15m instead of the usual ~7m.</p> <p>Of course, I wasn't the only one to have this idea. Konrad had the same idea as he entered the game, and wrote an x86-32 bot with this strategy; our bots didn't meet, so we had a draw. His bot didn't have a buggy emulation, so he had less draws and arrived first. I was second, so I got another prize! Yeah !</p> <p>Special mention to Dimitris who managed to be third here with a strategy that didn't use any stack escape. And he had more wins than my bot ! (I won because I had more draws).</p> <p>Of course, this very long tournament iteration triggered a reaction from the organizers, and <a href="https://github.com/radareorg/r2wars/commit/15905b2860d332ad647dac8b7f9c15ea4c5538b4">execution out of the main arena had to be forbidden</a>. No more stack execution.</p> <h2>Ending</h2> <p>So, with the my last strategy not working anymore, I re-uploaded my first competing bot, and called it a day. I had already won top-3 twice, and I didn't need any more prizes. I tried other strategies for a while, but didn't submit them and came back to fixing ESIL emulation of <code>str</code> pre-index and post-index addressing. I sent a <a href="https://github.com/radare/radare2/pull/11447">pull request to radare</a> once it somehow worked, and went on to follow more conferences.</p> <p>The next round saw my strategy being relatively bad, with <a href="https://twitter.com/T_PAGEFLT/status/1038369497546149889">place 7 over 12</a>. Fighting bots of different architectures is sometimes a disadvantage: x86 has more compact instructions, it has an instruction to dump all registers permitting very wide writes, etc.</p> <p>I still took the opportunity to update it before the last round to write 0xff instead of 0x00 to improve the win-rate against x86 bots:</p> <div class="highlight"><pre><span></span><span class="nf">adr</span> <span class="no">x0</span><span class="p">,</span> <span class="no">start</span> <span class="nf">mov</span> <span class="no">x3</span><span class="p">,</span> <span class="mi">1008</span> <span class="nf">neg</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x0</span> <span class="nf">mov</span> <span class="no">x4</span><span class="p">,</span> <span class="no">x5</span> <span class="nf">ldp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x0</span><span class="p">]</span> <span class="nf">stp</span> <span class="no">x1</span><span class="p">,</span> <span class="no">x2</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">]</span> <span class="nf">br</span> <span class="no">x3</span> <span class="nl">start:</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">stp</span> <span class="no">x5</span><span class="p">,</span> <span class="no">x4</span><span class="p">,</span> <span class="p">[</span><span class="no">x3</span><span class="p">,</span> <span class="p">-</span><span class="mi">16</span><span class="p">]!</span> <span class="nf">b</span> <span class="no">start</span> </pre></div> <p>I really want to thank the organizer of this tournament, skuater. He couldn't be present at r2con since he missed his flight, but still streamed the tournament, reacted to bugs we found, provided very nice support over Telegram, told us when our bots weren't building, etc. Kudos !</p> <p>I didn't get to watch the last tournament of r2wars because I had to leave early. As I finish writing these lines, my plane lands in Paris. I open my phone, to see that <a href="https://github.com/radare/radare2/pull/11447#issuecomment-419655483">my PR was merged</a>. And that Konrad just sent me this picture, showing the results of the last r2wars tournament:</p> <p><img alt="jpg" src="/images/r2con2018/final.jpg"></p>Kernel Recipes 2017 day 3 notes2017-09-29T00:00:00+02:002017-09-29T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2017-09-29:/kernel-recipes-2017-day-3.html<p>This is continuation of <a href="kernel-recipes-2017-day-1.html">day 1</a> and <a href="kernel-recipes-2017-day-2.html">day 2</a> of Kernel Recipes 2017.</p> <h1>Using Linux <code>perf</code> at Netflix</h1> <p>by Brendan Gregg</p> <p>Brendan started with a ZFS on Linux case study, where it was eating 30% of the CPU resources, which it should never be doing. He started by generating a …</p><p>This is continuation of <a href="kernel-recipes-2017-day-1.html">day 1</a> and <a href="kernel-recipes-2017-day-2.html">day 2</a> of Kernel Recipes 2017.</p> <h1>Using Linux <code>perf</code> at Netflix</h1> <p>by Brendan Gregg</p> <p>Brendan started with a ZFS on Linux case study, where it was eating 30% of the CPU resources, which it should never be doing. He started by generating a flame graph with perf, through Netflix's <a href="https://github.com/Netflix/vector">Vector</a> dashboard tool. It was confirmed instantly, despite the initial hunch. This was then quickly thought to be the container teardown cleanup using lots of resources. The only issue here, is that this particular project never used ZFS. It was in fact the free code path trying to get real entropy to free empty lists. It was later fixed in ZFS.</p> <p>A particular point underlined is that when profiling, you want to see everything, from the kernel, to userspace C or Java code. perf allows doing that, because it has no blind spots, is accurate and low overhead.</p> <p>This is useful at Netflix, because they scale the number of instances based on the percentage of CPU usage. At Netflix scale, a small performance improvement might lead to a scale-down saving the company a lot of money. While perf can do many things, Netflix uses it to profile CPU usage 95% of the time.</p> <h2>perf basics</h2> <p>perf originated from implementation of CPU Performance Monitoring Counters (PMCs) in Linux, and supports many features.</p> <p>The main workflow is to do a <code>perf list</code> to look at the available tracepoint events, then <code>perf stat</code> to count particular events. <code>perf record</code> allows capturing and dumping the events to the file system, <code>perf report</code> or <code>perf script</code> is used to analyze a dumped perf data. <code>perf top</code> can be used to look at events in real-time.</p> <p>Brendan maintains a list of <a href="http://www.brendangregg.com/perf.html#OneLiners"><code>perf</code> one-liners</a>, useful to explore and learn about perf capabilities.</p> <p>Brendan came up with <a href="https://github.com/brendangregg/FlameGraph">Flame Graphs</a> when he was profiling a MySQL issue. It's a perl script that converts input data to svg. To use it with <code>perf</code>, use <code>stackcollapse-perf.pl</code> with <code>perf script</code>, and feed the output into <code>flamegraph.pl</code></p> <h2>Gotchas</h2> <p>An important thing is to have working stack traces and symbol resolving working. To fix stack traces you should either use frame-pointer based stack walking, libunwind or DWARF. You probably want <code>-fno-omit-frame-pointer</code> into your gcc option lists for C code. For Java, you might want to use <code>perf-map-agent</code> to do symbol resolution and de-inlining.</p> <p>When you go to instruction-level, the problem is that resolution isn't really precise, so you don't really know which one you're executing. This is because of modern out-of-order CPU architecture. Intel's PEBS helps with this issue.</p> <p>When using VMs, you might want to have you hypervisor (Xen, etc.) enable PMCs for your OS and handle this properly. For containers, <code>perf</code> might have issues finding the symbol files, since they are in a different namespace; this is fixed in 4.14.</p> <p>In conclusion, there's a lot to say about perf, and this talk only scratched the surface of what's possible; Brendan pointed us to the many resources available about it online.</p> <h1>The Serial Device Bus</h1> <p>by Johan Hovold</p> <p>While serial buses are ubiquitous, the TTY layer failed at modeling the associated resources with a serial line.</p> <p>The TTY layer exposes a character device to userspace. It supports line discipline for switch modes, handling errors, etc.</p> <p>It's possible to write drivers on top in userspace, and Johan used gpsd as example of this. But you need to know in advance the associated Port and resources aren't necessary accessible. And you lose the ability to interact with other subsystems in the kernel. Another example of this is bluetooth, where you register further devices (hci0) in order to be able to control the line-discipline and properly initialize ports.</p> <p>To initialize the bluetooth, you use <code>hciattach</code> to configure a tty as bluetooth device, then the hci device appears, and then you use <code>hciconfig</code> to manage this device. The problem with this type of ldisc drivers is that you lose control over some information to userspace, and you don't have the full picture for GPIOs, and other resources for handling power management for example.</p> <h2>Serial Device Bus</h2> <p><code>serdev</code> was originally written by Rob Herring; it was created as bus for UART-attached device. It was merged in 4.11, but enabled in 4.12 follwing some issues.</p> <p>The new bus name is "serial"; it refers to <code>servdev</code> controllers and clients (or slaves). The only controller available is the TTY-port controller. The hardware description happens in the Device Tree.</p> <p>serdev allows a new architecture, with simpler interaction and layering, without the need to have userspace change the mode of a TTY first, since all the necessary data is in the Device Tree. For bluetooth, this would mean hci0 would appear at dt probe time, making it possible to use <code>hciconfig</code> directly.</p> <p>There are currently three bluetooth drivers using this infrastructure in the kernel, as well as one ethernet driver (qca_uart).</p> <p>The main limitation is that it's serial-core only. While it only supports Device Tree, this is being worked on to add ACPI. Hotplug support isn't solved either. Multiplexing for supporting multiple slaves patches have been posted.</p> <h1>eBPF and XDP seen from the eyes of a meerkat</h1> <p>by Éric Leblond</p> <p>Suricata is an open-source Intrusion Detection System that relies on kernel features. It starts with dumping all packets at the IP level with linux raw sockets, then does stream reconstruction and application protocol analysis. It works at 10GB/s in normal use in enterprise networks. It analyses the data, and output JSON, or even a web dashboard.</p> <p>Suricata uses linux raw sockets with <code>AF_PACKET</code> in memory-mapped fan-out mode for multi-threaded processing.</p> <p>One issue Suricata encountered was the asymmetrical hash being changed in Linux 4.2, breaking ordering so that Suricata couldn't properly analyze the streams. This was fixed later in 4.6.</p> <h2>eBPF</h2> <p>eBPF came to the rescue by enabling Suricata to customize the hash function, and then properly tag packets so that they go to the proper thread (load-balanced), hence preserving ordering.</p> <p>Another issue related to load-balancing, is the big flow handling, that is hard to handle without losing packets or ordering. One solution is to discard select packets, by bypassing certain packets as soon as possible in the kernel to reduce performance impact.</p> <p>Suricata implemented a new "stream depth" bypass that allows to start discarding after the flow started, while still capturing the most interesting part at the beginning.</p> <p>For the kernel part of this bypass implementation, nftables did not work because it was too late in the process, after <code>AF_PACKET</code> handling. An eBPF filter using maps helped Suricata achieve this.</p> <p><code>bcc</code> didn't match Suricata requirements, so they used <code>libbpf</code> which is hosted inside the kernel in <code>tools/lib/bpf</code>. Eric says it's easy enough to use.</p> <h2>XDP</h2> <p>The eXtreme Data Path (XDP) project was started to give access to raw packet data from the network card, before it reaches the Linux network subsystem, creating an skb. You can even interact with it using an eBPF filter. This needs modified drivers, and many are already supported; in 4.12 there's even a generic driver usable for development, but less performant.</p> <p>Eric started integrating XDP in Suricata, and found that it meant doing more parsing since it was raw packets. <code>libbpf</code> support isn't done yet either. To hand over the capture to userspace, the strategy is to use the perf event system, with its memory mapped ring buffer.</p> <p>This is still a bit fresh, Eric says, but promising and very efficient.</p> <h1>HDMI CEC Status Report</h1> <p>by Hans Verkuil</p> <p>The Voyager space probe sent in 1977 communicates at 1477 bits per second, and CEC is a bus that communicates at 400 bits per second, making Hans the maintainer of the slowest bus in the Kernel.</p> <p>CEC is an option part of HDMI that provides high level functions and communications for Audio and Video products. It's a 1-line protocol. It has physical addresses, the TV always being 0, and inputs have others. Then there are logical addresses from 0 to 15. </p> <h2>Features</h2> <p>CEC allows waking up, shutting down a device (TV or else), switch sources, getting remote passthrough. You can tell also tell other devices the name of your device. You can also configure the Audio Return Channel (ARC) to send the audio from the sink (TV) to a device through the HDMI Ethernet pins.</p> <p>Inside the kernel, the CEC framework implements most of the features. The drivers only need to implement the low-level CEC adapter operations. It handles core messages automatically, but you can also get them if you enable passthrough. If you need to assemble or decode CEC messages, there's a BSD and GPL-licensed header-only implementation in <code>cec-funcs.h</code> that can be used by applications. The framework driver API is pretty compact and simple to implement.</p> <p>The userspace API has various messages to set a physical or logical address, set the mode of the fd, etc. </p> <p>The Hotplug Detect use case is complex, since it depends on the status of the HDMI Hotplug Detect Pin (HDP). If the pin is down, some devices won't be able to send CEC messages. Some TVs turn off HPD, but still receive CEC messages. Hans says that the most reliable way to wakeup a TV is to just send a message, regardless of the HPD status. It's out-of-spec, but is the only way to make it work.</p> <p><code>cec-ctl</code> is the tool that implements the userspace API and allows interacting with the framework from the command line.</p> <p>In kernel 4.14, many devices are now supported, including the Raspberry Pi. It can now be emulated with the <code>vivid</code> driver. It passed CEC 1.4 and 2.0 compliance tests. This makes Linux the only OS with built-in CEC support, Hans says.</p> <p>In the pipeline, is support for many new devices, as well as a brand new <strong>cec-gpio</strong> driver allowing to do bit-banging of CEC over a GPIO. It also allows injecting errors, but this should come later.</p> <h1>20 years of Linux Virtual Memory</h1> <p>by Andrea Arcangeli</p> <p>Virtual Memory(VM) is practically unlimited and costs virtually nothing, virtual pages point to physical pages, which is the real memory.</p> <p>In x86, the pagetable format is a radix tree. With traditional 3 levels of pages tables you can have 256TiB of memory; with 5-level page tables, you can address 128PiB of memory, but it has a performance impact.</p> <p>The VM algorithms in Linux use heuristics to solve a hard problem of using the memory as best as possible. One such choice is to have overcommit by default. Or to use all free memory as cache.</p> <p>In the VM, the basic structure is struct page. It's currently 64 bytes, and is using 1.56% of all memory in a given system.</p> <p>MM is the memory of a process, and is shared by threads. <code>virtual_memory_area</code> VMA is inside the MM. The LRU cache is combination of two lists of recently used pages, and uses an active and inactive optimum balancing algorithm. The status of those lists is visible in <code>/proc/meminfo</code>.</p> <p>Reverse mapping of the objects (objrmap) is used as well to find reverse references of pages to processes.</p> <p>There are other LRUs for anonymous and file-based mappings, or cgroups.</p> <h2>Trends</h2> <p>Automatic NUMA Balancing helps running various workloads, without having to adapt it to NUMA mode with hard bindings.</p> <p>Transparent Hugepages are a way to automatically use huge pages if an application uses lots of memory, instead of manually with hugetlbfs.</p> <p>The MMU notifier allows reducing page pinning, making it possible to swap-out DMAed memory with proper driver interactions.</p> <p>HMM or Unified Virtual Memory allows going even furthers for GPU and seamless computing, without requiring cache-coherency.</p> <p>Andrea showed auto-NUMA balancing benchmarks, and it improves transactions as much as 10%. A remark from the audience showed that in some pathological cases, the performance might actually be worse, but the feature can be disabled.</p> <h2>Huge Pages</h2> <p>With hugepages, you can go from 4KiB pages to 2MiB pages. This allows completely removing a pagetable level, and thus improving performance in some cases. But it has a cost when clearing pages, making it less cache friendly. In the last case, a huge improvement in performance was seen when clearing the faulting sub-page last, so that it's still in the cache.</p> <p>Transparent Hugepage (THP) works by simply sending 2M pages when the mmap region is 2M aligned, and the request is big enough. It is tunable in <code>/sys/kernel/mm/transparent_hugepage</code>; it can be disabled, enabled only for madvise, or always. The THP defragmentation/compaction is also tunable.</p> <p>Since Linux 4.8, it's possible to use THP with tmpfs and shmem. This is also tunable and disabled by default.</p> <h2>KSM and userfaultfd</h2> <p>Virtual memory deduplication (KSM) is practically unlimited, affecting migration during compaction for example; with KSMscale, a maximum limit is set on per-physical pages dedup, the default is 256, so that a given KSM would only be referenced by 256 virtual pages; this is tunable. Answering a question from the audience, Andrea said that if you care about cross-VM sidechannel attacks, you should probably disable KSM after disabling HyperThreading.</p> <p>userfaultfd allows userspace more visibility and control over page-faulting. It enables postcopy live migration with VMs (efficient snapshotting). It can be used to drop write bits for with JITs, and has many other uses.</p> <p>Andrea concluded that he is amazed with the room for innovation to continue further improvements, after 20 years of working with the Linux memory management.</p> <h1>An introduction to the Linux DRM subsystem</h1> <p>by Maxime Ripard</p> <p><a href="http://free-electrons.com/pub/conferences/2017/kr/ripard-drm/ripard-drm.pdf">Presentation slides</a></p> <p>In the beginning, there was the framebuffer. That's how fbdev was born, to do very basic graphics handling. Then, GPUs came along, getting bigger and bigger. In parallel in the embedded space, piles of hack were accumulated in display engines to accelerate some operations.</p> <p>At first, <em>DRM</em> was only for GPUs' needs, without any kind of modesetting. It required to map device registers to userspace so that it would do it. But since Kernel Mode-Setting (<em>KMS</em>), this has moved back into the kernel.</p> <p>fbdev is now obsolete, and dozens of ARM drm drivers have been merged since 2011.</p> <p>Traditionally in embedded devices, there were two completely different devices for the GPU and the display engine. In Linux, there's the divide between DRM and KMS.</p> <p>KMS has <em>planes</em>, that can be used for double-buffering. It also has the <em>CRTC</em>, that does the composition. <em>Encoders</em> take the raw data from the CRTC, and convert it to a useful hardware bus format (HDMI, VGA). <em>Connectors</em> output the data, handle hotplug events and EDIDs.</p> <p>In the DRM stack, <em>GEM</em> can be used to allocate and share buffers without copy with the kernel. <em>PRIME</em> can interact with <em>GEM</em> and dma-buf to also handle buffers shared with hardware.</p> <p>Vendors also have their own solutions, like ARM's Mali proprietary driver. Blob access for userspace is tightly controlled.</p> <h1>Build farm again</h1> <p>by Willy Tarreau</p> <p>This is a followup of <a href="https://anisse.astier.eu/kernel-recipes-2016-notes.html">last year's</a> <a href="https://lwn.net/Articles/702375/">presentation</a>. The old build farm had shortcomings: it wasn't reliable (HDMI sticks), had a bad power supply, and heating issues. Yet the RK3288 was quite powerful, so Willy wanted to try again with the same CPU.</p> <p>He got 10 MiQi boards, which are even faster thanks to dual-channel DDR3, although still having shortcomings when combining them with foam. Willy fixed the heatsink, by using a 3M thermal tape. Instead of microUSB, Willy simply soldered thicker cables directly on the board. And to solve the switch attrition issue, he tried a Clearfog-A1 board.</p> <p>distcc was updated to the latest version for more flexibility, and bumped settings in order to saturate all the cores on all CPUs. LZO compression helped reducing upload time. He also found that there was a hardcoded limit of 50 parallel jobs in distcc, and fixed it. </p> <p>He improved the distcc distribution using haproxy in front with the leastconn algorithm, this helped a lot.</p> <p>Using the cluster in addition to his local beefy machine, he went from 13 minutes for kernel builds to 4m45s.</p> <p>To help with monitoring, Willy submitted a new <code>led-activity</code> LED trigger for the kernel to change the blinking speed depending on CPU usage.</p> <p>To build haproxy, he went from 11s to 3s with the added farm. With up to 200 builds a day, it saves less than half an hour per day.</p> <p>Feedback was sent to MiQi's maker; patches to distcc. The quest for a good USB power supply continues. Willy is now exploring alternative boards for even faster builds.</p> <p><em>(That's it for Kernel Recipes 2017! See you next year!)</em></p>Kernel Recipes 2017 day 2 notes2017-09-28T00:00:00+02:002017-09-28T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2017-09-28:/kernel-recipes-2017-day-2.html<p>This is continuation of <a href="kernel-recipes-2017-day-1.html">yesterday's live blog</a> of Kernel Recipes 2017.</p> <h1>Linux Kernel Self Protection Project</h1> <p>by Kees Cook</p> <p><a href="http://outflux.net/slides/2017/kr/kspp.pdf">Presentation slides</a></p> <p>The aim of the project is more than protecting the kernel.</p> <h2>Background</h2> <p>Kees' motivation for working on Linux, is the two billion Android devices running a Linux. The majority …</p><p>This is continuation of <a href="kernel-recipes-2017-day-1.html">yesterday's live blog</a> of Kernel Recipes 2017.</p> <h1>Linux Kernel Self Protection Project</h1> <p>by Kees Cook</p> <p><a href="http://outflux.net/slides/2017/kr/kspp.pdf">Presentation slides</a></p> <p>The aim of the project is more than protecting the kernel.</p> <h2>Background</h2> <p>Kees' motivation for working on Linux, is the two billion Android devices running a Linux. The majority of those are running a 3.4 kernel.</p> <p>CVE lifetimes — the time between bug introduction and fix — are pretty long, averaging many years.</p> <p>Kees says the kernel team is fighting bugs, they are finding them, but just doing that isn't enough. The analogy Kees gave was that the Linux security is in the same place the car industry was in the 60s, where most work done was on making sure the car worked, but not necessarily that they were safe.</p> <p>Killing bug classes is better than simply fixing bugs. There's some truth in the upstream philosophy that all bugs might be security bugs. Shutting down exploitation targets and methods is more valuable in the long term, even it has a development cost.</p> <p>Modern exploit chains are built on a series of bugs, and just breaking the chain at one point is enough to stop or delay exploitation.</p> <p>There are many out-of-tree defenses that have existed over the years: PaX/grsec, or many articles presenting novel methods that were never merged upstream. Being out-of-tree is not anything special, since the development mode in Linux is to fork. Distros integrate custom mitigations, like RedHat's ExecShield, Ubuntu's AppArmor, grsecurity or Samsung's Knox for Android.</p> <p>But in the end, upstreaming is the way to go, Kees says. It protects more people, reduces maintenance cost, allowing to focus on new work instead of playing catch-up.</p> <p>Many defenses are the powerful because it's they're not the default, and aren't widly examined. Kees gave an example of custom email server configuration that were very effective to fight spam <em>because</em> they're not the default, otherwise the spammers would adapt.</p> <p>Kees then showed another example with grsecurity, where the stack clash protection was not upstreamed, not reviewed, and was in the end weaker than the solution finally merged upstream.</p> <h2>Kernel self protection project</h2> <p>In 2015, Kees announced this project because he realized he wouldn't be able to do all the upstreaming work by himself. It is now an industry-wide project, with many contributors.</p> <p>There are various type of protections: probabilistic protections reduce the probability of success of an exploit. Deterministic protection completely block an exploitation mechanism.</p> <p>Stack overflow and exhaustion is an example of bug stack that was closed down upstream with vmap stack. Kees is still porting a pax and grsecurity gcc plugin to work on that. The stack canary is essential as well, Kees said. For instance, it mitigates the latest BlueBorne vulnerability.</p> <p>Integer over/underflow protection went inside the kernel with the new refcount patches. Buffer overflows are mitigated upstream through Hardened user copy or recent FORTIFY_SOURCE integration. Format string injection was mitigated in 3.13 when the %n format option was completely removed.</p> <p>Kernel pointer leak isn't entirely plugged, despite various fixes. Uninitialized variable was mitigated through porting of the structleak PaX gcc plugin. Kees says it's more than an infoleak, and this might be exploited in some cases.</p> <p>Use-after-free was mitigated with page zero poisoning in Linux 4.6, and freelist randomization in 4.7 and 4.8.</p> <h2>Exploitation</h2> <p>The basic is to find the kernel in memory (e.g through kernel pointer leaks). To mitigate this, there's various types of kASLR or the ported grsecurity randstruct plugin.</p> <p>A very basic protection is to make sure executable memory cannot be writable, and this was merged for various architectures a long time ago.</p> <p>Function pointer overwrite is a very standard exploitation method, and this was mitigated by the pax constify plugin, and then the ro_after_init annotation in the kernel.</p> <p>Mitigating userspace execution is still a work in progress on x86, but arm64 already fixes for that.</p> <p>The next stages are mitigating user data reuse, and reused code chunks (ROP), PaX has a RAP closed-source technology to do this.</p> <h1>Understanding the Linux Kernel via ftrace</h1> <p>by Steven Rostedt</p> <p>Steven started by saying that this talk is really fast, and you should watch it three times to understand it.</p> <p>Ftrace is an infrastructure with several features. Ftrace is the exact opposite of security hardening: it gives visibility in the kernel, provides instrumentation to do live-kernel patching, and of course rootkits.</p> <p>Ftrace is already in the kernel. It was usually initially interacted with through <code>debugfs</code>, but it now has its own fs, <code>tracefs</code>, mountable in <code>/sys/kernel/tracing</code>. All files and even documentation are in there, so it's usable through echo and cat because Steve wanted that busybox be enough to control these features. This is were the described files are in the rest of the talk.</p> <p>The basic file is <code>trace</code>, showing the raw data. Then there's <code>available_tracers</code>. The default tracer is the <code>nop</code> one, which does nothing. The most interesting one is the <code>function</code> tracer, that shows every called function in the kernel. The most beautiful one, according to Steve is the <code>function_graph</code> tracer that follows the call graph.</p> <p>The <code>tracing_on</code> file controls the writes the ring buffer. Tracing infrastructure is still here, but the ring buffer isn't filled with data. It's there for temporary pauses of tracing.</p> <p>There are few files that allow limiting ftrace to filter the output: <code>set_ftrace_filter</code> for example matches the function names, and supports glob matching, appending, or clearing.</p> <p>The file <code>available_filter_functions</code> shows the available functions; it does not include all kernel functions, depending on gcc instrumentation(inline functions, and annotated non-traceable functions (timers, ftrace itself, boot time code).</p> <p>When using the <code>function</code> tracer, it shows the function calls as well as the parent.</p> <p>The filter file <code>set_ftrace_pid</code> limits function executed by a given task. If you have multiple threads, it's the thread id.</p> <p>To trace syscalls, you need to know that the definition macros add a <code>sys_</code> prefix to the syscall names. If you want to trace the read syscall, you should trace the <code>SyS_read</code> function, because the upper case function comes first. You can find it in the <code>available_filter_functions</code> file.</p> <p>The <code>set_graph_function</code> filter helps when you want to trace starting from a given point, and follow the call graph, accross function pointer boundaries, giving you insight that's harder to get with just the code. Steven gave an example with the <code>sys_read</code> syscall, where you can know exactly which function is called, even when you have the file_operations structure making code reading harder, but the graph is very clear. You can combine this with <code>set_ftrace_notrace</code> to set a boundary of functions or <code>set_graph_notrace</code> for call graphs you're not interested in, to ease reading the call graph and reduce the ftrace performance impact.</p> <p>There are many options in the <code>options</code> directory or the <code>trace_options</code> file. Steven likes the <code>func_stack_trace</code> option: it creates a stack trace of traced functions. Be careful, if you don't set a filter, it's going to bring your machine to a knee. Also remember to turn it off when done. <code>sym_offset</code> or <code>sym_addr</code> options show the function relative and absolute locations in memory.</p> <p>When you set a filter starting with <code>:mod:module_name</code>, it will trace all the functions in a given module.</p> <p>Function triggers are useful when you want a start a tracing, stop tracing, or even add a stacktrace when a function is it. For example you do set a filter with <code>function_name:stacktrace</code>, and it will give you stacktrace everytime this particular function is called.</p> <p>When interrupted, you might not want to see the interrupt function graph: there's a default-on option <code>funcgraph-irqs</code> that does just that if you turn it off.</p> <p>It's possible to limit the graph depth of the <code>function_graph</code> tracers with the <code>max_graph_depth</code> option.</p> <p>You can also trace with events. The events are listed by subsystems in the <code>events</code> directory. The most commonly used ones are <code>sched</code>, <code>irq</code> or <code>timer</code> families of events. You enable events separately of the specific tracers. If you only want events, use the <code>nop</code> tracer, but this can be combined with the others.</p> <p>There are two useful options to control event and function tracing: <code>event-fork</code> and <code>function-fork</code> allow to continue tracing children of a traced process.</p> <p>Finally, Steve introduced the <code>trace-cmd</code> program, that wraps all the custom <code>echo</code>s and <code>cat</code>s in a single program. <code>trace-cmd</code> has nice tricks to make sure you only stack-trace a single function, and can do all you can do without it with a simpler interface.</p> <h1>Introduction to Generic PM domains</h1> <p>by Kevin Hilman</p> <p>Two years ago, Kevin did an introduction on various power management subsystems at Kernel Recipes. This talk focuses on PM domains.</p> <p>The driver model starts with the <code>struct dev_pm_ops</code>. You control the global system suspend through <code>/sys/power/state</code>, and this then calls the appropriate driver callbacks. It's very powerful, but also fragile since any driver failing will stop the whole chain. This is static power management or system-wide suspend.</p> <p>The focus of this talk is the Dynamic power management, in particular for devices.</p> <h2>Dynamic power management</h2> <p>It starts with runtime PM, a per-device idle mode, one device at a time. It's handled by the driver, based on activity. In this mode, devices are independent, and one device cannot affect other drivers. When using <code>powertop</code>, the "device stats" tell you how long your device is idle.</p> <p>The runtime PM core keeps a usage count for driver uses. When the count hits 0, the core calls runtime_suspend on a device. If you have a device on a bus_type, it sits between you and the runtime PM core. In driver callbacks, one can ensure context is saved, and the wakeups are enabled, restore context on resume, etc.</p> <p>PM domains map the architecture of power domains inside modern SoCs, where various hardware blocks are grouped in domains that can be turned on and off independently, to the Linux kernel.</p> <p>PM domains are similar to bus types in the kernel, but orthogonal since some devices might be in the same domain but different buses.</p> <h2>genpd</h2> <p>Generic PM domains (genpd) are the reference implementation of PM domains, to be able to do the grouping and actions when a device becomes idle or active.</p> <p>In order to implement a genpd, you first implement the power_on/power_off function. It's typically messaging a power domain controller on a separate core, but might be related to clock management or voltage regulators. This is then described in a Device Tree node, allowing to reorder domains for different chip revisions.</p> <p>Power domains have a notion of governors, allowing custom decision making before cutting power. It allows flexibility relative to the ramp up/down delays for example. It is usually implemented in the genpd, but there are two built-in governors like Always-on or Simple QoS governors. You can attach runtime system-wide or per-device QoS constraints to control the governors.</p> <p>There has been a lot of work recently upstream, like IRQ-safe domains, or always-on domains. Statistics and debug instrumentations were also added recently.</p> <p>Under discussion is a way to unify CPU and devices power domain management. Upstream is also interested in having a better interaction between static and runtime PM. Support for more complex domains, in order to have the same driver for an IP block whether it's used through ACPI or genpds, is still in the works.</p> <h1>Performance Analysis Superpowers with Linux BPF</h1> <p>by Brendan Gregg</p> <p><a href="https://www.slideshare.net/brendangregg/kernel-recipes-2017-performance-analysis-with-bpf">Presentation slides</a></p> <p>Boldly starting the presentation with a demo, Brendan showed how to analyze how top works, with <code>funccount</code> and <code>funcslower</code>, <code>kprobe</code>, <code>funcgraph</code> and other ftrace-based tools he wrote.</p> <p>He then switched to an eBPF frontend called <code>trace</code>, that was used to dig into the arguments of a kernel function. You can leverage eBPF even more with other tools like <code>execsnoop</code> or <code>ext4dist</code>.</p> <h2>eBPF and bcc</h2> <p>BPF comes from network filtering, originally used with tcpdump. It's a virtual machine in the kernel.</p> <p>BPF sources can be tracepoints, kprobes, or uprobes. It uses the perf event rig buffer for efficiency. You can use maps as an associative array inside the kernel. The general tracing philosophy is to have a very precise filter to only get the data you need, instead of dumping all the data in userspace, and filtering it later.</p> <p>Many features were added recently to eBPF, and it keeps being improved.</p> <p>BPF Compiler Collection (BCC) is the most used BPF frontend. It allows you to write BPF program in C instead of assembly, and load the programs. You can then combine this with a python userspace.</p> <p><a href="https://github.com/ajor/bpftrace"><code>bpftrace</code></a> is a new in-development frontend, with a simple-to-use philosophy.</p> <p>Installing bcc on your distro is becoming easier as it gets packaged. There are <a href="https://github.com/iovisor/bcc#tools">many tools</a>, each with a different use giving visibility into a different kernel part.</p> <p>Heatmaps are very useful to visualize event distribution. Flamegraphs are also very powerful when combined with kernel stacktraces generation. It's now even possible to merge userspace and kernelspace stacktraces for analysis.</p> <h2>Future work</h2> <p>Support for higher level languages to write BPF programs like <code>ply</code> or <code>bpftrace</code> is in progress.</p> <p>In conclusion, eBPF is very useful to understand Linux internals, and you should use it.</p> <h1>Kernel ABI Specification</h1> <p>by Sasha Levin</p> <p>What's an ABI ? ioctls, syscalls, and the vDSO are examples of the Linux ABI.</p> <p>Sasha repeated the ABI promise from Greg's talk yesterday. The issue, he says, is that kernel lacks tools to detect a broken ABI.</p> <p>Sometimes basic syscall argument checks are forgotten, and discovered as a security vulnerability. Sometimes, some interfaces have undefined behaviour, making the ABI stability uncertain.</p> <p>Breakage is sometimes difficult to fix when detected late, because new userspace might depend on the new behaviour.</p> <p>In the end, some userspace programs like glibc, strace, or syzkaller might rewrite their understanding of the kernel ABI, and those might be out of sync. Man pages might not document everything either, and they're not a real documentation of the ABI Contract.</p> <h2>ABI Contract</h2> <p>Right now it's in the form of kernel code. Unfortunately, code evolves, so it's not an optimal format for this.</p> <p>The goal is to fix many issues at the same time: ensure backwards compatibility, prevent kernel to userspace errors, document the contract, and encourage re-use. Sasha looked for a format that would only require writing this once, and be machine readable. <code>syzkaller</code>'s description looked like a good starting point. He wanted this to be reusable by userspace tools that need this information. And finally, he wanted to use this as a tool to help ABI fixes and fast breakage detection.</p> <p>It also helps re-assuring the distribution that the ABI promise is really kept. In Sasha's view, it would also greatly help the security aspect of things, since the ABI is the main interface by which the kernel is attacked.</p> <p>The hard part is to determine the format of this contract, document all syscalls and ioctls and write the tools to test it out.</p> <p>Sasha already started with a few system calls, and is currently looking for help to get the ball rolling.</p> <h1>Lightning Wireguard talk</h1> <p>by Jason A. Donenfeld</p> <p>Jason's background is in breaking VPNs. He wanted to create one that was more secure. That's how Wireguard was born.</p> <p>Wireguard is UDP based, and uses modern cryptographic principles. The goals is to make it simple and auditable. To prove his point, he showed that it clocks at 3900 lines of code, while OpenVPN , Strongswan or SoftEther have between 116730 and 405894 lines of code each.</p> <p>It uses normal interfaces, added through the standard <code>ip</code> tool. Jason says it's blasphemous because it breaks through the layering assumptions barriers, as opposed to IPsec for example.</p> <p>A given interface has a 1 to N mapping between Public keys and IP addresses representing the peers. To configure the cryptokey routing, you use the <code>wg</code> tool for now. Once merged, the intention to have this merged into the <code>iproute</code> project.</p> <p>In Wireguard, the interface appears stateless, while under the hood, session state, connections are handled transparently.</p> <p>The key distribution between peers is left to userspace.</p> <p>Wireguard works well with network namespaces. You can for example limit a container to only communicate through a wireguard interface.</p> <p>As a design principle, wireguard has no parsing. It also won't interact at all with unauthenticated packets, making it un-scannable unless you have the proper peer private key.</p> <p>Under the hood, it uses the Noise Protocol Framework (used by Whatsapp) by Trevor Perrin, with modern algorithms like Chacha20, Blake2s, etc. It lacks crypto agility, but support a transition path.</p> <p>To conclude, Jason says that Wireguard is the fastest, and lowest latency available VPN out there.</p> <h1>Modern Key Management with GPG</h1> <p>by Werner Koch</p> <h2>What's new</h2> <p>GnuPG 2.2 was released a few weeks ago, while 2.1 has been around for nearly 3 years. There's now easy key discovery going through key servers to search keys associated with an email address.</p> <p>You can now use gpg-agent over the network, so that you don't have to upload your private keys to a server.</p> <p>In the pipeline for version 2.3 is SHA2 fingerprinting, an AEAD mode, and new default algorithms. The goal is also to help upper applications to integrated GPG in there projects. Werner says he also wants to make the Gnuk hardware open usb token easier to buy in Europe. Improving documentation is also planned.</p> <p>GPG will be moving to ECC. While this is a well researched-field, some curves (specific ECC implementation) have a pretty bad reputation according to Werner, and some of those are required by NIST, or European standards. The new de-facto standard curves are Curve25519 and Curve448-Goldilocks.</p> <p>An advantage of ECC key signatures is that they are much shorter than RSA signature, and faster to compute for signing. Verification is slower though.</p> <h2>User experience</h2> <p>The command line interface is being improved with new <code>--quick-</code> options, that are simpler to use. There's now a quick command to generate a key, update the expiration time, add subkeys, update your email address (uid), revoke the old address, sign key, verify a key locally for key signing parties.</p> <p>The main issue with key servers is that they can't map an address to a key. Anyone can publish a key with a given email. The proper way to handle this is through the email server, but this isn't solved yet. Werner's opinion is that the Web-of-Trust is a too complex tool, he believes that Trust On First Use (TOFU) is a better paradigm.</p> <p>There are two GPG interfaces: one for humans, and one for scripting. You should always use the scripting ones with you programs, it's more stable.</p> <p>There are now import/export filters in GPG to reduce the size impact of keys with lots of signatures.</p> <p>You can now <code>ssh-add</code> keys into the <code>gpg-agent</code>. Only caveat, is that in this case, GnuPG is storing the key forever in its private key directory instead of just in memory.</p> <p>In conclusion, GPG isn't set in stone, and it keeps improving and evolving. The algorithms, user interface, scriptability are getting better.</p> <p><em>(That's it for today ! Continue reading on the <a href="kernel-recipes-2017-day-3">last day</a> !)</em></p>Kernel Recipes 2017 day 1 live-blog2017-09-27T00:00:00+02:002017-09-27T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2017-09-27:/kernel-recipes-2017-day-1.html<p>Following <a href="kernel-recipes-2016-notes.html">last year attempt</a>, I'm doing a live blog of Kernel Recipes 6th edition. There's also a <a href="https://air.mozilla.org/embedded-recipes-27-sept-morning/">live stream at Air Mozilla</a></p> <h1>What's new in the world of storage for Linux</h1> <p>by Jens Axboe</p> <p>Jens started with the status of blk-mq conversions: most drivers are now converted: stec, nbd, MMC …</p><p>Following <a href="kernel-recipes-2016-notes.html">last year attempt</a>, I'm doing a live blog of Kernel Recipes 6th edition. There's also a <a href="https://air.mozilla.org/embedded-recipes-27-sept-morning/">live stream at Air Mozilla</a></p> <h1>What's new in the world of storage for Linux</h1> <p>by Jens Axboe</p> <p>Jens started with the status of blk-mq conversions: most drivers are now converted: stec, nbd, MMC, scsi-mq, ciss. There are about 15 drivers left, but Jens says it isn't over until floppy.c is converted, re-offering the prize he offered two years ago.</p> <p>blk-mq scheduling was the only missing feature, in order to tag I/O request, have better flush handling, or help with scalability. To address this, blk-mq-sched was added in 4.11, with the "none" and "mq-deadline" algorithms. 4.12 saw the addition of BFQ and Kyber algorithms.</p> <p>Writeback throttling is a feature to prevent overwhelming the device with request, to keep peak performance high. It was inspired by the networking Codel algorithm. It was tested with io.go, and proven to improve latency tremendously on both NVMe and hard-drives.</p> <p>IO polling helps getting faster completion times, but it has a high CPU cost. A hybrid polling as added, adding predictive algorithms in the kernel to be able to wakeup the driver just before the IO completes. The kernel tracks IO completion time, and just sleeps for half the mean, allowing both fast completion time, and less CPU load leading to better power management. This is configurable through sysfs, with the proper fd configuration. Results show that adaptive polling is comparable in completion times with active polling, but with half the CPU cost.</p> <p>Faster O_DIRECT and Faster IO accounting were also worked on. IO accounting used to be invisible in profiling, but with the huge scaling efforts of the IO stack, it started showing at 1-2% in testing. In synthetic tests, disabling iostat started improving performance greatly. It was rewritten and merged in 4.14.</p> <p>A new mechanism called Write lifetime hints allows application to signal expected write lifetime with fcntl. It allows giving hint to flash based storage (supported in NVMe 1.3), of the total size of the write, making sure you won't get such a big write amplification associated with the internal Flash Translation Layer (FTL), when you do big writes. The device might make more intelligent decisions, better garbage collection internally. It showed improvements with RocksDB benchmarks.</p> <p>IO throttling was initially tied to CFQ, which isn't ideal with the new blk-mq framework. It now scales better on SSDs, supports cgroup2, and was merged for 4.10.</p> <p>Jens came back to a slide of 2015 Kernel Recipes were he predicted the future work, and all the feature previously discussed in this talk were completed in the two-year timespan.</p> <p>In the future, IO determinism is going to be focus of work, as well as continuous performance improvements.</p> <h1>Testing on device with LAVA</h1> <p>by Olivier Crête</p> <p>Continuous integration is as simple as "merge early, merge often" Olivier says. But the core of the value is more in Continuous Testing, and that's what most people think when they say CI.</p> <p>Upstream kernel code is properly reviewed, so why should it be tested, Olivier asked. Unfortunately, arm boards aren't easy to test, so the kernel used to rely on users to do the testing.</p> <p>That's until kernelci.org came along, doing thousands of compiles and boots every day, catching a lot of problems. kernelci.org is very good at breadth of testing, but not depth. If you have any serious project, you should do your own testing, with your own hardware and patches.</p> <p>Unfortunately, automation isn't ubiquitous, because the perceived value is low compared to cost. To overcome this, the first thing to have is a standardized build, single click build system, with no manual operation. The build infrastructure should be the same for everyone, and Olivier recommends using docker images.</p> <p>The second step is to close the CI loop, which is sending automated messages to the developer on failure as soon as possible. Public infrastructure in Gitlab, github or phabricator have support for CI, as well as blocking merging of anything that breaks the build.</p> <h2>LAVA</h2> <p>Linaro Automation and Validation Architecture (LAVA) is not a CI system. It just focuses on board management, making testing them easier. It can install images, do power control, supports serial, ssh, etc. It's packaged for Debian and has docker images available. It should be combined with CI system like Jenkins.</p> <p>The first thing to have is to have a way to Power on/off a board. You can find various power switch relay boards from APC, Energenie, devantech, or even other USB relays.</p> <p>LAVA supports different bootloaders: u-boot, fastboot, and others. The best strategy is to configure the bootloader for network booting.</p> <p>Lava is configured with a jinja2 template format, where you set various variables for the commands you need to connect to, reset, power on/off the board.</p> <p>Tests are defined by YAML files, and can be submitted directly through the API or via command line tools like lava-tool, lqa, etc. You specify the name of the job, timeouts, visibility, priority, and a list of actions to do.</p> <h2>Conclusion</h2> <p>You should do CI, Olivier says. It requires a one-time investment, and saves a lot of time in the end. According to Olivier, from nothing, a LAVA+Jenkins setup is at most two days of work. Adding a new board to an infrastructure, is done in one or two hours.</p> <h1>Container FS interfaces</h1> <p>by James Bottomley</p> <p>After an introduction on virtualization, hypervisor OSes. Within linux, there are two hypervisor OSes: Xen and kvm. Both use Qemu to emulate most devices, but they differ in approach. Xen introduced para-virtualization, modifying the OS to enhance emulation. But hardware advancements killed para-virt, except in a few devices. In James' opinion, the time lost in working with paravirt in Linux made it lose the enterprise virtualization market to VMWare.</p> <p>Container "guests" just run on the same kernel: there is one kernel that sees everything. The disadvantage is that you can't really run Windows on Linux.</p> <p>The container interface is mostly cgroups and namespaces. There are label-based namespaces, the first one being the network namespace. There are mapping namespace, mapping some resources to somewhere else, allowing those to be seen differently, like the PID namespace, which can map a given PID on the host to be PID 1 inside the container.</p> <p>Containers are used in Mesos, LXC, docker, and they all use the same cgroups and namespaces standard kernel API. There many sorts of cgroups(block IO, CPU, devices, etc.), but aren't a focus of the talk. James intends to focus on namespaces instead.</p> <p>James claims that you don't need any of the "user-friendly" systems, and you can just use the clone, unshare, and standard kernel syscall API to configure namespaces.</p> <h2>Namespaces</h2> <p>User namespaces are the tying it all together, allowing to run as root inside a contained environment. When you buy a machine in the cloud, you expect to run stuff on it as root. Since they give enhanced privileges to the user, the user namespaces were unfortunately the source of a lot of exploits, although there weren't any serious security breach recently since 3.14, James said.</p> <p>User namespaces also maps uids; in Linux, the shadow-utils provides a newuidmap and newgidmap for this. The user namespace hides unmapped uids, so they are inaccessible, even to "root" in the namespace. This creates an issue since a container image will mostly have the files with uid 0, which then should be mapped to the real kuid, and the fsuid accross the userspace/kernel/storage boundary.</p> <p>In kernel 4.8, the superblock namespace was added to allow plugging a usb key or running a FUSE driver in a container. But to be useful, you need a superblock, which isn't useful with bind maps, because you only have one superblock per underlying device.</p> <p>The mount namespace works by cloning the tree of mounts when you do <code>unshare --mount</code>; at first it's identical to the original one, but once you modify it it's different. But, all the modified mounts point to the same refcounted super_block structure. It might create issues when you add new mounts inside a sub-namespace, then this locks the other refcounted super_blocks from the host until you can umount the new mount, like the usb key you plugged in your container, that completely locks the mount namespace trees.</p> <p>James then did a demo, showing with <code>unshare</code> that if you first create a user namespace, you can then create mount namespaces, despite being unable to do it before entering the user namespace. It shows how you can elevate you privileges with user namespaces, despite not being root, from an outside view.</p> <p>It was then showed how you can create a file that is really owned by root by manipulating the mount points inside the user/mount namespace by using marks with shiftfs.</p> <p>shiftfs isn't yet upstream, and other alternatives are being explored to solve the issues brought by the container world.</p> <h1>Refactoring the Linux kernel</h1> <p>by Thomas Gleixner</p> <p>The main motivation for Thomas' refactoring over the years was to get the RT patch in the kernel, and to get rid of the annoyances.</p> <h2>CPU Hotplug</h2> <p>One of his pet peeves is the CPU hotplug infrastructure. At first, the notifier design was simple enough for the needs, but it had its quirks, like the uninstrumented locking evading lockdep, or the obscure ordering requirements.</p> <p>While CPU hotplug was known to be fragile, people kept applying duct tape on top of it, which just broke down when the RT patch started adding hotplug support. After ten years, in 2012, Thomas attempted to rewrite it but ran out of spare time. He picked it up again in 2015 and it was finalized in 2017.</p> <p>It started by analysing all notifiers, and adding instrumentation and documentation in order to explicit the order requirements. Then, one by one the notifiers were converted to states.</p> <p>The biggest rework, was that of the locking. Adding lockdep coverage unearthed at least 25 deadlock bugs, and running Steven Rostedt's cpu-hotplug stress test tool could find one in less than 10 minutes. Answering a question from Ben Hutchings in the audience, Thomas said that these fixes are unfortunately very hard to backport, leaving old kernel with the races and locks.</p> <p>The lessons learned are that if you find a bug, you expected to fix them. Don't rely on upstream to do that for you. There's a lot of bad code in the kernel, so don't assume you've seen the worse yet. You also shouldn't give up if you have to rewrite more things. Estimation in this context is very hard, and the original estimation of task was off by factor of three. In the end, the whole refactoring took 2 years, with about 500 patches in total.</p> <h2>Timer wheel</h2> <p>Its base concept was implemented in 1997, and extended over time. The purpose initially the base for all sort of timers, mostly for timeouts after 2005.</p> <p>Those timeouts aren't triggered most of the time, but re-cascading them caused a lot of performance issues for timers that would get canceled immediately after re-cascading. This is a process that holds a spin-lock with interrupts disabled, and therefore very costly.</p> <p>It took a 3 month effort to analyze the problem, then 2 month for a design and POC phase, followed by 1 month for implementation, posting and review process. Some enhancements are still in-flight.</p> <p>The conversion was mostly smooth, except for a userspace visible regression that was detected 1 year after the code was merged upstream.</p> <p>The takeout of this refactoring is to be prepared to do palaeontological research; don't expect anyone to know anything, or even care. And finally, be prepared for late surprises.</p> <h2>Useful tools</h2> <p>Git is the absolute necessary tool for this work, with grep/log and blame. And if you need to dig through historical code, use the tglx/history merged repository.</p> <p>Coccinelle is also very useful, but it's a bit hard to learn and remember the syntax.</p> <p>Mail archives are very useful, but they need to be searchable, as well as quilt, ctags, and of course a good espresso machine.</p> <p>In the end, this isn't for the faint of heart says Thomas. But it brings a lot of understanding on kernel history. It also gives you the skill to understand undocumented code. The hardest part is to fight the "it worked well until now" mentality. But, it is fun, for some definition of fun.</p> <h1>What's inside the input stack ?</h1> <p>by Benjamin Tissoires</p> <p>Why talk about input, isn't it working already, Benjamin asked. But the hardware makers are creative, and keep creating new devices with questionable designs.</p> <p>The usages keep evolving as well, with the ubiquitous move to touchscreen devices for example.</p> <h2>Components</h2> <p>The kernel knows about hardware protocols(HID), talks over USB, and sends evdev events to userspace.</p> <p>libinput was created on top of libevdev "because input is easy"; but it keeps being enhanced after three years, showing the simplicity of the task. It handles fancy things like gestures.</p> <p>The toolkits use libevdev, but they also handle gestures because of different touchscreen use cases.</p> <p>On top of that, the apps use toolkits.</p> <h2>The goood, bad and ugly</h2> <p>Keyboards are mostly working, so it's good. Except for that Caps Lock LED in a TTY being broken since UTF-8 support isn't in the kernel.</p> <p>Mice are old too, so they are a solved problem. Except for those featureful gaming mice, for which the libratbag project was created to configure all the fancy features.</p> <p>Most touchpads are still using PS/2, but extending the protocol to add support for more fingers. On Windows, the touchpads communicate over i2c (in addition to PS/2). Sometimes the i2c enumeration goes through PS/2, but other times through UEFI.</p> <h2>Security</h2> <p>There were a few security issues, with an issue on Chromebook where they allowed the webapp to inject HID events through the uhid driver, and this enabled exploiting a buffer overflow in the kernel.</p> <p>In 2016, the <a href="http://www.mousejack.com/">MouseJack</a> vulnerability enabled remotely hacking wireless mouses. Which meant you could remotely send key events to a computer. You could also force a device to connect to your receiver. A receiver firmware update was pushed through gnome software for Logitech mouses.</p> <h1>Linux Kernel Release Model</h1> <p>by Greg Kroah-Hartman <a href="https://github.com/gregkh/presentation-release-model">Slides</a></p> <p>While the kernel has 24.7M lines of code in more than 60k files, you only run a small percentage of that at a given time. There's a lot of contributors, and a lot of changes per hour. The rate of change is in fact accelerating.</p> <p>This is something downstream companies don't realize. They're getting behind faster than ever when not working with upstream.</p> <p>The release model is now that there's a new release every 2 or 3 months. All releases are stable. This time-based release model works really well.</p> <p>The "Cambridge Promise", is that the kernel will never break userspace. On purpose. This promise was formalised in 2007, and kept as best as possible.</p> <p>Version numbers mean nothing. Greg predict that every 4 years, the first number will be incremented, so that's we might see Linux 5.0 in 2019.</p> <p>The stable kernels are branched after each releases. They have publicly documented rules for what is merged, the most important one is that a patch has to be Linus' tree.</p> <p>Longterm kernels are special stable versions, selected once a year, that are maintained for at least 2 years. This rule is now even applied by Google for every future Android device. This makes Greg thinks he might want to maintain some of those kernels for a longer time. Since people care, the longterm kernels also have a higher rate of bugfixes.</p> <p>Greg says you should always have a mechanism to update your kernel (and OS). What if you can't ? Blame your SoC provider. He took for example a Pixel phone, where there's a 2.8M patch to mainline, for a total of 3.2M lines of running code. 88% of the running code isn't reviewed. It's very hard to maintain and update.</p> <p>Greg's stance is that all bugs can eventually be a "security" issue. Even a benign fix might become a security fix years later once someone realizes the security implications. Which is why you should always update to your latest stable kernel, and apply fixes as soon as possible.</p> <p>In conclusion, Greg says to take <strong>all</strong> stable kernel updates, and enable hardening features. If you don't use a stable/longterm kernel, your device is insecure.</p> <h1>Lightning talks</h1> <h2>Fixing Coverity Bugs in the Linux Kernel</h2> <p>by Gustavo A. R. Silva</p> <p>Coverity is a static source code analyzer. There are currently around 6000 issues reported by the tool for the Linux kernel; those are sorted in different categories.</p> <p>The first category is illegal memory access, followed by the medium category.</p> <p>Gustavo first worked on a missing break in a switch in the usbtest driver. Gustavo sent first a patch to fix the issue, then a second one to refactor the code following advices from the maintainer.</p> <p>Then he worked on arguments sent in the wrong order in scsi drivers. Following was an uninitialized scalar variable, and others. Gustavo showed many examples with obvious commenting or logic bugs. </p> <p>Tracking exactly which bugs were fixed was really useful to take note of similar issues. He sent in total more than 200 patches in three months, in twenty-six different subsystems.</p> <h2>Software Heritage: Our Software Commons, Forever</h2> <p>by Nicolas Dandrimont</p> <p>Open Source Software is important, Nicolas says. Its history is part of our heritage.</p> <p>Code disappears all the time, whether maliciously, or when a service like Google Code is shut down. </p> <p>Software Heritage is a project an open project to preserve all the open source code ever available. The main targets are VCS repositories, and source code releases. Everything is archived in the most (VCS)agnostic data model possible.</p> <p>The project heritage fetches the source code from many sources, and then deduplicates it using a Merkle tree. There are currently 3.7B source files from 65M projects. It's already the richest source code archive available, and growing daily.</p> <p>How to store all of this on a limited budget (100k€ hw budget). It all fits in a single (big) machine. The metadata is stored in PostGres, the files are in filesystems. XFS was selected, and they hit the bottlenecks pretty quickly.</p> <p>They are thinking of moving to scale-out object storage system like Ceph. The project wants to lower the bar for anyone wanting to do the same thing. They also have plans to use more recent filesystem features.</p> <p>Software Heritage is currently looking for contributors, sponsors, for this project.</p> <p><em>(That's it for day 1! Continued on <a href="kernel-recipes-2017-day-2.html">day 2</a> and <a href="kernel-recipes-2017-day-3.html">day 3</a>!)</em></p>Embedded Recipes 2017 notes2017-09-26T00:00:00+02:002017-09-26T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2017-09-26:/embedded-recipes-2017-live-blog.html<p>Following <a href="kernel-recipes-2016-notes.html">last year attempt</a>, I'm doing a live blog of Embedded Recipes 1st edition.</p> <p><img alt="jpg" src="/images/er2017/01-Anne.jpg"></p> <h1>Understanding SCHED_DEADLINE</h1> <p>by Steven Rostedt</p> <p>Every task starts as SCHED_OTHER, where each task gets a fair share of the CPU bandwidth.</p> <p>Then comes SCHED_FIFO, where it's first in, first out, a task will run until it …</p><p>Following <a href="kernel-recipes-2016-notes.html">last year attempt</a>, I'm doing a live blog of Embedded Recipes 1st edition.</p> <p><img alt="jpg" src="/images/er2017/01-Anne.jpg"></p> <h1>Understanding SCHED_DEADLINE</h1> <p>by Steven Rostedt</p> <p>Every task starts as SCHED_OTHER, where each task gets a fair share of the CPU bandwidth.</p> <p>Then comes SCHED_FIFO, where it's first in, first out, a task will run until it gives up the CPU. SCHED_RR shouldn't be used said SCHED_FIFO because it works between tasks of the same priority.</p> <p>Steve gave an example of a machine that runs two tasks, one of a nuclear power plant, and one of a washing machine. The point it to show that priorities should be thought of in a system-wide view when using Rate Monotonic Scheduling. It's not as simple as which task is most important.</p> <h2>Earliest Deadline First (EDF)</h2> <p>Earliest deadline first solves some of the issues of RMS, by allowing to run times without missing deadlines.</p> <p>Steve explained that sched_yield should never be used because it's almost always buggy. Except when using SCHED_DEADLINE of course, where it can be useful.</p> <p><img alt="jpg" src="/images/er2017/02-Steven.jpg"></p> <h2>Multi processors</h2> <p>Steve then introduced a simple example to show Dhall's effect. It shows you can't get over utilization of 1 when using EDF.</p> <p>If you want to partition EDF, it becomes similar to the packing problem, which is NP complete. A solution is to use global EDF, which constrains the problem, but can solve a special case, and get more than 1 of utilization when using multiple processors.</p> <h2>The limits of SCHED_DEADLINE</h2> <p>It has to run on all CPUs.</p> <p>It can not fork, because the tasks has been fixed.</p> <p>It's very hard to calculate the worst case execution time(WCET), and if you get it wrong, it breaks.</p> <p>Using cgroups, it's possible to configure SCHEAD_DEADLINE affinity, but it's still a long series of commands, and stuff in /proc to do, but this is being worked on, Steve says.</p> <p>It's possible to use Greedy Reclaim of Unused Bandwidth (GRUB) in order to utilize bandwidth left by some tasks, leaving more leeway to deal with WCET.</p> <h1>Proper APIs to HW video accelerators</h1> <p>by Olivier Crête</p> <p>There are various types of codecs: software, hardware, and then hardware accelerators. The last ones are the subject of Olivier's talk.</p> <p>Codecs can be used in a variety of contexts: players, encoders, streamers, transcoders, VoIP systems, content creation software, etc.</p> <p>The different use cases have different requirements: Broadcast production want high quality, and user generated content will have lower quality for example.</p> <p>Video calls care mostly about latency. When transcoding, you might care about latency if you're live, or about quality per bit if you want to store it.</p> <h2>Requirements</h2> <p>Exchange formats on the encoded side might need to support variance in packetization, byte stream, etc.</p> <p>The raw content might have different subsambling, color space, etc.</p> <p>Then, the memory layout might vary as well: is it planar (RGBRGBRGB) or packed (RRGGBBRRGGBB) ? Are there multiple planes (multiple DMAbuf fds )? Do you have alignment requirements in memory ? You might have tiled formats, with different tiled formats, compressed in-memory formats, padding, etc.</p> <p>The memory allocation can be internal or external. In Linux, you mostly care about DMAbuf.</p> <p>There might be attached metadata, per-frame like timestamps, or Vertical Ancillary Data (VANC): AfD. Inter-frame data like SCTE-35/104, like ad insertion points.</p> <p>A good API, Olivier says, should support push or pull modes for different uses. A good API should be a living, maintained project, with Open Source code as opposed to being just a specification.</p> <p><img alt="jpg" src="/images/er2017/03-Olivier.jpg"></p> <h2>Existing (wrong) solutions</h2> <p>OpenMAX IL is everywhere because it's required by Android. But no one implements the full OpenMAX, only the Android subset, validated through CTS. The spec isn't maintained at Khronos anymore, and the last library passing the full test suite was from 2011. It's a fragmented landscape.</p> <p>OpenMAX has a specific threading and allocation model. The whole framework isn't a good API according to Olivier.</p> <p>libv4l is a "transparent" wrapper over the kernel API, but it's tied to the kernel API rules, and has limited maintenance, Olivier says.</p> <p>VA-API is more interesting, albeit Intel-specific. Still, it requires complex code, and is video-only.</p> <p>GStreamer is a whole multimedia framework, with a specific way of working(threads, allocation, etc.), not a HW acceleration API. It's not designed for low latency.</p> <p>FFmpeg/libav is kind of OK, Olivier says, but is not focused on the hardware side. MFT on Windows is close to what Olivier is seeking, but tied to Windows.</p> <h2>Simple Plugin API (SPA)</h2> <p>This is library, coming from Pipewire, matches all of Olivier's requirements: no pipeline, no framework, registered buffers, synchronous or asynchronous modes, externally-provided thread contexts, and not limited to codecs.</p> <p>It's available on github:</p> <p>https://github.com/PipeWire/pipewire/tree/master/spa</p> <p>It can work outside PipeWire, although it hasn't been picked-up elswhere yet.</p> <h1>Introduction to the Yocto Project - OpenEmbedded-core</h1> <p>by Mylène Josserand</p> <h2>Why use a build system ?</h2> <p>There are many constraints in embedded systems to match. You can try building everything manually, but despite the flexibility, it's a dependency hell, and lacks reroducibility. Binary distributions are less flexible, harder to customize, and not available on all architectures.</p> <p>A build system like Buildroot or Yocto is a middle ground between the two.</p> <p>In yocto, you have multiple tasks, to download, configure, compile, install, the builds. The tasks are grouped in recipes, and you manage recipes with Bitbake.</p> <p>Many common tasks are already defined in the OpenEmbedded core. Many recipes are available, organised in layers.</p> <p>OpenEmbedded is co-maintained by the Yocto Project and OE project. It's the base layer, the core of all the magic as Mylène says.</p> <h2>Workflow</h2> <p>Poky is a distribution built on top of the OpenEmbedded Core, and provides Bitbake.</p> <p>The general workflow is Download -&gt; Configure -&gt; Compile. You download the proper version you want with git clone.</p> <p>To add applications, add layers (compilations of recipes). There are folders in your poky directory. Always look at existing layers before creating a recipe. Do not edit upstream layer if you don't want breakage when updating.</p> <p>To configure the build, you first source the Bitbake environment, which moves you to a build folder, and gives you a set of commands in order to do the build. You can then edit the local.conf to set your MACHINE, which describes the hardware and can be found in specific BSP layers, and setup your DISTRO, which represents top-level configuration that will be applied on every build, and brings toolchains, libc, etc. And then the IMAGE, brings the apps, libs, etc.</p> <p><img alt="jpg" src="/images/er2017/04-Mylène.jpg"></p> <h3>Creating a layer</h3> <p>When needed, you might create a layer, whether you have custom hardware, or want to integrate your own-application. You can do that with the yocto-layer tool which does the heavy-lifting. It's a good practice to create multiple layers to share common tasks/recipes between projects.</p> <p>The recipe are created in .bb files, the format that bitbake understands. The naming of the file is application-name_version.bb, and a file is split in header, source, and tasks parts.</p> <p>Mylène says it's a good practice to always use remote repositories to host app sources to make development quicker. App sources should never be in the layer directly. The folder organization should always be the same in order to find the recipes faster.</p> <p>Sometimes, you might want to extend an existing recipe, without modifying it. It's possible with the Bitbake engine, when creating .bbappend files. All .bbappend files are version specific. They can be used to add patches, or customize the install process by appending a task.</p> <h3>Creating an image</h3> <p>An image is a top-level recipe, it has the same format as other recipes, with specific variables on top, like IMAGE_INSTALL to list the included package, or IMAGE_FSTYPES for the binary format of images you want (ext4, tar, etc.).</p> <p>It's a good practice to only install what you need for your system to work.</p> <h3>Creating a machine</h3> <p>The machine describes the hardware. It contains variables related to the architecture, like TARGET_ARCH for the architecture, or KERNEL_IMAGETYPE.</p> <h1>Mainline Linux on AmLogic SoCs</h1> <p>by Neil Armstrong</p> <p>The AmLogic SoC Family has multimedia capabilities, and is used in many products. They have different products, ranging from the Cortex-A9 to Cortex A53 CPUs.</p> <p>The SoCs are very cheap at ~7$ when compared to competitors.</p> <p>Amlogic SoCs are used in many different cheap Android boxes. They are also in community boards from ODroid, the Khadas VIM1/2, NanoPi K2 or Le Potato which has been designed by BayLibre.</p> <p>The <a href="https://libre.computer">Libre Computer board</a> has been backed on Kickstarter. Mainline support is done by BayLibre, with many peripherals already working.</p> <p>The upstream support started from 4.1 by independent hackers. From 4.7, BayLibre started working on it. The bulk of the work went in 4.10 and 4.12.</p> <p>The work was concentrated on 64bit SoCs (the latest ones), but the devices are very similar inside the family.</p> <h2>Drivers so far</h2> <p>Dynamic Voltage and Frequency Scaling is a complex part of the work, since it's done on a specific CPU in the SoC, but ARM changed the protocol after some time and did not publish the old one at first.</p> <p>SCPI (the DVFS driver on this SoC) is now supported on 4.10 though.</p> <p>Kevin Hilman wrote a new eMMC host driver from the original implementation and public datasheet. It's very performant.</p> <p>At the end of 2016, Amlogic did a new variant S905X of those SoCs, and supporting it was easily done through re-architecturing the Device Tree files.</p> <p>For CVBS (analog video support), support was integrated in 4.10. For HDMI, Amlogic integrated a Synopsys DesignWare HDMI Controller, and a clean dw-hdmi bridge has been published sharing the code between different SoCs family. The PHY was custom, as well as the HPD though.</p> <p>CEC support was merged using the CEC framework maintained by Hans Verkuil. </p> <p>The Mali GPU inside the SoC does not have an open driver. The open source kernel driver is available. But the userspace shared binary is delivered as a blob, that has to be compiled by the SoC vendor to customize it.</p> <p><img alt="jpg" src="/images/er2017/05-Neil.jpg"></p> <h2>Work in progress</h2> <p>There is still a lot of work for the Video Display: cursor plane, overlay planes, osd scaling, overlay scaling are missing for example.</p> <p>DRM Planes only have a single, primary plane, without scaling. Support for scaling, or planes with different sub-sampling (various YUVformats), overlay planes, is still missing.</p> <p>In Audio land, S/PDIF input and output is missing. I2S is working for output only through HDMI or external DAC, but the embedded stereo DAC in GXL/GXM or I2S input aren't support.</p> <p>Video Hardware Acceleration, while one of the best feature of the SoC, is still missing, Neil says. There's at least 6 month of development to have a proper V4L2 driver.</p> <h2>Community</h2> <p>There are a lot of hobbyist hacking on the Odroid-C2 board, and running LibreELEC and KODI. Many raspberry-pi oriented projects are also ported to Amlogic boards.</p> <p>There are upstream contributions from independent hackers. This is also helped by the growing of Single Board Computer (SBC) diversity with these SoCs.</p> <h1>Long-Term Maintenance, or How to (Mis-)Manage Embedded Systems for 10+ Years</h1> <p>by Marc Kleine-Budde</p> <p>Marc started by asking the audience who had Embedded Systems in the field, for how long, and which ones were still maintained. Then he asked who had to update to fix a vulnerability, and how long it took to deploy.</p> <p>The context of the talk are systems created by small teams, using custom hardware, and pushing out new products every few years, that need to be supported for more than 10 years.</p> <p>The traditional Embedded Systems Lifecycle starts with a Component Version Decision, followed by HW/SW development, then the maintenance starts. It's usually the longest phase.</p> <p>Marc showed graphs of vulnerabilities per-year in the Kernel, glibc and openssl. Despite most vulnerabilities being Denial of Services, it's still a lot. There's also the infamous "rootmydevice" in a proc file that was in a published linux-sunxi kernel from Allwinner.</p> <p>Don't trust you vendor kernel, Marc says.</p> <h2>Field Observations</h2> <ul> <li>vendor kernels are already obsolete at start of project</li> <li>the workflow for customized pre-built distributions isn't standard</li> <li>you get the worst of both world if you select "longterm" components but don't have an update concept</li> <li>if your update process isn't proven, it's bad</li> <li>there's a critical vulnerability in a relevant component at least twice a year</li> <li>upstream only maintain components for 2 to 5 years</li> <li>Server distros are made for admin interaction, and not suited to embedded systems.</li> </ul> <p>It all leads to the conclusion that Continous Maintenance is very important. </p> <p>Backporting, while simple at its core — you take a patch and apply it — doesn't scale. As you get more products, versions diverge, as you make local modifications test coverage is reduced, and after a few years, it's almost impossible to decide which upstream fixes are relevant.</p> <p>If you don't want your product to become part of botnet, you need to have a few safeguards. You need to have short time between incident and fix, have low risk of negative side effects, predict maintenance cost, and have this whole process scalable to multiple products.</p> <p>These are ingredients for a sustainable process: making sure you can upgrade in the field, review security announcements regularly, always use releases maintained by upstream, disable unused components and enable security hardening.</p> <p><img alt="jpg" src="/images/er2017/06-Marc.jpg"></p> <h2>Development Workflow</h2> <p>It's important to submit changes to upstream to reduce maintenance effort.</p> <p>You need to automate the processes as early as possible: use CI.</p> <p>When starting a new project, use the development version of upstream projects, so that when you reach completion, it's in stable state, and still maintained, as opposed to already obsoleted.</p> <p>Every month, do periodic maintenance: integrate maintenance releases in order to be prepared, review security announcements, and evaluate impact on the product.</p> <p>When you identify a problem, apply the upstream fix, and leverage your automated build, testing and deployment infrastructure to publish.</p> <p>Marc advises using Jenkins 2 with Pipeline as Code. For test automation, take there's kernelci.org or LAVA. For redundant boot, barebox as bootchooser, u-boot/grub can do it with custom scripts as well as UEFI. For the update system, there is RAUC, OSTree or Swupdate. Finally, there are now many different rollout schedulers like hawkBit, mender.io, resin.io, but you can also use a static server or custom application.</p> <h2>Conclusion</h2> <p>Marc says that simply ignoring the problem does not work. Don't try ad-hoc fixes, it doesn't scale. Customized server distributions aren't fitted to the embedded use case.</p> <p>What works is upstreaming, process automation and having a proper workflow.</p> <h1>Developing an embedded video application on dual Linux + FPGA architecture</h1> <p>by Christian Charreyre</p> <p>The application discussed in this talk has high real time and performance constraints. It must be able to merge and synchronize images issued by 2 cameras, with safety constraints. Target latency is less than 200ms, with boot time less than 5s.</p> <p>Christian says that in a previous video application, they worked on an ARM SoC with gstreamer, but it didn't match the safety requirements, so they decided to go with a hybrid FPGA+linux solution.</p> <p>Target hardware is a PicoZED, an System On Module based on a Xilinx Zynq, which embeds and ARM processor as well as an FPGA in the SoC. Its software environment is yocto-based, and does not use the Xilinx-provided solutions Petalinux or Wind River Pulsar Linux, because of their particular quirks. Yocto is now well known and Christian decided to pick-up the meta-xilinx layer and start from that instead. All necessary layers are from the OE layer Index. </p> <p>The FPGA development are made with the Eclipse-based Xilinx Vivado tool, which enables scripting with tcl.</p> <p>The AXI bus is used to communicate between the Linux host and the FPGA design. It allows adding devices accessible from Linux, extending the capabilities: for example, a new serial line, dedicated hardware. It also allows dynamically changing the video pipeline by changing the parameters.</p> <p><img alt="jpg" src="/images/er2017/07-Christian.jpg"></p> <h2>Boot mechanism</h2> <p>The PicoZed needs a First Stage Boot Loader (FSBL), before u-boot. This FSBL is generated by the Vivado IDE according to the design. The FSBL then starts u-boot, which starts Linux.</p> <p>The FPGA can't start alone, and it's code (bitstream) is loaded by the FSBL or u-boot. The Xilinx Linux kernel has a drivers for devices programmed in the FPGA. It uses device tree files to describe the specific configuration available at the moment. Vivado generated the whole device tree, not just the part for the Programmable Logic (FPGA), it merges the two in a single system.dts file.</p> <p>It's a good idea to automate the process of rebuilding the device tree after each change in Vivado, Christian says.</p> <p>The boot is comprised of several tasks before showing an image, making boot time optimization a complex problem: FSBL, u-boot, bitstream loading, kernel start, etc. Various techniques were used to reduce boot time. Inside u-boot, the bootstage report was activated, some devices init were disabled.</p> <p>Bootchart was used to profile Linux startup: the kernel size was reduced, the system console removed, and the init scripts reordered. Filesystem checks were bypassed by using a read-only filesystem. SPI bus speed was increased. Other techniques were used, and the 5 second goal was met.</p> <h2>Closing words</h2> <p>While the design of the system was done so that only the part on the FPGA is impacted by the certification process, the bitstream code is still updated through Linux on the network. Therefore code signing was used in the installer and updater mechanisms to protect the integrity of the system.</p> <p>According to Christian, the project has many unknown before starting, but those were surmounted. The splitted design constraint payed off. The choice of meta-xilinx layer is good one, because of its good quality. You only need to understand that the device tree is not built within the kernel; once you understand the general structure, it's working well, and the distribution is well tailored to the requirements.</p> <h1>Lightning talks</h1> <h2>Atom Linux</h2> <p>by Christophe Blaess</p> <p><a href="https://github.com/AtomLinux">Atom Linux</a> is a new embedded linux distro designed by Christophe. It's a binary distribution, but definitely embedded-oriented. It aims to be industrial-quality.</p> <p>Atom Linux targets small companies, that already have an embedded Linux project, but with poor embedded Linux knowledge. It aims to provide a secure update system (with rollback, factory defaults, etc.). It want to be power-failure proof with a read-only rootfs, and data backup.</p> <p>It's easy to configure Christophe says. The base system is already compiled. It provides a UI for configuration. It aims to make custom code integration simple by providing a toolchain in a VM or natively if needed.</p> <p><img alt="jpg" src="/images/er2017/08-Christophe.jpg"> The user starts by downloading the base image for his target, then installing the configuration tool. The user configures the base image with a few parameters. The configuration tool merges the prebuilt packages and the user custom code in a new root filesystem image.</p> <p>This image is then stored in the user's repository (update server), and at first boot, the system does an update.</p> <p>Currently, the base image builder works, as well as u-boot and the update shell scripts. The first version of the configuration tool is Qt-based, but it's very ugly according to Christophe. He still wants to improve the tool, and rewrite the base image builder as a Yocto layer. Christophe is looking for contributors and ask anyone interested to contact him.</p> <h2>Wayland is coming</h2> <p>by Fabien Lahoudere</p> <p>Fabien started that he is just a user, not a Wayland developer. Wayland is protocol for compositors to talk to its clients. It's aimed as a simpler replacement for X.</p> <p>Wayland is designed for modern devices, more performant, simpler to use and configure according to Fabien. It's also more secure, supported by toolkits, and the future of Linux distributions. For instance, it prevents keyloggers, that are very easy to implement with X11.</p> <p><img alt="jpg" src="/images/er2017/09-Fabien.jpg"> Wayland is more performant, because it has less ping/pong between the compositor and the clients. Weston is the reference implementation. It's a minimal and fast Wayland compositor. You can extend it by using libweston. There's also AsteroidOS and Maynard which are two embedded-oriented Wayland compositors.</p> <p>It's also possible to use a "legacy" X application through Xwayland. In fact, Fabien did his whole presentation on a small iMX6Solo based board running evince on top of wayland.</p> <p>Someone from the audience said they recently had to work with QtCompositor, and it was very simple to use.</p> <h2>Process monitoring with systemd</h2> <p>by Jérémy Rosen</p> <p>Jérémy says systemd is a very good tool for embedded systems. It cost about ~6Mb of disk space when built with yocto. It's already integrated in Yocto and Buildroot.</p> <p>systemd makes it easy to secure processes with capabilities, and limits system calls; it can bind mount files to control exactly what a process sees. It makes it easy to control resources with cgroups, as well as monitoring processes.</p> <p>Jérémy compared moving to systemd from init scripts is like going from svn to git. It requires to understand and re-learn a lot of things, but is really worth it in the end.</p> <p><img alt="jpg" src="/images/er2017/10-Jérémy.jpg"> systemd provide very fine grained control on how to kill a unit: which command to send, which signal to send when it doesn't work, what cleanup command to run, etc. You can define what is a normal or abnormal stop. It can restart an app automatically, and rate-limit this. You can also do coredump management, soft watchdog monitoring, it also monitors itself with a hardware watchdog.</p> <p>A fine-grained integration of how services work interact is also available. You can react to hardware changes, filesystem changes, use socket activation, etc.</p> <p>Jérémy said monitoring is a solved problem for him in embedded and he does not want to work on custom solutions anymore.</p> <p><em>That's it for Embedded Recipes first edition ! Congratulations on reading this far !</em></p> <p><img alt="jpg" src="/images/er2017/11-Speakers.jpg"></p>awk driven IoT2017-07-05T00:00:00+02:002017-07-05T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2017-07-05:/awk-driven-iot.html<p>With a Raspberry Pi or other modern single-board computers, you can make very simple toys. I started with the hello world of interactive apps: the <a href="https://en.wikipedia.org/wiki/Soundboard_(computer_program)">soundboard</a>. But even then, I was too ambitious.</p> <h1>The soundpad</h1> <p>Since the main target was kids. I wanted a simple, screen-less toy that would teach …</p><p>With a Raspberry Pi or other modern single-board computers, you can make very simple toys. I started with the hello world of interactive apps: the <a href="https://en.wikipedia.org/wiki/Soundboard_(computer_program)">soundboard</a>. But even then, I was too ambitious.</p> <h1>The soundpad</h1> <p>Since the main target was kids. I wanted a simple, screen-less toy that would teach the basics of interactivity, as well as serve as a platform for learning. A simple soundboard can be quite useful to learn animal calls for instance, so I was set.</p> <p>But I also wanted this toy to be wireless and interact with the house. For the first part, I decided to hook an old <a href="https://en.wikipedia.org/wiki/IControlPad">bluetooth game controller</a> I had lying around. I was able to detect its keys with <code>evtest</code> pretty quickly and make an inventory of all buttons keycodes:</p> <div class="highlight"><pre><span></span>304 305 306 307 308 312 313 314 315 316 317 318 </pre></div> <p>For the sounds, I reused the sounds present in the default raspbian scratch installation. There are a few wave files in <code>/usr/share/scratch/Media/Sounds/</code> that proved useful. I made a few directories with symbolic links to the samples I was interested in. Combining <a href="https://joeyh.name/code/moreutils/"><code>vidir</code></a> and the previous keycode list, I ensured each wave file name started with a keycode, like this for the Animal sounds:</p> <div class="highlight"><pre><span></span>304-Bird.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Bird.wav 305-Cricket.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Cricket.wav 306-Crickets.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Crickets.wav 307-Dog1.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Dog1.wav 308-Dog2.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Dog2.wav 312-Duck.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Duck.wav 313-Goose.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Goose.wav 314-Horse.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Horse.wav 315-HorseGallop.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/HorseGallop.wav 316-Kitten.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Kitten.wav 317-Meow.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Meow.wav 318-Owl.wav -&gt; /usr/share/scratch/Media/Sounds/Animal/Owl.wav </pre></div> <p>In order to interact with the house, I paired a bluetooth soundbar to the raspberry pi.</p> <p>Once all of this is setup, this is the entirety of the code for the first iteration of the working soundpad (soundboard + joypad):</p> <div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span> <span class="nb">cd</span> <span class="nv">$1</span> stdbuf -o0 evtest /dev/input/event0<span class="p">|</span> awk -W interactive <span class="s1">&#39;</span> <span class="s1">/EV_KEY/ { if ( $NF == 1) { system(&quot;paplay &quot; $8 &quot;-*.wav&amp;&quot;) }}&#39;</span> </pre></div> <ul> <li><code>stdbuf</code> is very useful when playing with pipes where the input command is blocking, but you still want interactivity. It allows you to control i/o buffering.</li> <li><code>evtest</code> parses input events.</li> <li><code>awk -W interactive</code> has the same role as <code>stdbuf -i0</code>, but for <code>mawk</code>'s internal buffering (it's not needed for GNU awk).</li> <li>when a matching line is found, <code>paplay</code> is used to play the audio through pulseaudio's bluez sink, that was previously configured as default. The filename corresponds to the button keycode.</li> </ul> <p>The last iteration has the same core code, but with a bit more setup: using <code>bluetoothctl</code> and <code>pactl</code> to make sure the controller and the soundbars are properly connected and configured mainly.</p> <p>It worked, for the most part, but was far from plug-and-play. The soundbar needed to be turned on and put in bluetooth mode. The wireless joypad had to be turned on. It needed constant re-setup of the bluetooth connections, because it lost the pairings regularly. And sometimes the audio would stutter horribly. I tried compiling a more recent version of bluez, to no avail.</p> <p>So after a few day of demos and sample playing, I binned this project about 9 months ago.</p> <h1>The music portal</h1> <p>Fast forward today, I had this thing bothering me about modern music and rhymes for kids. With Deezer &amp; Spotify, we have access to a library we could only dream of. But it's impossible for 2 year old child to operate, or even desirable.</p> <p>Even without online services, the only alternative would be to go back to the audio CDs. But the only functional CD player in our house is the CD-ROM drive in my Desktop computer; I therefore backup all our audio CDs in audio files. Playing those has the same level of complexity (and screen-interaction) as interoperating with streaming services, so it's back to square one.</p> <p>That's where the music portal comes in. It's a combination of a <a href="https://en.wikipedia.org/wiki/Mir:ror">Violet Mir:ror</a> I had lying around, and a Raspberry PI with a speaker hooked up.</p> <p>The Mir:ror is a very simple RFID reader. It's basically plug-and-play on Linux, since it sends raw HID events, with the full ID of the tags it reads, and it has audio and visual feedback. I also evaluated using a <a href="https://en.wikipedia.org/wiki/Skylanders">Skylanders</a> portal, which also sent raw HID events, but its data was much less detailed, with only two bytes of information in the HID events, and the need to do more work to <a href="https://github.com/silicontrip/SkyReader">get the full data</a>, and has no audio or visual feedback.</p> <p>So here is the code of the first version:</p> <div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span> sudo stdbuf -o0 hexdump -C /dev/hidraw0 <span class="p">|</span> awk -W interactive <span class="s1">&#39;</span> <span class="s1">/02 01 00 00 08 d0 02 1a 03 52 c1 1a 01 00 00 00/ { print &quot;file1 &quot;; play=1; file=&quot;file1.mp3&quot; ; }</span> <span class="s1">/02 01 00 00 08 d0 02 1a 03 52 c1 4b ad 00 00 00/ { print &quot;file2 &quot;; play=1; file=&quot;file2.mp3&quot; ; }</span> <span class="s1">/02 01 00 00 04 3f d7 5f 35 00 00 00 00 00 00 00/ { print &quot;dir1 &quot;; play=1; file=&quot;dir1/*.mp3&quot; ; }</span> <span class="s1">/ 02 02 00 00 0. |01 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00/ { print &quot;stop&quot;; system(&quot;killall -q mpg321&quot;); }</span> <span class="s1">{</span> <span class="s1">if (play) {</span> <span class="s1"> system(&quot;mpg321 -q &quot; file &quot; &amp; &quot;);</span> <span class="s1"> }</span> <span class="s1">play=0 ;</span> <span class="s1">}</span> <span class="s1">&#39;</span> </pre></div> <ul> <li>we use the same <code>stdbuf</code> and <code>awk -W interactive</code> trick as before. Fun fact: I rediscovered this <code>mawk</code> argument by reading the man page while doing this project because I had forgotten about it in only 9 months. I don't think I'll forget it again.</li> <li>Here we're matching full HID event lines. We don't even bother decoding the payload size, etc. Since it all fits on a line matchable by <code>awk</code>.</li> <li>I used <code>mpg321</code> because it has the less footprint when compared to <code>mpg123</code>, <code>gst-launch</code>, <code>mplayer</code>, <code>vlc</code>, and others.</li> <li>I used the same symbolic link structure because it's much easier than putting the full file names in the script.</li> <li>We handle "tag" removal as well as portal shutdown. The Mir:ror automatically shuts down when turned face down.</li> <li>There are race conditions hiding here. It's not a big deal, it's just a prototype.</li> </ul> <p>What could I use after I setup the two included Nanoztags ? I could put RFID stickers on objects; or I could use my visa card; or anything that has an RFID/NFC feature (like my phone). But there are better, available off-the-shelf choices: <a href="https://en.wikipedia.org/wiki/Toys-to-life">toys-to-life</a> like Skylanders ! There are already made for kids, are very sturdy, and I managed to snag a few on clearance at ~1€ a piece !</p> <p>Make sure the Raspberry Pi is connected to your wireless network, so you can add new songs remotely, and throw in a <a href="https://www.freedesktop.org/software/systemd/man/systemd.service.html">systemd.service</a> for automatic starting, and the toy is finished:</p> <div class="highlight"><pre><span></span><span class="k">[Unit]</span> <span class="na">Description</span><span class="o">=</span><span class="s">Music Portal</span> <span class="k">[Service]</span> <span class="na">Type</span><span class="o">=</span><span class="s">simple</span> <span class="na">ExecStart</span><span class="o">=</span><span class="s">/home/pi/musicportal.sh</span> <span class="na">User</span><span class="o">=</span><span class="s">pi</span> <span class="na">Group</span><span class="o">=</span><span class="s">pi</span> <span class="na">WorkingDirectory</span><span class="o">=</span><span class="s">/home/pi</span> <span class="na">StandardOutput</span><span class="o">=</span><span class="s">journal+console</span> <span class="na">StandardError</span><span class="o">=</span><span class="s">journal+console</span> <span class="na">Restart</span><span class="o">=</span><span class="s">always</span> <span class="na">RestartSec</span><span class="o">=</span><span class="s">3</span> <span class="k">[Install]</span> <span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span> </pre></div> <p>And it's truly plug-and-play: you just need to plug the Raspberry Pi, and it powers the speaker through USB, as well as the Mir:ror.</p> <p>Here's a video of the final result:</p> <iframe width="560" height="315" src="https://www.youtube.com/embed/I1Vc38DaTQY" frameborder="0" allowfullscreen></iframe> <p>Last but not least, the title of this article is <em>awk driven <strong>I</strong>oT</em>. So I integrated <a href="https://github.com/plietar/librespot">librespot</a>, and I can now play songs and rhymes from this online streaming service ! Success ✔</p>Go Time2017-02-19T00:00:00+01:002017-02-19T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2017-02-19:/go-time.html<p>For the Go 1.8 Release Party in Paris I gave a lightning talk on monotonic clocks. It's essentially a talk version of the <a href="https://golang.org/design/12914-monotonic">Russ Cox's design document on monotonic clocks</a>, which is really well written and sourced. You should go read it !</p> <p>Here are the <a href="https://anisse.github.io/gotime">slides export</a> and their …</p><p>For the Go 1.8 Release Party in Paris I gave a lightning talk on monotonic clocks. It's essentially a talk version of the <a href="https://golang.org/design/12914-monotonic">Russ Cox's design document on monotonic clocks</a>, which is really well written and sourced. You should go read it !</p> <p>Here are the <a href="https://anisse.github.io/gotime">slides export</a> and their <a href="http://github.com/anisse/gotime">source</a>.</p>Embedded Linux Conference Europe 20162016-10-21T00:00:00+02:002016-10-21T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2016-10-21:/embedded-linux-conference-europe-2016.html<p>I was in Berlin last week for ELCE, and it was great. It was a nice mix of talks on many different subjects, and as always you come back with lots of new ideas and improved motivation.</p> <p>As you know, I took some notes for <a href="kernel-recipes-2016-notes.html">Kernel Recipes 2016</a>, and lots …</p><p>I was in Berlin last week for ELCE, and it was great. It was a nice mix of talks on many different subjects, and as always you come back with lots of new ideas and improved motivation.</p> <p>As you know, I took some notes for <a href="kernel-recipes-2016-notes.html">Kernel Recipes 2016</a>, and lots of people at ELCE told me they were glad to have read them. I have since updated the article with videos and LWN links for future readers.</p> <p>Since I was recovering from attending dotGo on monday, and then flying to Berlin, I did not take notes at ELCE this year. But I stumbled during the conference on Arnout Vandecappelle, Buildroot contributor and fellow Embedded Engineer. He took great notes which he posted on <a href="https://mindlinux.wordpress.com/">his company's blog</a>:</p> <ul> <li><a href="https://mindlinux.wordpress.com/2016/10/11/irqs-the-hard-the-soft-the-threaded-and-the-preemptible-alison-chaiken/">IRQs: the Hard, the Soft, the Threaded and the Preemptible – Alison Chaiken</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/11/survey-of-open-hardware-2016-john-hawley-intel/">Survey of Open Hardware 2016 – John Hawley, Intel</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/12/building-a-bards-farm-continuous-integration-and-remote-control-antoine-tenart-quentin-schulz-free-electrons/">Building a Bards Farm: Continuous Integration and Remote Control – Antoine Tenart &amp; Quentin Schulz, Free Electrons</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/12/demystifying-systemd-for-embedded-systems-gustavo-sverzut-barbieri-profusion-embedded-systems/">Demystifying Systemd for Embedded Systems – Gustavo Sverzut Barbieri, ProFUSION Embedded Systems</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/12/running-ubiubifs-on-mlc-nand-richard-weinberger-sigma-star-boris-brezillon-free-electrons/">Running UBI/UBIFS on MLC NAND – Richard Weinberger, sigma star &amp; Boris Brezillon, Free Electrons</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/12/reconfigurable-computing-architecture-for-the-linux-kernel-vince-bridgers-yves-vandervennet-intel-ex-altera/">Reconfigurable Computing Architecture for the Linux Kernel – Vince Bridgers &amp; Yves Vandervennet, Intel (ex-Altera)</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/13/using-sched_deadline-steven-rostedt-red-hat/">Using SCHED_DEADLINE – Steven Rostedt, Red Hat</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/13/the-internet-of-things-and-life-beyond-linux-wolfgang-mauerer-technical-university-regensburgsiemens-ag/">The Internet of Things and Life Beyond Linux – Wolfgang Mauerer, Technical University Regensburg/Siemens AG</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/13/modernizing-the-nand-framework-the-big-picture-boris-brezillon-free-electrons/">Modernizing the NAND Framework: The Big Picture – Boris Brezillon, Free Electrons</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/13/update-on-shared-logging-between-the-kernel-and-the-bootloader-sean-hudson-mentor-graphics-inc/">Update on Shared Logging between the Kernel and the Bootloader – Sean Hudson, Mentor Graphics, Inc</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/13/gpio-for-engineers-and-makers-linus-walleij/">GPIO for Engineers and Makers – Linus Walleij</a></li> <li><a href="https://mindlinux.wordpress.com/2016/10/13/open-source-tools-for-fpga-development-marek-vasut-denx-software-engineering/">Open-Source Tools for FPGA Development – Marek Vašut, DENX Software Engineering</a></li> </ul> <p>I attended a few of those talks, and I can tell you he did a great recap.</p> <p>See you next year !</p>Kernel Recipes 2016 notes2016-09-28T00:00:00+02:002016-09-28T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2016-09-28:/kernel-recipes-2016-notes.html<p><strong>Update 2016-10-21:</strong> I've added links to the <a href="https://www.youtube.com/playlist?list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm">videos</a> and <a href="https://lwn.net/Archives/ConferenceByYear/#2016-Kernel_Recipes">LWN articles</a>, which are of much higher quality than these live notes.</p> <p>This year I'm trying a live blog of <a href="http://kernel-recipes.org/">Kernel Recipes 2016</a>, live from Paris, at Mozilla's headquarters. You can watch the <a href="https://air.mozilla.org/kernel-recipes-2016-09-28-AM-Session/">live stream here</a>.</p> <h1>The kernel report</h1> <p>by Jonathan …</p><p><strong>Update 2016-10-21:</strong> I've added links to the <a href="https://www.youtube.com/playlist?list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm">videos</a> and <a href="https://lwn.net/Archives/ConferenceByYear/#2016-Kernel_Recipes">LWN articles</a>, which are of much higher quality than these live notes.</p> <p>This year I'm trying a live blog of <a href="http://kernel-recipes.org/">Kernel Recipes 2016</a>, live from Paris, at Mozilla's headquarters. You can watch the <a href="https://air.mozilla.org/kernel-recipes-2016-09-28-AM-Session/">live stream here</a>.</p> <h1>The kernel report</h1> <p>by Jonathan Corbet; <a href="https://www.youtube.com/watch?v=9DM3DaQkbGw&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=2">video</a></p> <p>We've had 5 kernel releases since last November, with 4.8 coming out hopefully on Oct 2nd. There were between 12 and 13k changesets for each releases. About 1.5k devs contributed to each release.</p> <p>The number of developers contributing to each release is stable, growing slowly. For each new releases, there are about 200 first-time contributors.</p> <p>The development process continues to run smoothly, and not much is changing.</p> <h2>Security</h2> <p>Security is a hot topic right now. Jon showed an impressive list of CVE numbers, estimating that the actual number of flaws is about double that.</p> <p>The current process for fixing security flaws is like a game of whack-a-mole: there are more and more new flaws, and in the end it's not sure you can keep up.</p> <p>The distributors also aren't playing their part pushing updates to users.</p> <p>So vulnerabilites will always be with us, but what is possible is eliminating whole classes of exploits. Examples of this include: - Post-init read-only memory in 4.6 - Use of GCC plugins in 4.8 - Kernel stack hardening in 4.9 - Hardened usercopy in 4.8 - Reference-count hardening is being worked on.</p> <p>A lot of this originates in grsecurity.net, some of it is being funded by the Core Infrastructure Initiative.</p> <p>The catch is that there are performance impacts, so it's a tradeoff. So can we convince kernel developers it's worth the cost ? Jonathan is optimistic that the mindsets are changing towards a yes.</p> <h2>Kernel bypass</h2> <p>A new trend is to bypass the kernel stack, for instance in the network stack for people doing High-Frequency-Trading.</p> <p>Transport over UDP (TOU) is an example of this, enabling applications to make transport protocols in userspace. The QUIC protocol in Chrome is an example of this.</p> <p>The goal here is to be able to make faster changes in the protocol. For instance, TCP Fast Open has been available for a long time in the kernel, but most devices out there (Android, etc.) have such an old kernel, that nobody is using this.</p> <p>Another goal is to avoid middlebox interference (for example, they mess with TCP congestion, etc.). So here, the payload is "encrypted" and not understood by those middleboxes, so they can't interfere with it.</p> <p>The issue with TOU is that we risk having every app (Facebook, Google, etc.) speaking its own protocol, killing interoperability. So the question is will the kernel still be a strong unifying force for the net ?</p> <h2>BPF</h2> <p>The Berkeley Packet Filter is a simple in-kernel virtual machine. Users can load code in the kernel with the bpf() syscall.</p> <p>It's safe because, there are a lot of rules and limitations to make sure BPF programs do not pose a problem: they can't loop, access arbitrary memory, access uninitialized memory, or leak kernel pointers to user space for example.</p> <p>The original use car of BPF was of course to filter packets. Nowadays it allows system call restriction with seccomm(), perf events filtering, or tracepoint data filtering and analysis. This is finally the Linux "dtrace".</p> <h2>Process</h2> <p>A lot has changed since 2.4. At the time distributors backported lots of code and out-of-tree features.</p> <p>Since then, the "upstream first" rule, or the new regular release (every 10 weeks or so) helped solve a lot of problems.</p> <p>Yet, there are still issues. For instance, a phone running the absolute latest release of Android (7.0), is still running kernel 3.10, which was released in June 2013 and is 221k+ patches behind mainline.</p> <p>So why is this ? Jonathan says that Fear of mainline kernel is a reason. With the rate of change there's the possibility of new bugs and regressions.</p> <p>Jon then showed a table compiled by Tim Bird showing that most phones have a vast amount of out-of-tree code to forward port: between 1.1M and 3.1M lines of inserted codes!</p> <p>Out-of-tree code might be because upstreaming can take a long time. For example, wakelocks or USB changing aren't upstream. Other changes like scheduler rewrites are simply not upstreamable. The kernel moves to slowly for people shipping phones every 6 months.</p> <p>This is a collision of two points of views: manufacturers say that "nobody will remember our product next year", while kernel developers say they've been here for 25 years and intend to continue be here. This is quite a challenge that the community will have to overcome.</p> <h2>GPL enforcement</h2> <p>To sue or not to sue ?</p> <p>Some say that companies will not comply without the threat of compliance. Other say that lawsuits would just shut down any discussions with companies that might become contributors in the future.</p> <p>Contributions stats show that the absolute maximum of independent contributors is about 15%, and that the rest of contributions are coming from people being paid by companies to do so. Therefore alienating those companies might not be the best idea.</p> <p>Corbet put it this way: do we want support for this current device eventually, or do we want support from companies indefinitely ?</p> <h1>entry_*.S : A carefree stroll through kernel entry code</h1> <p>by Borislav Petrov; <a href="https://www.youtube.com/watch?v=f0mEz0XQQMY&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=3">video</a></p> <p>There are a few reasons for entry into the kernel: system calls, interrupts(software/hardware), and architectural exceptions (faults, traps and aborts).</p> <p>Interrupts or exceptions entry need and IDT (Interrupt Descriptor table). The interrupt numbers indexes to it for example.</p> <p>Legacy syscalls had quite an overhead due to segment-based protections. This evolved with the long mode, which requires a flat memory model with paging. Borislav then explains how the setup the MSRs to go into the syscall.</p> <p>The ABI described is x86-specific (which Borislav is a maintainer of), with which registers to setup (rcx, rip, r11) in order to do a long mode syscall. Borislav explains what the kernel does on x86. Which flags should be set/reset ? Read his slides (or the kernel code) for a nice description.</p> <h2>entry_SYSCALL_64 …</h2> <p>… is the name of the function that takes 6 arguments in registers that is run once we're in the kernel. </p> <p>SWAPGS is then called, GS and FS being one of the only segments still used. Then the userspace stack pointer is saved.</p> <p>Then the kenel stack is setup (with a per-cpu-varible) appropriately reading cpu_tss struct.</p> <p>Once the stack is setup, user pt_regs is constructed and handed to helper functions. A full IRET frame is setup in case of preemption.</p> <p>After that the thread info flags are looked at in case there's a special situation that needs handling, like ptraced' syscalls.</p> <p>Then the syscall table is looked at, using the syscall number in RAX. Depending on the syscall needs, it's called more or less differently.</p> <p>Once the syscall has been called, there is some exit work, like saving the regs, moving pt_regs on stack, etc.</p> <p>A new thing on the return path is SYSRET, being faster than IRET which is implemented in microcode (saving ~80ns in syscall overhead). SYSRET does less checks. It depends on the syscall, whether it's on slowpath or fastpath.</p> <p>If the opportunistic SYSRET fails, the IRET is done, after restoring registers and swapping GS again.</p> <p>On the legacy path, for 32-bit compat syscalls, there might be a leak of 16bits of ESP, which is fixed with per-CPU ministacks of 64B, which is the cacheline size. Those ministacks are RO-mapped so that IRET faults are promoted and get their own stack[…].</p> <h1>cgroup v2 status update</h1> <p>by Tejun Heo; <a href="https://www.youtube.com/watch?v=RLqXG4ArPe4&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=4">video</a></p> <p>The new cgroup rework started in Sep 2012 with gradual cleanups.</p> <p>The experimental v2 unified hierarchy support was implemented in Apr 2014.</p> <p>Finally, the cgroup v2 interface was exposed in 4.5.</p> <h2>Differences in v2</h2> <p>The resource model is now consistent for memory, io, etc. Accounting and control is the same.</p> <p>Some resources spent can't be charged immediately. For instance, an incoming packet might consume a lot of CPU in the kernel before we know to which cgroup to charge these resources.</p> <p>There's also a difference in granularity, or delegation. For example, what to do when a cgroup is empty is well defined, with proper notification of the root controllers.</p> <p>The interface conventions have been unified, for example for weight-base resource sharing, the interfaces are consistent accross controllers.</p> <h2>Cpu controller controversy</h2> <p>The CPU controller is still out of tree. There are disagreements around core v2 design features, see <a href="http://lwn.net/Articles/697366/">this LWN article</a> for details.</p> <p>A disagreement comes from page-writeback granularity, i.e how to tie a specific writeback operation to a specific thread as opposed to a resource domain.</p> <p>Another main reason is process granularity. The scheduler only deals with threads, while cgroups don't have thread-granularity, only process-level granularity. This is one of the major disagreements.</p> <p>The scheduler priority control (nice syscall) is a very different type of interface to the cgroup control interface (echo in a file).</p> <p>Discussion on this subject is still ongoing.</p> <h2>The rest</h2> <p>A new pids controller was added in 4.3. It allows controlling the small resource that is the PID space (15 bits) and prevent depletion.</p> <p>Namespace support was added in 4.6, hiding the full cgroup path when you're in a namespace for example. There are still other bugs.</p> <p>An rdma controller is incoming as well.</p> <h2>Userland support</h2> <p>systemd 232 will start using cgroup v2, including the out-of-tree cpu controller. It can use both cgroup v1 and v2 interfaces at the same time.</p> <p>libvirt support is being worked on by Tejun Heo as well, which is currently deploying it with systemd at Facebook.</p> <p>We've had some interesting questions from the audience with regards to some old quirks and security issues in cgroups, but Tejun is quite optimistic that v2 will fix many of those issues and bugs. </p> <p>Old userland tools will probably be broken once cgroup v2 is the norm, but they are fixable.</p> <h1>from git tag to dnf update</h1> <p>by Konstantin Ryabitsev; <a href="https://www.youtube.com/watch?v=vohrz14S6JE&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=5">video</a></p> <p><a href="http://mricon.com/git2dnf">How is the kernel released ? (presentation)</a></p> <h2>Step 1: the git tag</h2> <p>It all starts with a signed git tag pushed by Linus. The transport is git+ssh for the push.</p> <p>It connects to git master, a server in Portland Oregon maintained by the Linux Foundation.</p> <p>The ssh transport passes the established connection to a gitolite shell. gitolite uses the public key of the connection (through an env variable) to identify the user. Then the user talks to the gitolite daemon.</p> <p>Before the push is accepted, a two-factor authentication is done via 2fa-val. This daemon allows the user to validate an IP address for a push. It uses the TOTP protocol. The 2fa token is sent through ssh by the user. It allows the user to authorize an IP address for a certain period of time (usually 24h).</p> <p>Once the push is accepted, gitolite passes control to git for the git protocol transfer.</p> <p>As a post-commit hook, the "grokmirror" software is used to propagate changes to the frontend servers.</p> <p>grokmirror updates a manifest that is served through httpd (a gzipped json file), on a non-publicly accessible server.</p> <p>On a mirror server connected through a VPN, the manifest is checked for changes every 15 seconds, and if there's a change, the git repo is pulled.</p> <p>On the frontend, the git daemon is running, serving updates the repo.</p> <h2>Step 2: the tarball</h2> <p>To generate the tar, the git archive command is used. The file is then signed with gpg.</p> <p>kup (kernel uploader) is then used to upload the tarball. Or it can ask the remote to generate the tarball itself from a given tag, saving up lots of bandwidth. Only the signature is then uploaded. Then the archive is compressed and put in the appropriate public directory.</p> <p>kup uses ssh transport as well to authentify users. The kup server store the tarball in a temporary storage.</p> <p>The tarball is then downloaded by the marshall server, and copied over nfs to the pub master server.</p> <p>The pub master server is mounted over nfs on rasperry pi that watches directory changes and updates the sha256sums file signatures. On marshall, builder server checks if the git tag and tarball are available and then runs pelican to update the kernel.org frontpage.</p> <p>Finally, to publicly get the tarballs, you shouldn't use ftp. It is recommended to use https or rsync, or even https://cdn.kernel.org which uses Fastly.</p> <h1>Maintainer's Don't Scale</h1> <p>by Daniel Vetter; <a href="https://www.youtube.com/watch?v=gZE5ovQq9g8&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=6">video</a>, <a href="https://lwn.net/Articles/703005/">LWN article</a></p> <p><em>I took break here so you'll only find a summary of the talk. <a href="https://kernel-recipes.org/en/2016/talks/maintainers-dont-scale/">Talk description here</a></em> </p> <p>Daniel exposes the new model adopted by the open source intel graphics team to include every regular contributor as Maintainer. His trick ? Give them all commit access.</p> <p>The foreseen problems failed to materialize. Everything now works smoothly. Can this process be applied elsewhere ?</p> <h1>Patches carved into stone tablets</h1> <p>by Greg Kroah-Hartman; <a href="https://www.youtube.com/watch?v=L8OOzaqS37s&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=7">video</a>, <a href="https://lwn.net/Articles/702177/">LWN article</a></p> <p>Why do we use mail to develop the kernel? <a href="https://github.com/gregkh/presentation-stone-tools/blob/master/stone-tools.pdf">presentation</a></p> <p>Because it is faster than anything else. There are 7 to 8 changes per hour. 75 maintainer took on average 364 patches.</p> <p>There are a lot of reviewers.</p> <p>A good person knows how to choose good tools. So Greg reviews a few tools.</p> <p>Github is really nice: free hosting, drive-by contributors, etc. It's great for small projects. The downside is that it doesn't scale for large projects. Greg gives kubernetes as an example: there are 4000+ issues, 500+ outstanding pull requests. Github is getting better at handling some issues, but still requires constant Internet access, while the kernel has contributors that don't have constant Internet access.</p> <p>gerrit's advantage is that project managers love it, because it gives them a sense of understanding what's going on. Unfortunately, it makes patches submissions hard, it's difficult to handle patch series, and doesn't allow viewing a whole patch at once if it touches multiple files. It's slow to use, but it makes local testing hard, people have to work around it with scripts. Finally, it's hard to maintain as a sysadmin.</p> <h2>email</h2> <p>Plain text email has been around since forever. It's what the kernel uses. Everybody has access to email. It works with many types of clients. It's the same tool you use for other types of work. A disadvantage is gmail, exchange, outlook: many clients suck. Gmail as a webserver is good.</p> <p>Read Documentation/email_clients.txt in order to learn how to configure yours.</p> <p>Another advantage of email, is that you don't need to impose any tool. Some kernel developers don't even use git ! Although git works really well with email: it understands patches in mailbox format (git am), and you can pipe emails to it.</p> <p>Project managers don't like it though because they don't see the status.</p> <p>But there's a simple solution: you can simply install Patchwork, which you plug into your mailing list, and it gives you a nice overview of the current status. There's even a command line client.</p> <p>Why does it matter ? Greg says it's simple, has a wide audience, it's scalable, and grows the community by allowing everybody to read and understand how the development process works. And there are no project managers.</p> <p>Kubernetes and docker (github-native projects) are realizing this.</p> <p>Greg's conclusion is that email is currently the best (or less worse?) tool for the job.</p> <h1>Why you need a test strategy for your kernel development</h1> <p>by Laurent Pinchard; <a href="https://www.youtube.com/watch?v=aksNeWAsDPg&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=8">video</a></p> <p>Laurent showed us an example of how a very small, seemingly inconsequential change might introduce quite a bug. There's a need to test everything before submitting.</p> <p>The toolbox used when he started to test his v4l capture driver is quite simple and composed of a few tools ran in the console, in two different telnet connections.</p> <p>He quickly realized that running the commands every time wouldn't scale. After writing a script simplifying the commands, he realized running the script in each of the 5 different terminal connection wouldn't scale either.</p> <p>After this, he automated even further by putting images to be compared in a directory and comparing them with the output. But the test set quickly grew to over a gigabyte of test files.</p> <p>Instead of using static files, the strategy was then to generate the test files on the fly with an appropriate program.</p> <p>He then ran into an issue where the hardware wasn't sending data according to the datasheet. While looking at the data, he discovered he had to reverse engineer how he hardware worked for a specific image conversion algorithm (RGB to HSV).</p> <p>The rule of thumb Laurent advises is to have one test per feature. And to add one test for each bug. Finally, to add a test for each non-feature. For example, when you pass two opposite flags, you should get an error.</p> <p>The test suite Laurent developed is called vsp-tests and is used to test the specific vsp driver he has been working on. There are many other kind of tests in the kernel(selftests, virtual drivers...), or outside of it (intel-gpu-tools, v4l2-compliance, linaro lava tests...).</p> <p>While there are many test suites in the kernel development, there's no central place to run all of these. </p> <p>Regarding CI, the 0-Day project now monitors git trees and kernel mailing lists, performs kernel builds for many architectures, in a patch-by-patch way. On failure it sends you an email. It also runs coccinelle, providing you a patch to fix issues detected by coccinelle. Finally, it does all that in less than one hour.</p> <p>kernelci.org is another tool doing CI for kernel developers. There will be a talk about it on the next day.</p> <p>There's also Mark Brown's build bot and Olof Johansson's autobuilder/autobooter.</p> <p><em>That's it for day one of Kernel Recipes 2016 !</em></p> <h1>Man-pages: discovery, feedback loops and the perfect kernel commit message</h1> <p>by Michael Kerrisk; <a href="https://www.youtube.com/watch?v=TbKtpLHjG1I&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=9">video</a></p> <p><a href="http://man7.org/conf/kr2016/man-pages-discovery-feedback-and-commit-messages-Kernel-Recipes-2016-Kerrisk.pdf">Presentation slides</a></p> <p>Michael has been contributing man pages since around 2000. There around ~1400 pages.</p> <p>When providing an overview, there a few challenges : providing a history of the API, the reason for the design, etc.</p> <p>One of Michael's recent goals has been preventing adding new buggy Linux API. There are a few examples of this. One of the reasons is lack of prerelease testing.</p> <p>There are design inconsistencies, like the different clone() versions. Behavioral inconsistencies might also creep up, like the mlock() vs remap_file_pages() differences in handling pages boundaries.</p> <p>Many process change APIs have different combination of rules for matching credentials of the process that can do the changes.</p> <p>Another issue is long-term maintainability, in which an API must make sure it's extensible, and work on making sure the flags are properly handled, and bad combinations are rejected.</p> <p>We don't do proper API design in Michael's opinion. And when it fails, userspace can't be changed, and the kernel has to live with the problems forever.</p> <h2>Mitigations</h2> <p>In order to fix this, Unit tests are a good first step. The goal is to prevent regressions. But where should they be put ? One of the historical home of testing was the Linux Test Project. But those are out of trees, with only a partial coverage.</p> <p>In 2014, the kselftest project was created, lives in-tree, and is still maintained.</p> <p>A test needs a specification. It turns out specifications help telling the difference between the implementation and intent of the programmer. It's recommended to put the specification at the minimum in the commit message, and at best send a man-page patch.</p> <p>Another great mitigation is to write a real application. inotify is good of example of that: it took Michael 1500 lines of code to fully understand the limitations and tricks of inotify. For example, you can't know which user/program made a given file modification. The non-recursive monitoring nature of inotify also turned out to be quite expensive for a large directory. A few other limitations were find while writing an example program.</p> <p>The main point is that you need to write a real-world application if you're writing any non-trivial API in order to find its issues.</p> <p>Last but not least, writing a good Documentation is a great idea: it widens the audience, allows easier understanding, etc.</p> <h2>Issues</h2> <p>A problem though is discoverability of new APIs. A good idea is to Cc the linux-api mailing list. Michael runs a script to watch for changes for example. It's an issue, because sometimes ABI changes might happen unvoluntarily, while there are a complete no-no in kernel development.</p> <p>Sometimes, we get silent API changes. One example was an adjustment of the posix mq implementation that was discovered years after. By then it was too late to reverse. Of course, this API had no unit tests.</p> <p>The goal to fix this is to get as much feedback as possible <strong>before</strong> the api is released to the public. You should shorten the feedback loop.</p> <h2>Examples</h2> <p>The example of recent cgroup change was given, where improvement of the commit message over the versions gave people a better understanding of the problem that was corrected. It make life easier of the reviewer, for userspace developer, etc.</p> <p>The advice to the developer for a commit message is to assume the less knowledge as possible for the audience. This needs to be done at the beginning of the patch series so many people can give feedback.</p> <p>The last example is from Jeff Layton's OFD locks who did a near perfect API change proposal: well explained, example programs, publicity, man-page patch, glibc patch and even going as far as proposing a POSIX standard change.</p> <p>In response to a question in the audience about the state of process to introduce Linux kernel changes, Michael went as far as to propose that there be a full-time Linux Userspace API maintainer, considering the huge amount of work that needs to be done.</p> <h1>Real Time Linux: who needs it ? (Not you!)</h1> <p>by Steven Rostedt; <a href="https://www.youtube.com/watch?v=4UY7hQjEW34&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=10">video</a></p> <h2>What is Real Time ?</h2> <p>It's not about being fast. It's about determinism. It gives us repeatability, reliability, known worse case scenario and knows reaction time.</p> <p>Hard Real Time is mathematically provable, and has bounded latency. The more code you have, the harder it is to prove.</p> <p>With soft Real Time you can deal with outliers, but have unbounded latency.</p> <p>Examples of hard real time include airplane engine controls, nuclear power plants, etc. Soft real time include a video systems, video games, and some communication systems. Linux is today a Soft Real Time system.</p> <h2>PREEMPT_RT in Linux</h2> <p>It's not a Soft Real Time system because it doesn't allow for outliers or unbounded latency. But it's not Hard Real Time either because it can't be mathematically proven. Steven says it's Hard Real Time "Designed".</p> <p><em>If</em> it had no bug Linux would be a Hard Real Time system. It's used by financial industries, audio recordings (jack), navigational systems.</p> <p><strong>Lots</strong> of feature from PREEMPT_RT has been integrated in the kernel. Examples include highres timers, deadline scheduler, lockdep, ftrace, mostly tickless kernel etc. It allowed people to test SMP-related bugs with only one CPU, since it changed the way spinlocks worked, giving Linux years of advance in SMP performance.</p> <p>But every year PREEMPT_RT also keeps evolving and getting bigger. Missing features still in PREEMPT_RT include Spin locks to sleeping mutexes.</p> <p>Latency always happens. When you have an interrupt, it might run and steal processor time to high priority thread. But with threaded interrupts, you can make sure the "top half" runs for as little time as possible, just to wake up the appropriate thread that will handle the interrupt. </p> <h2>Hardware matters</h2> <p>The hardware needs to be realtime(cache/TLB misses, etc.) as well, but this is topic of Steven's next talk. Come back tomorrow !</p> <h1>kernelci.org : 2 million kernel boots and counting</h1> <p>by Kevin Hilman; <a href="https://www.youtube.com/watch?v=kSe5GMJvqOI&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=11">video</a></p> <p>Kevin showed his growing collection of boards sitting in his home office, that is in fact part of <a href="https://kernelci.org/">kernelci.org</a>.</p> <p>Over the last years, the number of different boards supported by device trees has exploded, while board files have been slowly removed. The initial goal was therefore to test as many boards as possible, while trying to keep up with the growing number of new machines.</p> <p>It started with automation of a small board farm, and then grew into kernelci.org, that builds, boots and reports on the status through web, mail or RSS.</p> <p>Many trees are being tested, with many maintainers requesting that their tree being part of the project.</p> <p>The goal of kernelci.org is to boot those kernel. Building is just a required step. There are currently 31 unique SoCs, across four different architectures, with 200+ unique boards.</p> <p>A goal is to quickly find regressions on a wide range of hardware. Another goal is to be distributed. Anyone having a board board farm can be contributing. There are currently labs at Collabora, Pengutronix, BayLibre, etc. And all of this done in the Open, by small team, none of its member working full-time on it.</p> <p>Once booted a few test suites are run, but no reporting or regression testing is done, and this is only done a small subset of platforms. The project is currently looking for help in visualization and regression detection, since the logs of these tests aren't automatically analyzed. They also would like to have more hardware dedicated to long-running tests.</p> <p>They have a lot of ideas for new features that might be needed, like comparing size of kernel images, boot times, etc.</p> <p>The project is also currently working on energy regressions. The project uses the ARM energy probe and BayLibre's ACME to measure power during boot, tests, etc. The goal is to detect major changes, but this is still under development. Data is being logged, but not reported or analyzed either.</p> <p>How to help ? A good way to start might be just try it, and watch the platforms/boards you care about. The project is looking for contributors in tools, but also for people to automate their lab and submit the results. For the lazy, Kevin says you can just send him the hardware, as long as it's not noisy.</p> <p>Kevin finally showed his schematics to plug many boards, using an ATX power supply, with usb-controled relays and huge USB hubs. The USB port usage explodes since in the ARM space, many boards need USB power supply, and then another USB port for the serial converter.</p> <h1>Debian's support for Secure Boot in x86 and arm</h1> <p>by Ben Hutchings; <a href="https://lwn.net/Articles/703001/">LWN article</a></p> <p>Secure Boot is an optional feature in UEFI that protects against persistent malware if implemented correctly. The only common trusted certificate on PCs are for Microsoft signing keys. They will sign bootloaders on PCs for small fee, but for Windows ARM the systems are completely locked down.</p> <p>For GNU/Linux, the first stage needs an MS signature. Most distribution ship "shim" as a first stage bootloader that won't need updating often.</p> <p>For the kernel, you can use Matthew Garrett's patchset to add a 'securlevel' feature, activated when booted with Secure Boot, that makes module signatures mandatory, and disables kexec, hibernation and other peek/poke kernel APIs. Unfortunately this patch is not upstream.</p> <p>The issue with signatures is that you don't want to expose signing keys to build daemons. You need to have reproducible builds that can't depend on anything secret, therefore you can't auto-build the signed binary in a single step. Debian's solution is to have an extra source package. The first one from which you build the unsigned binary, and a second one in which you put signatures you generated offline.</p> <p>This new package is called <strong>linux-signed</strong>. It contains detached signatures for a given version, and a script to update them for a new kernel version. This is currently done on Ben's machine, and the keys aren't trusted by grub or shim.</p> <p>Signing was added to the Debian archive software dak. This allows converting unsigned binaries to signed ones.</p> <p>While this was already done in Ubuntu, the process is different for Debian (doesn't use Launchpad). Debian signs kernel modules, has detached signatures (as opposed to Ubuntu's signed binaries), and supports more architectures than amd64. Finally, the kernel packages from Ubuntu and Debian are very different.</p> <p>Julien Cristau then came on stage to explain his work on signing with a PKCS#11 hardware security module (Yubikey for now). Signing with an HSM is slow though, so this is only done for the kernel image, not modules.</p> <p>You can find more information the current status of <a href="https://wiki.debian.org/SecureBoot">Secure Boot on the Debian wiki</a>. The goal is to have all of this ready for the stretch release, which freezes in January 2017.</p> <h1>The current state of kernel documentation</h1> <p>by Jonathan Corbet; <a href="https://www.youtube.com/watch?v=UHbq1SzmfUE&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=18">video</a></p> <p>Documentation is unsearchable, and not really organized. There is no global vision, and everything is a silo.</p> <p>Formatted documentation (in source-code) is interesting because it's next to the code. It's generated with "make htmldocs", and is complex multi-step system developed by kernel developers. It parses the source files numerous times for various purposes, and is really slow. The output is ugly, and doesn't integrate with he rest of Documentation/ directory.</p> <p>How to improve this ? Jon says it needs to be cleaned up, while preserving text access.</p> <p>Recently, asciidoc support was added in kernel comments. It has some advantages but adds a dependency on yet-another tool.</p> <p>Jon suggests that it would have been better to get rid of DocBook entirely, and rework the whole documentation build toolchain instead of adding new tools on top.</p> <p>To do this, Jon had a look at Sphinx, a documentation system in Python using reStructuredText. It is designed for documenting code, generating large documents, is widely supported.</p> <p>After posting a proof of concept, Jani Nikula took responsibility and pushed it into a working system. It now supports all the old comments, but also supports RST formatting. To include kerneldoc comments, Jani Nikula wrote an extension module to Sphinx.</p> <p>All this work has been merged for 4.8, and there are now Sphinx documents for the kernel doc HOWTO, GPU and media subsystems.</p> <p>Developers seem to be happy for now, and a new manual is coming in 4.9: Documentation/driver-api is conversion of the device drivers book. Of course, this is just the beginning, as there are <em>lots</em> of files to convert to the new format, and Jon estimates this might take years until it's done.</p> <p>For 4.10, a goal would be to consolidate the development process docs (HOWTO, SubmittingPatches, etc.) into a document. The issue here is that some of this files are really well-known, and often pointed-to, and this would break a lot of "links" in a way.</p> <h1>Landlock LSM: Unprivileged sandboxing</h1> <p>by Mickaël Salaün; <a href="https://www.youtube.com/watch?v=OJ9LuNEP-D8&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=12">video</a>, <a href="https://lwn.net/Articles/703876/">LWN article</a></p> <p>The goal of landlock is to restrict processes without needing root privileges.</p> <p>The use case is to be used by sandboxing managers (flatpak for example).</p> <p>Landlock rules are attached to the current process via seccomp(). They can also be attached to a cgroup via bpf()</p> <p>Mickaël then showed a demo of the sandboxing with a simple tool limiting the directories a given process can access.</p> <p>The approach is similar to Seatbelt or OpenBSD Pledge. It's here to minimize the risk of sandbox escape and prevent privilege escalation.</p> <p>Why do existing features do no fit with this model ? The four other LSMs didn't fit the needs because they are designed to be controlled by the root/admin user, while Landlock is accessible to regular users.</p> <p>seccomp-BPF can't be used because it can't filter arguments that are pointers, because you can't dereference userland memory to have deep filtering of syscalls.</p> <p>The goal of Landlock is to have a flexible and dynamic rule system. It of course has hard security constraints: it aims to minimize the attack surface, prevent DoS, and be able to work for multiple users by supporting independent and stackable rules.</p> <p>The initial thought was to extend the seccomp() syscall, but then it was switch to eBPF. The access rules are therefore sent to the kernel with bpf().</p> <p>Landlock uses LSM hooks for atomic security checks. Each rule is tied to one LSM hook. It uses map of handles, a native eBPF structure to give rules access to kernel objects. It also exposes to eBPF rules filesystem helpers that are used to handle tree access, or fs properties (mount point, inode, etc.).</p> <p>Finally, bpf rules can be attached to a cgroup thanks to a patch by Daniel Mack, and Landlock uses this feature.</p> <p>Rules are either enforced with the process hierarchy, with the seccomp() interface to which Landlock adds a new command; or via cgroups for container sandboxing.</p> <p>The third RFC <a href="http://lwn.net/Articles/700607/">patch series for Landlock is available here</a>.</p> <h1>Lightning talks</h1> <h2>the Free Software Bastard Guide</h2> <p>by Clement Oudot; <a href="https://www.youtube.com/watch?v=yHjtjoqE7V0&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=13">video</a></p> <p>This is a nice compilation of things not to do as user, developer or enterprise. While the talk was very funny, I won't do you the offense of making a summary since I'm sure all my readers are very disciplined open source contributors.</p> <p><a href="http://fr.slideshare.net/coudot/kr2016-the-free-software-bastard-guide">(slides)</a></p> <h2>Mini smart router</h2> <p>by Willy Tarreau; <a href="https://www.youtube.com/watch?v=hzvVtV_zmHw&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=14">video</a></p> <p>This is about a small device made by Gl-inet. It has an Atheros SoC (AR9331) with a MIPS processor, 2 ethernet ports, wireless, 64MB of RAM and 16MB of flash.</p> <p>The <a href="http://www.haproxy.com/download/aloha/pocket/">documentation and sources for the Aloha Pocket</a>, a small distro running on the hardware.</p> <h2>Corefreq</h2> <p>by Cyril</p> <p>Corefreq measures Intel CPUs frequencies and states. It gives you a few hardware metrics. You can lean more on <a href="https://github.com/cyring/CoreFreq">Corefreq github page</a>.</p> <p><em>That's it for day two of Kernel Recipes 2016 !</em></p> <h1>Speeding up development by setting up a kernel build farm</h1> <p>by Willy Tarreau; <a href="https://www.youtube.com/watch?v=vwQ-KcjskRw&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=15">video</a>, <a href="https://lwn.net/Articles/702375/">LWN article</a></p> <p>Some people might spend a lot of time building the Linux kernel, and this hurt the development cycle/feedback loop. Willy says during backporting sessions, the build time might dominate the development time. The goal here is to reduce the wait time.</p> <p>In addition, build times are often impossible to predict when you might have an error in the middle breaking the build.</p> <p>Potential solutions include, buying a bigger machine or using a compiler cache, but this does not fit Willy's use case.</p> <p>Distributed building is the solution chosen here. But as a first step, this required a distributed workload, which isn't trivial at all for most project. Fortunately, the Linux kernel fits this model.</p> <p>You need to have multiple machines, with the exact same compiler everywhere. Willy's proposed solution is to build the toolchain yourself, with crosstool-ng. You then combine this with distcc, which is a distributed build controller, with low overhead.</p> <p>Distcc still does the preprocessing and linking steps locally, which will consume approx 20% to 30% of the build time. And you need to disable gcov profiling.</p> <p>In order to measure efficiency of a build farm, you need to compare performance. This requires a few steps to make sure the metric is consistent, as it might depend on number of files, CPU count, etc. Counting lines of code after preprocessing might be a good idea to have a per-line metric. </p> <h2>Hardware</h2> <p>In order to select suitable machines, you first need to consider what you want to optimize for. Is it build performance at given budget, number of nodes, or power consumption ?</p> <p>Then, you need to wonder what impacts performance. CPU architecture, DRAM latency, cache sizes and storage access time are all important to consider. </p> <p>For the purpose of measuring performance, Willy invented a metric he calls "BogoLocs". He found that dram latency and L3 cache are more important for performance than CPU frequency.</p> <p>To optimize for performance, you must make sure your controller isn't the bottleneck: its CPU or network access shouldn't be saturated for instance.</p> <p>PC-type machines are by far the fastest, with their huge cache and multiple memory channels. However, they can be expensive. A good idea might be to look at gamer-designed hardware, that provides the best cost-to-performance ratio.</p> <p>If you're optimizing for a low number of nodes, buy a single dual-socket high-frequency, x86 machine with all RAM slots populated.</p> <p>If you're optimizing for hardware costs, a single 4-core computers can cost $8 (NanoPi). But there are a few issues: there are hidden costs (accessories, network, etc.), it might be throttled when heating, some machines are unstable because of overclocking, while only achieving up to 1/16th performance of a $800 PC.</p> <p>You can also look at mid-range hardware (NanoPI-M3, Odroid C2), up to quad-core Cortex A9 at 2GHz. But then they run their own kernel. "High-range" low cost hardware are often sold as "set-top-boxes" (MiQi, RKM-v5, etc.) Some of these can even achieve 1/4th performance of a $800 PC. But there are gotchas as well, with varying build quality, high power draw, thermal throttling.</p> <p>The MiQi board at $35 is Willy's main choice according to his performance measurements (or its CS-008 clone). It's an HDMI dongle that can be opened and used barebones. You don't need to use a big linux distribution, a simple chroot is enough for gcc and distcc.</p> <p><a href="http://wiki.ant-computing.com/Choosing_a_processor_for_a_build_farm">All the data from this presentation is on a wiki</a>.</p> <h1>Understanding a real-time system: more than just a kernel</h1> <p>by Steven Rostedt; <a href="https://www.youtube.com/watch?v=w3yT8zJe0Uw&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=16">video</a></p> <p>Real-time is hard. Having preempt-rt patched kernel, is far from enough. You need to look at the hardware under, and the software on top of it, and in general have holistic view of your system.</p> <p>A balance between a Real-Time system versus a "Real-Fast" system needs to be found.</p> <p>You have to go with a Real-Time hardware if you want a real-time system. It's the foundation, and if you don't have it, you can forget about your goal.</p> <h2>Non-real-time hardware features</h2> <p>Memory cache impacts determinism. One should find the worst-case scenario, by trying to run without the cache.</p> <p>Branch prediction misses can severely impact determinism as well.</p> <p>NUMA, used on multi-CPUs hardware, can cause issues whenever a task tries to access memory from a remote node. So the goal is to make sure a real-time task always uses local memory.</p> <p>Hyper-Threading on Intel processors (or AMD's similar tech) is recommended to be disabled for Real-Time.</p> <p>Translation Lookaside Buffer is a cache for page tables. But this means that any miss would kill determinism. Looking for the worst-case scenario during testing by constantly flushing the TLB is needed for a real-time system.</p> <p>Transactional Memory allows for parallel action in the same critical section, so it makes things pretty fast, but makes the worst case scenario hard to find when a transaction fails.</p> <p>System Management Interrupt (SMI), puts the processor in System Management Mode. On a customer box, Steven was able to find that every 14minutes, an interrupt was eating CPU time, that was in fact an SMI for ECC memory.</p> <p>CPU Frequency scaling needs to be disabled (idle polling), while not environmental friendly, it's a necessity for determinism. </p> <h2>Non-real-time software features</h2> <p>When you're using threaded interrupts, you need to be careful about priority, especially if you're waiting for important interrupts, like network if you're waiting for data.</p> <p>Softirqs need to be looked at carefully. They are treated differently in PREEMPT_RT kernels, since they are run in the context of who raises them. Except when they are raised by real Hard interrupts like RCU or timers.</p> <p>System Management Threads like RCU, watchdog or kworker also need to be taken into account, since they might be called as side-effect of a syscall required by the real-time application.</p> <p>Timers are non-evident as well and might be triggered with signals, that have weird posix requirements, making the system complex, also impacting determinism.</p> <p>CPU Isolation, whether used with the isolcpus kernel command line parameter, or with cgroup cpusets can help determinism if configured properly.</p> <p>NO_HZ is good for power management thanks to longer sleeps, but might kill latency since coming out of sleep can take a long time, leading to missed deadlines.</p> <p>NO_HZ_FULL might be able to help with real-time once ready, since it can keep the kernel from bothering real-time tasks by removing the last 1-second tick.</p> <p>When writing an RT Task, memory locking with mlockall() is necessary to prevent page fault from interrupting your threads. Enabling priority inheritance is a good idea to prevent some types of locking situations leading to unbounded latency.</p> <h1>Linux Driver Model</h1> <p>by Greg Kroah-Hartman; <a href="https://www.youtube.com/watch?v=AdPxeGHIZ74&amp;list=PLQ8PmP_dnN7L5OVT95uXJAE78qcGCcDVm&amp;index=17">video</a></p> <p><a href="https://github.com/gregkh/presentation-driver-model">Presentation files</a> </p> <p>Greg says nobody needs to know about the driver model.</p> <p>If you're doing reference counting, use struct kref, it handles lots of really tricky edge cases. You need to use your own locking though.</p> <p>The base object type is struct kobject, it handles the sysfs representation. You should probably never use it, it's not meant for drivers.</p> <p>On top of that struct attribute provides sysfs files for kobjects, also to never be managed individually. The goal here is to have only one text or binary value per file. It prevents a problem seen in /proc where multiple values per file broke lots of applications when values were added, or unavailable.</p> <p>kobj_type handles sysfs functions, namespaces, and release().</p> <p>struct device is the universal structure, that everyone sees. It either belongs to a bus or a "class".</p> <p>struct device_driver handles a driver that controls a device. It does the usual probe/remove, etc.</p> <p>struct bus_type binds devices and drivers, matching, handles uevents and shutdown. Writing a bus is a complex task, it requires at least 300 lines of code, and has lots of responsibilities, with little helper functions.</p> <p>Creating a device is not easy either, as you should set its position in the hierarchy (bus type, parent), the attributes and initialize it in two-step way to prevent race conditions.</p> <p>Registering a driver is a bit simpler (probe/release, ownership), but still complex. struct class are userspace-visible devices, very simple to create (30-40 lines of code). A class has a lot of responsibilities, but most of those are handled by default, so not every driver has to implement them.</p> <p>Greg says usb is not a good example to understand the driver model, since it's complex and stretches it to its limits. The usb2serial bus is good example.</p> <p>The implementation relies on multiple levels of hierarchy, and has lots of pointer indirections throughout the tree in order to find the appropriate function for an operation (like shutdown())</p> <p>Driver writers should only use attribute groups, and (almost) never called sysfs_*() functions. Greg says you should never use platform_device. This interface is abused of using a real bus, or the virtual bus.</p> <p>Greg repeated that raw sysfs/kobjects should never be used.</p> <h1>Analyzing Linux Kernel interface changes by looking at binaries</h1> <p>by Dodji Seketeli; <a href="https://lwn.net/Articles/703890/">LWN article</a></p> <p>What if we could see changes in interfaces between the kernel and its modules just by looking at the ELF binaries ? It would be a kind of diff for binaries, and show changes in meaningful way.</p> <p>abidiff already does <em>almost</em> all of this userspace binaries. It builds an internal representation of an API corpus, and can build differences. Dodji shows us here how does abidiff works. </p> <p>Unfortunately, there's nothing yet for the Linux Kernel. Dodji entertains the idea of a "kabidiff" tool that would work like abidiff, but for the Linux kernel.</p> <p>For this to work, it would need to handle special Linux ELF symbol sections. For instance, it would restrain itself to "__export_symbol" and "__export_symbol_gpl" sections. It would also need to support augmenting an ABI corpus with artifacts from modules.</p> <p>In fact, work on this has just started in the dodji/kabidiff branch of libabigail.git.</p> <h1>Video color spaces</h1> <p>by Hans Verkuil; <a href="https://lwn.net/Articles/703142/">LWN article</a></p> <p>struct v4l2_pix_format introduced in kernel 3.18 is the subject of the talk.</p> <p>Hans started by saying that Color is an illusion, interpreted by the brain.</p> <p>A colorspace is actually the definition of the type of light source, where the white point is, and how to reproduce it.</p> <p>Colorspaces might be linear, but neither human vision or early cRTs were. So to convert from a linear to non-linear colorspace, you define a transfer function.</p> <p>In video, we otfen use the Y'CbCr (YUV) colorspace. To convert to and from RGB is possible. You can represent all colors in all colorspaces, as long as you don't do quantization (cut of values &lt;0 and &gt;1), which is why you should always do it last.</p> <p>There are a few standards to describe colorspaces: Rec 709, sRGB, SMPTE 170M, and lately BT 2020 used for HDTVs.</p> <p>Typically, colorspace names might be confusing, the conversion matrices might be buggy, and applications would just ignore colorspace information. Sometimes, hardware uses a bad transfer function.</p> <p>In the end Hans found that only half of the vl2_pix_format structure fields were useful.</p> <p>Hans showed examples of the difference of transfer functions between SMPTE 170M and Rec.709. The difference between Rec. 709 and sRGB, or betweer Rec.709 and BT.601 Y'CbCr is more visible. Those example would be impossible to see on a projector, but luckily the room at Mozilla's has huge LCD screens. But even there, it's not enough, since with Full/Limited Range Quantization, a light grey color visible on Hans' screen, was simply white while displayed on the big screen and recording stream. Some piece of the video chain was just doing quantization "bad".</p> <h1>State of DRM graphics driver subsystem</h1> <p>by Daniel Vetter; <a href="https://lwn.net/Articles/703574/">LWN article</a></p> <p>The Direct Rendering Management (drm) subsystem is slowly taking over the world.</p> <p>Daniel started by saying that the new kerneldoc toolchain (see above talk by Jonathan Corbet) is really nice. Everything with regards to the new atomic modesetting is documented. Lots of docs have been added.</p> <p>Some issues in the old userspace-facing API are still there. Those old DRI1 drivers can't be removed, but have been prefixed with drm_legacy_ and isolated.</p> <p>The intel-gpu-tools tests have been ported to be generic, and are starting to get used by on many drivers. Some CI systems have been deployed, and documentation added.</p> <p>The open userspace requirement has been documented: userspace-facing api in DRM kernel code requires an open source userspace program. </p> <p>Atomic display drivers have allowed flicker-free modesetting, with check/commit semantics. It has been implemented because of hardware restrictions. It also allows userspace to know in advance if a given modification would be possible. You can then write userspace that can try approaches, without becoming too complex.</p> <p>20 drivers and counting have been merged with an atomic interface, which 2 or 3 per release, as opposed to one per year (1 per 4 or 5 releases) in the 7 years before atomic modesetting. There's a huge acceleration in development, driving lots of small polish, boiler-plate removals, documentation and new helpers.</p> <p>There's a bright future, with the drm api being used in android, wayland, chromeos, etc. Possible improvements include a benchmark mode, or more buffer management like android's ion.o</p> <p>A generic fdbev implementation has been written on top of KMS.</p> <p>Fences are like struct completion, but for DMA. Implicit fences are taken care of by the kernel. Explicit fences can be passed around by userspace. Fences allows synchronisation between components of video pipeline, like a decoder and an upscaler for example.</p> <p>With upcoming explicit fencing support in kernel and mesa, you can now run Android on upstream code, with graphics support.</p> <p>The downside right now is the bleak support of rendering in open drivers. There are 3 vendor-supported, 3 reverse-engineered drivers, and the rest is nowhere to be seen.</p> <h1>The new hwmon device registration API</h1> <p>by Jean Delvare</p> <p>The hwmon subsystem is used for hardware sensors available in every machine, like temperature sensors for example.</p> <p>hwmon has come a long way. 10 years go, it became unmaintanable, with lots of device-specific code in userspace libraries.</p> <p>The lm-sensors v2 in 2004 was based on procfs for kernel 2.4, and sysfs for kernel 2.6.x.</p> <p>In 2006, there was no standard procfs interface. Therefore, for lm-sensors v3, a documentation was written, standards were enforced, and the one-value per sysfs file rule was adopted. No more device-specific code in libsensors and applications was allowed. Support for new devices could finally be added without touching user-space.</p> <h2>kernel-space</h2> <p>Once the userspace interface was fixed, it did not mean the end of the road.</p> <p>It turned out that every driver implemented its own UAPI. So in 2005, a new hwmon sysfs class was submitted. It was quite simple, and all drivers were converted to the new subsystem at once.</p> <p>It worked for a while, but wasn't sufficient. In 2013, a new hwmon device registration API was introduced: hwmon_register_with_groups. It gives the core flexibility, and allows it to validate the device name. Later this year a new API was added to help unregister and cleanup.</p> <p>Finally, in July 2016 a new registration API proposal was proposed, moving hwmon attributes in core, and doing the heavy lifting of setting up sysfs properly. This patchset is still under review and discussion. Driver conversion won't be straightforward at all, but still deletes more code.</p> <p>In conclusion, a good subsystem should help drivers, integrate well into the kernel, and offer a standard interface. It should provide a smaller binary size and have fewer bugs. But there are still concerns with regards to performance issues, and added complexity because of too many registration functions.</p> <p><em>That's it for Kernel Recipes 2016 ! Congratulations if you managed to read everything !</em></p>Making a Twitter bot that looks for hashes2016-09-09T18:00:00+02:002016-09-09T18:00:00+02:00Anisse Astiertag:anisse.astier.eu,2016-09-09:/making-a-twitter-bot-that-looks-for-hashes.html<p><em>This is a followup to <a href="what-do-you-find-when-you-search-twitter-for-hashes.html">What do you find when you search Twitter for hashes ?</a></em></p> <h1>Why ?</h1> <p>I'm not sure I remember how it started.</p> <p>It all started four years ago. Jon Oberheide was still an independent security researcher and not yet CTO of a successful product company. He posted some …</p><p><em>This is a followup to <a href="what-do-you-find-when-you-search-twitter-for-hashes.html">What do you find when you search Twitter for hashes ?</a></em></p> <h1>Why ?</h1> <p>I'm not sure I remember how it started.</p> <p>It all started four years ago. Jon Oberheide was still an independent security researcher and not yet CTO of a successful product company. He posted some hashes on twitter. I was perplex at first but then I quickly understood that it was to serve as a proof in case someone disputed his research's finding later (and the timing at which he found his results). He was posting hash proofs. And then Matthew Garrett did it too.</p> <p>At the time Twitter wasn't very reliable for accessing old tweets (they vastly improved). I thought maybe by finding these hash proofs and indexing them, we could serve as an independent verifier. Nowadays all the kids put their hashes in the Bitcoin blockchain, and there are even services to do it from your browser.</p> <h2>So how to do it ?</h2> <p>This ought to be easy, right? The initial idea was to just do a simple search of random characters in the hexadecimal space, and hope that they are in hashes ? Well, not really. At first I thought it could be done, but it can't, because twitter search only works on full words, since it's tokenizing for indexing purposes. Which means you can't search for part of words hoping to stumble upon hashes. So much for using n-grams.</p> <p>Therefore, I had to use the public sample stream, and filter <em>every</em> tweet in order to find relevant ones.</p> <h2>Firehose ? Not likely.</h2> <p>Twitter has a special stream that contains all the tweets being posted, called "Firehose". Few people get access to it. There are two other streams: Gardenhose, containing 10% on the tweets, and Spritzer, the sample stream containing 1% of the tweets. The bot currently runs on Spritzer, and Gardenhose was requested, but I never got an answer. It's part of the <a href="http://gnip.com">monetization</a> <a href="http://support.gnip.com/apis/firehose/overview.html">strategy</a>. No place here for hacker/hobbyists.</p> <p>So only 1% of tweets(I have tried to verify that with other public data, it seems about right despite my initial thoughts) that's why the bots haven't been talking much together yet. It also means there's a 99% chance of missing your tweet. And that development iteration speed is a hundred time slower.</p> <h1>How does it work ?</h1> <p>The initial version used a naive regex, but had too many false positives, from repeated characters, to magnet links of P2P files. Now it's much harder to match.</p> <p>The regex is currently matching MD5, SHA1, SHA256 and SHA512 sizes. Most uses are covered.</p> <p>I added a naive exclusion filter (all letters or all numbers), which might not detect extremely well crafted hashes a researcher might be working on. This is out of scope for hashproofs, the anti-spam measures are already pretty strong and might miss interesting content.</p> <h2>Current approach</h2> <p>The first stage is a simple regex <code>[a-f0-9]{32,128}</code> . I wanted it as simple as possible because it is run on every tweet, and should be as fast as possible.</p> <p>The second stage is a much more complex regex (harder to match), with specific sizes of various hashes.</p> <p>Then there are lots of manually crafted filters to fight off spam. Blocked keywords. Users banned automatically. Embedded images and most links are blocked.</p> <p>Finally there is entropy measurement, making sure we have a hash and not a mindless series of characters.</p> <h2>Performance research</h2> <p>To improve performance, I built-in quite a few tools. For example, there's a command allowing to dump the sample stream in temporary file (that you're not allowed to keep). This file is then used to measure performance in a repeatable fashion (there's no contradiction here, right ?), and isolated from the network.</p> <p>I implemented different version of the core line processing, some of which are still in the tests. I was trying to see how to speed up the code. But after some profiling, I realized that most of the time was spent in json processing. Moving to ultrajson(ujson) cut the processing time by 5, compared to python2's cjson module.</p> <h2>Bot detection and spam fighting algorithm</h2> <p>What I did was initially mostly manual: keyword based, username and client based. I kept adding new keywords and banning new clients, but it didn't scale.</p> <p>I then implemented an analysis of a match users's timeline. Within the last 200 tweets, if it had more than 5% of hashes, it was probably a bot. It greatly cut the spam at first, and since it's implementation in 2013 has detected 14k+ accounts posting more that 5% of hashes, and 2.7k+ accounts posting more 50%.</p> <p>There was still a LOT of things passing through (including porn). But the strategy is to use automatic (algorithmic) filtering, not manual. I had to resolve to blocking most outgoing URLs, meaning ther's nothing to spam for. I had to filter tweets containing images.</p> <p>Earlier this year, I discovered a spam network selling followers used the new Twitter Cards to embed links &amp; images without having an URL in the tweet, so I added a filter for that too. For some reason, they were posting lots of hashes. Maybe adding entropy helps circumvent Twitter's detection systems.</p> <h2>Challenges</h2> <p>The code is not py3k compatible for historical reasons (used to need requests-oauth, but moved since to requests-oauthlib (which at some point was inside requests)), although I love py3k. I also had to use ur"" strings, which were ported in python 3.3, which wasn't available at the time. The porting shouldn't be very hard.</p> <p>It was very hard to deal with twitter intermittent service. I developed a watchdog specifically to detect hangs, and then auto-restart. It's the easy way out, but has allowed the bot to work quite well, with months-long uptimes between the updates.</p> <p>As I said earlier, it's hard to debug with a very slow stream that make errors appear a hundred times more slowly.</p> <p>Finally, this "light" stream means there's a 99% chance of missing your tweet. Unless you have lot of followers that RT you, but then you don't need hashproofs, do you ?</p> <h2>Potential improvements</h2> <ul> <li>Follow user stream and watch for hashes. The bot already auto follows people below a certain rate already for good potential feed.</li> <li>use a hashtag (e.g #hashproof) that security researcher can use so that their important tweets are seen.</li> </ul> <h2>Gimme the code, gimme the data</h2> <p>Today I'm publishing the source code for <a href="https://github.com/anisse/hashbot">hashbot on Github</a>. The data is available there as well and <a href="what-do-you-find-when-you-search-twitter-for-hashes.html">analyzed in the earlier article</a>.</p> <h1>Who noticed ?</h1> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/mjg59/status/189781717698101248"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/ochsff/status/189793467872968705"></a></blockquote> <p>I actually implemented Georg's suggestion and all hashes were entropy checked after this.</p> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/bcrypt/status/486855164708810754"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/blinry/status/679276045229563905"></a></blockquote> <p>Yeah, spam was this bad (and still is to an extent).</p> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/opsecanimals/status/733577781238366208"></a></blockquote> <p>It was also noticed by @adulau <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/adulau/status/773831987127787521"></a></blockquote> He asked about the code. Which is why you're seeing this here today.</p> <h1>A few successful findings</h1> <p>There out to be some after all ? Here are a few:</p> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/ioerror/status/365860766517174272"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/OpenCryptoAudit/status/476195439323406336"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/mikko/status/260827290404020224"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/PurpleTeamiOS/status/442341601072136192"></a></blockquote> <h1>Lessons from the project</h1> <p>Always test, makes for robust code.</p> <p>Always benchmark, you might have surprises, cf ultrajson that gave 5x performance speed up.</p> <p>A watchdog is essential when interacting with an external, long-lived service. Twitter has been stopping the stream while keeping the TCP socket open many times, which would mean a hang of the bot.</p> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>What do you find when you search Twitter for hashes ?2016-09-09T17:00:00+02:002016-09-09T17:00:00+02:00Anisse Astiertag:anisse.astier.eu,2016-09-09:/what-do-you-find-when-you-search-twitter-for-hashes.html<p>This image:</p> <p><img alt="jpg" src="/images/accents-of-blue_small.jpg"></p> <p>This is what I found with <a href="https://anisse.astier.eu/making-a-twitter-bot-that-looks-for-hashes.html">hashbot, a twitter bot that looks for hashes.</a></p> <h2>What is this image ?</h2> <p>Posted with the hash "2f404a288d1b564fadee944827a39a14" by japanese accounts (of which @furueru_zekkei used to be the top poster, now suspended).</p> <p>After a bit of research on google images and more, I …</p><p>This image:</p> <p><img alt="jpg" src="/images/accents-of-blue_small.jpg"></p> <p>This is what I found with <a href="https://anisse.astier.eu/making-a-twitter-bot-that-looks-for-hashes.html">hashbot, a twitter bot that looks for hashes.</a></p> <h2>What is this image ?</h2> <p>Posted with the hash "2f404a288d1b564fadee944827a39a14" by japanese accounts (of which @furueru_zekkei used to be the top poster, now suspended).</p> <p>After a bit of research on google images and more, I found that this image is a photo of the <a href="http://www.dpchallenge.com/image.php?IMAGE_ID=186442">White Desert in New Mexico, by Greg Riegler</a>. This might or might not be the same Greg Riegler as <a href="http://gregriegler.com/">here</a>.</p> <h2>Why is this ?</h2> <p>Bots. There a lots of them. The Internet is made of bots.</p> <p>This is what you were most likely to find until 2015 (with a 10% chance).</p> <p>How do I know that ? Well I searched. But this is a story for another post.</p> <h1>What else do you find ? Bots bots bots.</h1> <p>Along this, I found many japanese bots mentionning @null</p> <p>Porn posting bots. The internet is made of them. For some reasons they post hashes... maybe to make sure their tweets are unique and not detected as a spam network ?</p> <p>Occasionnal git and mercurial commit IDs.</p> <p>Security researcher posting proof-of-work. This was the initial motivation behind <a href="https://twitter.com/hashproofs">hashproofs</a>.</p> <p>iPhone UDIDs. Apparently there's a 'market' on Twitter between devs and users to enable iPhones with beta builds:</p> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/Fiddop/status/344225558365876225"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/UsmanUP/status/469233226679353344"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/fmiquiza/status/759891924912254976"></a></blockquote> <p>Giveaway of various activation codes for games, digital products.</p> <p>People crowd-sourcing password hashs, and bots <a href="https://twitter.com/PlzCrack">running rainbow table queries</a>.</p> <p>Bitcoin transaction IDs:</p> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/MikeBeas/status/441866558382821376"></a></blockquote> <p>Torrent hashes:</p> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/corentin_stn/status/354294100431876096"></a></blockquote> <p>Some things just impossible to understand: <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/IvaniaDelRey/status/250016102892072960"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/lakeeffect_kid/status/233214857716060160"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/aperthure/status/242521145868439553"></a></blockquote> <blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><a href="https://twitter.com/Rueangritufo/status/269656715690115072"></a></blockquote></p> <p>LOTS of bots posting more than 5% of tweets containing hashes (found a lot) These won't appear in the results, but <a href="https://github.com/anisse/hashbot/blob/master/results/banned-usernames-2016-09-09">here is the list</a>.</p> <p>I realize how ironic it is to criticize Twitter for having a lot of bots, because the same conditions that allowed all these bots (the API), also permitted this research (as compared to a scraping bot that would have to be updated more often). Of course, hashproofs isn't really spamming, and just acts as a "curator", and does a job that would be impossible to do for a human (i.e analyzing lots of tweets/s).</p> <p>The full list of results can be found on <a href="https://twitter.com/hashproofs">hashproofs' Twitter feed</a>.</p> <h2>Give me the data</h2> <p>I published the <a href="https://github.com/anisse/hashbot">code on Github</a> and the <a href="https://github.com/anisse/hashbot/tree/master/results">full results of the four-year research</a>. (WARNING: contains spam and porn links)</p> <p>This should give you the full data you need to re-analyze the results or run you own hashbot instance (with a better algorithm? or access to a better stream ?)</p> <h1>Unveiling a few bot networks</h1> <p>As I explained earlier, hashproofs analyzes the timeline of users for every matching tweets. If the percentage of matching tweets they have is above a certain arbitrary level (5%), the username is banned locally. If it's over 50%, the account is blocked. That's why you'll find two different lists in the results. One is from Twitter, listing the ids of blocked account. The other is the content of the "banlist" state file of the bot.</p> <p>By analyzing the list of blocked users, I found a few legitimate bots (e.g posting commits on twitter, running rainbow tables, see earlier). I also found a lot of spam bots, some of which were taken car of by twitter. I also discovered that spammers tend to rename their accounts, and my younger self only thought of tracking the usernames, not the account ids, so that's why you'll see discrepancies if you try to have the two lists match.</p> <p>You'll also see that even regular users rename their account if you look at historical data from 2014.</p> <p>Here are a few excerpt from the banlist that show twitter handles that I doubt have been created by legitimate users:</p> <div class="highlight"><pre><span></span> 3924fe95e2cd5f8 68c59dbbb15c5a4 6298c2a08ef9b3b a33262acc8e5c77 b2dc44d67994d44 21332a575639f58 […] Cloud404aa cloud405aa cloud406aa cloud407aa […] 000xxx_6wy 000xxx_897 000xxx_dr3 […] Death_ldo Death_y7s Death_mew Death_ojy </pre></div> <p>All of those are in sequence, which means they were detected by hashproofs one after the other. There are many other examples like this if you want to look at all the 14k+ automatically banned handles.</p> <p>If you're interested in the <a href="making-a-twitter-bot-that-looks-for-hashes.html">historical and technical details, read on to the following article</a>.</p> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>Unofficial witty cloud module documentation with nodemcu firmware2016-02-25T00:00:00+01:002016-02-25T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2016-02-25:/unofficial-witty-cloud-module-documentation-with-nodemcu-firmware.html<p>I wanted to try my hand with ESP8266 modules, so I got <a href="http://www.aliexpress.com/item/ESP8266-serial-WIFI-Witty-cloud-Development-Board-ESP-12F-module/32557886226.html">a witty cloud development board</a>. It's running a proprietary firmware from gizwits which I <a href="/static/witty-flash-backup.bin.gz">backed up</a> if anyone wants to look at it.</p> <p>The board is in two parts: programming board("cape") with ch340g usb serial and 3 …</p><p>I wanted to try my hand with ESP8266 modules, so I got <a href="http://www.aliexpress.com/item/ESP8266-serial-WIFI-Witty-cloud-Development-Board-ESP-12F-module/32557886226.html">a witty cloud development board</a>. It's running a proprietary firmware from gizwits which I <a href="/static/witty-flash-backup.bin.gz">backed up</a> if anyone wants to look at it.</p> <p>The board is in two parts: programming board("cape") with ch340g usb serial and 3.3V converter (plus flash and reset buttons); and main board with the esp module, ams1117 3.3V voltage regulator, a button, a blue led, an rgb led, and a light sensor(photo resistor). All this for the <a href="http://www.aliexpress.com/item/ESP8266-serial-WIFI-Witty-cloud-Development-Board-ESP-12F-module/32557886226.html">price</a> of a nodemcu board, but in a smaller form factor.</p> <p>One of the greatest things of the ESP8266 ecosystem is <a href="https://github.com/nodemcu/nodemcu-firmware">nodemcu-firmware</a>, an environment allowing you to program the microcontroller in lua, greatly simplifying the prototyping and familiarization.</p> <p>After backing up the flash with <a href="https://github.com/themadinventor/esptool">esptool</a> (see <code>esptool read_flash</code>), I flashed the latest release of nodemcu-firmware. Then, using <a href="https://github.com/kmpm/nodemcu-uploader">nodemcu-uploader</a>, one can access the lua REPL (<code>nodemcu-uploader terminal</code>) and uploads lua scripts (<code>nodemcu-uploader --baud 9600 upload init.lua</code>); <code>init.lua</code> being the first script being run at powerup.</p> <h1>Quick doc</h1> <p>I reverse-engineered the various goodies that are on board, since I didn't find any documentation on this specific board online:</p> <p>Blue LED: use the PWM 4. High duty cycle = OFF.</p> <div class="highlight"><pre><span></span><span class="c1">-- Use a LED with a 500Hz PWM</span> <span class="kr">function</span> <span class="nf">led</span><span class="p">(</span><span class="n">pin</span><span class="p">,</span> <span class="n">level</span><span class="p">)</span> <span class="n">pwm</span><span class="p">.</span><span class="n">setup</span><span class="p">(</span><span class="n">pin</span><span class="p">,</span> <span class="mi">500</span><span class="p">,</span> <span class="n">level</span><span class="p">)</span> <span class="n">pwm</span><span class="p">.</span><span class="n">start</span><span class="p">(</span><span class="n">pin</span><span class="p">)</span> <span class="kr">end</span> <span class="c1">-- Control the Blue LED: 0 -&gt; 1023 higher means light off</span> <span class="kr">function</span> <span class="nf">blueLed</span><span class="p">(</span><span class="n">inverted_level</span><span class="p">)</span> <span class="n">led</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="n">inverted_level</span><span class="p">)</span> <span class="kr">end</span> <span class="n">blueLed</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="c1">-- test at high intensity</span> </pre></div> <p>RGB LED: use PWMs 8, 6, 7. High duty cyle = ON.</p> <div class="highlight"><pre><span></span><span class="c1">-- Control an RGB LED: three 0-&gt;1023 values; higher means more light</span> <span class="kr">function</span> <span class="nf">rgb</span><span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="n">led</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="n">r</span><span class="p">)</span> <span class="n">led</span><span class="p">(</span><span class="mi">6</span><span class="p">,</span> <span class="n">g</span><span class="p">)</span> <span class="n">led</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="kr">end</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">500</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1">-- test RED</span> </pre></div> <p>Button: GPIO 2. button pressed = 0 level.</p> <div class="highlight"><pre><span></span><span class="c1">-- launch connect() on button press</span> <span class="n">gpio</span><span class="p">.</span><span class="n">mode</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">gpio</span><span class="p">.</span><span class="n">INPUT</span><span class="p">)</span> <span class="n">gpio</span><span class="p">.</span><span class="n">trig</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s2">&quot;down&quot;</span><span class="p">,</span> <span class="n">connect</span><span class="p">)</span> </pre></div> <p>Light sensor: use the ADC.</p> <div class="highlight"><pre><span></span><span class="c1">-- Print light sensor value</span> <span class="nb">print</span><span class="p">(</span><span class="n">adc</span><span class="p">.</span><span class="n">read</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> </pre></div> <h1>Going further</h1> <p>I then discovered the <a href="http://nodemcu.readthedocs.org/en/dev/en/">official nodemcu-firmware documentation</a> currently points to the dev branch; which has many new modules and functions I wanted to use (like the wifi event monitor or http module) that weren't available in master yet. I used the <a href="http://nodemcu-build.com/">nodemcu cloud builder</a>, a service provided by a kind community member to build a custom version of nodemcu-firmware on the dev branch and the modules I needed enabled.</p> <p>This allows to do this kind of code, that connects to wifi on a button press, and reacts with a simple HTTP request:</p> <div class="highlight"><pre><span></span><span class="kr">function</span> <span class="nf">connect</span><span class="p">()</span> <span class="c1">-- if wifi is already connected (config saved), launch job directly</span> <span class="kr">if</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">status</span><span class="p">()</span> <span class="o">==</span> <span class="n">wifi</span><span class="p">.</span><span class="n">STA_GOTIP</span> <span class="kr">then</span> <span class="n">doOnlineJob</span><span class="p">()</span> <span class="kr">return</span> <span class="kr">end</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1">-- turn orange</span> <span class="kr">for</span> <span class="n">event</span><span class="o">=</span><span class="n">wifi</span><span class="p">.</span><span class="n">STA_IDLE</span><span class="p">,</span><span class="n">wifi</span><span class="p">.</span><span class="n">STA_GOTIP</span> <span class="kr">do</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">eventMonReg</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">monCallback</span><span class="p">)</span> <span class="kr">end</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">config</span><span class="p">(</span><span class="s2">&quot;mynetworkssid&quot;</span><span class="p">,</span> <span class="s2">&quot;mynetworkpassword&quot;</span><span class="p">)</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">eventMonStart</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="c1">--the event mon polls every 100ms for a change</span> <span class="kr">end</span> <span class="kr">function</span> <span class="nf">monCallback</span><span class="p">(</span><span class="n">prevState</span><span class="p">)</span> <span class="n">state</span> <span class="o">=</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">status</span><span class="p">()</span> <span class="kr">if</span> <span class="n">prevState</span> <span class="o">==</span> <span class="kc">nil</span> <span class="kr">then</span> <span class="n">prevState</span> <span class="o">=</span> <span class="s2">&quot;unknown&quot;</span> <span class="kr">end</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Wifi status &quot;</span> <span class="o">..</span> <span class="n">prevState</span> <span class="o">..</span> <span class="s2">&quot; -&gt; &quot;</span> <span class="o">..</span> <span class="n">state</span><span class="p">)</span> <span class="n">blueLed</span><span class="p">(</span><span class="n">state</span><span class="o">*</span><span class="mi">204</span><span class="p">)</span> <span class="c1">-- led intensity depends on status, with success = OFF</span> <span class="kr">if</span> <span class="n">state</span> <span class="o">==</span> <span class="n">wifi</span><span class="p">.</span><span class="n">STA_GOTIP</span> <span class="kr">then</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">200</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="c1">--blue/green-ish, wifi OK</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Got IP &quot;</span> <span class="o">..</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">getip</span><span class="p">())</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">eventMonStop</span><span class="p">(</span><span class="s2">&quot;unreg all&quot;</span><span class="p">)</span> <span class="c1">-- stop event monitor</span> <span class="n">doOnlineJob</span><span class="p">()</span> <span class="kr">end</span> <span class="kr">if</span> <span class="n">state</span> <span class="o">==</span> <span class="n">wifi</span><span class="p">.</span><span class="n">STATION_NO_AP_FOUND</span> <span class="ow">or</span> <span class="n">state</span> <span class="o">==</span> <span class="n">wifi</span><span class="p">.</span><span class="n">STATION_CONNECT_FAIL</span> <span class="kr">then</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1">-- red/fail</span> <span class="n">wifi</span><span class="p">.</span><span class="n">sta</span><span class="p">.</span><span class="n">eventMonStop</span><span class="p">(</span><span class="s2">&quot;unreg all&quot;</span><span class="p">)</span> <span class="c1">-- stop event monitor</span> <span class="kr">end</span> <span class="kr">end</span> <span class="kr">function</span> <span class="nf">doOnlineJob</span><span class="p">()</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">150</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span> <span class="c1">-- working, purple</span> <span class="n">http</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="s2">&quot;http://example.invalid/api/pushed&quot;</span><span class="p">,</span> <span class="kc">nil</span><span class="p">,</span> <span class="s1">&#39;{&quot;hello&quot;: &quot;from_esp_witty_42&quot;}&#39;</span><span class="p">,</span> <span class="kr">function</span><span class="p">(</span><span class="n">status_code</span><span class="p">,</span> <span class="n">body</span><span class="p">)</span> <span class="kr">if</span> <span class="n">status_code</span> <span class="o">==</span> <span class="kc">nil</span> <span class="ow">or</span> <span class="n">body</span> <span class="o">==</span> <span class="kc">nil</span> <span class="kr">then</span> <span class="nb">print</span><span class="p">(</span><span class="n">status_code</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="n">body</span><span class="p">)</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1">--fail red</span> <span class="kr">return</span> <span class="kr">end</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Got code &quot;</span> <span class="o">..</span> <span class="n">status_code</span> <span class="o">..</span> <span class="s2">&quot; answer &quot;</span> <span class="o">..</span> <span class="n">body</span><span class="p">)</span> <span class="kr">if</span> <span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span> <span class="kr">then</span> <span class="n">rgb</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">200</span><span class="p">)</span> <span class="c1">--success, blue</span> <span class="kr">end</span> <span class="kr">end</span><span class="p">)</span> <span class="kr">end</span> </pre></div> <p>This is reproducing the software function of the <a href="https://aws.amazon.com/fr/iot/button/">DASH/IoT Button</a>, <a href="https://makeit.netflix.com/the-switch">Netflix Switch</a> or <a href="https://flic.io/">Flic</a>.</p> <p>There are <a href="http://benlo.com/esp8266/esp8266Projects.html#thebutton">a few projects</a> that will <a href="https://www.hackster.io/noelportugal/ifttt-smart-button-e11841">guide you</a> through the <a href="http://deqingsun.github.io/ESP8266-Dash-Button/">hardware part</a> of building a button with an ESP module.</p> <p>PS: Be careful of big https cert chains, there's a <a href="https://github.com/nodemcu/nodemcu-firmware/blob/7ff8326cc9ed430abcc215be651ab6b8588dc57b/app/http/httpclient.c#L384">hardcoded limit of 5120 bytes for the SSL buffer</a> in the firmware, that might make the handshake fail.</p> <p>PPS: <strong>2016-07-01</strong> I did a <a href="https://docs.google.com/presentation/d/1x2xQUi3j3OUK7PLJu5Mx89TK0mD3az2KCAk8yDGhv_4/pub?start=false&amp;loop=false&amp;delayms=60000#slide=id.p">talk on ESP8266 modules</a> at the Paris Embedded <a href="http://www.meetup.com/ParisEmbedded/events/231095289/">Meetup #9</a>.</p>Bépo-android2015-01-27T00:00:00+01:002015-01-27T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2015-01-27:/bepo-android.html<p><a href="https://play.google.com/store/apps/details?id=fr.bepo.clavierexterne">This</a> is a small project I recently <a href="https://github.com/anisse/bepo-android/">released on github</a> and <a href="https://play.google.com/store/apps/details?id=fr.bepo.clavierexterne">Google Play</a>. It aims at catering to the needs of people using the bépo layout, and wanting to use it for <a href="http://bepo.fr/wiki/BepoAndroid">physical keyboards on Android</a>.</p> <p><a href="http://bepo.fr">Bépo</a> is a french dvorak-like keyboard layout; it was designed by a community of …</p><p><a href="https://play.google.com/store/apps/details?id=fr.bepo.clavierexterne">This</a> is a small project I recently <a href="https://github.com/anisse/bepo-android/">released on github</a> and <a href="https://play.google.com/store/apps/details?id=fr.bepo.clavierexterne">Google Play</a>. It aims at catering to the needs of people using the bépo layout, and wanting to use it for <a href="http://bepo.fr/wiki/BepoAndroid">physical keyboards on Android</a>.</p> <p><a href="http://bepo.fr">Bépo</a> is a french dvorak-like keyboard layout; it was designed by a community of enthousiasts, and is now included in Xorg. Its platform support is pretty good on the three main PC OSes, but <a href="http://bepo.fr/wiki/Android">limited on Android</a>. For physical keyboards, there's a paid app that supports the bépo layout (as part of whole package of other keymaps) but it requires you to use it as your input method, whereas since Android 4.1 it's possible to have custom keyboard layouts exported by apps and managed by the system.</p> <p>It's this facility that is used by <a href="https://github.com/anisse/bepo-android/">bepo-android</a> (or <a href="https://play.google.com/store/apps/details?id=fr.bepo.clavierexterne">Bépo clavier externe</a>): a <a href="http://developer.android.com/reference/android/hardware/input/InputManager.html#ACTION_QUERY_KEYBOARD_LAYOUTS">simple intent property</a> you declare in your manifest to tell the system that you're exporting keyboard layouts, with which you point to an xml file listing all your keyboard layouts, each pointing to a single .kcm file. bepo-android is currently exporting a <a href="https://github.com/anisse/bepo-android/blob/master/clavierexterne/src/main/res/raw/bepo.kcm">single layout file</a>, generated for bépo.</p> <p>The bépo project has this wonderful tool called the <a href="http://git.tuxfamily.org/dvorak/pilotes.git/tree/configGenerator">configGenerator</a> that allows regenerating the whole set of keymap files, images of the layout, configuration files for all platforms in a single command; which runs a few shell, perl and python scripts. This allowed the project to move fast when making modifications, but is today used mostly by enthousiasts creating their own variants (I used to have such a variant, but now I just use the official one). I created <a href="https://github.com/anisse/bepo-android/blob/master/android.py">such a script for the android platform</a>. So the script generates the necessary .kcm file, and could be re-run with your own customized bépo layout as source. An alternative possible use would be to use its ability to read xkb files to export Xorg keymaps to Android. This would probably need adaptation to add support for more exotic languages and characters.</p> <p>Android has this two-tier keyboard management system. First, there is the <a href="https://source.android.com/devices/tech/input/key-layout-files.html">key layout</a>, which maps evdev keycodes into key names. The key names currently match the ones Linux's input.h. There's a default key layout you can use as a base, <a href="https://android.googlesource.com/platform/frameworks/base.git/+/android-5.0.2_r1/data/keyboards/Generic.kl">Generic.kl</a>. Then there's the <a href="https://source.android.com/devices/input/key-character-map-files.html">key character map</a>, that will tell the system which unicode character to input when you press a given key. This is at least two order of magnitudes simpler than Xorg's system; although xkb is also much more powerful.</p> <p>As you might have guessed, the file generated for this project is a key character map, that also uses the undocumented "<a href="https://android.googlesource.com/platform/frameworks/native/+/android-5.0.2_r1/libs/input/KeyCharacterMap.cpp#694">map</a> <a href="https://android.googlesource.com/platform/frameworks/native/+/android-5.0.2_r1/libs/input/KeyCharacterMap.cpp#795">key</a>" directive, to remap a part of the key layout to make sure it's not changing under us; for instance Archos has a different key layout for its keyboards than the default.</p> <p>One of the tradeoff I had to make was to have the underlying key layout a QWERTY, that then generated bépo characters. This decision was due to the fact that we cannot attribute key names to all keys: ÉÀÈÊÇ, etc. don't have key labels (found in <a href="https://android.googlesource.com/platform/frameworks/base/+/android-4.1.2_r2.1/include/androidfw/KeycodeLabels.h">KeycodeLabels.h</a> or <a href="https://android.googlesource.com/platform/frameworks/native/+/android-5.0.2_r1/include/input/InputEventLabels.h">InputEventLabels.h</a> depending on your AOSP version), so you can't simply remap all keys in order to have a bépo key layout; you'd have holes in it. I therefore had to resort to using QWERTY as base, as it's the one in <a href="https://android.googlesource.com/platform/frameworks/base.git/+/android-5.0.2_r1/data/keyboards/Generic.kl">Generic.kl</a>. I'm also secretly hoping it might help with badly-programmed games that have keyboard support, assume qwerty, and don't allow key remapping. If those exist on Android. (but they are legion on the web, which is very annoying).</p> <p>The bépo key character map also maps the <a href="https://source.android.com/devices/input/key-character-map-files.html#behaviors">documented special diacritical dead keys</a>; this is quite useful, but not as complete as xkb's many dead keys; and not nearly as powerful Xorg's Compose; so not all bépo dead and compose keys are supported.</p> <p>This currently only works with devices having Android 4.1+, provided the manufacturer didn't botch the external keyboard support, as Asus did on my Fonepad 7 (K019), and as a user reported a Samsung Galaxy Note 10.1 to be. OEMs do that to allow synchronisation between the virtual and physical keyboard layout, but this is just wrong if it removes the user's ability to chose his own keymap.</p> <p>I have yet to hear from other non-working devices; after a month or so, the app hasn't seen much traction (Google Play says there are less than 50 installs); so maybe in the niche that is bépo, there isn't much interest in typing stuff on Android. We could even wonder if people would ever do productive work on this platform. But that's a debate for another day.</p> <p>At least I scratched my itch =)</p> <p>Get <a href="https://android.googlesource.com/platform/frameworks/base.git/+/android-5.0.2_r1/data/keyboards/Generic.kl">Bépo clavier externe on Google Play</a>.</p>Testing a NAS hard drive over FTP2013-12-20T00:00:00+01:002013-12-20T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2013-12-20:/testing-a-nas-hard-drive-over-ftp.html<p>So I have this NAS made by my ISP, that does a lot of things; but recently, I started having issues with its behavior. Recorded TV shows had lag/jitter while replaying, and the same happened with other types of videos I put on it. I narrowed it down to …</p><p>So I have this NAS made by my ISP, that does a lot of things; but recently, I started having issues with its behavior. Recorded TV shows had lag/jitter while replaying, and the same happened with other types of videos I put on it. I narrowed it down to the hard drive, which was sometime providing read speeds of less than 300 KiB/s. I cannot open it to test the hard drive more thoroughly, using mhdd or ATA SMART tests. I'll have to innovate a little.</p> <p>In this post, in the form of an ipython3 notebook(<a href="/static/FTP%20fun.ipynb">source</a>), I'm going to test the hard drive over ftp, by doing a full hard drive fill, and then a read. I'm going to :</p> <ul> <li>measure the read and write speed to see if the problem is still present after I formated it.</li> <li>make sure what I wrote is the same as what I read</li> <li>I'll have to make sure that I can generate data fast enough</li> <li>And I'll try to make the data look "random" so that I don't stumble upon some compression in the FTP -&gt; fs &gt; hard drive chain.</li> </ul> <p>If everything is well I'll just get on with it: formatting the hard drive fixed the issue. Otherwise, it's might be a hardware problem, and I'll have to exchange it.</p> <p>To generate the data, I'll use an md5 hash for its nice output which looks fairly random, and this is very hard to compress. I chose md5 because it's fast. I'll use a sequential index as the input so that it's deterministic and I can fairly easily re-generate the input data for comparison.</p> <div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">hashlib</span> <span class="n">h</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s2">&quot;md5&quot;</span><span class="p">)</span> <span class="c1">#Generate a deterministic hash</span> <span class="k">def</span> <span class="nf">data</span><span class="p">(</span><span class="n">i</span><span class="p">):</span> <span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">i</span><span class="p">))</span> <span class="k">return</span> <span class="n">h</span><span class="o">.</span><span class="n">digest</span><span class="p">()</span> <span class="kn">import</span> <span class="nn">time</span> <span class="k">def</span> <span class="nf">testdata</span><span class="p">():</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">10000</span> <span class="n">size</span> <span class="o">=</span> <span class="n">h</span><span class="o">.</span><span class="n">digest_size</span> <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="n">data</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="n">end</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span> <span class="n">speed</span> <span class="o">=</span> <span class="n">n</span><span class="o">*</span><span class="n">size</span><span class="o">/</span><span class="p">(</span><span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;We generated </span><span class="si">%d</span><span class="s2"> bytes in </span><span class="si">%f</span><span class="s2"> s </span><span class="si">%d</span><span class="s2"> B/s&quot;</span><span class="o">%</span><span class="p">(</span><span class="n">n</span><span class="o">*</span><span class="n">size</span><span class="p">,</span> <span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">,</span> <span class="n">speed</span><span class="p">))</span> <span class="n">testdata</span><span class="p">()</span> </pre></div> <pre> We generated 160000 bytes in 1.610000 s 99378 B/s </pre> <p>Ouch. I use a slow machine, and it's far from the at least 60MiB/s I need to thoroughly test the hard drive. Let's see if I can find a faster hash.</p> <div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">testallhashes</span><span class="p">():</span> <span class="k">global</span> <span class="n">h</span> <span class="k">for</span> <span class="nb">hash</span> <span class="ow">in</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">algorithms_available</span><span class="p">:</span> <span class="n">h</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="nb">hash</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="nb">hash</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">&#39; &#39;</span><span class="p">)</span> <span class="n">testdata</span><span class="p">()</span> <span class="n">testallhashes</span><span class="p">()</span> </pre></div> <pre> SHA1 We generated 200000 bytes in 2.570000 s 77821 B/s SHA512 We generated 640000 bytes in 6.780000 s 94395 B/s RIPEMD160 We generated 200000 bytes in 2.870000 s 69686 B/s SHA224 We generated 280000 bytes in 3.480000 s 80459 B/s sha512 We generated 640000 bytes in 6.770000 s 94534 B/s md5 We generated 160000 bytes in 1.620000 s 98765 B/s md4 We generated 160000 bytes in 1.350000 s 118518 B/s SHA256 We generated 320000 bytes in 3.500000 s 91428 B/s ripemd160 We generated 200000 bytes in 2.870000 s 69686 B/s whirlpool We generated 640000 bytes in 19.590000 s 32669 B/s dsaEncryption We generated 200000 bytes in 2.580000 s 77519 B/s sha384 We generated 480000 bytes in 6.800000 s 70588 B/s sha1 We generated 200000 bytes in 2.570000 s 77821 B/s dsaWithSHA We generated 200000 bytes in 2.580000 s 77519 B/s SHA We generated 200000 bytes in 2.580000 s 77519 B/s sha224 We generated 280000 bytes in 3.490000 s 80229 B/s DSA-SHA We generated 200000 bytes in 2.570000 s 77821 B/s MD5 We generated 160000 bytes in 1.600000 s 99999 B/s sha We generated 200000 bytes in 2.570000 s 77821 B/s MD4 We generated 160000 bytes in 1.350000 s 118518 B/s ecdsa-with-SHA1 We generated 200000 bytes in 2.570000 s 77821 B/s sha256 We generated 320000 bytes in 3.490000 s 91690 B/s SHA384 We generated 480000 bytes in 6.780000 s 70796 B/s DSA We generated 200000 bytes in 2.590000 s 77220 B/s </pre> <p>Well, no luck. I'll just use a big buffer and have it loop around.</p> <div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">bigbuffer</span><span class="p">():</span> <span class="k">global</span> <span class="n">h</span> <span class="n">h</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s2">&quot;md5&quot;</span><span class="p">)</span> <span class="n">buf</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">()</span> <span class="n">count</span> <span class="o">=</span> <span class="mi">2</span><span class="o">**</span><span class="mi">18</span> <span class="o">//</span> <span class="n">h</span><span class="o">.</span><span class="n">digest_size</span> <span class="c1"># we want a 256KiB buffer</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span> <span class="n">buf</span> <span class="o">+=</span> <span class="n">data</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">return</span> <span class="n">buf</span> <span class="k">assert</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">bigbuffer</span><span class="p">())</span> <span class="o">==</span> <span class="mi">262144</span><span class="p">)</span> <span class="c1"># verify the length</span> </pre></div> <p>That's for the basics.</p> <div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">CustomBuffer</span><span class="p">:</span> <span class="sd">&quot;&quot;&quot;</span> <span class="sd"> A wrap-around file-like object that returns in-memory data from buf</span> <span class="sd"> &quot;&quot;&quot;</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">limit</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">buf</span> <span class="o">=</span> <span class="n">bigbuffer</span><span class="p">()</span> <span class="bp">self</span><span class="o">.</span><span class="n">bufindex</span> <span class="o">=</span> <span class="mi">0</span> <span class="bp">self</span><span class="o">.</span><span class="n">fileindex</span> <span class="o">=</span> <span class="mi">0</span> <span class="bp">self</span><span class="o">.</span><span class="n">bufsize</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">buf</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">limit</span> <span class="o">=</span> <span class="n">limit</span> <span class="k">def</span> <span class="nf">readloop</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="o">=</span><span class="mi">8096</span><span class="p">):</span> <span class="n">dat</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">buf</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">bufindex</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">bufindex</span> <span class="o">+</span> <span class="n">i</span><span class="p">]</span> <span class="n">end</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">bufindex</span> <span class="o">+</span> <span class="n">i</span> <span class="k">while</span> <span class="n">end</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">bufsize</span><span class="p">:</span> <span class="n">end</span> <span class="o">-=</span> <span class="bp">self</span><span class="o">.</span><span class="n">bufsize</span> <span class="n">dat</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">buf</span><span class="p">[:</span><span class="n">end</span><span class="p">]</span> <span class="bp">self</span><span class="o">.</span><span class="n">bufindex</span> <span class="o">=</span> <span class="n">end</span> <span class="k">return</span> <span class="n">dat</span> <span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="o">=</span><span class="mi">8096</span><span class="p">):</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">limit</span> <span class="o">==</span> <span class="bp">None</span><span class="p">:</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">readloop</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">fileindex</span> <span class="o">&gt;=</span> <span class="bp">self</span><span class="o">.</span><span class="n">limit</span><span class="p">:</span> <span class="k">return</span> <span class="nb">bytes</span><span class="p">()</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">fileindex</span> <span class="o">+</span> <span class="n">i</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">limit</span><span class="p">:</span> <span class="n">dat</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">readloop</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">limit</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">fileindex</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">fileindex</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">limit</span> <span class="k">return</span> <span class="n">dat</span> <span class="bp">self</span><span class="o">.</span><span class="n">fileindex</span> <span class="o">+=</span> <span class="n">i</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">readloop</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">def</span> <span class="nf">testreadcbuf</span><span class="p">():</span> <span class="n">f</span> <span class="o">=</span> <span class="n">CustomBuffer</span><span class="p">(</span><span class="mi">2548</span><span class="p">)</span> <span class="k">assert</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">2048</span><span class="p">))</span> <span class="o">==</span> <span class="mi">2048</span><span class="p">)</span> <span class="k">assert</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">())</span> <span class="o">==</span> <span class="mi">500</span><span class="p">)</span> <span class="n">testreadcbuf</span><span class="p">()</span> <span class="k">def</span> <span class="nf">testcbuf</span><span class="p">(</span><span class="n">limit</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="n">f</span> <span class="o">=</span> <span class="n">CustomBuffer</span><span class="p">(</span><span class="n">limit</span><span class="p">)</span> <span class="n">l</span> <span class="o">=</span> <span class="mi">0</span> <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10000</span><span class="p">):</span> <span class="n">l</span> <span class="o">+=</span> <span class="nb">len</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">())</span> <span class="n">end</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span> <span class="n">speed</span> <span class="o">=</span> <span class="n">l</span><span class="o">/</span><span class="p">(</span><span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;We generated </span><span class="si">%d</span><span class="s2"> bytes in </span><span class="si">%f</span><span class="s2"> s </span><span class="si">%d</span><span class="s2"> B/s&quot;</span><span class="o">%</span><span class="p">(</span><span class="n">l</span><span class="p">,</span> <span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">,</span> <span class="n">speed</span><span class="p">))</span> <span class="n">testcbuf</span><span class="p">()</span> </pre></div> <pre> We generated 80960000 bytes in 0.780000 s 103794871 B/s </pre> <p>That's more in line with what we need.</p> <h1>the FTP stuff</h1> <div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">ftplib</span> <span class="kn">from</span> <span class="nn">ftpconfig</span> <span class="kn">import</span> <span class="n">config</span><span class="p">,</span> <span class="n">config_example</span> <span class="c1"># ftp credentials, etc</span> <span class="k">print</span><span class="p">(</span><span class="n">config_example</span><span class="p">)</span> </pre></div> <pre> {'password': 'verylongandcomplicatedpassword', 'host': '192.168.1.254', 'path': '/HD/', 'username': 'boitegratuite'} </pre> <div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">ftpconnect</span><span class="p">():</span> <span class="n">ftp</span> <span class="o">=</span> <span class="n">ftplib</span><span class="o">.</span><span class="n">FTP</span><span class="p">(</span><span class="n">config</span><span class="p">[</span><span class="s1">&#39;host&#39;</span><span class="p">])</span> <span class="n">ftp</span><span class="o">.</span><span class="n">login</span><span class="p">(</span><span class="n">config</span><span class="p">[</span><span class="s1">&#39;username&#39;</span><span class="p">],</span> <span class="n">config</span><span class="p">[</span><span class="s1">&#39;password&#39;</span><span class="p">])</span> <span class="n">ftp</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">config</span><span class="p">[</span><span class="s1">&#39;path&#39;</span><span class="p">])</span> <span class="k">return</span> <span class="n">ftp</span> <span class="k">def</span> <span class="nf">transfer_rate</span><span class="p">(</span><span class="n">prev_stamp</span><span class="p">,</span> <span class="n">now</span><span class="p">,</span> <span class="n">blocksize</span><span class="p">):</span> <span class="n">diff</span> <span class="o">=</span> <span class="n">now</span> <span class="o">-</span> <span class="n">prev_stamp</span> <span class="n">rate</span> <span class="o">=</span> <span class="n">blocksize</span><span class="o">/</span><span class="p">(</span><span class="n">diff</span><span class="o">*</span><span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">)</span> <span class="c1"># store in MiB/s directly</span> <span class="k">return</span> <span class="p">[</span><span class="n">now</span><span class="p">,</span> <span class="n">rate</span><span class="p">]</span> <span class="k">def</span> <span class="nf">store</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">2</span><span class="o">**</span><span class="mi">25</span><span class="p">,</span> <span class="n">blocksize</span><span class="o">=</span><span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">):</span> <span class="n">values</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">watch</span><span class="p">(</span><span class="n">block</span><span class="p">):</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="n">values</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">transfer_rate</span><span class="p">(</span><span class="n">t1</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">t2</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">block</span><span class="p">)))</span> <span class="n">t1</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">t2</span> <span class="n">ftp</span> <span class="o">=</span> <span class="n">ftpconnect</span><span class="p">()</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">CustomBuffer</span><span class="p">(</span><span class="n">size</span><span class="p">)</span> <span class="n">t1</span> <span class="o">=</span> <span class="p">[</span><span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()]</span> <span class="k">try</span><span class="p">:</span> <span class="n">ftp</span><span class="o">.</span><span class="n">storbinary</span><span class="p">(</span><span class="s2">&quot;STOR filler&quot;</span><span class="p">,</span> <span class="n">buf</span><span class="p">,</span> <span class="n">blocksize</span><span class="o">=</span><span class="n">blocksize</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="n">watch</span><span class="p">)</span> <span class="n">ftp</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="k">except</span> <span class="n">ConnectionResetError</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Connection severed by peer&quot;</span><span class="p">)</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Transfer interrupted:&quot;</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="k">return</span> <span class="n">values</span> <span class="n">values</span> <span class="o">=</span> <span class="n">store</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">27</span><span class="p">)</span> </pre></div> <p>Now trying to show those values !</p> <div class="highlight"><pre><span></span><span class="o">%</span><span class="n">pylab</span> <span class="n">inline</span> <span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="kn">as</span> <span class="nn">plt</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> <span class="n">a</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">values</span><span class="p">)</span><span class="o">.</span><span class="n">transpose</span><span class="p">()</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">a</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;rate (MiB/s)&quot;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;time (s)&quot;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span> </pre></div> <pre> Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline]. For more information, type 'help(pylab)'. </pre> <p><img alt="png" src="/images/FTP%20fun_14_1.png"></p> <p>Ok, we have what we wanted: we can measure the write speeds.</p> <p>Now let's check read speeds.</p> <div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">reread</span><span class="p">(</span><span class="n">blocksize</span><span class="o">=</span><span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">):</span> <span class="n">values</span><span class="o">=</span><span class="p">[]</span> <span class="n">verif</span> <span class="o">=</span> <span class="n">CustomBuffer</span><span class="p">()</span> <span class="n">i</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">def</span> <span class="nf">watch</span><span class="p">(</span><span class="n">block</span><span class="p">):</span> <span class="n">t2</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()</span> <span class="n">values</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">transfer_rate</span><span class="p">(</span><span class="n">t1</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">t2</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">block</span><span class="p">)))</span> <span class="n">dat</span> <span class="o">=</span> <span class="n">verif</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">block</span><span class="p">))</span> <span class="k">if</span> <span class="n">dat</span> <span class="o">!=</span> <span class="n">block</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;ERROR !!!! Data read isn&#39;t correct at block&quot;</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="n">t1</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">t2</span> <span class="n">i</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span> <span class="n">ftp</span> <span class="o">=</span> <span class="n">ftpconnect</span><span class="p">()</span> <span class="n">t1</span> <span class="o">=</span> <span class="p">[</span><span class="n">time</span><span class="o">.</span><span class="n">perf_counter</span><span class="p">()]</span> <span class="k">try</span><span class="p">:</span> <span class="n">ftp</span><span class="o">.</span><span class="n">retrbinary</span><span class="p">(</span><span class="s2">&quot;RETR filler&quot;</span><span class="p">,</span> <span class="n">blocksize</span><span class="o">=</span><span class="n">blocksize</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="n">watch</span><span class="p">)</span> <span class="n">ftp</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Transfer interrupted:&quot;</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="k">return</span> <span class="n">values</span> <span class="k">def</span> <span class="nf">plot_transfer_speed</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">title</span><span class="p">):</span> <span class="k">def</span> <span class="nf">average</span><span class="p">(</span><span class="n">arr</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span> <span class="n">end</span> <span class="o">=</span> <span class="n">n</span> <span class="o">*</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">arr</span><span class="p">)</span><span class="o">//</span><span class="n">n</span><span class="p">)</span> <span class="k">return</span> <span class="n">numpy</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">arr</span><span class="p">[:</span><span class="n">end</span><span class="p">]</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">n</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span> <span class="n">a</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="o">.</span><span class="n">transpose</span><span class="p">()</span> <span class="n">a0</span> <span class="o">=</span> <span class="n">average</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">max</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">//</span><span class="mi">300</span><span class="p">))</span> <span class="n">a1</span> <span class="o">=</span> <span class="n">average</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="nb">max</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="o">//</span><span class="mi">300</span><span class="p">))</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">a0</span><span class="p">,</span> <span class="n">a1</span><span class="p">)</span> <span class="c1">#plt.setp(lines, aa=True)</span> <span class="n">plt</span><span class="o">.</span><span class="n">gcf</span><span class="p">()</span><span class="o">.</span><span class="n">set_size_inches</span><span class="p">(</span><span class="mi">22</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;MiB/s&quot;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;seconds&quot;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="n">title</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span> <span class="n">rval</span> <span class="o">=</span> <span class="n">reread</span><span class="p">()</span> <span class="n">plot_transfer_speed</span><span class="p">(</span><span class="n">rval</span><span class="p">,</span> <span class="s2">&quot;Read speed&quot;</span><span class="p">)</span> </pre></div> <p><img alt="png" src="/images/FTP%20fun_17_0.png"></p> <p>We have all the pieces now. Let's do the filling and plot the data.</p> <div class="highlight"><pre><span></span><span class="c1"># About 230Gio is the effective size of the disk (almost 250Go)</span> <span class="n">val</span> <span class="o">=</span> <span class="n">store</span><span class="p">(</span><span class="mi">230</span><span class="o">*</span><span class="mi">2</span><span class="o">**</span><span class="mi">30</span><span class="p">,</span> <span class="mi">10</span><span class="o">*</span><span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">)</span> </pre></div> <pre> Connection severed by peer </pre> <div class="highlight"><pre><span></span><span class="n">plot_transfer_speed</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="s2">&quot;Write speed&quot;</span><span class="p">)</span> </pre></div> <p><img alt="png" src="/images/FTP%20fun_20_0.png"></p> <div class="highlight"><pre><span></span><span class="n">rval</span> <span class="o">=</span> <span class="n">reread</span><span class="p">(</span><span class="mi">10</span><span class="o">*</span><span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">)</span> <span class="n">plot_transfer_speed</span><span class="p">(</span><span class="n">rval</span><span class="p">,</span> <span class="s2">&quot;Read speed&quot;</span><span class="p">)</span> </pre></div> <p><img alt="png" src="/images/FTP%20fun_22_0.png"></p> <h1>That's it !</h1> <p>We could also plot the read speed against the disk byte index instead of time, which would maybe be more interesting. This is left as an exercise for the reader.</p> <p>Regarding the data, it's far from what we could get with ATA SMART (using smartctl or skdump/sktest), but it's interesting nonetheless. We can see the read speed falls sometimes, which may be indicative of localized hard drive problem.</p> <p>What does not appear here is that I've ran the tests multiple times to make sure the data is correct. And both the read and write are long, multi-hour tests.</p> <p>Also, the simple fact of making a write test may fix an existent problem, by making the disk's firmware aware of the existence of bad blocks. This is amplified by the fact that I ran the tests multiple times.</p> <p>Finally, what could be improved is having a better way to display a high number of data points. I've used here the average method, which might not show how low the read/write speed can go locally. Maybe displaying the data using a vector format would be better (svg, python-chaco ?).</p> <p>Regarding the decision to dispose of the hard drive or the NAS, I think I'll keep it for now until it dies, but I'll start putting my data on a external HDD (plugged to the ISP box), and only trust the internal hard drive with low priority stuff like the occasional video recording.</p>Embedded Linux Conference Europe 2013 notes2013-10-30T00:00:00+01:002013-10-30T00:00:00+01:00Anisse Astiertag:anisse.astier.eu,2013-10-30:/embedded-linux-conference-europe-2013-notes.html<p>So I was in Edinburgh this year, and I took notes as I usually do. These are intended for personal consumption (do no expect LWN-style reports), but as more people were asking me to share them, I thought why not do it in public ?</p> <h1>Embedded Linux timeline</h1> <p>by Chris Simmonds …</p><p>So I was in Edinburgh this year, and I took notes as I usually do. These are intended for personal consumption (do no expect LWN-style reports), but as more people were asking me to share them, I thought why not do it in public ?</p> <h1>Embedded Linux timeline</h1> <p>by Chris Simmonds</p> <p>Busybox was started by Bruce Perens to solve the floppy installation problem. The first Linux Router was described in "Arlan Wireless Howto".</p> <p>Linux gained portability to other architectures over time:</p> <ul> <li>1995: MIPS</li> <li>1996: m68k, ppc</li> <li>1998: m68k Dragon Ball Palm Pilot : creation of uClinux (no mmu)</li> <li>1999: ARM</li> </ul> <p>Flash memory support was added by David Woodhouse in 1999 (MTD layer), then JFFS by AXIS for their IP cameras.</p> <h2>Devices</h2> <p>Things really started in 1999: AXIS IP camera, TiVo DVR, Kerbango Internet Radio(Threecom). Lot of media coverage at that time.</p> <p>Companies sprung out to service embedded linux: Timesys, MontaVista, Lineo, Denx.</p> <p>The handhelds.org project aimed at porting of linux to Compaq iPaq H3600. Cross-compiling being a pain in the arse, they had a cluster of ~16 iPaqs to compile code.</p> <p>In 2001, there was the infamous Unobtainium : handset prototype at Compaq based on iPaq hardware with GSM/CDMA/Wifi/Bluetooth, camera, accelerometer, 1GiB of storage: it was really the first smartphone prototype. Never shipped.</p> <p>At the same time, Sharp made the Zaurus running Linux 2.4.10 (software made by Lineo).</p> <p>In 2003, Motorola made the A760 handset, first Linux handset (MontaVista) In 2005, the Nokia 770, the first Internet Tablet running Maemo Linux.</p> <h2>Buildtools and software</h2> <p>In 2001 buildroot was created from the needs of the uClinux project: it's still the oldest and simplest build system. Then OpenEmbedded in 2003. Then in 2004 Poky Linux based on OE by OpenedHand, then Yocto. In my opinion Chris has a narrow view of the build systems choice (what about Debian?)</p> <p>Real-time was at first achieved with sub-kernels, like Xenomai. Then Native Real-time : Linux/RT 1.0 by Timesys in 2000. Then the voluntary preempt patch (Ingo Molnar &amp; Andrew Morton). Robert Love kernel preemption patch in 2001. In 2003 Linux 2.6 includes voluntary preempt. In 2005 PREEMPT_RT was started, in 2013, not all of it is merged yet.</p> <p>In the end: Linux is the "default" embedded OS.</p> <h1>How not to write x86 platform drivers</h1> <p>by Darren Hart</p> <p>This talk was mostly a feedback around getting the Minnowboard mainlined properly.</p> <p>At Intel a platform is CPU + chipset, or a SoC. In Linux, it represent things that are not on a real bus, or things that cannont not be enumerated, leading to board fils drivers.</p> <p>The Minnowboard uses a 32bit UEFI Firmware. One of the first designs to make use of all Queensbay(Intel SoC) GPIOs. The UART clock is special (50mhz). Low-cost Ethernet phy with no EEPROM for macs. The Minnowboard is a dynamic baseboard, which is very different from what Intel usually does: it supports daughter cards.</p> <p>There are three main sources of GPIOs on this board (5 core, 8 suspend, 12 pch), 4 user buttons, 2 user LEDs, phy reset, then expansion GPIOs.</p> <h2>Board files</h2> <p>MinnowBoard used board files at first because they are simple to use.</p> <p>Those were rejected. Why ?</p> <ul> <li>not automatically enumerated and loaded</li> <li>adds maintenance</li> <li>independent drivers had to be board aware</li> </ul> <p>All this leads to "evil" vendor trees.</p> <p>UART clock is firmware dependent. Previous code used DMI detection, which isn't nice.</p> <p>Ethernet is complicated: aggressive power saving meaning you must wake it up open. How to id the PHY ? You could use SMBIOS/DMI, DT, ACPI; in the end PCI subsystem ID were used. Initialized with platform_data.</p> <p>The MAC: no EEPROM, so had to solve how to get a MAC. Was done in firmware in the end: read the SPI flash, then write PCI registers.</p> <p>To preserve the platform on should not create vendor trees. The complexity of core kernel vs drivers is inverted: core kernel has simple primitives, but complex algorithms. Drivers are the opposite: simple to understand, but hard to organize, how they fit together.</p> <h2>GPIO, take 2: ACPI 5.0</h2> <p>A lot of things can be done with the new ACPI standard, like identify GPIO resources. You can't do keybindings, default trigger, etc. Some vendors (like Apple) do already, but with their own proprietary additions.</p> <p>One needs to write ASL for the DSDT. You might want to have dictionaries to describe your hardware, which needs to be standardized. Right now ACPI reserved method are used (_PRP).</p> <h1>Device trees for dummies</h1> <p>by Thomas Pettazoni (<a href="http://free-electrons.com/pub/conferences/2013/elce/petazzoni-device-tree-dummies/petazzoni-device-tree-dummies.pdf">Slides</a>)</p> <p>Before DT, all the information was inside the kernel. Little information in ATAGS. Now all information is in DT.</p> <p>Device Tree is a tree data structure to describe hardware that cannot be enumerated. You compile a DTS (source) into a DTB(binary). In arm, all DTS are in arch/arm/boot/dts and automatically compiled for your board. Device Tree bindings is the "standard" for how you should describe the hardware using the DT language, and what the driver understands. All bindings should be documented and reviewed. DT should not describe configuration, just hardware layout. The problem isn't solved yet for configuration.</p> <p>The talk had a lot of nice syntax examples to learn how to write DTS. For example, Thomas explained the importance of compatible string, which are used to match DTS node device with a driver.</p> <p>Should DT be an ABI ? Hard question. While it was the original idea, maybe it shouldn't. Current discussions seem to want to relax the stable ABI rule.</p> <h1>Use case power management</h1> <p>by Patrick Titiano (<a href="http://events.linuxfoundation.org/sites/events/files/slides/Use-Case%20Power%20Management%20Optimization%20ELC-E%20Presentation.pdf">Slides</a>)</p> <p>First rule about PM: shutdown anything not used. You need to track running power resources: it starts with the clock tree.</p> <p>Things to monitor:</p> <ul> <li>C-states/idle states stats.</li> <li>Operating point statistics</li> <li>CPU &amp; HW load</li> <li>memory bandwidth : most often a bottleneck</li> </ul> <p>You need to instrument both software and hardware. It means you need resistors points in the PCB, temp sensors. You need to automate everything, otherwise you're not comparing apple to apples. All measurements should be automated to be easily reproduced.</p> <p>You need to have power model of your raw soc consumption and characteristics, then you need to assess this model to verify that the target is realistic.</p> <p>Voltage is more important than frequency. It's easier to reduce consumption than to find better way to dissipate energy.</p> <p>Battery is king. You need a full system view, because you should optimize the biggest offenders first. Take care of inter-dependent stuff.</p> <h1>Android debugging tools</h1> <p>by Karim Yaghmour (<a href="http://opersys.com/downloads/android-platform-debug-dev-clean-131030.pdf">Slides</a>)</p> <p>Android usually runs on an SoC. It uses bionic, a different libc, it has a Hardware Abstraction Layer that allows proprietary drivers in userspace. Toolbox is lesser busybox clone in BSD.</p> <p>Binder is a standard component, object IPC, that almost defines system android. Every system services uses that.</p> <p>To debug, it's handy to load AOSP in eclipse. You can load all the OS classes and apps in the editor to be able to trace anything and browse AOSP while seeing classes, call chains, etc. It's more powerful for browsing and live debugging than common editors. You still have to build AOSP by hand (type <code>make</code>/<code>lunch</code>) to generate an image.</p> <p>A few tools:</p> <ul> <li><code>latencytop</code>, <code>schedtop</code>, etc.</li> <li><code>dumpsys</code></li> <li><code>service</code></li> <li><code>logcat</code></li> <li><code>dumpstate</code> (root app), <code>bugreport</code> : dumps system state (in /proc, etc.)</li> <li><code>watchprop</code></li> </ul> <p>Logging goes through the logger driver.</p> <p>Interfaces with the system:</p> <ul> <li><code>start</code>/<code>stop</code> : stops <code>zygote</code>, which means you can shutdown/start the interface and all the java stuff</li> <li><code>service call statusbar</code> {1,2,5 s16 alarm_clock i32 0} : you can call methods directly that are defined in an aidl file, by using their implicit sequence number. It's useful to bypass the java framework and call the services directly. see in android/os/IPowerManager.aidl for example, or android/internal/statusbar/IStatusBar.aidl for the previous example</li> <li><code>am</code> : tool to call intents. e.g am start -a android.intent.action.VIEW -d http://webpage.com . Very powerful tool to call intents</li> <li><code>pm</code> : calls to the package manager</li> <li><code>wm</code> : calls to the windows manager</li> </ul> <p>When working with AOSP sources, source <code>build/envsetup.sh</code>, it has very handy functions, like:</p> <ul> <li><code>godir</code>: jumps to a dir a file is in</li> <li><code>mm</code> : rebuilds the tree</li> <li><code>jgrep</code>/<code>cgrep</code>/<code>resgrep</code> : grep for specific files (java/c/resource)</li> <li><code>croot</code> : jump up to aosp root</li> </ul> <p>Take care when working it AOSP, it's BIG (about 8GB)</p> <p>When debugging, you have to use a different debugger depending on the use case:</p> <ul> <li><code>ddms</code> for dalvik level stuff</li> <li><code>gdb</code>/<code>gdbserver</code> for HAL stuff</li> <li>JTAG for kernel</li> </ul> <p>DDMS talks JDWP (Java Debug Wire Protocol). Use the one from AOSP, not eclipse. It's very powerful to debug (java) system processes live.</p> <p><code>gdbserver</code>: you have to configure your app's Android.mk to have -ggdb, and disable stripping. You also have to do port forwarding with adb in order to access gdbserver:</p> <div class="highlight"><pre><span></span>adb forward tcp:2345 tcp:2345 </pre></div> <p>You can use the prebuilts arm-eabi-gdb, but Multi-thread might not be supported.</p> <h3>logging</h3> <ul> <li>logcat works</li> <li>ftrace is supported through systrace</li> <li>atrace (device dependent ?)</li> <li>perf is not well supported on ARM</li> </ul> <h1>Embbedded build systems showdown</h1> <p>A nice panel, where we had representatives of different build systems: - DIY : Tim Bird, Sony - Android : Karim Yaghmour, Opersys - Buildroot : Thomas Petazzoni, Free Electrons - Yocto : Jeff Osier-Mixon, Intel</p> <p>It was very friendly. The take-out is that each system addresses different use cases. Yocto is big-company friendly, because it has metadata and licence management built-in. Tim Bird said he had a personal preference for Buildroot as a developer, although a division of his company recently switch to Yocto for its projects.</p> <p>Karim's opinion was that although Android wasn't community friendly, that it couldn't integrate with anything external, it was king in term of market traction, and that it might be the most used system in any kind of embedded device in 4 to 5 years.</p> <h1>Best practices for long-term support and security of the device-tree</h1> <p>by Alison Chaiken (<a href="http://events.linuxfoundation.org/sites/events/files/slides/DT_ELCE_2013.pdf">Slides</a> <a href="http://she-devel.com/DT_ELCE_2013.pdf">at Author's</a>)</p> <p>DT make life a bit easier, although there are pitfalls. Best practices could help with that matter.</p> <p>Updates are hard without DT. How about with DT ? Should you update the DTB ?</p> <p>DTs are supposed to be for HW description, but there already many configuration items in DT: MTD partition tables, boot device selection, pinmux. Alison gave example about automotive and battery technology that's evolving that would allow updating electric car's battery. Cars have a lot of processors; e.g 2014 Mercedes S-Class will have 200MCUs on Ethernet, CAN-FD and LIN.</p> <p>One thing to be careful about is Kconfig and DTS matching.</p> <p>One pitfall you might have, is unintentionally breaking DT or device by changing something in a driver or another device. Example about Koen Kooi's post who said you might blow an HDMI transceiver on some board if you boot with micro SD, because micro SD uses a higher voltage by default.</p> <p>You can use .its to bundle DTS, kernel and other blobs in one .itb file. Support was added in u-boot to sign .itbs by ChromeOS engineers.</p> <p>One option floating around, presented by Pantelis, is to use DTS runtime overlays as an update method, similar to unionfs.</p> <p>DTS schema validator looks like a good thing to have, like Stephen Warren's very recent proposal.</p> <h1>Android on a non-mobile embedded system</h1> <p>by Arnout Vandecapelle</p> <p>The main motivation is the reduced time to market, and the wealth of available app developers.</p> <p>It's interesting because it's still linux, but there are few differences (bionic libc, special build system)</p> <p>My own impressions: lots of generic stuff, from someone who just recently went into android. Like most of this stuff, doesn't come from Google, so it has little "new" information in it. It was a nice conference if you've never heard of AOSP and have only been using other embedded distros/build systems.</p> <h1>BuildRoot : What's new ?</h1> <p>by Peter Korsgaard</p> <h2>BuildRoot</h2> <p>BuildRoot is an Embedded Linux build system. It's one of the oldest out there, and is fairly well documented. It has an active community. It's relatively simple, and that's a focus of the project to Keep It Simple (Stupid). For example, there's no binary package management.</p> <p>It's Kconfig-based for configuration, and uses make for building.</p> <p>Buildroot is package-based, and a build step just runs every package build. It's therefore a meta build system. A package is composed of :</p> <ul> <li>a config (in kconfig format) for dependencies, description, etc. You need to include this config under the parent config option.</li> <li>a makefile (Package.mk) with the build steps</li> </ul> <p>Buildroot is using git for its source code, patches are posted on ML and managed in Patchwork.</p> <p>Buildroot activity has been growing over the years: more emails on ML(~1000/month), more contributors (30-40 each month). Developer days are held 2 times a year (this year at FOSDEM and ELCE).</p> <p>Buildroot is used in many products(Barco, Google fiber), and SDKs (Atmel, Synopsys, Cadence, Imagination…)</p> <h2>What's new ?</h2> <p>It supports more architectures (ARC, Blackfin, Microblaze, Xtensa…), and the variant support has been improved (ARM softfp/hardfp/neon…), as well as the nommu support.</p> <p>Buildroot now supports more toolchains: C library (glibc, eglibc, uclibc), and external toolchains.</p> <p>Buildroot has 30% more packages than last year. A lot of stuff has been added (gstreamer, EFL, wayland, systemd, perf, python3, nodejs…). A GSoC student worked on adding ARM proprietary GPU and video drivers support.</p> <p>QA has been improved as well, with continuous integration/regression testing. The development cycle is now 3 months, with 1 month of stabilization.</p> <p>License compliance has been added: every package should have a license, and "make legal_info" generates all the necessary stuff.</p> <p>There's a new Eclipse CDT plugin. Popular boards got their own defconfigs to ease starting.</p> <p>A lot of configuration options were added to the menuconfig. New options to add a rootfs overlay, or last-second hook scripts.</p> <p>Upcoming work includes external packages overlays, SELinux support, updated systemd/udev, and whatever else gets submitted.</p>Hello, World!2013-10-20T00:00:00+02:002013-10-20T00:00:00+02:00Anisse Astiertag:anisse.astier.eu,2013-10-20:/hello-world.html<p>So I finally did it. You're reading it right now. My personnal website/blog.</p> <p>I should be posting here about things that cross my mind as well as various projects I've been working on. And maybe even new projets I didn't even start yet.</p> <h3>The design</h3> <p>First of all, kudos …</p><p>So I finally did it. You're reading it right now. My personnal website/blog.</p> <p>I should be posting here about things that cross my mind as well as various projects I've been working on. And maybe even new projets I didn't even start yet.</p> <h3>The design</h3> <p>First of all, kudos to Pascal Navière, a very talented web designer that did the design of this site(CSS, DOM structure, etc.), which I then modified. All bugs are therefore my own additions.</p> <h3>The tech</h3> <p>The DNS you used to access this website is hosted by <a href="https://gandi.net">gandi</a>. The website itself resides at <a href="http://kimsufi.com">OVH</a>, who used to sell the world's cheapest VPS (they're currently out of stock for all their products, but I won't go into that). The SSL certificate is provided by <a href="http://www.startssl.com/">StartSSL</a>.</p> <p>On this VPS, <a href="http://debian.org">Debian</a> Wheezy, with <a href="http://nginx.org">nginx</a> serving the actual pages. Pages which are all old scholl static <a href="http://www.w3.org/TR/html-markup/">HTML</a>(5), generated by <a href="http://docs.getpelican.com">Pelican</a>.</p> <p>On my machine pelican is run with <a href="http://python.org">python</a> 3.3, in a <a href="http://docs.python.org/3/library/venv.html#creating-virtual-environments">venv</a> where <a href="http://python-distribute.org/distribute_setup.py">distribute was installed</a>. The content is edited with <a href="https://vim.org">vim</a> on <a href="http://fedoraproject.org/">Fedora</a> 19.</p>