Runtime Checks

Compressed Apache Arrow Tables Over HTTP

Wed, 23 Oct 2024 22:50:34 -0300

Returning Arrow tables over HTTP is relatively straightforward. The Arrow IPC streaming format is repurposed to the HTTP use case and you can return each serialized IPC batch as a chunk of the HTTP response — Transfer-Encoding: chunked — to enable streaming clients.

Arrow IPC streams have a MIME type registered with IANA so you can also set the appropriate Content-Type header on your HTTP responses.

Content-Type: application/vnd.apache.arrow.stream

You can find Python examples of this in the arrow-experiments repository: https://github.com/apache/arrow-experiments/tree/main/http/get_simple/python.

Things get trickier when you try to support compression. Arrow IPC streams are not compressed by default and not all client implementations of the IPC protocol can parse streams with compressed buffers.

We will get back to IPC compression in a moment, but first let’s see how we can achieve compression of the data by compressing the HTTP response body itself.

Compressing HTTP Responses

All clients that can handle compressed HTTP responses can benefit from this. A server can look at the Accept-Encoding header of the request and decide to compress the response body if the client supports it. Chrome, Firefox, and most modern browsers support gzip and deflate compression as well as modern codecs like br (Brotli) and zstd (Zstandard)¹.

Web browsers and HTTP client libraries will automatically decompress the response body if the Content-Encoding header is set to the codec used to compress the response body.

Content-Encoding: gzip

When Chrome makes an HTTP request, it sets the Accept-Encoding: gzip, deflate, br, zstd header to indicate that it can handle compressed responses in any of these formats. If you’re using a library to make HTTP requests, make sure it sets this header correctly so the server can compress the response body without worrying about compatibility with the client.

Using Arrow IPC Compression

The buffers placed in a serialized Arrow IPC stream can be compressed using the LZ4 or Zstandard codecs. Arrow IPC compression has nothing to do with the HTTP protocol — the compression and decompression is done by the Arrow library itself as it parses the stream. Note, though, that not all Arrow implementations support compression.²

To maximize the chances of compression without compatibility issues, you can ask the clients to communicate their compression preferences in the request headers. This can be achieved by passing parameters to the MIME type in the Accept request header.³

Accept: application/vnd.apache.arrow.stream; codecs="lz4, zstd"

This way, the server can decide to compress the IPC stream buffers using LZ4 or Zstandard without fear that client won’t be able to decode it. If the client doesn’t specify a codec, the server can disable compression altogether, fall back to HTTP compression (based on the Accept-Encoding header), or choose a compression codec at the risk of incompatibility.

You can find examples of this in Python at https://github.com/apache/arrow-experiments/pull/35.

To make troubleshooting easier, the server may indicate which compression codec was used to compress the IPC stream, even though the Arrow IPC stream reader picks up which codec was used from the data stream itself.

Content-Type: application/vnd.apache.arrow.stream; codecs=lz4

A Note on Double-Compression

Applying both IPC buffer and HTTP compression to the same data is not recommended. The extra CPU overhead of decompressing the data twice is not worth any possible gains that double compression might bring. If compression ratios are unambiguously more important than reducing CPU overhead, then a different compression algorithm⁴ that optimizes for that can be chosen.

Conclusion

Returning Arrow tables over HTTP is a powerful way to share data between applications. Unsurprisingly, columnar formats like Arrow are very friendly to compression and serialization. This is why Arrow works well as a network transfer format. As tooling evolves you can expect to see more and more Arrow tables being shared over HTTP instead of CSV or JSON.

Zstdandard is not supported on Safari as of 2024. ↩
Check the implementation status table in the Arrow documentation to see if the Arrow implementation you’re using supports buffer compression. ↩
This is similar to clients requesting video streams by specifying the container format and the codecs they support like this Accept: video/webm; codecs="vp8, vorbis". ↩
Compression codecs can be instantiated with different compression levels trading off compression ratio for speed. ↩

Root Cause vs. Contributing Factors

Mon, 08 Apr 2024 22:01:03 -0300

In multiplication, to get a non-zero product, all factors must be greater than 0. If one factor is 0, nothing else matters.

\[\begin{align} 5 \times 2 \times 229 \times 0.2 &= 458.0\\ 5 \times 2 \times 229 \times 0.0 &= 0.0 \end{align}\]

This xz/liblzma incident and the public post-morten discussion is a great illustration of why incident post-mortens shouldn’t be carried as a search for a root cause, but as a comprehensive enumeration of all contributing factors.

You can model successful attacks as the observed occurrence of an event with non-zero probability — a possible incident that occurred.¹ In that model, the probability of an incident occurring is the product of all the contributing factors’ probabilities.

The nature of multiplication allows anyone to pick one factor and proclaim:

If the probability of this factor were 0, none of this would have happened, this must be the root cause.

The contributing factor you elect as the root cause says more about you, than it says about the shape of the effective solution or mitigating measures.

An open-source contributor will say the root cause is contributor burnout. An engineer that never worked on open-source will claim that open-source itself is the problem. A VP at a big corporation will say the problem is hobbyists not taking software maintenance seriously enough. A security researcher will blame the insufficient threat models that software distribution systems assume.

Getting the chances of that root cause happening down to zero would fully prevent the incident from ever happening again, except that it might be impossible to get it down to zero, and reducing all the contributing factors as much as possible can have a bigger impact on the final product than an imperfect attempt at reducing the chances of a single contributing factor. Because a root cause really is just an arbitrarily chosen contributing factor. ²

Time factors are necessary for a more advanced model. ↩
“Hindsight biases post-accident assessments of human performance” ↩

std::optional and non-POD C++ types

Sun, 19 Sep 2021 11:31:34 -0300

C++ 17 has introduced the std::optional<T> template class that is analogous to the Maybe/Optional monad implementation in many other languages. “Analogous” is doing a lot of work in this statement because the C++ type checker is not going to help you avoid dereferencing an empty optional like Rust, Haskell, Scala, Kotlin, TypeScript and many other languages will do.

That does not make it useless. As with many things in C++, we will be careful™ when using it and write only programs that do not dereference an empty optional.

In languages that deal mostly with reference types, an optional type can be implemented as an object that wraps a reference and a tag bit that tells if the optional has some data or nothing.¹ In C++ on the other hand, the std::optional<T> will inline the value type T onto itself. That means the general behavior of an optional of T depends a lot on the specifics of the type it’s wrapping.

For integer, floats, characters, the use of std::optional<T> doesn’t bring many surprises. In this post I want to look at what happens when non-POD types are wrapped in an optional. For this, I will write a class that prints a different message on each special function call:

class Object {
 private:
  std::string _s;

 public:
  Object() { puts("default-constructed"); }
  ~Object() { puts("destroyed"); }

  explicit Object(const std::string &s) : _s(s) { puts("constructed"); }

  Object(const Object &m) : _s(m._s) { puts("copy-constructed"); }
  Object &operator=(const Object &m) {
    puts("copy-assinged");
    _s = m._s;
    return *this;
  }

  Object(Object &&m) : _s(std::move(m._s)) { puts("move-constructed"); }
  Object &operator=(Object &&m) {
    puts("move-assigned");
    _s = std::move(m._s);
    return *this;
  }

  void dump() const { puts(_s.c_str()); }
};

And write a function that returns an optional of this class — a common use-case of optionals. The returned optional will contain a value if the string argument is non-empty, and be std::nullopt otherwise.

std::optional<Object> maybe(const std::string &s) {
  if (s.empty()) {
    return std::nullopt;
  }
  return Object(s);
}

The good

When using a std::optional<Object>, neither the Object constructors or destructors have to be called if the variable never gets populated with a value.

To see this in action, consider program1

void program1(const std::string &s) {
  const std::optional<Object> o = maybe(s);
  if (o) {
    o->dump();
  } else {
    puts("<empty>");
  }
}

and its output when called with an empty string

<empty>

The ugly

Things get more involved when program1 is called with "Hello!" and the optional gets populated

constructed
move-constructed
destroyed
Hello!
destroyed

The return Object(s) line in maybe, calls Object::Object(const std::string &) to create the Object that then gets moved into the storage within the std::optional<Object>. If Object didn’t have a move-constructor, it would be copied here. At the end of the scope of maybe, the “moved-from” temporary Object instance is destroyed, and at the caller — program1 — Object::~Object has to be implicitly called again to destroy the Object within the std::optional<Object> instance.

This situation can be improved if we tell std::optional<Object> to forward the arguments to Object::Object so it can construct Object in the optional’s storage area right away without a temporary

   if (s.empty()) {
     return std::nullopt;
   }
-  return return Object(s);
+  return std::optional<Object>(s);

With this change, the output of program1("Hello!") becomes

constructed
Hello!
destroyed

Only one constructor invocation and one destructor invocation. A win!

However, most functions returning a std::optional<T> are calling some function that returns T in the code path that instantiates and returns the optional. That takes us back to the same situation of duplicated constructor/destructor invocations.

Object makeObject(const std::string &s);

std::optional<Object> maybe(const std::string &s) {
  if (s.empty()) {
    return std::nullopt;
  }
  return makeObject(s);
}

To improve this and keep the logic of makeObject separate from maybe, we would have to change makeObject to allow the perfect-forwarding of the parameters from maybe, to makeObject, to std::optional<Object>::optional, to Object::Object!

Another common way of writing these functions is by declaring a variable of type T, performing some operations on it, and then returning it wrapped in an optional. This has the same problem we started with.

std::optional<Object> maybe(const std::string &s) {
  if (s.empty()) {
    return std::nullopt;
  }
  Object o(s);
  doSomething(o);
  return o;
}

The bad

These problems might not be a big deal in most situations, but if you insist on returning non-PODs wrapped in a optional, make sure that:

The wrapped type should be cheaply movable, otherwise your program might be copying it on every function call due to innocent-looking code;
Define your destructors outside the class declaration so they don’t get inlined by the compiler in both functions — the caller and the callee that returns the optional — to avoid binary size increase.

The specific situation in which I’ve seen this really affect binary size and possibly performance is when functions are written to return instances of classes generated by the Google Protocol Buffers compiler.

Google Protocol Buffers for C++ was designed before C++11 (i.e. before move-semantics was added to the language). Its APIs and generated code are designed to make it possible to use the classes without ever invoking copy constructors. It’s a good API.

If you never invoke a function, it doesn’t need to be in the compiled binary. You can notice a sudden increase in the binary size of your program when a single call to a big function is added to the codebase. Returning an optional of a Protocol Buffers object is enough to instantiate a lot of code that could otherwise never be needed.

Let’s take a look at the generated code based on a Protocol Buffers message called Date

class Date PROTOBUF_FINAL : public ::PROTOBUF_NAMESPACE_ID::Message {
 public:
  inline Date() : Date(nullptr) {}
  virtual ~Date();

  Date(const Date& from);
  Date(Date&& from) noexcept
    : Date() {
    *this = ::std::move(from);
  }

  inline Date& operator=(const Date& from) {
    CopyFrom(from);
    return *this;
  }
  inline Date& operator=(Date&& from) noexcept {
    if (GetArena() == from.GetArena()) {
      if (this != &from) InternalSwap(&from);
    } else {
      CopyFrom(from);
    }
    return *this;
  }
  ...

The move-constructor (by calling operator=(Date &&)) can potentially call InternalSwap and CopyFrom. The latter is called when the objects can’t be swapped because they are allocated in different arenas and have to be to be copied instead. By using the move-constructor of this object, both the moving (swapping) and copying functions are instantiated in the binary. This explains why returning the optional of a Protocol Buffers class increases the binary size of a program that, before doing that, didn’t have a need for InternalSwap and the CopyFrom function.

Recommendation

By adopting a more C-like way of initializing structures, the unnecessary use of move-constructors and extraneous destructor calls can be avoided. This pattern fits nicely with the code generated by Protocol Buffers.

Let’s add a new member function to the Object class

void set(const std::string &s) { _s = s; }

and write an alternative to program1 — program2

[[nodiscard]] bool maybe(const std::string &s, Object &out_object) {
  if (s.empty()) {
    return false;
  }
  out_object.set(s);
  return true;
}

void program2(const std::string &s) {
  Object o;
  if (maybe(s, o)) {
    o.dump();
  } else {
    puts("<empty>");
  }
}

The maybe function was rewritten to take an output parameter and return a boolean. out_object is changed in-place and doesn’t have to be moved into an optional and then destroyed within maybe. As expected, program2("Hello!") generates a cleaner output

default-constructed
Hello!
destroyed

program2 does not have to ever call the move-constructor, so it can be discarded by the linker and the destructor is called only once. If the destructor was inlined, it would be inlined once in the program, not twice.

Conclusion

Optionals are far from zero-cost abstractions in C++ and if this cost matters to you, taking output parameters and returning non-discardable booleans is an advantageous alternative solution to a function returning std::optional<T> when objects of type T are expensive² to move and/or destroy.

Languages like TypeScript can statically determine if an object is set based on its position in the control flow of the program. ↩
In the sense of run-time and size of the code in the binary. ↩

Demystifying JOIN Algorithms

Tue, 29 Jan 2019 03:10:53 -0300

Joins are an important class of operations in databases and data pipelines. There are many kinds of joins in SQL databases, but this post will focus only on the INNER JOIN. ¹

It is possible to make the JOIN between two large tables very efficient if the result is going to be small. That being a common scenario in practice, makes JOINs an interesting topic of research.

A naive JOIN algorithm can be very slow because JOINs are essentially nested loops.

NestedLoop JOIN based on sequential scans

for row_i in table_a:
  for row_j in table_b:
    if (row_i.key == row_j.key):
      emit_join_result(row_i, row_j)

A better alternative to the algorithm above is the HashJoin. In the first step of a HashJoin, a hash table of one of the JOIN operands — table_a — is built. Then a cheap lookup can replace the expensive innermost loop doing a sequential scan on the whole table_b for each row of table_a.

HashJoin

for row_i in table_a:
  hash_table[row_i.key] = row_i

for row_j in table_b:
  row_i = hash_table[row_j.key]
  if (row_i AND row_i.key == row_j.key):
    emit_join_result(row_i, row_j)

This code snippet above is very common in hand-written backend and UI code stitching data together to show something useful to users. So it should be no surprise that this is what databases commonly do to solve the same problem.

A Practical Example

Let’s look at an e-commerce database containing two tables: product and cart_item.

The product table contains all the product the store sells.

`product` table

 id |    name
----+-------------
   ...
| Iogurt
| Tea
| Sugar
| Chocolate
| Butter
| Detergent
| Panettone
| Coffee
| Shampoo
| Toothpaste
   ...

The cart_item contains products added to the cart of all the users.

`cart_item` table

  id  | user_id | product_id | quantity_added
------+---------+------------+----------------
|     776 |         34 |              4
|     494 |         35 |              2
|     494 |         34 |              2
|     494 |         36 |              1
|     494 |         37 |              3
|     494 |         38 |              1
|     494 |         39 |              2
|     494 |         40 |              4
|     494 |         41 |              2
|     494 |         42 |              2
|     574 |         34 |              2
|     574 |         36 |              1
|     574 |         37 |              4
|     574 |         43 |              1
                    ...

An useful view of the cart of a user will need more information about the product other than just the id (e.g. the name of the product). That is when a JOIN becomes necessary.

JOIN example

SELECT cart_item.user_id,
       cart_item.product_id,
       cart_item.quantity_added,
       product.name
FROM cart_item
JOIN product ON cart_item.product_id = product.id;

Results of the JOIN example query

 user_id | product_id | quantity_added |   name
---------+------------+----------------+------------
|         34 |              4 | Iogurt
|         34 |              2 | Iogurt
|         35 |              2 | Tea
|         36 |              1 | Sugar
|         36 |              1 | Sugar
|         37 |              3 | Chocolate
|         37 |              4 | Chocolate
|         38 |              1 | Butter
|         39 |              2 | Detergent
|         40 |              4 | Panettone
|         41 |              2 | Coffee
|         42 |              2 | Shampoo
|         43 |              1 | Toothpaste
                     ...

PostgreSQL might use a HashJoin to execute this query. The EXPLAIN command shows the query plan.

HashJoin query plan

Hash Join
  Hash Cond: (product.id = cart_item.product_id)
  ->  Seq Scan on product
  ->  Hash
        ->  Seq Scan on cart_item

The sequential scan on cart_item is necessary to build the hash table mapping cart_item.product_id to the cart_item record. That is the first for loop in the pseudo-code above. The second for loop is the sequential scan on product that queries the hash table with each product.id it finds.

If you mess with the PostgreSQL query planner really hard ², you can convince it to use the NestedLoop join based on nested sequential scans of both tables.

NestedLoop query plan based on sequential scans

Nested Loop
  Join Filter: (cart_item.product_id = product.id)
  ->  Seq Scan on product
  ->  Seq Scan on open_cart_item cart_item

And this is the most naive way to JOIN two tables as shown in the first code snippet — two nested loops doing sequential scans.

Indexing

The hash table built in the HashJoin is a form of indexing (temporary index) that speeds up the second for loop. Hash tables allow efficient random queries (i.e. keys used in a sequence of lookups do not obbey any particular order) for an specific key value.

An issue with the HashJoin query plan above is that the hash table is built at the time of the query. Indexing in the context of databases more commonly means that an index is already built when a query comes in (persistent index). This requires knowing which index is more suitable for the queries that will come later. Creating all possible indexes is not a good strategy because they would be many and indexes are not free. When the table changes, indexes have to be updated to reflect the changes. So each index adds an extra cost to writes and deletes.

Let’s disable HashJoin ³ and see how our JOIN query is planned.

NestedLoop query plan using a single index

Nested Loop
  ->  Seq Scan on cart_item
  ->  Index Scan using product_pkey1 on product
        Index Cond: (id = cart_item.product_id)

This query plan execution is similar to the second for loop in the HashJoin algorithm, but instead of using a temporary hash table that needs to be built at the time of the query, it uses the readily available primary key index of the product table on the id column.

Pseudo-code of the NestedLoop query plan

for cart_item_j in cart_item:
  product_i = product_pkey1.lookup(cart_item_j.product_id)
  if (product_i.id == cart_item.product_id):
    emit_join_result(product_i, cart_item_j)

Indexing with B-Trees

Classical relational databases were designed at a time when RAM was expensive and small. A lot of effort went into making sure that queries could be performed without bringing too much data into memory. Optimizing the disk access patterns is crucial for good performance of the algorithms operating on data stored in disks. Unlike main memory, disk are very bad at serving random access since the cost of seeking to the right position is considerable. ⁴

Sequentially scanning the randomly sorted product_ids in the cart_item table ⁵ and querying the product_pkey1 index stored in disk every time is not very good because it might require the disk to seek to random locations far too often.

The product_pkey1 index stored in disk is not a hash table, but a B-Tree. B-Trees (and other types of ordered indexes) allow more interesting queries than the specific key lookup provided by hash tables. In addition to key lookups, B-trees allow range queries.

Queries a B-Tree can efficiently resolve

Specific key lookup

  key = ?

Range queries

  key <  ?
  key <= ?
  key >= ?
  key >  ?

B-Trees shine when a range of keys is read sequentially. This access pattern minimizes the number of disk seeks and number of page loads into memory (storage blocks containing sorted keys).

Let’s create a B-Tree index for the product_id column in the cart_item table and see what happens to the query plan. ⁶

NestedLoop query plan using two indexes

Nested Loop
  ->  Index Scan using cart_product_id_idx on cart_item
  ->  Index Scan using product_pkey1 on product
        Index Cond: (id = cart_item.product_id)

Now the two indexes are being used. This is very good, because the scan for product_id on the cart_product_id_idx index will not be in the random order they appear in the cart_item table. The lookups for product by the product id will happen in the order of the product_pkey1 index. This will maximize the use of the page cache. Entry 43 of the product_pkey1 B-Tree is very likely to be in the same page of the entry 42 and so on. This locality is what guarantees the Nested Loop will not be quadratic in time. The cost of the innermost lookup will be amortized. Some will be slower than others when a page has to be loaded, but on average they will take a constant time. This is how the algorithm looks like in pseudo-code.

Pseudo-code of the NestedLoop query plan using two indexes

for (product_id, cart_item_row_id) in cart_product_id_idx:
  product_i = product_pkey1.lookup(product_id)
  emit_join_result(cart_item[cart_item_row_id], product_i)

Joining Sorted Relations

MergeJoin can take advantage of the two JOIN operands (relations) being sorted to efficiently produce output. The pseudo-code for the MergeJoin is not as straightforward as NestedLoop and HashJoin, but is not very complicated.

Pseudo-code for a simple MergeJoin

// Cursors for sequential scanning of table_a and table_b
cursor_a = 0
cursor_b = 0

while cursor_a.hasNext and cursor_b.hasNext:
  if cursor_a.key < cursor_b.key:
    cursor_a.next
  elif cursor_a.key == cursor_b.key:
    emit_join_result(cursor_a.row, cursor_b.row)
    cursor_a.next()
    cursor_b.next()
  else:
    cursor_b.next()

On every loop iteration at least one record is consumed from one of the two operands. This guarantees that MergeJoin will take time proportional to the sum of the size of the two sorted JOIN operands.

Since B-Tree indexes are sorted, they lend themselves very well to MergeJoin. This is a possible query plan to our example query. The query planner is taking advantage of two sorted indexes for the MergeJoin – cart_product_id_idx and product_pkey1.

Pseudo-code for a simple MergeJoin

Merge Join
  Merge Cond:
  ->  Index Scan using cart_product_id_idx on open_cart_item
  ->  Index Scan using product_pkey1 on product

If a persistent index is not available (or data is small enough to be completely in-memory), PostgreSQL might sort one or both operands to be able to run a MergeJoin on two sorted relations. This is possible query plan when the cart_product_id_idx index is not available.

Pseudo-code for a simple MergeJoin

Merge Join
  Merge Cond:
  ->  Sort
        Sort Key: open_cart_item.product_id
        ->  Seq Scan on open_cart_item
  ->  Index Scan using product_pkey1 on product

Conclusion

JOINs are essentially nested loops.
Multiple techniques are used to improve the performance of JOINs.
When datasets are small or no appropriate index is being maintained, query planners will likely decide to use memory to make queries run faster. The query executor can build hash tables (or sort records in memory, or…) and use HashJoin or MergeJoin to more efficiently join the two operands according to the join-condition in the query.
Maintaining index data structures like B-Trees up to date allows Nested Loop algorithm to run much faster even when the dataset doesn’t conveniently fit in memory. Indexed JOINs take time proportional to the size of the result and are not affected by the size of the tables.

There’s much more about JOINs and techniques to speed them up, but these are the basics that will allow you to start thinking about how data layout and choice of algorithms can greatly affect query performance.

Understanding the three classical JOIN algorithms – NestedLoop, HashJoin, MergeJoin – how they take advantage of different indexes and how they behave when there is no index can give you a lot of insight on how databases run queries.

Papers I've Read in 2017

Sat, 30 Dec 2017 18:30:53 -0300

In 2017 I decided to print some PDFs to read instead of just have them in an ever-growing folder of Papers to Read. Here is the list of the ones I managed to read this year.

Out of the Tar Pit / Ben Moseley, Peter Marks / 2006

This is my favorite paper. It is a bit long (around 60 pages), but well worth reading. It is very thorough in defining what software complexity is. The authors make the distinction between essential and accidental complexity (following Fred Brook’s ideas on the topic) and postulate that, to this day, most complexity in software is still accidental. To make things more concrete, they give an outline for a potential complexity-minimizing approach. The outline might throw some readers off. Some reviewers criticize the paper focusing only on this outline. I think that is a mistake. The given outline is one of the many possible solutions and many applicable insights can be found in the less concrete part of the paper.

See The Morning Paper’s review of Out of the Tar Pit.

Some thoughts on security after ten years of qmail 1.0 / Daniel J. Bernstein / 2007

In this paper, djb describes his approaches to writing secure and bug-free software.

Kqueue: A generic and scalable event notification facility / Jonathan Lemon / 2001

This paper describes the FreeBSD event notification facility: kqueue. The main application of such facility is in servers that rely on asynchronous I/O to be able to handle many clients concurrently (think Java NIO, Netty, Node.js, Nginx) without a thread/process per connection. kqueue is very elegant and can do much more than just asynchronous I/O. It is a pity that the Linux equivalent is much less nicer than kqueue.

Real-world Concurrency / Bryan Cantrill, Jeff Bonwick / 2008

Although we assert that less—much less—code needs to be parallel than some might fear, it is nonetheless true that writing parallel code remains something of a black art. We also therefore give specific implementation techniques for developing a highly parallel system. As such, this article is somewhat dichotomous: we try both to argue that most code can (and should) achieve concurrency without explicit parallelism, and at the same time to elucidate techniques for those who must write explicitly parallel code.

Solving Linear Arithmetic Constraints for User Interface Applications / Alan Borning, Kim Marriott, Peter Stuckey, Yi Xiao / 1997

This paper describes an incremental algorithm to efficiently solve linear constraints. This is the kind of algorithm behind systems like AutoLayout — a layout system that is widely used by iOS developers.

Software Engineering at Google / Fergus Henderson / 2017

A description of Google’s software engineering practices.

Mosh: An Interactive Remote Shell for Mobile Clients / Keith Winstein, Hari Balakrishnan / 2012

This paper describes how Mosh works. Great read for anyone building interactive networking software that needs to work in the presence of intermittent connection and keep perceived latency to a minimum.

On the Criteria to be Used in Decomposing Systems into Modules / David Parnas / 1972

A classical CS paper from the 70s. I think what it describes is known to anyone developing software in 2017. It still is a valuable reading for the historical value.

Spanner, TrueTime & The CAP Theorem / Eric Brewer / 2017

This paper describes how Spanner can exist if the CAP theorem is still true. Spanner is Google’s highly available global-scale distributed database that provides strong consistency for all transactions.

The Triumph of Types: Principia Mathematica’s Impact on Computer Science / Robert Constable / 2011

I read this when I was writing the Where do Type Systems Come From post which I obviously recommend you to read.

An Axiomatic Basis for Computer Programming / Tony Hoare / 1969

Another classical CS paper. Reading it will give you an initial idea of what software verification is about.

State the Problem Before Describing the Solution / Leslie Lamport / 1978

This one-page note was recommended by Fabian Giesen in his Papers I like series.

How Complex Systems Fail / Richard Irvin Cook / 2002

This paper is also from Fabian Giesen’s Papers I like series.

Time, Clocks and the Ordering of Events / Leslie Lamport / 1978

Another paper from Leslie Lamport. Probably his most cited paper.

It describes how to build distributed state machines (AKA distributed systems) given that only partial ordering of events is knowable in such systems (i.e. it is not possible to know for every event, what events happened before it).

In a distributed system, it is sometimes impossible to say that one of two events occurred first. The relation “happened before” is therefore only a partial ordering of the events in the system. We have found that problems often arise because people are not fully aware of this fact and its implications.

To this day, many systems are built as if it was possible to have complete ordering of events in distributed systems. This causes many subtle bugs that annoy users and puzzle programmers.

Holistic Configuration Management at Facebook / Chunqiang (CQ) Tang, Thawan Kooburat, Pradeep Venkat, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, Robert Karl / 2015

This paper describes how Facebook manages configuration (changing, testing, deploying…) of its backend services and mobile clients. My interest in reading this paper is in how they manage configuration of mobile clients in a reliable, scalable and backwards-compatible way. That is described in part 4 — Gatekeeper.

The Morning Paper also has a review of this paper.

If you like reading lists of papers people like with quick and short reviews, check out The Morning Paper by Adrian Colyer and Fabian Giesen’s excellent Papers I Like series.

Monorepo First, Submodules Later

Tue, 11 Jul 2017 17:17:48 -0300

When starting a new side-project I always wonder if parts of the codebase can be shared as an open-source project. A library extracted from the project might be of interest to other developers and it’s nice to have good stuff on your Github profile.

That possibility may compel you to spend a lot of time thinking about the repository structure. You may create a git submodule for every library that doesn’t exist yet but might be useful to you in other projects or to other people and you don’t want to lose the git history of that code. Trying to predict what is shareable before anything is done is almost impossible. When starting a project we have to try to do the minimum amount of metawork possible. There’s a solution to this problem that won’t require that much of upfront planning.

I start everything as monorepo and use git filter-branch to extract folders as separate repos if necessary (e.g. I want to extract a lib)
— Felipe O. Carvalho (@_Felipe) July 10, 2017

Don’t introduce any first-party git submodule in the beginning. Make sure things are cleanly separated in directories and get started.

[...]
* 493fcbc HAMT: Clone and operator=
* a33bf67 HAMT: Add destructor for the HAMT
* 9604360 HAMT: Start passing entries around instead of key and values
* 578fcbf HAMT: Move BitmatTrie and Node out of the HAMT class
* c8a6e1b HAMT: Start making it look like std::unordered_map
* 591df2a HAMT: Use KeyEqual to compare keys
* 16d3bb0 HAMT: Little pedantic tweaks
* fe8bec4 HAMT: Get rid of the fancy allocators
* 9295225 HAMT: Use std::{pair,hash} and move allocators out of the HAMT class
* df1a923 HAMT: Simplify alloc size calculation logic
* 07520de HAMT: Collect Fibonacci statistics correctly
* 9b3e627 HAMT: Make insert return the entry node pointer
* 960e5db HAMT: Rename some things
* 7d3993e HAMT: Make some stuff private
* bbde709 HAMT: Super wasteful allocation
* 429f897 HAMT: Fibonacci allocation and free lists
* 3799eca HAMT: Basic implementation
* 9777aa4 chronos: Fix lint errors in chronos.cpp
* e8d15f4 sqlite: Fix lint errors
* 4b796be chronos: Fix indentation
* ad7902f chronos: Use tuple for queries using the SQLite wrapper
* d04407e chronos: Add projects to the database through the UI
* 575ad64 chronos: Prototype with User and projects on the left sidebar
* 6bf5612 sqlite: Initial commit to the SQLite wrapper
* c67fe58 foc: SmallVector from LLVM (and tests!)
* fbecc32 cpplint.py
* 0750734 chronos: SQLite database schema
* 29eb494 sqlite: SQLite amalgamation source files
* 92c376a Copy examples/opengl3_example to chronos
* 3a369f6 Add googletest as a submodule
* bf1da9a Add imgui as a submodule
* bc3e92e Initial commit: .gitignore

Now you may notice that you have a directory with cool data structures that you want to extract and publish as a library. You can create a new repository that contains only that directory and commits that touch it using git filter-branch. Here’s how I did it for a project of mine.

To start, I made a copy of the repository to a new directory called foc_libraries.

~/code $ ls chronos/
CMakeLists.txt  chronos         cpplint.py      foc             sqlite_wrapper  vendor
~/code $ cp -r chronos foc_libraries

This new repository will be the repository of the library based on the foc in chronos. I use git filter-branch on master to filter out anything that’s not in the original foc directory.

~/code/foc_libraries (master)$ git filter-branch --prune-empty --subdirectory-filter foc/ master
Rewrite 1299f7e3c0e2ab9f469a72e2880700a301386639 (51/51)
Ref 'refs/heads/master' was rewritten

git filter-branch removed all the commits that weren’t related to the foc directory and the foc_libraries repository now contains only the files from the original foc directory in chronos.

~/code/foc_libraries (master)$ git ls-files
allocator.h
array_ref.h
hash_array_mapped_trie.h
hash_array_mapped_trie_test.cpp
hash_array_mapped_trie_test_helpers.h
none.h
small_vector.h
small_vector_test.cpp
support.h

The commit log of foc_libraries doesn’t have any commit related to the original application.

[...]
* 8c46e22 - HAMT: Move BitmatTrie and Node out of the HAMT class
* e2fd8cd - HAMT: Start making it look like std::unordered_map
* 67896cc - HAMT: Use KeyEqual to compare keys
* 1237a32 - HAMT: Little pedantic tweaks
* 1868375 - HAMT: Get rid of the fancy allocators
* 8259c1d - HAMT: Use std::{pair,hash} and move allocators out of the HAMT class
* 90d5771 - HAMT: Simplify alloc size calculation logic
* ae69ffa - HAMT: Collect Fibonacci statistics correctly
* adf5651 - HAMT: Make insert return the entry node pointer
* 2e8dfaa - HAMT: Rename some things
* 1addd89 - HAMT: Make some stuff private
* 11156b8 - HAMT: Super wasteful allocation
* 85b2d73 - HAMT: Fibonacci allocation and free lists
* 9c6dc21 - HAMT: Basic implementation
* 9f0a6c5 - foc: SmallVector from LLVM (and tests!)

Now I can upload foc_libraries to Github.

~/code/foc_libraries (master)$ git remote add origin git@github.com:felipecrv/foc_libraries.git
~/code/foc_libraries (master)$ git push -u origin master
Counting objects: 186, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (86/86), done.
Writing objects: 100% (186/186), 67.43 KiB | 0 bytes/s, done.
Total 186 (delta 124), reused 106 (delta 100)
remote: Resolving deltas: 100% (124/124), done.
To git@github.com:felipecrv/foc_libraries.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin

After publishing the foc_libraries repository I can go back to chronos and replace the foc directory with a git submodule.

Remove the foc/ directory

~/code/chronos (master)$ git rm -r foc
rm 'foc/allocator.h'
rm 'foc/array_ref.h'
rm 'foc/hash_array_mapped_trie.h'
rm 'foc/hash_array_mapped_trie_test.cpp'
rm 'foc/hash_array_mapped_trie_test_helpers.h'
rm 'foc/none.h'
rm 'foc/small_vector.h'
rm 'foc/small_vector_test.cpp'
rm 'foc/support.h'

Add the submodule as foc/

~/code/chronos (master)$ git submodule add https://github.com/felipecrv/foc_libraries.git foc
Cloning into 'foc'...
remote: Counting objects: 186, done.
remote: Compressing objects: 100% (62/62), done.
remote: Total 186 (delta 124), reused 186 (delta 124), pack-reused 0
Receiving objects: 100% (186/186), 67.43 KiB | 0 bytes/s, done.
Resolving deltas: 100% (124/124), done.
Checking connectivity... done.

Review the changes

~/code/chronos (master)$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 28 and 1 different commit each, respectively.
  (use "git pull" to merge the remote branch into yours)
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   .gitmodules
        new file:   foc
        deleted:    foc/allocator.h
        deleted:    foc/array_ref.h
        deleted:    foc/hash_array_mapped_trie.h
        deleted:    foc/hash_array_mapped_trie_test.cpp
        deleted:    foc/hash_array_mapped_trie_test_helpers.h
        deleted:    foc/none.h
        deleted:    foc/small_vector.h
        deleted:    foc/small_vector_test.cpp
        deleted:    foc/support.h

Commit the changes

~/code/chronos (master)$ git commit -m "Extract foc/ as a git submodule"
[master b2d4ce0] Extract foc/ as a git submodule
 11 files changed, 4 insertions(+), 3925 deletions(-)
 create mode 160000 foc
 delete mode 100644 foc/allocator.h
 delete mode 100644 foc/array_ref.h
 delete mode 100644 foc/hash_array_mapped_trie.h
 delete mode 100644 foc/hash_array_mapped_trie_test.cpp
 delete mode 100644 foc/hash_array_mapped_trie_test_helpers.h
 delete mode 100644 foc/none.h
 delete mode 100644 foc/small_vector.h
 delete mode 100644 foc/small_vector_test.cpp
 delete mode 100644 foc/support.h

Some adaptations might be needed on both repositories, but that’s pretty much all there is to it. Happy gitting!

Where do Type Systems Come From?

Fri, 07 Jul 2017 18:32:02 -0300

Every time I want to quickly understand something about an advanced type system or programming language concept I get lost when I see something like this on Wikipedia:

Linear type systems are the internal language of closed symmetric monoidal categories, much in the same way that simply typed lambda calculus is the language of Cartesian closed categories. More precisely, one may construct functors between the category of linear type systems and the category of closed symmetric monoidal categories.

Why there’s so much research around types if perfectly applying them to programming languages is impractical? ¹ It turns out mathematicians are the ones behind the development of most of the work related to type systems – type theory.² One might think they do this to support programming languages. After all, it’s part of the job of mathematicians to formalize what other disciplines developed with a more practical mindset. Infinitesimal calculus was greatly advanced by Isaac Newton to solve immediate problems in physics for example. But the development and study of type theory pre-dates programming languages! What problems were mathematicians trying to solve when they developed type theory, a curious programmer might ask.

The development of a theory of types started with Bertrand Russell in the beginning of the 20th century in Appendix B of The Principles of Mathematics (1903). It was developed as part of the effort to to describe new foundations for mathematics from which all mathematical truths could in principle be proven using symbolic logic.

Bertrand Russell – Widely held to be one of the 20th century's premier logicians.

In short, type theory was developed to be an alternative to set theory as the foundation of mathematical proofs in symbolic logic due to its ability to solve some contradictions stemming from naive set theory. ³ Later, type theory referred to a whole class of formal systems. There are many type theories.

In part I of his Mathematical Logic as Based on the Theory of Types paper published in 1908, Russell lists many of such contradictions and finds a commonality in them.

In all the above contradictions (which are merely selections from an indefinite number) there is a common characteristic, which we may describe as self-reference or reflexiveness.

The simplest form of this kind of contradiction is the person who says “I am lying;” if she is lying, she is speaking the truth and if she is speaking the truth, she is lying. Russell rewrites “I am lying” many times to find the root cause of the contradiction.

“I am lying” becomes “There is a proposition which I am affirming and which is false”. All propositions that say “there is so-and-so” may be regarded as saying that the negation of “so-and-so” is not always true. We can now rewrite “I am lying” to “It is not true of all propositions that either I am not affirming them or they are true” in other words “It is not true for all propositions p that if I affirm p, p is true.” The “I am lying” paradox results, according to Russell, from regarding that as a proposition. This makes it evident that the notion of “all propositions” is illegitimate; for otherwise, there must be propositions (such as the above) which are about all propositions, and yet can not, without contradiction, be included among the propositions they are about. Whatever we suppose to be “all propositions,” statements about them generate new propositions which must lie outside “all propositions” to avoid a contradiction. Regarding “all propositions” as a meaningless phrase will avoid us many contradictions. Similarly, in set theory, we should not consider “the set of all sets” to be a legitimate set.

In part IV, Russell proposes the hierarchy of types

A type is defined as the range of significance of a propositional function, i.e., as the collection of arguments for which the said function has values. Whenever an apparent variable occurs in a proposition, the range of values of the apparent variable is a type, the type being fixed by the function of which “all values” are concerned. The division of objects into types is necessitated by the reflexive fallacies which otherwise arise.

That’s the equivalent of writing type annotations for programming functions. And the goal is avoiding bugs instead of logical contradictions.

It’s called hierarchy of types because it separates propositions into orders:

A proposition containing no apparent variable we will call an elementary proposition. […] The terms of elementary propositions we will call individuals; these form the first or lowest type.

[…]

Elementary propositions together with such as contain only individuals as apparent variables we will call first-order propositions. These form the second logical type.

[…] We can thus form new propositions in which first-order propositions occur as apparent variables. These we will call second-order propositions; these form the third logical type. Thus, e.g, if Epimenides asserts that “all first-order propositions affirmed by me are false,” he asserts a second-order proposition; he may assert this truly, without asserting truly any first-order proposition, and thus no contradiction arises.

For a programming analogy, let’s try to create a function capable of running all functions

function runFunction(func) {
  return func(1);
}

function addTwo(param) {
  return 2 + param;
}

function multiplyByTwo(param) {
  return 2 * param;
}

function program1() { return runFunction(addTwo); }
function program2() { return runFunction(multiplyByTwo); }
function program3() { return runFunction(runFunction); }

All programs above are syntactically valid. program1 and program2 will finish (they halt). program3 also finishes, but will be due to an error caught by JavaScript interpreter checks.

>> program1()
3
>> program2()
2
>> program3()
TypeError: func is not a function

program3 fails because runFunction can only run first-order functions and runFunction is a second-order function – a function that takes a first-order function as a parameter.

If we redefine runFunction in Typescript as

function runFunction(func: (param: number) => any) {
  return func(1);
}

The type-checker will yell at us about program3

function program3() { return runFunction(runFunction); }
                                         ^
Argument of type '(func: (param: number) => any) => any' is not assignable to parameter of type '(param: number) => any'.
  Types of parameters 'func' and 'param' are incompatible.
    Type 'number' is not assignable to type '(param: number) => any'.
function runFunction(func: (param: number) => any): any

Every programmer knows that type systems can’t prevent all bugs. Similarly, type theory wasn’t enough to describe new foundations for mathematics from which all mathematical truths could in principle be proven using symbolic logic. It wasn’t enough because this goal in its full extent is unattainable.

Even though theoretically, type theories and type systems are not enough to prevent all the problems in logic and programming, they can be improved and refined to prevent an increasingly number of problems in the practice of logic and programming. That means the research for better practical type systems is far from over!

References

Russell, Bertrand, “The Principles of Mathematics”, 1903.

Russell, Bertrand, “Mathematical Logic as Based on the Theory of Types”, 1908, American Journal of Mathematics.

Coquand, Thierry, “Type Theory”, The Stanford Encyclopedia of Philosophy.

Constable, Robert, The Triumph of Types: Principia Mathematica’s Impact on Computer Science,, 2011, Proceedings of 10th Annual Oregon Programming Languages Summer School.

Notes

In fact, automatic proof assistants rely on type theories to achieve correctness (e.g. Coq). ↩
Making a distinction between type theory and type system is necessary. A programming language type system is analogous to a type theory. A type system is a feature of a programming language. It’s concerned with how to represent types in a machine and other things. ↩
A quick explanation of these paradoxes can be found in the Stanford Encyclopedia of Phylosophy. ↩

id Software Programming Principles

Sat, 25 Feb 2017 11:00:02 -0300

id Software co-founder John Romero tells the early story of the game company in this GDC 2016 talk and lists the programming principles that guided them towards the rapid development of many games including Doom and Quake with a very small team.

Some of these principles resembles today’s common Agile practices while others do not. I like them and id definitely had good results, so the principles should at least be considered and adapted to different contexts.

No prototypes. Just make the game. Polish as you go. Don’t depend on polish happening later. Always maintain constantly shippable code.

Continuous delivery and integration?

It’s incredibly important that your game can always be run by your team. Bulletproof your engine by providing defaults upon load failure.

Don’t break the build.

Keep your code absolutely simple. Keep looking at your functions and figure out how you simplify further.

Spend time simplifying code.

Great tools help make great games. Spend as much time on tools as possible.

We are our own best testing team and should never allow anyone else to experience bugs or see the game crash. Don’t waste others’ time. Test thoroughly before checking in your code.

As soon as you see a bug, you fix it. Do not continue on. If you don’t fix your bugs your new code will be built on a buggy codebase and ensure an unstable foundation.

Use a superior development system than your target.

This was very true in the 90s when consumer computers could be so much slower than high-end ones.

Write your code for this game only - not for a future game. You’re going to be writing new code later because you’ll be smarter.

Don’t write so many libraries.

Encapsulate functionality to ensure design consistency. This minimizes mistakes and saves design time.

Less moving parts.

Try to code transparently. Tell your lead and peers exactly how you are going to solve your current task and get feedback and advice. Do not treat game programming like each coder is a black box. The project could go off the rails cause delays.

Communicate.

Programming is a creative art form based in logic. Every programmer is different and will code differently. It’s the output that matters.

Respect differences and focus on the results.

Por que 0.4 - 0.5 = 0.09999999999999998 em Javascript?

Sat, 04 Feb 2017 13:02:02 -0300

Ontem o Marco Gomes perguntou por que 0.4 - 0.5 é 0.09999999999999998 em vez de 0.1 em Javascript. Esse post é uma tentativa de resposta à essa pergunta.

Existe um número infinito de números reais entre 0.0 e 1.0 então não dá pra representar todos os números reais com precisão infinita já que a memória do computador é finita. Ao contrário do que disseram nos comentários, dá sim pra representar 0.4, 0.5 e 0.1 em 64 bits (tamanho do float no Javascript) e em até muito menos que 64 bits o que garante mais 9’s na aproximação de 0.1, mas não está livre dos mesmos problemas.

Vou usar representação de 32 bits para todos os exemplos abaixo para não ter que lidar com números muito longos. O Javascript usa 64 bits.

Quando você digita “0.4” solicitando que o computador represente o número real 0.4 em um número finito de bits no padrão IEEE 754 (padrão usado por CPUs e linguagens modernas) o computador decide codificar 0.4 como \(2^{-2} \times 1.600000023841858\) (é o que cabe em 32 bits seguindo as regras do IEEE 754).

Essa representação no formato \(2^{expoente} \times mantissa\) é chamada de ponto flutuante e o padrão IEEE 754 define como esses números são transformados em operações.

Se você usar uma calculadora de alta precisão vai ver que

\[2^{-2} \times 1.600000023841858 = \\0.4000000059604645\]

Se você pedir para imprimir esse número de novo verá “0.4” porque a rotina que converte pontos flutuantes de 32 bits para texto seguindo as regras do IEEE 754 decide que “0.4” é o número mais próximo daquilo que está tentando-se representar com expoente igual a -2 e mantissa igual a 1.600000023841858. Mesmo assim, dá pra dizer que existe um erro de 0.0000000059604645 nessa representação.

Representar “0.5” é muito menos problemático porque computadores são muito bons em dividir por 2 e 0.5 é 1 dividido por 2.

\[0.5 = 2^{-1} \times 1.0\]

Fazendo a subtração usando as representações em ponto flutuante de 32 bits de 0.4 e 0.5 em uma calculadora de alta precisão:

\[0.4000000059604645 - 0.5 = \\-0.0999999940395355\]

Esse resultado contém o erro que vem da representação do 0.4. Como a cada operação com floats IEEE 754, os números são arredondados para o máximo de precisão possível com um número finito de bits (32 nesses exemplos), então o resultado visível será -0.099999994.

Por que esse pequeno erro agora se tornou “visível”? Porque a precisão de pontos flutuante é variável! Dá pra representar -0.0999999940395355 com um pouco mais de precisão do que 0.4000000059604645 “movendo o ponto” e o IEEE 754 aproveita isso. -0.099999994 é representado como \(2^{-4} \times 1.5999999046325684\). Esse -4 é a “flutuação do ponto” e dá a possibilidade de retornar mais uma casa decimal com confiança. -0.099999994 é mais próximo de -0.1 do que -0.09999999, mas 0.400000005 (9 casas) é menos próximo de 0.4 do que 0.40000000 (8 casas).

Um exemplo de número que é muito amigável à representação de ponto flutuante é 0.0000000000009094947017729282379150390625. Ao contrário de 0.4 pode ser representado sem erro algum por \(2^{-40} \times 1.0\).

Em ponto flutuante decimal (em vez de binário comum em computadores) é muito fácil representar 0.0000000000000000000000000000000000000001 como \(10^{-40} \times 1.0\) mas é bem mais problemático de se representar com ponto flutuante binário:

\[2^{-126} \times 0.008507013320922852 = \\ 0.0000000000000000000000000000\\ 000000000000999994610111476009\\ 580469753702428423236444030435\\ 167589607897177528261778434170\\ 992113649845123291015625\]

Representar números reais em computadores não é fácil e é necessário entender as limitações de cada representação (pontos flutuantes, ponto fixo, frações, fórmulas simbólicas…) para minimizar problemas em aplicações práticas.

Running the Xv6 Operating System

Mon, 22 Feb 2016 17:02:02 -0300

Being able to easily run and debug a simple operating system can be really useful when you want to learn how low level components are implemented. Xv6 is a very simple Unix-like operating system that allows you to do just that.

sillysaurus2 exemplified this in the Hacker News’ thread on Xv6:

Have you ever:

Wondered how a filesystem can survive a power outage?

Wondered how to organize C code?

Wondered how memory allocation works?

Wondered how memory paging works?

Wondered about the difference between a kernel function and a userspace function?

Wondered how a shell works? (How it parses your commands, or how to write your own, etc)

Wondered how a mutex can be implemented? Or how to have multiple threads executing safely?

How multiple processes are scheduled by the OS? Priority, etc?

How permissions are enforced by the OS? Security model? Why Unix won while Multics didn’t (simplicity)?

How piping works? Stdin/stdout and how to compose them together to build complicated systems without drowning in complexity?

So much more!

I credit studying xv6 as being one of the most important decisions I’ve made; up there with learning vim or emacs, or touch typing. This is foundational knowledge which will serve you the rest of your life in a thousand ways, both subtle and overt. Spend a weekend or two dissecting xv6 and you’ll love yourself for it later. (Be sure to study the book, not just the source code. It’s freely available online. The source code is also distributed as a PDF, which seems strange till you start reading the book. Both PDFs are meant to be read simultaneously, rather than each alone.)

Download, Compile, and Run

You can download Xv6’s source code, compile, and run under QEMU using the following commands:

# Download the code
git clone https://github.com/mit-pdos/xv6-public.git xv6
cd xv6

# Compile
make

# Install QEMU
sudo apt-get install qemu

# Run under QEMU
make qemu

It looks like this:

There are very few commands available but you can see UNIX here:

xv6...
cpu1: starting
cpu0: starting
sb: size 1000 nblocks 941 ninodes 200 nlog 30 logstart 2 inodestart 32 bmap start 58
init: starting sh
$ ls
.              1 1 512
..             1 1 512
README         2 2 1973
cat            2 3 13320
echo           2 4 12428
forktest       2 5 8144
grep           2 6 15020
init           2 7 13084
kill           2 8 12580
ln             2 9 12424
ls             2 10 14812
mkdir          2 11 12572
rm             2 12 12560
sh             2 13 23564
stressfs       2 14 13280
usertests      2 15 58584
wc             2 16 13848
zombie         2 17 12196
console        3 18 0
$

Remote Debug Xv6 Under QEMU using GDB

Xv6’s Makefile has a rule to make this very easy (qemu-gdb):

$ make qemu-gdb
...
sed "s/localhost:1234/localhost:26000/" < .gdbinit.tmpl > .gdbinit
*** Now run 'gdb'.
qemu-system-i386 -serial mon:stdio -hdb fs.img xv6.img -smp 2 -m 512  -S -gdb tcp::26000

Execution stopped before the first instruction and is now waiting for GDB to connect and supervise the execution (breakpoint, continue…). Open GDB from another shell:

$ gdb kernel
...
Reading symbols from kernel...done.
+ target remote localhost:26000
warning: A handler for the OS ABI "GNU/Linux" is not built into this configuration
of GDB.  Attempting to continue with the default i8086 settings.

The target architecture is assumed to be i8086
[f000:fff0]    0xffff0: ljmp   $0xf000,$0xe05b
0x0000fff0 in ?? ()
+ symbol-file kernel
(gdb)

Now you can set breakpoints, resume execution, and do whatever you want. Here I set a breakpoint at the exec function:

(gdb) breakpoint exec
Breakpoint 1 at 0x80100b5f: file exec.c, line 12.
(gdb) continue
Continuing.
[New Thread 2]
[Switching to Thread 2]
The target architecture is assumed to be i386
=> 0x80100b5f <exec>:   push   %ebp

Breakpoint 1, exec (path=0x1c "/init", argv=0x8dfffec8) at exec.c:12
12      {
(gdb) continue
Continuing.
=> 0x80100b5f <exec>:   push   %ebp

Breakpoint 1, exec (path=0x87d "sh", argv=0x8dffeec8) at exec.c:12
12      {
(gdb) continue
Continuing.
[Switching to Thread 1]
=> 0x80100b5f <exec>:   push   %ebp

Breakpoint 1, exec (path=0x19c0 "ls", argv=0x8dfbeec8) at exec.c:12
12      {
(gdb) p argv[0]
$5 = 0x19c0 "ls"
(gdb) p argv[1]
$6 = 0x0
(gdb) backtrace
#0  exec (path=0x19c0 "ls", argv=0x8df2bec8) at exec.c:12
#1  0x801062ba in sys_exec () at sysfile.c:417
#2  0x80105619 in syscall () at syscall.c:133
#3  0x801067d7 in trap (tf=0x8df2bfb4) at trap.c:43
#4  0x801065cc in alltraps () at trapasm.S:23
#5  0x8df2bfb4 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) continue
Continuing.

You can see that /init was the first process, it spawned sh and I entered ls in the console myself. Things are magically simple!

Runtime Checks

Compressed Apache Arrow Tables Over HTTP

Compressing HTTP Responses

Using Arrow IPC Compression

A Note on Double-Compression

Conclusion

Root Cause vs. Contributing Factors

std::optional and non-POD C++ types

The good

The ugly

The bad

Recommendation

Conclusion

Demystifying JOIN Algorithms

A Practical Example

Indexing

Indexing with B-Trees

Joining Sorted Relations

Conclusion

Further Reading

Papers I've Read in 2017

Out of the Tar Pit / Ben Moseley, Peter Marks / 2006

Some thoughts on security after ten years of qmail 1.0 / Daniel J. Bernstein / 2007

Kqueue: A generic and scalable event notification facility / Jonathan Lemon / 2001

Real-world Concurrency / Bryan Cantrill, Jeff Bonwick / 2008

Solving Linear Arithmetic Constraints for User Interface Applications / Alan Borning, Kim Marriott, Peter Stuckey, Yi Xiao / 1997

Software Engineering at Google / Fergus Henderson / 2017

Mosh: An Interactive Remote Shell for Mobile Clients / Keith Winstein, Hari Balakrishnan / 2012

On the Criteria to be Used in Decomposing Systems into Modules / David Parnas / 1972

Spanner, TrueTime & The CAP Theorem / Eric Brewer / 2017

The Triumph of Types: Principia Mathematica’s Impact on Computer Science / Robert Constable / 2011

An Axiomatic Basis for Computer Programming / Tony Hoare / 1969

State the Problem Before Describing the Solution / Leslie Lamport / 1978

How Complex Systems Fail / Richard Irvin Cook / 2002

Time, Clocks and the Ordering of Events / Leslie Lamport / 1978

Holistic Configuration Management at Facebook / Chunqiang (CQ) Tang, Thawan Kooburat, Pradeep Venkat, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, Robert Karl / 2015

Monorepo First, Submodules Later

Where do Type Systems Come From?

References

Notes

id Software Programming Principles

Por que 0.4 - 0.5 = 0.09999999999999998 em Javascript?

Running the Xv6 Operating System

Download, Compile, and Run

Remote Debug Xv6 Under QEMU using GDB

Useful Resources