Pinu planeet

September 16, 2017

Raivo LaanemetsNow, 2017-09

This is an update on things related to this blog and my work.

Blogging

Since the last update I have been returning to a more stable schedule of writing. I now usually write 1 or 2 articles per week and do most of the writing on weekends. I'm soon hitting the milestone of 200 articles and will review the blog in the similar way as I did for the previous milestone.

Work

I have moved back to my home office. Construction work has practically ended and the noise is not an issue anymore. During the couple of noisy days I have been sitting in a local pub with my laptop and drinking coffee.

The last 2 months have gone into a single project. It is a system to replace lots of manual work of managing transportation orders. The client has been pleasure to work with. Their office has been very welcoming and they have really good coffee. Most of my work has gone into the user interface. The backend is built using Node.js and MySQL and the frontend is using server-side JSX to generate HTML and browser-side React for dynamic forms.

Open Source and side projects

Since the last update I have built one side project. It's an app to ask questions about a software project. I have had the idea to build it for a long time. The basic system for questions was inspired by Expert Systems and the Bethesda dialogue system. I built the initial prototype in Prolog but later rewrote it as a browser-side application. It uses a tree-like data structure (actually a directed acyclic graph) where nodes are questions or topics and branches are activated by positive answers.

I rewrote the UI of my feed reader app in React. I wanted to test out React and apply some changes to the reader app. The original UI was written in Knockout.js and a bit hackish. I wrote the original UI many years ago when I did not have much experience in structuring large browser-side applications. At the moment I'm very satisfied with the rewritten code.

September 14, 2017

TransferWise Tech BlogOn the Logging in Mobile Applications

Here I describe an approach from iOS point of view. An approach for Android platform is no different. I can recommend to use Timber framework as an equivalent for CocoaLumberjack.

Logs? Phew! Just write something like:

print("Expected `foo` to be 'bar', got: \(foo)")  

Later, when you debug an issue, the message in the console could give you a hint for the solution. Simple enough. But it’s good only for a “Hello World” application.

Why?


1st of all, logs should be available when you need them desperately. Having a large number of users is a guarantee that every possible crash will happen including those under really strange circumstances. Unable to reproduce the issue, with only a stack trace in your hands you would clearly benefit from an additional source of information — application logs. Is it possible to attach logs to a crash report? Yes. For example, Crashlytics Custom Logs can be used for that.

Let’s assume that in the 3rd party SDK, which you are using to collect crash reports there is a func cloudLog(_ msg: String) method (like CLSLogv method in Crashlytics). Say this method follows the Unix philosophy and the only thing it does is sending a provided message to the 3rd party no matter what the current build configuration is. But we would want to use this handy method only in production builds in order to avoid the noise from the 3rd party when debugging.

So it makes sense to introduce your own method like this:

func myLog(_ msg: String) {  
#if DEBUG
    print(msg)
#else
    cloudLog(msg)
#endif
}

Hmm, didn’t we just start writing our own logging framework?

Our new myLog(_:) method works for most cases. But what if the application you are working on handles sensitive information? Banking, dating or health apps are good examples. If you are logging payloads of your network requests and responses (that would be handy, so probably you are) via previously introduced myLog method you might be sending sensitive information to the 3rd party. For some organisations that’s a huge no-no.

So what can we do about that? Something like this should work:

func myLog(_ msg: String, isSensitive: Bool = true) {  
#if DEBUG
    print(msg)
#else
    if !isSensitive {
        cloudLog(msg)
    }
#endif
}

Hmm, myLog keeps growing…


2ndly, sometimes a desire to log is in itself an indication of something unexpected happening and usually an additional action should be taken in such a case.

To be more specific imagine your application is talking to your beloved server and suddenly a terrible thing happens — incompatible JSON is served. Shouldn’t we do the following?

myLog("Error, look how terrible that JSON is: \(json)")`  

Not bad. But the problem won’t be noticed unless your application crashes right after. A crash is better than nothing in such a case, but the user might benefit from an offline data or remaining parts of the working functionality. A much better option would be to silently report the problem, receive the report, fix the API, meanwhile showing “Something went wrong. Sorry! We are already working on it.” message to the user. Is there a ready solution to report such a problem? Yes. For example, Crashlytics Caught Exceptions can be used for that. I will reference those as non-fatals — issues that are not crashing the application but should be reported.

Let’s assume that there is a func reportNonFatalError(_ error: Error) method to report problems silently (like recordError in Crashlytics).

We don’t want to report the issue in debug builds (to avoid the noise), but we still want to log to the console a message describing the problem. As for reporting, reusing this log message looks like a good idea. So the straightforward way to do that is like this:

let message = "Error, look how terrible that JSON is: \(json)"  
#if DEBUG
myLog(message)  
#else
reportNonFatalError(makeError(withMessage: message))  
#endif

The above code looks too cumbersome. Much better is to rewrite our myLog method like this:

enum LogBehavior {  
    case writeToLog(isSensitive: Bool)
    case reportAsNonFatal
}
struct WrappedStringError: Error {  
    var value: String
    var localizedDescription: String {
        return value
    }
}
func myLog(_ msg: String, behavior: LogBehavior = .writeToLog(isSensitive: true)) {  
#if DEBUG
    print(msg)
#else
    switch behavior {
    case let .writeToLog(isSensitive):
        if !isSensitive {
            cloudLog(msg) 
        }
    case .reportAsNonFatal:
        reportNonFatalError(WrappedStringError(value: msg))
    }
#endif
}

Now myLog‘s signature seems too complicated and its implementation is like a trash bin full of 3rd party specific logic. Is there a better way? Yes!


We better use some logging framework. Something like CocoaLumberjack. The idea behind it is really simple, but powerful. On the launch of the application you plug in some loggers (you can create your own or select from the predefined ones). When you log the message, it is broadcasted to every logger you plugged in. It’s up to a particular logger how to process the message. Given the lack of restriction on the number of loggers you can plug in you can untangle logging spaghetti using single responsibility principle.

But let’s give some examples.

The most simple but useful logger is a console logger which just prints a message to the console.

You might want to use file logger which persists logs on disk and rotates them. File logger is quite useful for feedback functionality — ask the user if he want to attach logs when composing email to customer support. A problem description with recent log is much more helpful than just the description. Of course you should not store any sensitive data in such logs.

Passing log messages to cloud log or reporting non-fatals is a good enough job for a separate logger as well.

Okay. Seems pretty straightforward. But how can we pass the information to a logger if the message is sensitive or if it should be reported as a non-fatal?

To answer that question let’s look at how we actually log. We do it the same way as we did before when using standard print, but now we have to flag the message as error, warning, info, debug or verbose one. Too many of them, isn’t it a complication? A bit it is, but a handy one. Of course those types impose specific semantics on messages and could be used to filter or color them appropriately in the console. But they can be also used to make a decision on what particular logger should do with a particular message. For example, the non-fatals logger should ignore all the messages except for of error type.

No more words, just look at the following logging policy scheme:

Build Console Non-fatal Cloud Log When To Use
Error Debug

Something bad has happened that should never happen. App is not crashing but a part of the functionality is not working. The situation should be reported.

E.g., a wrong JSON format, parsing errors when a source format is unsupported, some system service is not available but we expect it to be.

Release
Warning Debug

Something bad has happened that should never happen. App is not crashing but a part of the functionality is not working. The situation should be reported.

E.g., a wrong JSON format, parsing errors when a source format is unsupported, some system service is not available but we expect it to be.

Release
Info Debug

Something expected happened. Messages of that level are useful to understand the reason for a crash or an unclear bug.

E.g., a view controller is presented or dismissed, user interactions, some service finished an initialization, a network request succeeded, a DB is saved successfully.

Release
Debug Debug Messages of that level are useful to log sensitive information such as a network response content if it’s really needed. Such log messages should not litter the console.
Release
Verbose Debug Messages that are littering the console, should be manually turned on.
Release

Important details:

  • log sensitive information only on debug and lower levels (that’s only verbose);

  • report only error messages as non-fatals;

That’s roughly the scheme we are using in TransferWise for our iOS application. Of course some amount of work is needed to setup such an approach for an existing application, but it will make the maintenance simpler. For applications that are suffocating from the tech debt it’s one of the first steps to do. But even if your application is pretty healthy, efforts made to keep a finger on the pulse are a thousand times worth it!

DDLogInfo(“Happy logging!")

September 10, 2017

Raivo LaanemetsWho is a Software Consultant

It does matter what you call yourself. It sends out a signal of what you do and how you can be useful. By the offers I have been getting recently I have come to a conclusion that my current title (software developer or full-stack freelancer) has not been descriptive enough. The new offers have not matched up with what I have been doing for the existing clients.

What and how I do

  • Solving problems by developing custom software.
  • Develop in collaboration with the client.
  • Providing services directly to the client.
  • Working with multiple clients at once.
  • Choosing my own clients and projects.
  • Managing the development process.
  • Working remotely or on-site.
  • Develop using popular platforms and libraries (mostly JavaScript).
  • It is a full-time job and not a side business.

I believe this is the best described as a "Software Consultant". Lots of other titles are approriate in some aspect but are not more descriptive as they are too generic, too specific, or have negative connotations in some communities.

More information:

September 05, 2017

Raivo LaanemetsCryptopocalypse in Estonia

A day ago (Tuesday, 6th September) Estonian Information System Authority was informed about a security vulnerability in the electronic chip of Estonian ID cards.

Use of electronic signatures is very common in Estonia. This includes all sorts of legal contracts. Estonia uses the same electronic cryptographic system for online electronic voting. An electronic signature from an ID card is fully binding. It has the same power as a manually written signature.

The vulnerability affects 750000 ID cards issued in the last 3 years. Estonian population is 1.4 million. Estonian ID card is mandatory for every citizen.

Henrik Roonemaa, a well-known Estonian journalist has put together an article from various sources of information. It is published in Estonian. The important points are:

  • The cost of breaking one card for signing is 40k EUR.
  • At least one card has been broken for demonstration purposes.

This shows the severity of the situation. The vulnerability is believed due to the process of generating private and public keys inside the card chip. The weakness allows to deduce the private key from the public key. The list of all public keys was publicity available until Tuesday when it was taken down. Full details of the vulnerability have not yet published.

You get the full power to forge a perfect signature once you gain access to the private key. The electronic signature would be extremely hard to disprove as such attack was deemed practically impossible by experts.

Estonian government says that the issue won't affect the Estonian state as a whole. They say that it would only happen when someone put 60 billion EUR into work to break most of the 750000 signatures. Estonia is worth way more than 60 billion EUR. The government has not yet addressed the issues of individual citizens getting targeted by attacks using this vulnerability.

New cards will have to be issued to fix the vulnerability.

Update 2017-09-10

It has not been confirmed that a card has been actually broken. The issue might then be a lot less severe than thought. The text has been removed from the article.

Update 2017-09-13

It takes about 10-12 hours to crack a card according to the former Estonian president Toomas Hendrik Ilves. Ilves says that this is based on facts. The information comes from an article published in the daily newspaper Postimees. The article also states that the vulnerability is between the chip hardware and the software directly communicating with the chip.

September 04, 2017

Raivo LaanemetsA solution to dreaded "Resolving Host" in Chrome

Sometimes Chrome hangs for a couple of seconds with the "Resolving Host" message on the status bar. The problem is very annoying as it delays page loading for multiple seconds. It happens randomly, not on every site, not on every page open.

I started to get this behavior in the last couple of months. Googling reveals that this could be solved by changing various settings inside Chrome. None of that worked for me. One of the most popular solutions is to use Google DNS servers. I thought it would have no effect on me because I was already using them.

For some time I was trying to debug my hardware. I was suspecting my router and cables and network card. I checked various machine logs every time I had the issue occur in the browser. I found nothing.

Finally I decided to apply strace to the DNS lookup. Strace is a debugging utility for Linux. I thought I might not be a first one to do this and found a blog post by Jeff Blaine. I applied his command on my machine. The command is:

strace -f getent hosts www.puppetlabs.com

The command spewed out a verbose log but otherwise completed fast. Then I ran it again and again until it started to hang. It can be seen that it connects to a Google DNS server 8.8.8.8 and awaits a response from it. By the log it looks like sometimes the server won't respond. The third argument for the hanging call in the log, 5000, does seem to be a timeout in milliseconds. After the timeout expires, it tries to make a new request.

connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
poll([{fd=3, events=POLLOUT}], 1, 0)    = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "\223\254\1\0\0\1\0\0\0\0\0\0\3www\npuppetlabs
  \3com\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=3, events=POLLIN}], 1, 5000 <- HANGING HERE

The solution was to simply replace Google DNS servers with OpenDNS servers. They have no such behavior and the issue has now completely disappeared. You need consult your OS or distro manual or read online guides how to change DNS server settings for your computer. I currently have no idea what causes the problems with Google DNS.

September 03, 2017

Raivo LaanemetsChrome 59/60+ font rendering in Linux

Chrome 59 on Linux renders fonts differently than the rest of the system. This happens when you use subpixel hinting on your system. Chrome 59+ seems to ignore your settings. It can be fixed by tweaking the FreeType configuration. FreeType is a font rendering system used by most Linux distributions.

The issue can be fixed by setting an environment variable before launching Chrome:

export FREETYPE_PROPERTIES="truetype:interpreter-version=35"
google-chrome

I'm using an Application Launcher in MATE desktop to start Chrome. To automatically set the environment variable, I created a wrapper script chrome-wrapper.sh in my home directory and edited the launcher settings to invoke the wrapper script instead of google-chrome directly. The full contents of the script:

#!/usr/bin/env bash
export FREETYPE_PROPERTIES="truetype:interpreter-version=35"
/opt/google/chrome/google-chrome &

With medium or full hinting this brings back skinny font rendering in the browser. Without the fix, the text looks similar to text on Windows 10.

More information:

August 17, 2017

TransferWise Tech Blog4 fallacious reasons why we estimate

‘Estimate’ as defined in this article is: A guess about the future regarding the time or effort that a task(s) will take to be completed. In other words, anything that can be used to foresee when tasks will be done.

I have to confess, I’m not very good at estimating. Because of that, for the past 3 or 4 years, I’ve been trying to use different approaches other than estimates. Although I lost my faith in estimation long ago, as a software engineer, I still receive requests for my opinion. Regardless if I end up giving an estimate or not, I always ask the person why they want one.

In this post, I'll share some of the given reasons the team decided weren’t worth spending the energy to create estimates. However, keep in mind this is merely my own point of view and my own particular circumstances; this may not apply to your scenario.

1. I need an estimate because… I need to measure productivity

Although I understand measuring productivity could work well for repeatable activities, it's hard to believe it works well for abstract and, ultimately, non-repeatable tasks like software development. For non-repeatable activities - meaning it’s never been done before - no one really knows how much time a task should take. Thus, the common approach to "measure productivity" is to compare the estimates against what, in fact, happened. This can cause a number of problems, including the fact that once developers realize they are being judged by their own estimates, they start to be very careful about the process of creating those same estimates, usually spending a lot of precious time completing an activity which doesn’t accrue additional value to the product that they are building.

From my experience, seeking to measure productivity is usually more intense when the side requiring the estimate doesn't have a profound trust that the other side will work very hard. That's why I strive to create an environment with mutual trust between the engineers and the different stakeholders. One of the most powerful techniques in this sense is to have a genuine interest in understanding the reasons behind the requirements for estimates. We engineers should always keep in mind that software development is usually one part of many others inside the company. After having worked for outsourcing, consultancy and product companies I believe that creating a place where people really trust each other is easier when software engineers and stakeholders are both part of the same organization. That's one of the reasons why more and more companies are now considering software development as a core business - thus creating really strong software engineer teams.

2. I need an estimate because… someone else is asking me to provide one

It's incredibly common to hear this reason in environment with traditional hierarchy where decisions are made top-down. I strongly believe that "agile transformations are not about transforming IT, they are about transforming organizations".

I remember once working for a product where our project manager asked us for an upfront estimate regarding a large feature that would be a game changer in the product. We tried to make it clear it was just a guess, and then provided a 3-month estimate.

Without us realizing it, the marketing team then created a beautiful campaign and started giving hints for the customer that the feature would be launched soon. Unfortunately, due to a variety of reasons, the feature was released with a huge delay. At this point, the marketing team came and asked: "Why did you tell us it would take 3 months? If you didn't know upfront, why didn't you just say you didn't know?"

Cases like this helped me learn not to provide estimates unless someone specifically tells you the reason. In this case, had we realized that marketing would take our estimate as something solid to use to begin promoting to customers, we would have acted quite differently.

The interesting thing is we are so ingrained to the culture of giving estimates that we end up giving them even when no one is asking us! Having in mind how dangerous a reckless estimate can be, I would not provide an estimate unless someone is explicitly asking you one. You can still have a general time frame in your mind, but it may be better to keep it to yourself, thus avoiding problems like the anchoring effect.

3. I need an estimate because… having one motivates us to work harder

This is a dangerous statement. Although it may encourage the team to stay in the office for longer hours, it’s not a healthy long-term approach. I've seen many teams working very hard to finish tasks before the estimated date. Many have ended up forgetting about other important things in the process, like code quality and a focus on simplicity. The cost of this cannot be underestimated. It will really come to the surface later when the team realizes that, as a result, they have accrued significant technical debts.

4. I need an estimate because… I need one to prioritize my backlog

One of the most important and complex questions in software development is, “What should we work on next?” Although I believe that in order to prioritize we should take into consideration the size and effort of the tasks, I'm not convinced we really need estimates to do that. This statement may seem contradictory, but I’ll show you how I've been prioritizing without using estimates.

First - backlog prioritization is essentially a sorting process. As software developers, we know the only requirement to order a list of items is to be able to compare elements. In other words, we don't need to know the absolute value of an item, but we need to know how to compare them. The trick as it relates to estimates is that although we aren’t very good at determining absolute numbers, we are better comparing things of the same nature.

To illustrate this principle, let's consider these three images:

alt

Would you be able to tell me how many people we have in each picture? More than that, would you bet your money on your estimates? On the other hand, if I asked you to sort the pictures according to the number of people in each one, I'm sure you’d be able to do it. I've been using the same principle to prioritize my backlog.

How are estimates used in TransferWise?

Before answering this question, it's important to understand how teams are organized in TransferWise. As Nilan, our VP of growth, explained, "We hire smart people and trust them”. Teams are autonomous – no one can tell them what to do.

For example, we have a currencies team. It decides each quarter which currencies it's going to launch. No one tells the team what currencies to launch. Anyone can challenge the team on what it's doing and the team should have a rationale for explaining why it's focussing where it is.

As a result, it's completely up to team members to decide whether they want to use estimates as a tool to help them or not. In my current team, there's no planning poker session or any other form of estimates for individual tasks. The closest process we have to the traditional way of creating estimates is that we have a quarterly planning where we prioritize our backlog. After that, we decide the topics we’ll detail a bit more so that other people in the company would be able to understand where we’re headed and why we came to that conclusion.

After all, is there a good reason why I should use estimates?

To be honest, I don't know the answer to this question. The first thing I try to keep in mind is, "estimations are valuable when it helps you make a significant decision". Thus, the question I usually ask myself is, “What is the decision we need to make? Can we possibly make this decision without an estimate?”

Even more important than answering these questions is to open our minds to new ideas. In this way, I’d like to ask you, "What are the scenarios where you’re using estimates and they aren’t actually required?"

I'd love to hear from you.

August 13, 2017

Kiibi lühiuudiste blogiDragon viib kosmosesse superarvuti

Rahvusvahelise kosmosejaama arvutusvõimsus pole just suurem asi – astronaudid askeldavad seal vahel sülearvutitega, kuid teadusandmed saadetakse kohe edasi Maale ja arvutada on mõistlikum arvutuskeskustes. Kuid nüüd saadab NASA eksprimendi käigus kosmosesse HPE Apollo 40 klassil põhineva vedelikjahutusega arvuti. Uue arvuti teadusprojekti pikkuseks on üks aasta, mis on ka umbes see aeg, millega kaugemas tulevikus kosmoselaevad Marsile lendama võivad hakata. Kuna side Marsiga orbiidi vastaskülgedel võtab aega umbes 20 minutit (ehk edasi-tagasi 40 minutit), siis võib selliste pikamaa lendude jaoks arvutusvõimsusest kasu olla, et eriolukordades õigeid otsuseid teha saaks. Kosmosesse peaks SpaceX Dragon startima juba saabuval esmaspäeval, 14. augustil.

Põhjalikumalt pajatab Neowin ja HPE.


August 07, 2017

Anton ArhipovXRebel for standalone apps with embedded Jetty

XRebel was designed to work specifically with Java web applications. Currently, it relies on Servlet API to serve its UI. It has been tested with a range of application servers (Apache Tomcat, WildFly, WebSphere, and others) and some full stack frameworks as well, including Spring Boot and JavaSpark.

Sometimes however, you might want to use XRebel with a standalone Java process that is not using Servlet API. Hence, XRebel cannot display its embedded UI to render the data it collected from the application. What can we do about this? Well, XRebel is tested on Jetty and it works quite well there. Jetty is also often used as an embedded HTTP server. So why don’t we use this trick and embed Jetty into our standalone app to serve XRebel UI.

Here’s our example application:

<script src="https://gist.github.com/antonarhipov/d10f3abf5b6d8e9265f33fc54c7a6178.js"></script>

Just for the demonstration purposes, it executes an HTTP request every 3 seconds and reads the response. Quick and dirty.

To embed Jetty server we need is to add a dependency to Jetty container and initialize the server when the application starts. The required dependencies are jetty-server and jetty-servlet:

<script src="https://gist.github.com/antonarhipov/4fc7220a9d6246e417b37773080159bb.js"></script>

Then we can start Jetty by adding the following code snippet into the static initializer of the class:

<script src="https://gist.github.com/antonarhipov/73072de7bced1a99cf1c07b1d19104a2.js"></script>

To enable XRebel agent, you need to start the application with -javaagent VM argument pointing to the location of xrebel.jar. For instance:

-javaagent:/Users/anton/tools/xrebel/xrebel.jar -Dxrebel.traces.all=true

I also added -Dxrebel.traces.all=true VM property to enable tracing of non-HTTP activities by XRebel. Tracing of non-HTTP activities in XRebel is disabled by default, hence I need to add this parameter in order to see profiling data for the periodic tasks (if I wish).

Once I launch the application, it will boot up the embedded Jetty instance on port 8080. The standalone XRebel UI is deployed by the agent to /xrebel context. Hence, if we open http://localhost:8080/xrebel in the browser we will see the following:


As you can see, it is quite easy to use XRebel with the standalone apps with this little trick. Just start an embedded Jetty instance on some port and you will be able to see what is your application doing. Perhaps, you can spot a sneaky bug with it before it gets to production! :)

If you want to play with the example application, I have it at GitHub. Download XRebel and start the application with the appropriate VM arguments to enable the agent. It will be fun! :)


August 04, 2017

Raivo LaanemetsA set of questions

I'm often approached by potential clients wanting to build something. Usually they have a vague vision of the final result but not a concrete execution plan to reach it. My job is to reach the successful result but there are always constraints out of my control.

I do not want to drive the potential clients away but sometimes the overhead of arranging and attending meetings is not worth it once I discover that I won't be able to meet some of the project requirements or I won't fit into the existing team. This is a wasted time for both my potential client and myself.

The most common reasons for turning down a project have been:

  1. Timeline is unsuitable.
  2. The budget is way underestimated.
  3. Timezone makes communication hard.
  4. Type of technology (desktop, mobile, embedded, etc) is not my expertise.
  5. Gut feeling.

Timeline

The most common reason is a timeline. A project should have a start and an end. It is often helpful to divide the project into shorter periods each with a definite length.

I always have a queue of projects. These projects pass many of the constraints I point out here and often have negotiated deadlines. A single project taking longer than planned is much worse than a project ending prematurely. A project running longer requires re-negotiation of every other project in the queue and forces me to break promises I have made to my clients. My clients will suffer and I do not want that to happen.

Budget

Estonia is not the most expensive country to live in but the difference with Western Europe countries is not 2-3 times, it's more likely 20-30%. The employment taxes are relatively high. On top of the personal income tax is the hefty 33% social tax, increasing the employee cost. Experienced developers are in high demand. Estonia is not a country to find cheap developers. Clients with a certain budget range are preferred. A budget is nearly always underestimated but knowing the budget helps to set expectations. Sometimes low-quality work is expected. This is not something I would like to do.

Communication

A software project cannot be carried out without effective communication. This is especially important in a remote project. Some time ago I had a remote project with a client in US West Coast. I shifted my workday to late hours and she shifted hers to early hours. This was very exchausting and took a heavy toll on the life outside the project. Eventually we gave up on this schedule but important questions had to often wait an answer a day which slowed down the project.

Technology

The choice of the technology is often more important in short projects. There is enough time in a 3-year project to learn whatever language or platform but you can't realistically create a mobile app in a week if you haven't built any before.

Couple of years ago I worked on a desktop application. We chose Qt as the application framework because it was cross-platform and had a suitable free license. Qt uses C++ and C++ is a complete beast to learn. I had no experience with C++. I was able to get the initial prototype out in 3 days by copy/pasting together source code from examples. In the following months I picked up proper C++ and added all those const qualifiers I had missed before.

One of my clients needed an Android app recently to be done in a week. I thought I give it a chance. I spent 3 days trying to install Android development tools. I always got some weird errors. Once I got it running the emulator turned out to be extremely slow and useless. The time pressure did not help either. I had to admit defeat. The project would have been doable in the timeframe by someone knowing the tools already.

The choice of the technology is also important for longer projects to ensure project continuity. Example of a such negative choice would be Flash which is currently being removed from browsers and is completely dead in 1-2 years.

Gut feeling

I must be able to envision myself using the product or service, getting a great value out of it. Is it actually possible to accomplish it given all the hard constraints? That is the ultimate indicator for me.

Throughout the career I have had only a single client who gave me too much information. Every day I received 3-4 page essays containing bug reports, weekly plans, long term future plans, what-if scenarios and dreams all mixed together. In every other case it has been the opposite.

Business ideas

I'm careful about discussing business ideas. Ideas are considered worthless but frequently the client has some existing business tied to the idea. The existing business can contain crucial details that are important for the project's financial success. These details are trade secrets and the client should not be put into a situation where these are to be accidentally revealed.

A not-go decision for a project does not mean that you should reject the client. I have had clients with multiple projects and I have proceeded with some and turned down some others.

Simple questions

Some time ago I was studying a frontent framework and wanted to make an app with it. I decided to make an app asking questions about a potential project. It is available here. Using the app does not mean you have to work with me. It contains simple questions with yes/no answers and does not ask anything about your business idea. You get a shareable link with the answers and a downloadable PDF file.

July 27, 2017

Raivo LaanemetsStack Review 2017

In 2017 the typical setup of my project stack has been:

Backend

  • Node.js 8+ with the available ES6+ features.
  • Express 4 with either EJS or JSX-based templates.
  • MySQL database accessed through the mysql package.

The applications are based on a common project template. I do not maintain the template as a separate project. It is improved through project to project and taylored to specific needs. The template covers:

  • Directory structure.
  • A set of always needed dependencies.
  • Better static file serving with cache busting and CDN support.
  • Trivial async/await wrapper for Express.
  • jsend-style API response generator.
  • Useful defaults and helpers for templates (default app title, date formatting, etc).
  • Sessions with signed cookies.
  • MySQL transactional async/await wrapper for queries.

Frontend

  • Bootstrap 3.
  • Vue.js 2+ or React 15+.
  • Webpack 3+.

The Bootstrap framework provides the common approach to styling. Most web designers are familiar with it, even if not, the documentation is very good and accessible.

Both Vue.js and React are decent view libraries. I have used them both. Vue.js uses HTML as the templating language and is more suitable for projects where the majority of markup is coming from the backend and it needs interactivity. React wants to generate and fully control the markup itself.

I use Webpack for building browser-side JavaScript bundles. It has iterative compilation, and integrates really well with Babel.

Other things

I generally avoid non-web application projects as these are not my expertise. Web scrapping is one of the related topics. I have mostly used PhantomJS so far.

There are some problem domains that benefit from relational and logic programming with Prolog. For example, industrial planning is a such domain. Besides that, I have successfully used SWI-Prolog for developing web applications, including this blog. I currently maintain two Node.js/SWI-Prolog bridge interfaces.

My opinion on the development of mobile applications is to get them done by the platform expert. The Android development tools are extremely complex, iOS tools are a bit better but still require specific knowledge. Recently I have been playing around with DroidScript. DroidScript is a JavaScript-based development system for Android but I have not used it in a project much larger than a Hello World.

For a desktop application project I would pick Electron. Electron is a fusion of Node.js and Chromium runtimes. So far I have developed one project with it. The process is very similar to creating a web application and that is something I know very well.

July 25, 2017

Four Years RemainingSorting in Linear Time

Every student of computer science, who has managed to keep even a tiny shred of attention at their algorithms course, should know that sorting n numbers is a task that requires at least \Omega(n \log n) time in general. There are some special cases, such as sorting small integers, where you can use counting sort or radix sort to beat this baseline, but as long as your numbers are hypothetically arbitrarily large, you are stuck with the \Omega(n \log n) lower bound. Right?

Well, not really. One thing that many algorithms courses tend to skim over rather briefly is the discussion of the choice of the computation model, under which the algorithm of interest is supposed to run. In particular, the \Omega(n \log n) bound for sorting holds for the comparison-only model of computation — the abstract situation where the algorithm may only perform pairwise comparisons of the numbers to be sorted. No arithmetic, bit-shifts or anything else your typical processor is normally trained to do is allowed. This is, obviously, not a very realistic model for a modern computer.

Let us thus consider a different computation model instead, which allows our computer to perform any of the basic arithmetic or bitwise operations on numbers in constant time. In addition, to be especially abstract, let us also assume that our computer is capable of handling numbers of arbitrary size. This is the so-called unit-cost RAM model.

It turns out that in this case one can sort arbitrarily large numbers in linear time. The method for achieving this (presented in the work of W. Paul and J. Simon, not to be confused with Paul Simon) is completely impractical, yet quite insightful and amusing (in the geeky sense). Let me illustrate it here.

Paul-and-Simon Sorting

The easiest way to show an algorithm is to step it through an example. Let us therefore consider the example task of sorting the following array of three numbers:

a = [5, 3, 9]

Representing the same numbers in binary:

[101, 11, 1001]

Our algorithm starts with a linear pass to find the bit-width of the largest number in the array. In our case the largest number is 9 and has 4 bits:

bits = max([ceil(log2(x)) for x in a])     # bits = 4
n = len(a)                                 # n = 3

Next the algorithm will create a (4+1)\cdot 3^2 = 45-bit number A of the following binary form:

 1 {5} 1 {5} 1 {5} 1 {3} 1 {3} 1 {3} 1 {9} 1 {9} 1 {9}

where {9}, {3} and {5} denote the 4-bit representations of the corresponding numbers. In simple terms, we need to pack each array element repeated n times together into a single number. It can be computed in linear time using, for example, the following code:

temp, A = 0, 0
for x in a:
    temp = (temp<<(n*(bits+1))) + (1<<bits) + x
for i in range(n):
    A = (A<<(bits+1)) + temp

The result is 23834505373497, namely:

101011010110101100111001110011110011100111001

Next, we need to compute another 45-bit number B, which will also pack all the elements of the array n times, however this time they will be separated by 0-bits and interleaved as follows:

 0 {5} 0 {3} 0 {9} 0 {5} 0 {3} 0 {9} 0 {5} 0 {3} 0 {9}

This again can be done in linear time:

temp, B = 0, 0
for x in a:
    temp = (temp<<(bits+1)) + x
for i in range(n):
    B = (B<<(n*(bits+1))) + temp

The result is 5610472248425, namely:

001010001101001001010001101001001010001101001

Finally, here comes the magic trick: we subtract B from A. Observe how with this single operation we now actually perform all pairwise subtractions of the numbers in the array:

A = 1 {5} 1 {5} 1 {5} 1 {3} 1 {3} 1 {3} 1 {9} 1 {9} 1 {9} 
B = 0 {5} 0 {3} 0 {9} 0 {5} 0 {3} 0 {9} 0 {5} 0 {3} 0 {9}

Consider what happens to the bits separating all the pairs. If the number on top is greater or equal to the number on the bottom of the pair, the corresponding separating bit on the left will not be carried in the subtraction, and the corresponding bit of the result will be 1. However, whenever the number on the top is less than the number on the bottom, the resulting bit will be zeroed out due to carrying:

A   = 1 {5} 1 {5} 1 { 5} 1 { 3} 1 {3} 1 { 3} 1 {9} 1 {9} 1 {9} 
B   = 0 {5} 0 {3} 0 { 9} 0 { 5} 0 {3} 0 { 9} 0 {5} 0 {3} 0 {9}
A-B = 1 {0} 1 {2} 0 {12} 0 {14} 1 {0} 0 {10} 1 {4} 1 {6} 1 {0}

The same in binary (highlighted groups correspond to repetitions of the original array elements in the number A):

A   = 1 0101 1 0101 1 0101|1 0011 1 0011 1 0011|1 1001 1 1001 1 1001
B   = 0 0101 0 0011 0 1001|0 0101 0 0011 0 1001|0 0101 0 0011 0 1001
A-B = 1 0000 1 0010 0 1100|0 1110 1 0000 0 1010|1 0100 1 0110 1 0000

Each "separator" bit of A-B is effectively the result of a comparison of every array element with every other. Let us now extract these bits using a bitwise AND and sum them within each group. It takes another couple of linear passes:

x = A-B >> bits
mask, result = 0, 0
for i in range(n):
    mask = (mask<<(n*(bits+1))) + 1
for i in range(n):
    result += x & mask
    x = x >> (bits+1)

The result is now the following number:

result = 10|000000000000001|000000000000011

It is a packed binary representation of the array r = [2, 1, 3]. The number 2 here tells us that there are two elements in a, which are less or equal than a[0]=5. Similarly, the number 1 says that there is only one element less or equal than a[1]=3, and the number 3 means there are three elements less or equal than a[2]=9. In other words, this is an array of ranks, which tells us how the original array elements should be rearranged into sorted order:

r = [result >> (n*(bits+1)*(n-i-1)) & ((1<<(n*(bits+1)))-1) 
                                          for i in range(n)]
a_sorted = [None]*n
for i in range(n):
    a_sorted[r[i]-1] = a[i]

And voilà, the sorted array! As presented above, the method would only work for arrays consisting of distinct non-negative integers. However, with some modifications it can be adapted to arbitrary arrays of integers or floats. This is left as an exercise to the reader.

The General Implications

There are several things one can learn from the "Paul-and-Simon sort". Firstly, it shows the immense power of the unit-cost RAM computational model. By packing arbitrary amounts of data into a single register of unlimited size, we may force our imaginary computer to perform enormously complex parallel computations in a single step. Indeed, it is known that PSPACE-complete problems can be solved in polynomial time in the unlimited-precision RAM model. This, however, assumes that the machine can do arbitrary arithmetic operations. If you limit it to only additions, subtractions and multiplications (but not divisions or bit-shifts), you still cannot sort integers faster than \Omega(n \log n) even using infinitely-sized registers (this is the main result of the Paul and Simon's article that inspired this post). Not obvious, is it?

Of course, real computers can usually only perform constant-time operations on registers of a fixed size. This is formalized in the w-bit word-RAM model, and in this model the "Paul and Simon sort" degrades from a O(n) into a O(n^3) algorithm (with O(n^2) memory consumption). This is a nice illustration of how the same algorithm can have different complexity based on the chosen execution model.

The third thing that the "Paul and Simon sort" highlights very clearly is the power of arithmetic operations on packed values and bitstrings. In fact, this idea has been applied to derive practically usable integer sorting algorithms with nearly-linear complexity. The latter paper by Han & Thorup expresses the idea quite well:

Excerpt from Han & Thorup, "Integer Sorting in O(n sqrt(log log n)) Expected Time and Linear Space".

In case you need the full code of the step-by-step explanation presented above, here it is.

July 20, 2017

Raivo LaanemetsRemote Worker award

My company, Infdot OÜ, has been given an award "Kaugtöö Tegija 2017" for practising remote work. 50 companies in Estonia received the award. I have been freelancing for almost 15 years and participated many times in a remote team. While there are obvious communication benefits of working on-site, many of these projects would not have been possible to carry out without the option of working remotely.

Kaugtöö Tegija 2017 award

July 12, 2017

Raivo LaanemetsWhat React does not solve

React goes a long way to provide a sane approach for building large-scale complex frontend solutions but it does not solve everything. One of such use cases is the manipulation of existing markup.

Sometimes I need to add some interactivity to server-generated HTML. For example, to implement inline form validation. The server-side HTML might be generated by an ExpressJS application or WordPress or just be a set of static HTML files. Majority of web sites actually work this way. The alternative approach is to generate HTML (or rather the DOM tree) on the client. This is what React and many other view libraries or frontend frameworks try to do.

React wants to generate and fully control your DOM. It has the JSX language extension to do it and a clever internal mechanism to make DOM interactions fast. However, it does not make much sense to use it for scripting the existing markup. This is the use case solved better with jQuery or nowadays with plain browser APIs if you do not target the old browsers. View libraries that bind to HTML are also a good choice here. Two of such libraries are Knockout and Vue.js.

July 11, 2017

Raivo LaanemetsPython RPi.GPIO threading broken

RPi.GPIO is a popular Python module for accessing hardware GPIO pins on devices like Raspberry Pi. Unfortunately it has a threading bug that causes Python-side event callbacks executed in parallel in completely unexpected way.

Last week I worked on an app that was supposed to measure time difference between two electrical signals received by two GPIO pins. The code is based on the event-based handling of signals by using the function GPIO.add_event_detect. The callbacks access some global variables and log messages into the standard output by using the standard logging module.

I connected switches to GPIO pins for testing and used them to test that everything works. I set 500ms software debounce time for the callbacks. However, while pressing the switches I saw weird anomalies in the logged messages. I then added messages at the start and at the end of the callbacks and it was a great surprise seeing that start/end messages were interleaved.

A part of the documentation says:

the callback functions are run sequentially, not concurrently. This is because there is only one thread used for callbacks, in which every callback is run, in the order in which they have been defined

It comes out that the issue is already reported. The offending block of code starts here. This is executed indirectly by GPIO.add_event_detect and if you call the function multiple times "quick enough" then the thread_running flag might not get set by the firstly started thread and multiple handler threads will be created.

I was able to work around the issue by adding a small delay between GPIO.add_event_detect calls. This helps to ensure that the firstly created thread has set the flag in the condition.

GPIO.add_event_detect(channel1, GPIO.FALLING,
    callback=gpioCallback, bouncetime=500)
time.sleep(1)
GPIO.add_event_detect(channel2, GPIO.FALLING,
    callback=gpioCallback, bouncetime=500)

I rarely work with threads and prefer async IO. I wrote a MSc thesis on static thread analysis for C language and haven't used them since, unless there is no other choice. Threading is really hard. I hope that this post helps to solve some of the Raspberry GPIO anomalies you might have.

July 09, 2017

Four Years RemainingThe Blockchain Consensus Problem

The Dark Side of the Bitcoin

Recall that Bitcoin is a currency, i.e. it is a technology, which aims to provide a store of value along with a payment medium. With all due respect to its steadily growing adoption, it would be fair to note that it is not very good at fulfilling either of these two functions currently. Firstly, it is not a very reliable store of value due to extreme volatility in the price. Secondly, and most importantly, it is a mediocre payment medium because it is slow and expensive.

A typical transfer costs around $2 nowadays and takes about an hour for a full confirmation (or longer, if you pay a smaller fee). When you need to transfer a million dollars, this looks like a reasonable deal. When you buy a chocolate bar at a grocery store (something one probably does more often than transferring a million), it is unacceptable. Any plain old bank's payment card would offer a faster and cheaper solution, which is ironic, given that Bitcoin was meant to be all friendly, distributed and free (as in freedom) while banks are, as we all know, evil empires hungry for our money, flesh and souls.

The irony does not end here. The evil banks typically provide some useful services in exchange for the fees they collect, such as an online self-service portal, 24h support personnel, cash handling and ATMs, some security guarantees, interests on deposits, etc. The friendly Bitcoin offers nothing of this kind. What is Bitcoin wasting our money on then? Electricity, mainly! The Proof of Work (PoW) algorithm employed in the Bitcoin's blockchain requires the computation of quintillions of random, meaningless hashes to "confirm" payments. The "miner" nodes, running the Bitcoin's network are collectively performing more than 5 000 000 000 000 000 000 (five quintillion or five exa-) hash computations every second, continuously consuming as much electricity as the whole country of Turkmenistan. The situation is even worse if you consider that Bitcoin is just one of many other "coins" built upon the PoW algorithm (Ethereum and Litecoin being the two other prominent examples), and their overall power consumption is only growing with each day.

Just think of it: most of the $2 fee a Bitcoin user needs to pay for a transaction will neither end up as someone's wage nor make a return on investment in someone's pocket. Instead, it will burn up in fossil fuels which generate power for the "miners", wasting precious resources of our planet, contributing to global warming and pushing poor polar bears faster towards extinction. Is all this mayhem at least a "necessary evil"? Sadly, it is not.

The Unnecessary Evil

Formally speaking, Proof of Work is an algorithm for achieving consensus among a distributed set of nodes which collectively maintain a common blockchain. Is it the only such algorithm? Of course not! Many alternative methods exist, most of them (if not all) are both faster and less energy-hungry. In fact, the only valuable property of PoW is its ingenious simplicity. In terms of implementation it may very well be among the simplest distributed blockchain consensus algorithms ever to be invented.

It is natural that a successful pioneering technology (such as the Bitcoin) is originally built from simple blocks. Progress comes in small steps and you cannot innovate on all fronts at once, after all. There must come a time, however, when the limitations of the initially chosen basic blocks become apparent and the technology gets upgraded to something more efficient. With more than $1 billion dollars in electricity bills paid by Bitcoin users last year for the inefficiency of PoW, Bitcoin has long surpassed this turning point, in my opinion.

Unfortunately, due to its pioneering status, enormous inertia, ongoing hype and the high stakes involved, Bitcoin continues to roll on its old wooden proof-of-work wheels with no improvement in sight, somewhy still being perceived as the leader in the brave new world of cryptocurrencies.

Are nearly-instant and nearly-free payment along with energy efficiency too much to ask from a real "currency of the future"? I do not think so. In fact, Bitcoin could be such a currency, if only it could switch from the evil Proof of Work to a different, fast and eco-friendly consensus algorithm.

Which algorithm could it be? Let me offer you an overview of some of the current options I am personally aware of, so you could decide for yourself.

The Eco-Friendly Blockchain Consensus

Consider a network of many nodes, which needs to maintain a common state for a chain of blocks. There seem to be roughly three general categories of algorithms which the nodes could employ for their purpose: Proof of Authority (PoA), Nakamoto Consensus, and Byzantine Fault Tolerance (BFT). Let us consider them in order.

Proof of Authority

Perhaps the most straightforward solution would be to nominate a fixed subset of nodes as "authoritative", and let any of them append new blocks by signing them cryptographically. To avoid conflicting updates, nodes may agree on a predefined round-robin signing order, honestly randomize their waiting intervals, or use some kind of a deterministic lottery for selecting the signer for next block, etc.

As this approach relies on a fixed subset of (reasonably) trusted nodes, it does not look robust and secure enough for a proper worldwide distributed blockchain. For example, in the limit case of a single trusted party it is equivalent to using a single service provider such as a bank. None the less, it is a convenient baseline and an important primitive, actually applicable to a wide range of real-life blockchain deployments. By relying on a set of well-behaving parties, a PoA blockchain actually sidesteps most of the complexities of a real distributed algorithm, and can thus be made to perform much faster than any of the "truly distributed" algorithms.

The Ethereum software provides an implementation of this approach for those who want to run private chains. PeerCoin relies on the PoA principle by having "checkpoint blocks" signed regularly by a trusted authority. Finally, the Delegated Proof of Stake algorithm makes PoA work on a larger scale by relying on voting. It is probably one of the most interesting practical implementations of the idea.

Delegated Proof of Stake

Delegated Proof of Stake (DPoS) is a consensus algorithm implemented in Graphene-based blockchains (BitShares, SteemEOS). It is a variant of Proof of Authority, where the small set of authoritative delegate nodes is elected by voting. When electing the delegates, each node can cast the number of votes, proportional to their account value (or "stakeholder share"), thus "delegating their stake in the network". The elected authorities then participate in a simple and fast round-robin block confirmation with each node given a two second window for confirming the next block.

The security of DPoS hinges on the assumption that the nodes with the most stake in the system should generally manage to elect a set of reasonable authorities, and in case of errors, the misbehaving authorities will not cause too much trouble and will be quickly voted out. At the same time, being internally a PoA implementation, the DPoS-based blockchains are by an order of magnitude faster in terms of transaction throughput than any other currently running public blockchains. Notably, they can also naturally support fee-less transactions.

Nakamoto Consensus

Consider the variation of PoA, where there are no pre-selected trusted nodes (i.e. all nodes may participate in the algorithm). Each time a new block needs to be added to the chain, let us pick the node who will gain the right to add it according to some deterministic "lottery" system. The consensus can then be achieved by simply verifying that the resulting blockchain is conforming to the lottery rules at all times, and the conflicting chains are resolved by always preferring the "harder" chain (according to some notion of "hardness").

For example, the infamous Proof-of-Work is an example of such a method. The "lottery" here is based on the ability of a node to find a suitable nonce value. The "hardness" is simply the length of the chain. Such "lottery" methods are sometimes referred to as "Nakamoto consensus algorithms". In terms of efficiency, Nakamoto consensus algorithms are among the slowest consensus algorithms.

Several alternatives to the "PoW lottery" have been proposed. Let us review some of them.

Proof of Stake

Proof of Stake (PoS), first implemented in the Nxt cryptocurrency, is a Nakamoto consensus technique, where the nodes with a greater balance on their account are given a higher chance to "win the lottery" and sign the next block. The actual technique used in Nxt is the following: before signing a block every node obtains a pseudo-random "lottery ticket number" x by hashing the last block data with its own identifier. If this number is smaller than

    \[\alpha \cdot \text{(account balance)}\cdot \text{(time since last block)},\]

(where \alpha is a block-specific constant), the node gets the right to sign the next block. The higher the node's balance, the higher is the probability it will get a chance to sign. The rationale is that nodes with larger balances have more at stake, are more motivated to behave honestly, and thus need to be given more opportunities to participate in generating the blockchain.

Proof of Stake is typically considered as the primary alternative to Proof of Work without all the wasteful computation, and it should, in principle, be possible to transition the whole blockchain from the latter to the former. In fact, this is what may probably happen to Ethereum eventually.

Proof of Space

In Proof of Space (PoSpace), a consensus mechanism implemented in Burstcoin, the "miners" must first pre-generate a set of "lottery ticket numbers" in a particular manner for themselves, save these numbers on a hard drive and commit the hash (the Merkle tree root) of this complete ticket set to the blockchain. Then, similarly to Proof of Stake, by hashing the last block's data, a miner deterministically picks one of his own "lottery tickets" for the next block. If the value of this ticket, discounted by the number of tickets in possession, is small enough, the miner gets the right to sign the block. The more tickets a miner generates and stores, the better are his chances. When signing the block, the miner must present a couple of special hashes which he can only know if he constantly stores his complete set of tickets (or fully recomputes a large part of it every time, which is impractical). Consequently, instead of spending energy on the "mining" process, the nodes must constantly dedicate a certain amount of disk space to the algorithm.

Although it is probably among the less widely known methods, from both technical and practical standpoint, it is one of the most interesting techniques, in my opinion. Note how it combines the properties of PoS (speed and energy efficiency) with those of PoW (ownership of a real-world resource as a proxy for decentralization).

Proof of Burn

The idea behind Proof of Burn is to allow the nodes to generate their "lottery ticket numbers" by irretrievably transferring some coins to a nonexistent address and taking the hash of the resulting transaction. The resulting hash, scaled by the amount of coins burned, can then be used to gain the right to sign blocks just like in other Nakamoto lottery systems. The act of wasting coins is meant to be a virtual analogue of spending electricity on PoW mining, without actually spending it. Blockchains based purely on Proof of Burn do not seem to exist at the moment. However, the technique can  be used alongside PoW, PoS or other approaches.

Proof of Elapsed Time

Presumably, some Intel processors have specialized instructions for emitting signed tokens, which prove that a given process called a particular function a certain period of time ago. The Hyperledger project proposes to build a consensus algorithm around those. Each "miner" will gain the right to sign a block after it waits for a certain period of time. The token which proves that the miner did in fact wait the allotted time, would act as a winning lottery ticket. I do not see how this method could work outside of the trusted Intel-only environment or how is it better than a trivialized Proof of Stake (not sure I even understood the idea correcty), but I could not help mentioning it here for completeness' sake.

Hybrid Nakamoto Consensus Systems

Some systems interleave PoW and PoS confirmations, or add PoA signatures from time to time to lock the chain or speed-up block confirmations. In fact, it is not too hard to invent nearly arbitrary combinations of delegation, voting, payments, authorities and lotteries.

Byzantine Fault Tolerance

The Practical Byzantine Fault Tolerance (PBFT) algorithm offers an alternative solution to the consensus problem. Here the blockchain state is tracked by a set of "bookkeeping" nodes, which constantly broadcast all changes among themselves and consider a change reliably replicated when it is signed and confirmed by given quorum (e.g. 2/3) of the bookkeepers. The algorithms of this type can be shown to be reliable if no more than a third of the nodes are dishonest. The Ripple, Stellar and Antshares are examples of blockchains based on such techniques. This algorithm allows much higher transaction throughputs than Nakamoto consensus (PoW, PoS, PoSpace), yet it still lags behind the speed of PoA or DPoS.

July 04, 2017

TransferWise Tech Blogpg_ninja, now open source

In my previous post I wrote on how we scaled the analytics database using PostgreSQL.
One of the key requirements was the possibility to replicate and obfuscate the data in real time from our main MySQL database.
The tool which is managing this particular task is pg_ninja, now available under the terms of the Apache 2.0 license.

Some history

In our previous implementation we operated a MySQL replica for our analytics database, with some views exposing obfuscated data to the analysts.

With PostgreSQL I needed a dedicated tool for the job, as that functionality was not present.

Luckily at that time I was playing with a MySQL-to-PostgreSQL migrator tool called pg_chameleon, which was just a proof of concept.

In my spare time I converted this little experiment into a tool capable of replicating data from MySQL to PostgreSQL.

I forked pg_chameleon, as a starting point for building pg_ninja for Transferwise.

After all, pg_chameleon was not fit for production and the obfuscation requirements were out of pg_chameleon's scope.

During the development I struggled a lot to find a robust way to convert the DDL from the MySQL dialect to PostgreSQL's. I decided first to not reinvent the wheel. However, after several failures in using the python sqlparse library, I decided to write my own implementation using regular expressions, taking the occasion to learn how to use them.

Learning regex required time, and so I wrote pg_ninja's DDL support in a very tricky way. This worked quite well for first few months.
However, this method showed some serious limitations and I decided to use pg_chameleon's regular expression tokenisation, which is now very efficient.

The project status

pg_ninja is compatible with Cpython 2.7.
At moment is not present a daemonisation process but is very simple to automate the start using a cron job.

Multiple replica sources are possible with separate configuration files.

The obfuscation strategy is managed in a separate yaml file with four different strategies.

  • normal: the value si converted in a sha256 hash. It's possible to specify the start and the length of a not ashed value.
  • date: the date value is converted to the 1st of January preserving the year only.
  • setnull: the value is set to null
  • numeric: the value is set to 0

Further improvements

pg_ninja will be distributed soon via pypi.

The code change to make pg_ninja compatible with python 3.3+ is not complex and will happen soon in the future.

In the future I'll write the docstrings on the libraries in order to get a proper API documentation.

The future

In my free time I'm currently exploring a complete rewrite of pg_chameleon with full daemonisation support, parallel copy when initialising and separate processes for read and replay.

When this project will be stable enough I'll build the same for pg_ninja.

Contributing to the project

PR on github are absolutely welcome, you are encouraged to fork the repository https://github.com/transferwise/pg_ninja

July 03, 2017

Raivo LaanemetsThe Feeds app UI rewritten in React

On the last weekend I rewrote my Feeds app (live) UI in React. I originally wrote the application many years ago and used KnockoutJS back then. This rewrite gave me some useful insight about writing a Single Page Application in React and gave me a chance to compare it to KnockoutJS.

KnockoutJS

KnockoutJS is a two-way binding MVVM library based on implicitly tracked observables. By the time of writing the original application code, it was targeting pre-ES5 JavaScript (IE6 support!) and did not have components support built-in. The application code was structured around a god object which kept the whole state and a set of templates that were rendered depending on the requested view (articles, feeds, feed articles etc.). This made the code relatively compact but also made adding new functionality harder than it should be.

KnockoutJS templates are based on DOM. Template binding is achieved by using special data-bind attributes. The binding syntax is similar to JavaScript. The syntax can be arbitrarily extended and cannot be linted.

React

React is also a view library but it uses different concepts than KnockoutJS. It does not use two-way binding. Input from the page DOM is "read" using events. There are no observable properties, state changes are detected by comparing immutable state objects. The code is structured around components where state changes are applied through the special setState method. Inter-component communication is handled through props (from parent to children) and events (from children to parent and from DOM to component).

React uses JSX (JavaScript Syntax eXtension?) to generate DOM. JSX is an XML-like language, designed to be embedded into JavaScript. It is compiled into JavaScript. Compared to KnockoutJS binding syntax, JSX has strict specification, is lintable, supported by IDEs/text editors, and has multiple alternative implementations besides React.

JSX requires a compilation step. In the React version of the Feeds application this is handled by Babel. In fact, this is the only transformation that I have enabled in Babel. With this application I only target evergreen browsers, all of which support enough ES6+ at the moment (I do use CommonJS modules with Webpack, not ES6 modules). JSX was the reason why I did not consider using React before. In 2013, when React was first released, JSX tooling was garbage. This has completely changed for now.

State management

A big topic about React is state management. The use of immutable data requires a specific approach. There are libraries for simplifying it, such as Flux and Redux.

The Feeds application does not have much global state (only the authenticated flag is global) and I did not use a state management library. Instead, I used the Container Components pattern where the application state resides in a few selected components and is loaded over AJAX. Example: ArticleList is a container component for a set of Articles.

Issues

The only issue I encountered during the rewrite was inconsistent charSet attribute on the meta tag. For some reason React did not warn me about the invalid attribute and chose to just not render it.

Feelings

I think that React has come a long way and I plan to use it in my new applications. I'm also experimenting with JSX on the server side where it provides much better abstractions than string-based templating languages such as EJS. A couple of months ago I wrote a proprietary Electron app with Vue.js, another popular view library. While Vue.js is closer to KnockoutJS (DOM templates, two-way data binding), there is something in React (maybe it's the use of immutable data structures, the simplicity of pure components, and JSX) that makes me more and more sympathetic towards it.

June 11, 2017

Raivo LaanemetsPkg and Node.js 8 performance

Pkg is a Node.js application compiler. It makes application distribution easier without requiring a complex infrastructure setup like Docker. It packages and creates a single executable that can also contain non-executable asset files.

However, compiled scripts suffer from a performance penalty under the V8 Crankshaft compiler pipeline. The performance hit is much smaller under Ignition+TurboFan. Ignition is the new JavaScript interpreter and TurboFan is the new JIT compiler in V8.

I made some benchmarks using Node 8 (will become LTS soon). The benchmark is a minimal Express.js application that outputs a page generated from random data using an EJS template. With this application the results were:

  • Under the Crankshaft compilation pipeline there is a performance penalty about 31%.
  • For Ignition+TurboFan pipeline the performance penalty is much smaller at about 1.4%
  • Without packaging, the Ignition+TurboFan pipeline is 6.7% slower than the Crankshaft.

The benchmark code is on GitHub.

Update 2017-06-14

The Crankshaft compilation pileline performance penalty can be avoided when the application source is included. There are certain conditions required for this. See discussion here for more information.

June 01, 2017

Anton ArhipovConferences I have visited in May'17

Riga DevDays


Riga DevDays - a perfect conference of an Estonian to visit: not too far away (45 minutes flight), nice city, very good event!

There, I have presented a talk about Java class reloading, which covers the different options for reloading Java classes and explains the fundamental differences of those.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/key/N2q4gQxX3ay878" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" width="595"> </iframe>

GeeCON, Krakow


I have presented at GeeCON before. The vibe of the event is quite energising! :) I have presented a talk about TestContainers which seemed to spark a lot of the interest from the attendees. Almost a full room and a lot of questions after the talk. Looks like integration testing is in demand these days!

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/key/19nHX2LHHABZlX" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" width="595"> </iframe>

JUG.ua & JEEConf, Kiev


The visit to Kiev (Kyiv) was super-productive. I've visited EMAP offices of the on-site presentation as well as a local JUG meetup just before the conference. Very good attendance: 100+ people came to the meetup. Interestingly enough, in Ukraine (as well as in Russia) people ask questions in an interesting way: they usually start the question with "What if ...". They are always curious to find the limitations of the technology, the approach, the method, etc - almost like trying to break things. I think this critical mindset is very helpful when you have to develop software these days.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/key/fGoz1yIgbaDe09" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" width="595"> </iframe>

At the JEEConf I have presented 3 talks: 2 on my own, and 1 with Anton Keks, helping to deliver the Kotlin Puzzlers talk. This was a very well organized conference: super-nice view in the center of Kiev, well crafted schedule with the interesting and useful talks, good athmosphere... I recommend :)


I had a pleasure to deliver a live coding session about Javassist, though I still have the slides just as a reference for those who attended the session. I don't find this talk to be very useful for the developers, however, attendees still find it interesting, so I'm puzzled with this a bit :) Here are the slides:


<iframe allowfullscreen="allowfullscreen" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/key/AGFO67Pwn0d5xO" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" width="595"> </iframe>

As for the Java class reloading talk, I had some time to update the content since Riga DevDays -- removed boring parts and added a few other things. Lots of "What if.." questions after the talk -- I love this crowd! :)


<iframe allowfullscreen="allowfullscreen" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/key/hUqhyi91cVo5B0" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" width="595"> </iframe>