Pinu planeet

March 16, 2018

Four Years RemainingA Program for Adding 10 Numbers

I have randomly stumbled upon a Quora question "Can you write a program for adding 10 numbers" yesterday. The existing answers competed in geeky humor and code golf, so I could not help adding another take on the problem.

Can you write a program for adding 10 numbers?

The question offers a great chance to illustrate how to properly develop software solutions to real-life problems such as this one.

First things first - let us analyze the requirements posed by the customer. They are rather vague, as usual. It is not clear what “numbers” we need to add, where and how should these “numbers” come from, what is really meant under “adding”, what should we do with the result, what platform the software is supposed to be running on, what are the service guarantees, how many users are expected, etc.

Of course, we do not want to discover that we misunderstood some of the requirements late in the development cycle, as this could potentially require us to re-do all of the work. To avoid such unpleasant surprises we should be planning for a general, solid, enterprise-grade solution to the problem. After a short meeting of the technical committee we decided to pick C# as the implementation platform. It is OS-independent and has many powerful features which should cover any possible future needs. For example, if the customer would decide to switch to a cluster-based, parallel implementation later along the way, we’d quickly have this base covered. Java could also be a nice alternative, but, according to the recent developer surveys, C# development pays more.

The Architecture

Let us start by modeling the problem on a higher level. The customer obviously needs to process (“add”) some data (“10 numbers”). Without getting into too much detail, this task can be modeled as follows:

interface IInputProvider {}
interface IOutput {}
interface ISolution {
    IOutput add10(IInputProvider input);    

Note how we avoid specifying the actual sources of input and output yet. Indeed, we really don’t know where the “10 numbers” may be coming from in the future - these could be read from standard input, sent from the Internet, delivered by homing pigeons, or teleported via holographic technology of the future - all these options are easily supported by simply implementing IInputProvider appropriately.

Of course, we need to do something about the output once we obtain it, even though the customer forgot to mention this part of the problem. This means we will also have to implement the following interface:

interface IOutputConsumer {
    void consumeOutput(IOutput output);

And that is it - our general solution architecture! Let us start implementing it now.

The Configuration

The architecture we work with is completely abstract. An actual solution would need to provide implementations for the IInputProviderIOutputConsumer and ISolution interfaces. How do we specify which classes are implementing these interfaces? There are many possibilities - we could load this information from a database, for example, and create a dedicated administrative interface for managing the settings. For reasons of brevity, we’ll illustrate a simplistic XML-based factory method pattern.

Namely, we shall describe the necessary implementations in the XML file config.xml as follows:

    <InputProvider class="Enterprise.NumberSequenceProvider"/>
    <OutputConsumer class="Enterprise.PeanoNumberPrinter"/>
    <Solution class="Enterprise.TenNumbersAddingSolution"/>

A special SolutionFactory class can now load this configuration and create the necessary object instances. Here’s a prototype implementation:

class SolutionFactory {
    private XDocument cfg;
    public SolutionFactory(string configFile) {
        cfg = XDocument.Load(configFile);
    public IInputProvider GetInputProvider() {
        return Instantiate<IInputProvider>("InputProvider");
    public IOutputConsumer GetOutputConsumer() {
        return Instantiate<IOutputConsumer>("OutputConsumer");
    public ISolution GetSolution() {
        return Instantiate<ISolution>("Solution");
    private T Instantiate<T>(string elementName) {
        var typeName = cfg.Root.Element(elementName)
        return (T)Activator.CreateInstance(Type.GetType(typeName));

Of course, in a real implementation we would also worry about specifying the XML Schema for our configuration file, and make sure it is possible to override the (currently hard-coded) “config.xml” file name with an arbitrary URI using command-line parameters or environment variables. In many real-life enterprise solutions in Java, for example, even the choice of the XML parsing library would need to be configured and initialized using its own factory pattern. I omit many of such (otherwise crucial) details for brevity here.

I am also omitting the unit-tests, which, of course, should be covering every single method we are implementing.

The Application

Now that we have specified the architecture and implemented the configuration logic, let us put it all together into a working application. Thanks to our flexible design, the main application code is extremely short and concise:

class Program {
    static void Main(string[] args) {
        var sf = new SolutionFactory("config.xml");
        var ip = sf.GetInputProvider();
        var oc = sf.GetOutputConsumer();
        var sol = sf.GetSolution();
        var op = sol.add10(ip);

Amazing, right? Well, it does not really work yet, of course, because we still need to implement the core interfaces. However, at this point we may conclude the work of the senior architect and assign the remaining tasks of filling in the blanks to the the main engineering team.

The Inputs and Outputs

Now that we have set up the higher-level architecture, we may think a bit more specifically about the algorithm we plan to implement. Recall that we need to “add 10 numbers”. We don’t really know what these “numbers” should be - they could be real numbers, complex numbers, Roman numerals or whatnot, so we have to be careful and not rush into making strict assumptions yet. Let’s just say that a “number” is something that can be added to another number:

interface INumber: IOutput {
    INumber add(INumber other);

We’ll leave the implementation of this interface to our mathematicians on the team later on.

At this step we can also probably make the assumption that our IInputProviderimplementation should somehow give access to ten different instances of an INumber. We don’t know how these instances are provided - in the worst case each of them may be obtained using a completely different method and at completely different times. Consequently, one possible template for an IInputProvider could be the following:

interface ITenNumbersProvider: IInputProvider {
    INumber GetNumber1();
    INumber GetNumber2();
    INumber GetNumber3();
    INumber GetNumber4();
    INumber GetNumber5();
    INumber GetNumber6();
    INumber GetNumber7();
    INumber GetNumber8();
    INumber GetNumber9();
    INumber GetNumber10();

Note how, by avoiding the use of array indexing, we force the compiler to require that any implementation of our ITenNumbersProvider interface indeed provides exactly ten numbers. For brevity, however, let us refactor this design a bit:

enum NumberOfANumber {
interface ITenNumbersProvider: IInputProvider {
    INumber GetNumber(NumberOfANumber noan);

By listing the identities of our “numbers” in an enum we still get some level of compile-time safety, although it is not as strong any more, because enum is, internally, just an integer. However, we god rid of unnecessary repetitions, which is a good thing. Refactoring is an important aspect of enterprise software development, you see.

The senior architect looked at the proposed interface at one of our regular daily stand-ups, and was concerned with the chosen design. “Your interface assumes you can provide immediate access to any of the ten numbers”, he said. But what if the numbers cannot be provided simultaneously and will be arriving at unpredictable points in time? If this were the case, an event-driven design would be much more appropriate:

delegate void NumberHandler(NumberOfANumber id, INumber n);
interface IAsynchronousInputProvider: IInputProvider {
    void AddNumberListener(NumberHandler handler);

The adding subsystem would then simply subscribe to receive events about the incoming numbers and handle them as they come in.

“This is all good and nice”, responded the mathematician, “but for efficient implementation of the addition algorithm we might need to have all ten numbers available at the same time”. “Ah, software design 101”, says the senior architect. We simply install an adapter class. It would pool the incoming data until we have all of it, thus converting the IAsynchronousInputProvider, used for feeding the data, into an ITenNumbersProvider, needed by the mathematician:

class SyncronizationAdapter: ITenNumbersProvider {
   private Dictionary<NumberOfANumber, INumber> nums;
   private ManualResetEvent allDataAvailableEvent;
   public SynchronizationAdapter(IAsynchronousInputProvider ainput){
       nums = new Dictionary<NumberOfANumber, INumber>();
       allDataAvailableEvent = new ManualResetEvent(false);
   private void HandleIncomingNumber(NumberOfANumber id, INumber n){
       nums[id] = n;
       if (Enum.GetValues(typeof(NumberOfANumber))
               .All(k => nums.ContainsKey(k)))
   public INumber GetNumber(NumberOfANumber noan) {
       return nums[noan];

Now the mathematician can work on his addition logic without having to know anything about the way the numbers are coming in. Convenient, isn’t it?

Note that we are still only providing the input interface specification (along with an adapter) here. The actual implementation has to wait until our mathematicians come up with an implementation of INumber and the data engineers decide on how to obtain ten of these in the most optimal way.

But what about IOutput? Let us assume that we expect to output a single number. This means that INumber must itself already be an instance of IOutput:

interface INumber: IOutput {
   INumber add(INumber other);

No need to implement anything, we just add an interface tag to INumber! See how object-oriented design techniques allow us to save development time!

The Order of Addition

OK, so we now have a concept of an INumber which has a (binary) addition operation defined, an ITenNumbersProvider which can provide ten INumber instances (conveniently abstracting away the IAsynchrhonousInputProvider which actually obtains the numbers), and our goal is to add them up to get an IOutput which is itself an INumber. Sounds easy, right? Not so fast! How exactly are we going to add these numbers? After all, maybe in some cases adding ((a+b)+c)+d)… can be less efficient or precise than (a+(b+(c+(d…. Or maybe the optimal addition strategy is to start from the middle and then add numbers in some order? There do exist nontrivial ways to add up numbers, you know. To accommodate for any possible options in the future (so that we wouldn’t have to rewrite the code unnecessarily), we should design our solution in a way that would let us switch our addition strategy easily, should we discover a better algorithm. One way to do it is by abstracting the implementation behind the following interface:

interface IAdditionStrategy {
   INumber fold(Func<NumberOfANumber, INumber> elements,
                Func<INumber, INumber, INumber> op); 

You see, it is essentially a functor, which gets a way to access our set of numbers (via an accessor function) along with a binary operator “op”, and “folds” this operator along the number set in any way it deems necessary. This particular piece was designed by Harry, who is a huge fan of functional programming. He was somewhat disappointed when we decided not to implement everything in Haskell. Now he can show how everyone was wrong. Indeed, the IAdditionStrategy is a core element of our design, after all, and it happens to look like a fold-functor which takes functions as inputs! “I told you we had to go with Haskell!”, says Harry! It would allow us to implement all of our core functionality with a much higher level of polymorphism than that of a simplistic C# interface!

The Solution Logic

So, if we are provided with the ten numbers via ITenNumbersProvider and an addition strategy via IAdditionStrategy, the implementation of the solution becomes a very simple matter:

class TenNumbersAddingSolution: ISolution {
   private IAdditionStrategy strategy;
   public TenNumbersAddingSolution() {
       strategy = ...
   public IOutput add10(IInputProvider input) {
       var tenNumbers = new SynchronizationAdapter(
       return strategy.fold(i => tenNumbers.GetNumber(i), 
                            (x,y) => x.add(y));

We still need to specify where to take the implementation of the IAdditionStrategy from, though. This would be a good place to refactor our code by introducing a dependency injection configuration framework such as the Autofac library. However, to keep this text as short as possible, I am forced to omit this step. Let us simply add the “Strategy” field to our current config.xml as follows:

    <Solution class="Enterprise.TenNumbersAddingSolution">
        <Strategy class="Enterprise.AdditionStrategy"/>

We could now load this configuration setting from the solution class:

    public TenNumbersAddingSolution() {
        var cfg = XDocument.Load("config.xml");
        var typeName = cfg.Root
        strategy = (IAdditionStrategy)Activator

And voilà, we have our solution logic in place. We still need to implement INumberIAdditionStrategyITenNumbersProvider and IOutputConsumer, though. These are the lowest-level tasks that will force us to make the most specific decisions and thus determine the actual shape of our final product. These will be done by the most expert engineers and mathematicians, who understand how things actually work inside.

The Numbers

How should we implement our numbers? As this was not specified, we should probably start with the simplest possible option. One of the most basic number systems from the mathematician’s point of view is that of Peano natural numbers. It is also quite simple to implement, so let’s go for it:

class PeanoInteger: INumber {
    public PeanoInteger Prev { get; private set; }
    public PeanoInteger(PeanoInteger prev) { Prev = prev; }
    public INumber add(INumber b) {
        if (b == null) return this;
        else return new PeanoInteger(this)

Let us have IOutputConsumer print out the given Peano integer as a sequence of “1”s to the console:

class PeanoNumberPrinter: IOutputConsumer {
    public void consumeOutput(IOutput p) {
        for (var x = (PeanoInteger)p; x != null; x = x.Prev)

Finally, our prototype IAdditionStrategy will be adding the numbers left to right. We shall leave the option of considering other strategies for later development iterations.

class AdditionStrategy: IAdditionStrategy {
    public INumber fold(Func<NumberOfANumber, INumber> elements,
                        Func<INumber, INumber, INumber> op) {
       return Enum.GetValues(typeof(NumberOfANumber))

Take a moment to contemplate the beautiful abstraction of this functional method once again. Harry’s work, no doubt!

The Input Provider

The only remaining piece of the puzzle is the source of the numbers, i.e. the IAsynchronousInputProvider interface. Its implementation is a fairly arbitrary choice at this point - most probably the customer will want to customize it later, but for the purposes of our MVP we shall implement a simple sequential asynchronous generator of Peano numbers {1, 2, 3, …, 10}:

class NumberSequenceProvider: IAsynchronousInputProvider {
    private event NumberHandler handler;
    private ManualResetEvent handlerAvailable;
    public NumberSequenceProvider() {
        handlerAvailable = new ManualResetEvent(false);
        new Thread(ProduceNumbers).Start();
    public void AddNumberListener(NumberHandler nh) {
        handler += nh;
    private void ProduceNumbers() {
        PeanoInteger pi = null;
        foreach (var v in Enum.GetValues(typeof(NumberOfANumber))
                              .Cast<NumberOfANumber>()) {
                pi = new PeanoInteger(pi);
                handler(v, pi);

Note that we have to be careful to not start publishing the inputs before the number processing subsystem attaches to the input producer. To achieve that we rely on the event semaphore synchronization primitive. At this point we can clearly see the benefit of choosing a powerful, enterprise-grade platform from the start! Semaphores would look much clumsier in Haskell, don’t you think, Harry? (Harry disagrees)

So here we are - we have a solid, enterprise-grade, asynchronous, configurable implementation for an abstractly defined addition of abstractly defined numbers, using an abstract input-output mechanism.

$> dotnet run

We do need some more months to ensure full test coverage, update our numerous UML diagrams, write documentation for users and API docs for developers, work on packaging and installers for various platforms, arrange marketing and sales for the project (logo, website, Facebook page, customer relations, all that, you know), and attract investors. Investors could then propose to pivot the product into a blockchain-based, distributed solution. Luckily, thanks to our rock solid design abstractions, this would all boil down to reimplementing just a few of the lower-level interfaces!

Software engineering is fun, isn’t it?

The source code for the developed solution is available here.

March 08, 2018

TransferWise Tech BlogBuilding modular apps using Spring

Building modular apps using Spring

When looking at most of the sample applications and code examples for Spring framework it may seem that everything should be defined as a bean. However, if we want to achieve nicely modular design then we should try to do the exact opposite - have each module expose as little as possible as a bean.

In essence beans are just like global variables. We cannot do anything to control or restrict who can access them. There are no namespaces that might give us glues if what we are accessing is something quite close to us or something we should not have any dependency on. By exposing every Service as a bean we effectively make it part of the API of given module. The larger the API the harder it is to make sure that different parts of our system stay loosely coupled.

If we have small application then perhaps this is not so big issue but the bigger our codebase gets the more dangerous this jungle of beans will become. We may have very nice and thought through package structure but by exposing all behavior in one single namespace we nullify all that effort.

Misreading framework examples

The problem with code examples is not anything specific to Spring. In fact I think Spring has awesome documentation which probably has no small role in its success. It's just that if we want to show some specific feature of framework X then it is quite obvious that we want to keep the code as simple and unopinionated as possible. Our goal is to show that framework can be used together with any kind of application design.

However, the problem is that for developers reading this documentation it is very easy to start taking these code examples as "the way how systems should be built". No matter how unopinionated and impersonal style was used in examples it will still inevitably suggest some kind of approach to system design. Of course defining beans is just one aspect that can be misread. There are many other similar simplifications like implementing a Service for each Entity or having package structure per stereotype and not per module. Both are good enough for sample application but inevitably fall short for any bigger production application.

Only expose API beans

So instead of making everything a bean we should only expose Services that form the API of given module.

These API beans are both Services that are being accessed by other modules inside our applications but also things that implement the public API of our application. If we think in terms of hexagonal architecture then all Adapters can be defined as Spring beans.

Spring supports this kind of design thanks to its Java config.

For example:

ProfileGateway profileGateway() {  
  return new ProfileGatewayWithFallback(
    new ClientBasedProfileGateway(
      new ProfilesFromProfileService(profileClient),
        new CachedProfiles(repository)

Here we define just one bean that wraps access to some external profile-service instead of having 4 beans. We only want profile-service to be accessed via profileGateway bean and we do not want these other 3 Services to be used anywhere outside of the profile module.

Define one configuration class per each module

Another aspect that I have often found myself at loss is deciding how to split different bean configurations. Obviously we could have everything in one configuration class but it grows quite quickly even for relatively small microservice.

Again thinking in terms of modules helps here. It makes sense to have separate @Configuration for each module. This also given additional benefit when running integration tests as we have logical subsets of our application that we can start.


If we look at things from testability perspective then it may seem that having everything as a bean gives us more flexibility to choose what parts of application we want to have wired together by Spring and what things to replace with mocks. One might argue that now that we have less beans we will be more constrained in what we can stub/mock. Yes that is true but it is not necessarily a bad thing.

I have often seen integration tests that launch dozens of beans and then seemingly randomly mock out just a few. The problem with such tests is that they are this strange amalgam between unit and integration tests. I find that very often these tests are better rewritten as plain unit tests without any framework dependency or as high level component tests that test the microservice as a whole.

Writing in-process component tests is very suitable for doing high level acceptance testing that complements unit tests as suggested in Growing Object-Oriented Software, Guided by Tests book.

For such component tests we can do all input and output through HTTP API (using Spring MVC test framework) and stub only Adapter beans to external systems. I have rarely seen need for stubbing things out on a more granular level.

Following is an example of placing an order for transferring 2000 EUR to GBP in a cross-currency money transfer application. Test is written using Groovy and Spock.

def "places new order and updates it"() {  
    // we set up some stubs for external services
    rateGateway.rate 1.234
    pricingGateway.fee 5.5

    // this is a wrapper around Spring MockMvc 
    // using Groovy JsonBuilder to put together the request
    def order = quoteService.placeOrder({
      profile 111
      sourceCurrency "EUR"
      targetCurrency "GBP"
      sourceAmount 1000

        sourceAmount 2000

    def fetchedOrder = quoteService.getById(

  then: ==
    fetchedOrder.profile == 111
    fetchedOrder.sourceAmount == 2000
    fetchedOrder.rate == 1.234    
    fetchedOrder.fee == 5.5 
    fetchedOrder.sourceCurrency == "EUR"
    fetchedOrder.targetCurrency == "GBP"   

* Used image of a spring without framework from pxhere

March 05, 2018

Raivo LaanemetsWorkaround to broken SublimeLinter 4 eslint support

SublimeLinter is breaking compatibility and provides no auto-migration while auto-updating through Package Control. This broke eslint support today morning. Workaround for now:

  1. Remove SublimeLinter and SublimeLinter-eslint from Package Control.
  2. Stop Sublime.
  3. Remove all references to SublimeLinter in ~/.config/sublime_text_3 (different path on other OS).
    • This might break your syntax highlighting theme setup as some versions of SublimeLinter move these files around and modify them.
  4. Download SublimeLinter 3.10.10 release and install manually.
  5. Download SublimeLinter-eslint 1.10.1 and install manually.

This will get you a predictable setup and less bad surprises when coming to work and see that all your tools are broken.

Looking through the issues of SublimeLinter on GitHub, it seems like lots of changes have been done to work around issues with linters of other languages (and their crappy package managers), none of which applies to eslint.

February 26, 2018

TransferWise Tech BlogReducing your Webpack bundle

The last couple of months, my team at Transferwise, has been working to deliver a new feature on our product. We call this feature Money Tracker and helps TransferWise users to track their money better. I've been developing it for the web using Angular as an external component. As a general approach for our front end all of our new features are created in their own repository, using ES2015, Webpack and Babel.

After deploying the first versions and going through bug fixing we found out that my newly created bundle from Webpack was 1mb. This is unacceptable size for an app as large as ours (or any app really). After a bit of fiddling around I managed to drop the size of it in less than 40kb. This is how I did it.


The first step is to find out why your bundle is that big. The hard way to do this is by searching in the bundle outcome of Webpack for know libraries. But this is tedious and you'll miss stuff for sure. The best way to do so is by using a Webpack analyser. I ended up using Webpack Visualizer by Chris Bateman.

To use it just do npm install --save-dev webpack-visualizer-plugin and then in your webpack.conf.js you need to add the following things:

var Visualizer = require('webpack-visualizer-plugin');

plugins: [new Visualizer()],  

Now if you run your Webpack build job locally you will get a stats.html file. That will look something like this:

Webpack visualizer graph

We can see a few things here. The size of my bundle (1mb 😱) and a very nice breakdown where is everything coming from. In my case I was bundling moment.js, bootstrap.css, bootstrap.js and an internal TransferWise library. All of these things for me were peer dependancies, so I shouldn't have bundled them.


This is the point that you need to understand where this files are coming from. There are two suspects here, libraries that you haven't marked as externals in your webpack.conf.js or libraries you have explicitly included in one of your files, e.g import moment from 'moment'. Lets look at these two things.

Webpack externals

This is a configuration in Webpack that tells the bundler that it shouldn't pack these things, because they will be already there to the application that is going to use your new module. This is excellent for these cases where you are developing a library or a component to be used by a bigger app. By using externals you greatly reduce your bundle size.

So in my moment example I should have added the following line in my webpack.conf.js:

// ...
externals: {  
  moment: 'moment'
// ...

Now, for bootstrap and the TransferWise library I had already an entry for them in externals, so I really wondered why they were still bundled. And that's how we go to the second point.

Explicit imports

In my package I have central index.js that works as the starting point of everything. Bootstraps the component, imports necessary libraries etc. In there I had these three lines:

import 'awesome-transferwise-library';  
import 'bootstrap/dist/js/bootstrap';  
import 'bootstrap/dist/css/bootstrap.css';  

These three lines were saying to Webpack that it should really bundle them as the application needs them. That wasn't necessary though. These libraries are marked and peer dependancies in my package.json so the app that uses my component should already have them. So I removed them, and they stopped being bundled. It was my bad for including them in the first place.

The only caveat here is that now your tests might be failing (mine did, that's one of the reasons why I put them there in the first place). To solve that just import the necessary libraries in your test configuration file.


So here's what I learn from this:

  • Use an analyser to see what is it that you are packaging
  • Avoid explicit import of libraries that are marked as external dependancies
  • Learn how to use Webpack external externals
  • Don't serve massive bundles, not everyone has a speedy connection. You are making the customer experience worst.

February 09, 2018

TransferWise Tech BlogPull Requests that make code reviewers smile

Pull Requests (PR) are central to most software development organisations these days. Modern source control tools such as Github or Continuous Integration ones like CircleCI have helped a lot making them more accessible. Yet, it seems developers consider PRs mostly as something that you need get through to get your code merged to master. It runs your tests and the merge button becomes available.

We are forgetting that the main purpose of a PR is to receive a code review.

Someone reviewing your code is doing you a favour: They take from their time doing more fun stuff such as coding and instead try to understand what you have been trying to do. They help you catch issues before they reach production. Probably your code reviewer will save you from several rollbacks and embarrassing postmortems.

It is in your interest to make it as easy as possible to give you a good code review.

Sometimes, a PR makes me feel like closing my browser tab and not giving a damn about it, because obviously the poster didn’t care about my time. For example:

  • The PR has no description at all. How can I know what you are trying to fix?

  • Worse, the PR topic is the automatically generated one from git based on the branch name. Arghh!!

  • The PR has one big commit with multiple different things, like indentation fixes, some refactorings, and the actual change.

  • The PR is made of plenty of "Fix tests" commits.

You don't care, so why should I?

So how can you help your code reviewers, besides obviously writing clean code? Have good commits and a good description.

Organize your individual commits cleanly. If you need to do some code cleanup before implementing your actual feature, put it in a separate commit. Your code reviewer can check the PR commit by commit, going quickly over your cleanup ones and spending more time on the part with the real changes. Git with interactive rebase and git add -p makes it easy to have well organized commits.

An obvious requirement for this to work is to have good commit messages, which describe the actual commit content.

Additionally, each commit needs to be self contained. After each commit, the application must be functional, with all tests passing. Otherwise reviewing commit by commit doesn’t make sense.

A PR description gives the reviewer background about the changes. It doesn’t repeat the relevant comments you have put in the code, instead it covers why the change is made, and any specific things about. For example, if the change needs some special care when releasing, the poster will indicate it in the description. The code reviewer won’t have to worry about it, knowing the poster has already considered it. A PR about some UI changes will be much easier to understand if you add some screenshots to it.

A link to the corresponding ticket in the bug tracking system is also strongly advised, but this never replaces a description. Let’s face it, often the information in the bug tracking system is a bit outdated already when the code has been implemented.

So give some love to your PRs, your reviewers will be grateful!

January 20, 2018

Raivo LaanemetsSummary of 2017


In 2017 I wrote less than on the previous years. I did not run out of topics, I have collected a huge list of them but under a time pressure I decided to do other things than writing.

Neither the blog engine (I use Blog-Core which I maintain myself) and the blog code itself got any upgrades during the year. I consider them to be done.


I worked on two major projects in 2017 and both of them were further developments of older projects.

Electronic voting system

In 2016 I worked on an online voting system for Estonian Free Party. We could not employ it back then as the legal regulation of using a such system was not clear enough. For March 2017 it was sorted out. The original system relied on electronic identification which was not enough and electronic signatures had to be used. Electronic signatures have full legal power in Estonia. I implemented the new approach and the system was then used for voting the board and the other organs of the organization.

Shipping order system

In 2016 I worked on a small in-browser app to optimize the placement of packages on a freight truck. The users liked the app a lot and we decided to enhance it by developing a multi-user centralized backend with loading scheduling and other useful additional features. The application was built on the Node.js Express framework with MySQL database on the backend and React-based forms on the frontend. A part of the system is also a custom mobile app to take photos of trucks. The mobile app was implemented using DroidScript.

Learning new tech skills

My top objective in learning new technology has always been finding new ways to improve my current work. I try to avoid jumping into something very new unless I'm convinced that it is going to bring benefits.

React - I had tried to use React before in 2013 but the tooling (especially the JSX support) was quite bad back then. I was really surprised that this was not the case anymore in 2017. I evaluated it by rewriting my own feed reading app UI to use React. I also quickly discovered that JSX is a very powerful alternative to the EJS templating system in Express web applications. I prefer it over KnockoutJS and Vue and would use it for all my new web applications.


Sometimes you also need to let go old stuff.

Slackware - I do not use it on my desktop and laptop anymore. I have been using Debian with MATE for 8 months for now and it has proved to be a more stable environment. It is also more compatible with various software as Slackware has made a decision to stay from technologies like systemd and PulseAudio. Besides, all servers that I maintain, are running Debian. Slackware is awesome if you have time and willingness to compile software yourself, and when you have general Do-It-Yourself mentality. The system won't get in your way. I have become too lazy for that.

Mail setup - my mail setup was the most complex system that I was running and I felt that I do not want to maintain it anymore. I switched to FastMail and I can tell for sure that their paid account is worth it. Dealing with a complicated setup of a dozen mail-related daemons is now an history for me.

Supervisord - I have installed all my new apps using the systemd process manager and I really see no point of using Supervisord anymore. It was a nice alternative to init scripts before systemd caught up in usage.

Home server maintenance - at the beginning of 2017 I dismantled my home server. The main reason was to reduce the risk of having a broken server during travelling without anyone at home to repair it. I ran simple web stuff on it and it was quite easy to migrate the stuff into VPS/cloud.

Bash scripting - I had a complex set of scripts for running my backup tasks. It contacted some JSON-based APIs and that was not a very nice thing to handle with shell scripting. I rewrote the scripts as a Node.js application and discovered many issues in my old shell scripts. I do not ever want to write any shell script longer than 5-10 lines. It is just not reliable for anything larger.

Makefiles - most build tasks in developing web apps take a single command and incremental build support is built-in. This is true for tools like Webpack and Sass. This is also platform-independent. There is no need to use an external dependency-based build tool anymore.

Open Source

I did develop and publish less than on the previous years. I also had no significant contribution to any existing Open Source project besides rewriting the UI of my own feeds app. I did work on bridging SWI-Prolog and Node.js for an interesting side project but at the end I found a language bridge too inconvenient for the project and proceeded with JavaScript only and built the important parts as client-side code. However, the work has outcome in these two packages:

Other things

Besides work I also took part of a large construction project to renovate the apartment buildings at the place where I live. The buildings, five at total, containing 102 apartments, were built at end of the Soviet era. These were mostly in a good shape but the heating cost was very high and the buildings lacked any proper ventilation. A chance to help out with the renovation process was the main reason for me to join the management board 2 years ago.

The hardest project ever

It was an extremely hard process from convincing the owners to accept the plan, negotiating with banks for a very big loan, negotiating for a state support, doing enormous amount of paperwork, to bearing the heavy noise once the actual construction work started.

Our job was also made much harder by a small group of owners who wanted to stop the process at any cost. They held meetings to turn other owners against the project and against the board. They tried to unseat the board. The plan and the management board was attacked in two articles published in a nationwide newspaper. One article's online version contained comments attacking board members personally. All members, including me, received threats over the phone. The group phoned our contacts at the bank and at the other partners to show the board as incompetent as possible. As the final step, they made an attempt to manipulate our relatives to influence us to stop the project.

The project was absolutely brutal. I had no previous experience with something of this scale and impact. The budget was 100 times larger than what I usually deal with. The amount of paperwork was staggering, on some weeks it took me couple of full days. This was combined with strict deadlines and those attacks above.

Our management board was exceptionally determined. Our partners and the majority of the owners never had doubt in us and showed lots of support. We were always ready to hear any question from an owner over the phone, by email, or when meeting in-person. We answered and explained all unclear aspects as detailed as necessary. Through conversations we convinced the owners that we are doing the right things and we are doing them in the right way.

The project ended 2 months ago and it was very successful. We achieved all project goals. We did not go over budget. In December we paid 59% less for heating than on the previous year. Together with the loan payback, my monthly bill is about the same as on the previous year. On the other hand, me and the rest of the 102 families enjoy proper heating (and you can set your room temperature!); working ventilation (no more humidity issues); all pipes and radiators are new, they won't fall apart in the near future; all new windows; new power distribution system that is not a fire hazard; and the buildings have a new beautiful outer finish. I'm very proud that the project came together and that I was involved.

In 2018

Considering work, I will likely continue some existing development project or start a new one with an existing client in next the 1-2 months.

My longer term plan is to either move to a higher-populated place and/or accept a full-time position. Berlin would be my first choice. The largest city in 100km distance at my current place is Tartu with its 100k population. For some time I have felt that my current location is a bit limiting opportunity-wise.

A full-time position would likely to mean relocation for me anyway. It's not much different in effort whether I move to the capital city in my country or to a large city in Western Europe. The latter, however, makes much higher salaries available. I have got some very good offers lately but I have been hesitant to accept anything as I believe that accepting a position first and relocating after will set me up to a situation where I gain local connections at the new place and through them a significantly better offer than the one I accepted right before relocation!

A full-time position would also require a hard choice of specialization. I'm not sure whether I should go to the backend route (I have a solid CS background; it has the highest pay but I do not enjoy it), frontend route (I enjoy UI work and I have done a lot of it), product management route (I love working with users directly and create something they need even if it involves no coding), or fullstack route (depends a lot on the company what it really means). In 2018 I might be finally make the hard choice or maybe I just keep doing consulting, which really is the only place where this skillset combination is optimal.

January 15, 2018

Anton ArhipovThe essential tools for software developer

As a software developer I have used a ton of different tools for different purposes. However, it got me thinking, if I really need a lot of tools and what are the essential tools for software developer and what tools are rather “nice to haves”.


First, we better be able to write code. Hence, the editor is absolutely required. For Java, I’ve used different IDEs, but mostly IntelliJ IDEA. NetBeans IDE is nice. Eclipse-based IDEs shine in one way or another. I usually prefer to install JBoss Developer Studio as it provides a solid support for Java EE projects.

For one-off editing, I still use Vim and can’t get rid of it. I’m not even using the full power of the editor. I’ve tried other awesome editors at different times: Notepad++, Crimson, Atom, Sublime Text, VS Code (which I do have installed on my machine). Still, those editors haven’t got much use by me for some unknown reason.

The cool thing about the editors is that there’s so much choice!

Version Control System

In 2004 I started at one financial company as a Java developer. The first task I was assigned to was something likes this: “We have an application running in production and it has a few bugs. We need to fix those, but we don’t have the source code. Please do something?”. A disaster. They didn’t use source control properly and managed to lose it.

Today, the scenario above is very unlikely. However, I still hear stories how people store the source code in folders with suffixes _1, _2, _3 on the network drive. Argh!

Which version control system would you pick? IMO, today Git is the dominant one. However, the way people often use Git (or any other distributed VCS) by avoiding branching and playing with feature flags, they probably would be just fine by using Subversion instead :)

Issue Tracker

An issue tracker is absolutely needed for the team to plan their work, tasks, issues, etc. Now the question is in the implementation. I’ve seen tasks being tracked in MS Excel, text files, sticky notes, and other modern tools like Jira, Mingle, PivotalTracker, YouTrack, ForBugz, Bugzilla, Mantis, Trello… the list is infinite!

The thing with issue trackers is that they should support the approach you take for running the project. For a product company there are quite a few reasons to track issues. First, obviously, you need to track your work, i.e. tasks. From the project management perspective, it’s nice to have the ability to track the tasks, i.e. have the project management facilities in the task tracker. Lots of teams are happy to have it all in one so it’s easier to get an overview of the progress, especially if the tool provides Kanban style boards.

Second, good products usually have users who are eager to submit questions or bug reports. For that, most of the teams are using some kind of helpdesk software. HelpScout was the one I have used at ZeroTurnaround and can recommend. Plus, some teams make use of forum software that also serves as a channel for communication. Also, some teams expose a public issue tracker so that the users would directly submit issues there.

The above means that the issue tracker you choose is better have the required integrations available if that’s a requirement.

Continuous Integration

You write code in the editor, you test the code on your machine, you push it to Git. Someone pulls the code down, tries to run it, and it doesn’t work. Sounds familiar? Yeah, reminds me the famous “Works on my machine” excuse. Automation is absolutely required for a healthy software project. Automated builds, automated tests, automated code style checks, etc.

Previously at ZeroTurnaround, I’ve seen that the CI server was a very critical piece of project infrastructure. If a source control system want offline, it wasn’t an end of the world to them. But if the build server was down, for many developers the work pretty much stalled. Now at JetBrains, I also see that their own product, TeamCity, is a very important service internally, with many teams relying on it. Dogfooding at its best!

Continuous integration was brought to the masses by eXtrame Programming practices. Today, CI is absolutely essential to any healthy software project. Years ago it all started with automating the test execution, building the artifacts, and providing the feedback quickly. Remember CruiseControl? Or Hudson, when it appeared? CI servers have been evolving the in the past years as the notion of Continuous Delivery appeared. Better visualisation of the process is required to cope with growing complexity. Also scaling the CI server has become an important aspect of the process.

With the latest trends in CI, the build servers are eagerly implementing the notion of build pipelines (build chains in TeamCity) which provide a good overview of the process. And also the trend of 2018, I think, is running the CI tasks in a Kubernetes cluster.

Artifact repository

The build server produces the artifacts. Plenty of them. Either the artifact is a final software package, or a reusable component, or a test execution report, etc. Where would you store those artifacts with all the metadata associated with it? Today, there isn’t much choice, actually.

JFrog’s Artifactory is the dominant solution for storing binaries and managing them. Plus, the final artifacts could be promoted directly to Bintray for distribution. Sonatype’s Nexus was originally the go to solution for Java/Maven projects and added support for some other technologies as well in the recent years. Apache Archiva and ProGet are the other options but that’s pretty much it.

The security aspect of software development becoming more and more critical. I think, automated security checks will become an absolute requirement for any software as well. The trend that has been ongoing for years now and binary repositories, such as Artifactory and Nexus, are actually integrating with the services that provide such vulnerability checks. So, don’t be that guy, use the binary repository!


I have listed 4 categories of tools that I think are essential to any software project:

  • an editor,
  • a version control system,
  • CI server,
  • and an artifact repository
I didn’t mention build tools, compilers, static code verifiers, etc, but those are the small tools that will be used by developers anyway. And if not, some of the tools I’ve mentioned can replace or provide such functionality as a bundled feature.

January 06, 2018

Anton ArhipovSetting up JRebel for WebSphere AS in Docker environment

Getting any Java application server up and running in the development environment is usually a fairly simple task. You can just download the zip archive, and start the contain either from command line or via IDE integration. Configuring JRebel agent for the server is also quite straightforward. However, there are some exceptions to that. For instance, if you’d like to try JRebel on WebSphere AS and you are using MacOS, then you will have to take another route.

WebSphere Application Server is available for Linux and Windows platforms, but not for MacOS. The good news is that there is a WebSphere Docker image that you can use for development.

The developerWorks' article demonstrates it pretty clearly, what needs to be done in order to get WebSphere running in Docker and deploy a web application. With the help of the article I have assembled a demo project that deploys a Petclinic application on WebSphere running with JRebel in Docker container.

Let me explain some interesting bits of the outcome.

First of all, we need to derive from the base image, package the application archive into the new image, and make sure that WebSphere will deploy the application when it starts:

FROM ibmcom/websphere-traditional:profile
COPY target/petclinic.war /tmp/petclinic.war
RUN -lang jython -conntype NONE -c "AdminApp.install('/tmp/petclinic.war', \
'[ -appname petclinic -contextroot /petclinic -MapWebModToVH \
[[ petclinic petclinic.war,WEB-INF/web.xml default_host]]]')"

As you’ve noticed, it is not enough just to copy the application archive to some folder. You also need to invoke a script to actually deploy the application: call by providing it a snippet of Jython code.

Next, as we want to enable JRebel, we also need to package the agent binary into the image, and also we need to modify JVM arguments of the application server. Hence, the corresponding Dockerfile will get a little more complicated:

FROM ibmcom/websphere-traditional:profile
COPY ["jrebel-7.1.2","/tmp/jrebel"]
RUN -lang jython -conntype NONE -c "AdminConfig.modify(AdminConfig.list('JavaVirtualMachine', \
AdminConfig.list('Server')), [['genericJvmArguments', \
'-Xshareclasses:none -agentpath:/tmp/jrebel/lib/']])"
COPY target/petclinic.war /tmp/petclinic.war
RUN -lang jython -conntype NONE -c "AdminApp.install('/tmp/petclinic.war', \
'[ -appname petclinic -contextroot /petclinic -MapWebModToVH \
[[ petclinic petclinic.war,WEB-INF/web.xml default_host]]]')"

The Dockerfile above packages JRebel distribution into the image. That’s an easy part. The hard part was to figure out how to configure JVM arguments. In WebSphere, JVM arguments are set via server.xml configuration which is quite unusual. Normally, a developer would use an administrative user interface to modify the parameters, but in our case we need the arguments to be in the right place right at the start. Hence we need to do some Jython scripting via wsadmin again.

Now that the Dockerfile is ready, we can build and run the new image. In the terminal:

$ docker build -t waspet .
$ docker run -d -p 9043:9043 -p 9443:9443 -v `pwd`:/tmp/petclinic -v ~/.jrebel:/home/was/.jrebel waspet

The docker run command above also maps a few directories: a project folder and JRebel’s home folder. We map the project folder because JRebel agent could then see if any resource is updated. JRebel’s home folder (~/.jrebel) includes cached resources, so that if we would have to restart the Docker image then the application will start faster the next time.

Now it is possible to use JRebel to update the application instantly, without restarting the application server or redeploying the application. For the full list of instructions, see the file in GitHub repository.

January 04, 2018

TransferWise Tech BlogHibernate and Domain Model Design

Hibernate and Domain Model Design

When I first learned about Hibernate about 12 years ago (I guess just about when it was at its peak) I found it to be one of the coolest libraries ever created. Hibernate made it possible to use OOD in practice! Build complex domain model graphs that were not only data structures but also contained some actual domain logic. And then you could just persist all of that.

Unfortunately my views on Hibernate/JPA have changed since then. Where I once saw it as an enabler I now see it more as a constraint - a source of accidental complexity. So much so that I would not consider Hibernate (or any other ORM tool) as the first choice for persistence anymore.

I feel that nowadays it is popular to criticize Hibernate and JPA in general. Unfortunately it seems that very often these opinions are based on some superficial experience and emotions. I do believe that emotional aspect and overall popularity is important when evaluating some technology but it is also important to try to understand the underlying rational reasoning.

I do not consider myself a Hibernate expert but I hope my experience is good enough to represent the opinion of someone who has given it a good try. I have been part of a dozen or so projects using JPA/Hibernate. From small 4 dev 4 month ones to big enterprise systems built by many different teams.


As I said the main reason why I immediately liked Hibernate was its support for building complex object graphs. I used to believe that it is the OO way to always use references to other objects instead of holding only IDs of associations or limiting model traversability in general.

This all changed when thanks to Vaughn Vernon I finally understood what was the idea behind Aggregates in DDD. I realized that referencing objects outside of your aggregate by ID actually makes your design much better and helps to avoid this huge tangled mess where everything is somehow linked to everything. Also I understood that if you are relying on lazy-loading for something in your aggregate then you have probably modeled it incorrectly. Aggregate is all about keeping invariants - something that you don't seem to need if you can do lazy-loading.

So in short Hibernate's support for complex object graphs and lazy loading is not something we inevitably need for implementing rich domain models.

Design constraints

Hibernate supports persisting quite complex models. However, there are some cases where it is hard to find any good solution.

It gets hard to use Hibernate without making any sacrifices to the design as soon as we do not want to model our domain classes 1:1 based on our database schema. Once we start thinking how to make our domain entities smaller, introduce interfaces and multiple implementations for value objects, use unidirectional associations where it makes sense it will require a lot of effort to find the right blog post or stackoverflow answer. Yes, it is possible to use things like PrePersist, PostLoad hooks or force Hibernate to use AccessType.PROPERTY but these are all tricks that are only needed for the persistence framework and will not add any other value.

Violation of Single Responsibility Principle

The simplest way how this becomes obvious is when we start writing tests for the ORM managed classes.

Even though Hibernate takes care of all DB interaction we still need to verify that it understands our mapping the way we intended. If we have an entity MyEntity which uses annotations for mapping information we will need to write a test like MyEntityPersistenceSpec. However, we also need to test business logic so we will have something like MyEntityDomainLogicSpec as well. Now we have 2 specs for testing 2 different aspects of one production class. This kind of split in tests is often good sign that we should split the production code as well. One way how to achieve that is by moving all state into MyEntityState object as suggested by Vernon but this is again an example of particular tooling forcing design decisions on us.

Even if we go old-school and externalize mappings into XML we still have the problem that we cannot be fully sure if the way how we modify our entities does not somehow affect persistence. All the collections in MyEntity are not under our full control. They are implementations provided by Hibernate. For example, we must be careful when we create new collections vs when we use clear, addAll.

Learning curve

The reality is that often we as developers use tools that we don't fully understand. As long as we are willing to make compromises in our design - use bidirectional associations, avoid interfaces, keep model relatively simple - all is fine. There is a lot of good documentation available for JPA and Hibernate. However, if we want to build something more advanced it gets much harder to find help. There are simply too many variations for building domain models.

I still believe that Hibernate is an amazing technology. What better proof can there be when something is so widely used for more than a decade. It's just that when I used to believe that Hibernate is the enabler for building good domain models then now I believe it is more suitable when you are OK to keep your domain model relatively close to your DB schema.

I think quite many systems are built using anemic domain model where full power of OOD is not utilized. Depending on the essential complexity of given problem domain this can have little or significant effect on the overall cost of maintenance. So in that sense I don't believe that complex domain model is always a "must have" in which case using Hibernate might still be ok. At least when none of the reasons for not using Hibernate are applicable.

Used image from Ridley Scott's Alien

January 03, 2018

Targo tarkvaraInformaatika õppekava: good, bad and ugly

Infoaja kultuurist ja olustikust 

Detsembri keskel toimus gümnaasiumi informaatika ainekava ümarlaud. Kohal oli igasugust huvitavat rahvast: ministeeriumist, ettevõtetest, kesk- ja ülikoolidest, teemaspetsiifilistest asutustest, nagu HITSA ja Innove. Ka mind oli kutsutud, ilmselt seetõttu, et mul on üldiselt kombeks igale poole nina toppida ja erinevatel teemadel jaurata.

Olen viimastel aastatel töötanud peamiselt idufirmades, kus kõigile probleemidele lähenetakse ülipragmaatiliselt ja peamine eesmärk on kõike muud kõrvale jättes „teeme asja ära“. Haridusvalla seltskonda sattudes pidin esmalt kohanema hoopis teistsuguse emotsionaalse õhkkonnaga.

Näide: ettekandele, mis minu arvates oli huvitav, informatiivne ja täiesti neutraalne, järgnes teise asutuse esindaja reaktsioon „Miks te meie tööd maha teete?“. Sarnaseid suuremaid ja väiksemaid solvumisi ja neile järgnevaid silumisi oli päris mitu. Sain ka ise oma esimese sõnavõtu järel kergelt vastu näppe ja olin hiljem ettevaatlikum.

Seetõttu jäi mind kummitama õppekavaväline küsimus: tegu oli ju põhimõtteliselt üsna otsese ja mõneti isegi kuiva teemaga – miks osalised nii pinges on ja miks siin nii suurt umbusaldust ja kriitikahirmu tuntakse?

Lihtsaim selgitus oleks muidugi, et õpetajatel on liiga vähe raha ja see muudab inimesi närviliseks. Samas näen ma võrreldavaid agressiivses kaitseasendis reaktsioone igal tasemel, professorite, akadeemikute ja ministriteni välja. Mul on seetõttu hüpotees, et meie haridus põhineb fundamentaalselt rohkem laitusel kui kiitusel, seda juba lasteaiast alates (võiks kasvõi statistikat teha, kui palju öeldakse koolipäevas kiitvaid sõnu võrreldes laitvatega). Seetõttu on ka õpilased ning lapsevanemad võitlusvalmis, pelgavad seda, mis koolist tuleb, ning hammustavad teinekord juba ennetavalt vastu. Tagajärjena koguvad haridustöötajad endasse kõvasti allergiat ja kasvatavad okkaid.

Kui ma pärast üritust mainisin, et väga huvitavad teemad, refereeriksin neid kusagil, kuulsin ka reaktsiooni:  „Ära tee, inimesed võivad solvuda.“ Keeruline olukord tõesti, kuid leian, et antud teema on siiski väärt laiemalt kui mõnekümnele inimesele valgustamist. Praegu jäävad mitmed teemad selliste pingete pärast üldse arutamata ja lahendamata.

Eelnevat arvestades ütlen seega juba ette ära, et kui ma kellelegi varbale astun, siis see pole minu kavatsus. Eesmärgiks on ikka see, kuidas asju paremaks teha.

The Good 

Minu jaoks kõige positiivsem oli see, et kohal oli palju inimesi, kes tõesti soovivad elu edendada ja asju paremaks muuta. Eestis on mitmeid asutusi ja ettevõtmisi, mis vankrit sikutavad, kuigi nende struktuur ja võimalused on väga erinevad. Need, kelle jaoks üritusele kohaletulek polnud otsene tööülesanne, ongi muidugi definitsiooni kohaselt entusiastid, nii mõnigi neist korraldab ainealast ringi või kursust. Ma ise korraldan informaatikaolümpiaadi, mille üle olen ka päris uhke, aga kõige kõvema panuse on minu arvates andnud Eno Tõnisson, kelle juhitud internetikursused ( on õpetanud programmeerimist juba tuhandetele ja tuhandetele inimestele.

Eno tegi ka ühe põhiettekannetest, kus oma mass-internetikuruste (MOOCide) kogemusi jagas. Kokku on neil kursustel osaletud üle 15 000 korra, kuigi seal on ilmselt korduvusi, sest paljud on osalenud mitmel kursusel. Arv näitab, et ühiskondlik huvi programmeerimise vastu on kõrge, ületades suuresti kõigis Eesti vastavates erialakoolides õppijate koguarvu.

Eesti Informaatikaõpetuse ajaloost rääkis Urmas Heinaste. Siin saan õnne tänada, et mu enda keskkoolitee jäi selle ajaloo algusjärku, kui keskkoolides õpetatigi konkreetselt programmeerimist. See andis mulle omal ajal päris tugeva tõuke ja nii mõnedki praeguste kõvade IT-firmade juhid on samast perioodist. 1996. aastal eraldi programmeerimise õpe lõppes ning pärast seda on informaatika sildi all õpetatud mitmesuguseid erinevaid asju.

Erinevatest õppemudelitest rääkis Mart Laanpere, kelle ettekandest jäi mulle enim meelde järgmine joonis:

Skeemi mõte on selles, et on kolm peamist jõudu, mis informaatika õppekava erinevates suundades sikutavad. Ülemine neist on ”akadeemiline” suund, mis täidab eelkõige vastava eriala (antud juhul arvutiteadust õpetavate) kõrgkoolide ja ettevõtete tellimust. Parempoolne peaks ette valmistama nö laiatarbe IT-oskusi, selliseid mida nõutakse näiteks arstilt, taksojuhilt või müüjalt. Lõpuks alumine osa täidab eelkõige kooli enda eesmärke, arendades IKT-oskusi lõimitud kujul, näiteks läbi arvuti kasutamise teistes ainetundides.

Ettekandja oli teinud ära huvitava taustatöö, paigutades skeemile selle, kus mitmed riigid praegu asuvad ja kuhu nad liikuda plaanivad. Kui Inglismaa puhul oli määravaks erialaste firmade tellimus ja surve, siis näiteks Soome asus end liigutama kuulujutu tõttu, et Eestis õpivad kõikesimeste klasside lapsed programmeerimist, ega soovinud naabrist maha jääda.

Enamik ürituse diskussioonist keskenduski sellele, milline võiks olla Eesti liikumissuund (A tähistab praegust asukohta).

Headest asjadest edasi rääkides tahan tunnustada Eesti haridusasutusi, eriti Tartu Ülikooli, kus hoolimata spetsiifilise rahastuse puudumisest juba mitu aastat laia avalikkust on programmeerimise juurde toodud. Teiselt poolt lähevad minu tänusõnad paljudele ettevõtetele, kes mitmesuguseid haridusalaseid ettevõtmisi on toetanud, olgu siis tegu õppejõudude hankimise, oma töötajate ülikoolidesse õpetama saatmise või õpilasürituste sponsoreerimisega.

The Bad 

Kuigi üritus oli hästi läbi viidud ja ettekandjad valdasid selgelt teemat, jäid minu hinge kripeldama mitu asjaolu.

Esimene oli see, et vaatlusalune kolme tõmbesuunaga mudel keskendub sellele, mida mingid välised osalised täna soovivad. On selge, et arsti või taksojuhi kutsestandard on suure tõenäosusega koostatud vastava valdkonna juhtide ehk siis eelmise või üle-eelmise põlvkonna esindajate poolt. IKT ettevõtted on pisut paremad, kuid olles viibinud üritustel, kus arutatakse näiteks nende tellimust ülikoolidele, tundub ka seal olevat kahetsusväärselt palju lühiajalist mõtlemist. Kui täna on vaja Java-programmeerijaid, siis seda nõuamegi, kui teatud metoodikat tundvaid süsteemianalüütikuid, siis nõuame neid jne.

Ideaalses maailmas oleks joonisel esindatud ka konkreetselt tulevikku vaatav jõud, kes mõtleks, millistest oskustest on kasu nt 15-20 aasta pärast. Omaaegne keskkoolis programmeerimise õpetamine oli küll nõukogudeaegse kampaania vili ning sai koosviibimisel ka arvestatava sarkasmi osaliseks, kuid see oli siiski oluliselt tulevikku vaatav. Kui toona oleks lähtutud pigem vabrikudirektorite ja kolhoosijuhtide tellimusest, siis kas näiteks 90. aastate alguses keskkoolis käinud Skype’i tegijad oleks samaks ajaks sama kogemuse saanud?

Teiseks olid valikud suuresti kinni mitte sisus, vaid vormis. Kas õpetus peaks olema iseseisva ainena või teistega lõimitult? Kas rühmaprojekti või tunnipõhise kava alusel? Algama kümnendas või üheteistkümnendas klassis?

Nii palju kui sisust juttu oli, jäi mulle tunne, et see on küllaltki eklektiline: samal pulgal 3D-modelleerimine, geoinformaatika, mehhatroonika, programmeerimine, robootika. Programmeerimise kohta öeldi korduvalt, et see on spetsiifiliselt neile, kes seda erialaselt õppima ja hiljem vastavasse ettevõttesse tööle läheks.

Viimases punktis jäin ma eriarvamusele: minu seisukohalt pole programmeerimine sugugi ainult kitsalt erialainimeste pärusmaa, vaid väga praktiline üldoskus ja ka märgatavalt laiema kasutuspinnaga kui teised loetletud oskused.

Kui 90. aastate alguses olid arvutid veel vähelevinud ja programmeerimisoskust raskem rakendada, siis tänapäeval kasutatakse neid igal erialal. Arvuti eripära on aga selles, et ta on masinana universaalne – lisaks tööjuhendis vajalikele ülesannetele saab teda panna tegema ka igasuguseid muid asju. Ning kui sa suudad arvuti enda heaks tööle panna, on see sama võimas kui isiklik assistent, kes osa su tööst endale võtab – sinu efektiivsus ja ühiskonnale toodav lisaväärtus kasvavad hüppeliselt. Siin võivad näideteks olla nii lasteaiaõpetaja, kes kohalolijate märkimiseks lihtsa veebisaidi loob, ametiasutuse asjaajaja, kes skaneeritud dokumentide laialijagamiseks väikese skripti koostab, kui ka põllumees, kes optimaalse väetisekoguse leidmiseks Exceli makro teeb. Need kõik on elust võetud näited ja igaüks neist suurendab mingis pisikeses lõigus ühiskonna toimimise efektiivsust. Tulemusena saavutaksime aga samasuguse sotsiaalse hüppe nagu siis, kui kogu rahvas lugema ja kirjutama õppis.

Selliseid võimalusi panna masinaid tegema ära rutiinseid tegevusi on tuhandeid, kuid enamik neist ei jõua kunagi seda järge ära oodata, et „päris“ programmeerijad suure raha eest vastava töö ära teeks. Analoogiliselt olid kunagi olemas üksikud kirjaoskajad inimesed, kes teiste eest kirju kirjutasid ja neile raamatuid ette lugesid, ilmselgelt jäid siis enamik vajalikest kirjadest kirjutamata ning raamatud lugemata. Praegu kardaks enamik inimesi selliste ülesannete lahendamist isegi üritada, kuid need, kes on programmeerimisega natukegi kokku puutunud, saaks juba kasvõi veebist leitud instruktsioonide varal hakkama. Ka Tartu Ülikooli MOOCide populaarsus kinnitab vastavat vajadust.

Osalejaskond jagunes selles osas kahte lehte, oli nii neid, kes mind toetasid, kui ka neid, kes ütlesid, et koolis programmeerimise õpetamine on sama praktiline kui alpinismi õpetamine – ainult entusiastidele. Mis tegelikult saama hakkab, näitab aeg.

The Ugly 

Räägitud teemadest olulisem oli aga ruumis viibinud elevant, kelle vari kõigile teemadele langes, aga kelle olemasolu sellegipoolest keegi ei maininud. Tõenäoliselt sellepärast, et keegi ei osanud sellega midagi peale hakata, kuid igasugusele plaanile on ta takistuseks ikka.

Tegu on mõistagi informaatikaõpetajate palga teemaga. Kui informaatikaõpetajad saaks riigieelarvest rohkem palka kui teised õpetajad, põhjustaks see ilmselgelt kõvasti nurinat. Kui tal on aga olemas head IT-alased oskused (ja selleks, et lapsi õpetada, võiks need ju olla, lisaks ka pedagoogilised teadmised), aga palka saab vähem kui nooremspetsialist suvalises IT-firmas, on ka narr eeldada, et see eriala teab kui populaarne oleks. Ja tõepoolest, TLÜ sulges hiljuti vastava õppekava nõudluse puudumise tõttu.

Teiseks oleme langenud omaenda „kuvandi loomise“ kampaania ohvriks. Ühelt poolt, nagu eeltoodud Soome näitest näha, teevad naaberriigid ära asjad, millest Eesti puhul vaid pressiteated eksisteerivad. Teiselt poolt on ka kohalikud otsustajad uskuma jäänud, et IT-s ongi kõik juba suurepärane ja vastavasse haridusse pole tarvis rohkem panustada. Drastiline näide: Statistikaameti andmetel olid avaliku sektori kulutused teadus- ja arendustegevusele 2016. aastal väiksemad kui 2008. aastal! (

Silmaga nähtav väljund neile probleemidele ongi, et osalised on vastastikku umbusklikud (olen olnud korduvalt tunnistajaks näiteks olukorrale, kus üldhariduskoolid ja ülikoolid takerduvad koostööd arutades sellesse, kelle kaudu mingi raha peaks liikuma), tirivad õhukesevõitu tekki enda poole ja jagelevad saadaoleva ressursi pärast, kui tegelikult peaks tegelema sellega, kuidas laiem ja paksem tekk hankida, mis kõiki korralikult kataks.

Mis on lahendus?

Õppekava ürituse üks meeldejäänud joon oli veel erinevate osaliste arv – haridusasutused, riigiasutused, sihtasutused, ettevõtted, liidud, seltsid. Igaüks neist korraldab oma üritusi ja projekte, mõnel on raha, teisel on inimesi, mõnel õnnelikul ka mõlemat. Tegevused on aga tavaliselt suhteliselt piiratud, ebastabiilse rahastusega. Ma käin isegi igal sügisel erinevatelt firmadelt olümpiaadi korraldamiseks raha küsimas.

Rääkisin teemast TÜ Arvutiteaduse Instituudi juhi Jaak Viloga, kes kinnitas samuti: „Probleem on Eestis, et kõik käigu justkui õhina-tuhina korras ja projektikestega. Õppekava võimalusi nii ei saa arendada.“

Tõepoolest, entusiastide tegutsemine on väga hea, kuid küsimuse struktuurseks lahendamiseks peaks enim panustama need, kes IT ja programmeerimise õpetamisest ka kõige rohkem kasu saavad. Peamisteks kasusaajateks on ühelt poolt IT-ettevõtted, kelle praegust potentsiaalset käivet ja kasumit kvalifitseeritud tööjõu nappus otseselt piirab, ning teiselt poolt Eesti riik, kellele vastava võimekusega spetsialistid mitmekordse keskmise jagu makse maksavad ja samal ajal riigi üldist infrastruktuuri efektiivsemaks muudavad.

Kahjuks esineb siin aga palju näpuga näitamist: ettevõtjad süüdistavad riiki ja haridusasutusi, et nood ei tee piisavalt õigeid asju, aga riik uriseb vastu, et miks ettevõtted ise ei panusta. Lahendus võiks peituda koostöös.

Esimene koostöötase saaks aset leida eraettevõtete vahel. Selle asemel, et igaüks midagi eraldi teeb, saaks luua Eesti IT-hariduse konsortsiumi, millel on võimalik tuhandete eurode asemel käsutada miljoneid ning sellega midagi tõsisemat korda saata.

Teine koostöö võimalus on ettevõtete ja riigi vahel. Olgu meil vaja palgata ülikoolidesse paar kõva õppejõudu või meelitada hulk spetsialiste koolidesse programmeerimist õpetama – vastastikune hea tahte avaldus ja riske maandav käitumine oleks see, kui näiteks ettevõtete konsortsium panustaks veerandi jagu rahast lõiku, mis neid enim aitaks ja riik kataks vastavatest kuludest ülejäänud kolmveerandi (viimase skeemi soovituse sain samuti Jaak Vilolt).

Veel üks vastus peitub taas pikemaajalises mõtlemises ja tänasest kaugemale vaatamises. Praegu räägitakse palju spetsialistide sissetoomisest. Esiteks ei suuda me sellel turul iial konkureerida selliste industriaalse võimsusega ajuimejatega nagu Silicon Valley või ka London või Berliin, mistõttu Eestisse jõuavad vaid globaalse intellekti teise ešeloni esindajad. Teiseks on see nagu kolmanda maailma riikide majanduskäitumine – selle asemel, et investeerida omaenda tööstusse, ostetakse teiste riikide valmistoodangut.

Nii nagu Saksamaa ehitas 20. sajandi keskel uuesti nullist üles kõva tööstuse ja tagas endale sellega pikaajalise juhtpositsiooni, on ka spetsialistide osas pikas plaanis efektiivsem ostmise asemel „ehitada ajutööstus“ ehk panustada vastavalt haridussüsteemi.

Selleks tuleb aga ettevõtetel mõelda kaugemale järgmise paari aasta kasumiaruandest ning riigivalitsusel kaugemale järgmistest valimistest. Vastasel korral ei jää ei tiigrist ega kuvandist lõpuks midagi järele.

Viimane lahendus on rohkem unistuste vallast, kuid siiski realistlik. Palju on räägitud sellest, kuidas Skandinaavia maade elanikud ei pane pahaks kõrgeid makse maksta, sest nad saavad selle eest vastu häid teenuseid, maksude maksmine on rohkem au- kui häbiasi. Analoogselt võiks Eesti eripäraks olla uhkus tulevikku (eriti haridusse) panustamise üle. Nii riik, omavalitsused kui ettevõtted saaks kiidelda ja võistelda, millise protsendi keegi oma eelarvest/kasumist haridusse ja tulevikuprojektidesse paneb. Maailmas, kus pahatihti elatakse pigem tuleviku arvelt, oleks “IT-riigi” kuvandist veel kõvem sõna “tulevikku panustav riik ja ühiskond”.

December 27, 2017

Raivo LaanemetsMigration to FastMail

I have migrated my mail setup to FastMail. It removed the need to run my own mailservers with all the required infrastructure including webmail. I finished the migration a week before Christmas and should have not missed any mails.

Maintaining my own setup was a too high effort. Before FastMail I also tried G Suite (Gmail) but it does not have mail folders and would have required me to drastically change the way I read my mail.

The migration was done over IMAP interface, provided by the Dovecot-imap package. It took about an hour to transfer all 8000 messages. I'm using the Standard plan which is 50$ per year and which provides 25GB of space for mail.


I added a cron job to my backup server to backup the FastMail account automatically. It uses the offlineimap utility. Here is the article where I got the idea. The original article is about Gmail but it's easy to adopt to FastMail.


I have some servers that send out mail. I have configured them to send messages through my FastMail account SMTP. Here is an article how to set up Postfix to do it.

December 23, 2017

Targo tarkvaraProtected: Informaatika õppekava: good, bad and ugly

This post is password protected. To view it please enter your password below:

December 17, 2017

Kuido tehnokajamSQL SERVER CLR serialization assembly

Kui juhtub, et SQL SERVER-i veebiteenuse poole pöörduv CLR hakkab pilduma viga Cannot load dynamically generated serialization assembly. In some hosting environments assembly load functionality is restricted, consider using pre-generated serializer. Please see inner exception for more information.  System.IO.FileLoadException: LoadFrom(), LoadFile(), Load(byte[]) and LoadModule() have been

December 05, 2017

Four Years RemainingThe Mystery of Early Stopping

Early stopping is a technique that is very often used when training neural networks, as well as with some other iterative machine learning algorithms. The idea is quite intuitive - let us measure the performance of our model on a separate validation dataset during the training iterations. We may then observe that, despite constant score improvements on the training data, the model's performance on the validation dataset would only improve during the first stage of training, reach an optimum at some point and then turn to getting worse with further iterations.

The early stopping principle

The early stopping principle

It thus seems reasonable to stop training at the point when the minimal validation error is achieved. Training the model any further only leads to overfitting. Right? The reasoning sounds solid and, indeed, early stopping is often claimed to improve generalization in practice. Most people seem to take the benefit of the technique for granted. In this post I would like to introduce some skepticism into this view or at least illustrate that things are not necessarily as obvious as they may seem from the diagram with the two lines above.

How does Early Stopping Work?

To get a better feeling of what early stopping actually does, let us examine its application to a very simple "machine learning model" - the estimation of the mean. Namely, suppose we are given a sample of 50 points \mathbf{x}_i from a normal distribution with unit covariance and we need to estimate the mean \mathbf{w} of this distribution.



The maximum likelihood estimate of \mathbf{w} can be found as the point which has the smallest sum of squared distances to all the points in the sample. In other words, "model fitting" boils down to finding the minimum of the following objective function:

    \[f_\mathrm{train}(\mathrm{w}) := \sum_{i=1}^{50} \Vert \mathbf{x}_i - \mathbf{w}\Vert^2\]

As our estimate is based on a finite sample, it, of course, won't necessarily be exactly equal to the true mean of the distribution, which I chose in this particular example to be exactly (0,0):

Sample mean as a minimum of the objective function

Sample mean as a minimum of the objective function

The circles in the illustration above are the contours of the objective function, which, as you might guess, is a paraboloid bowl. The red dot marks its bottom and is thus the solution to our optimization problem, i.e. the estimate of the mean we are looking for. We may find this solution in various ways. For example, a natural closed-form analytical solution is simply the mean of the training set. For our purposes, however, we will be using the gradient descent iterative optimization algorithm. It is also quite straightforward: start with any point (we'll pick (-0.5, 0) for concreteness' sake) and descend in small steps downwards until we reach the bottom of the bowl:

Gradient descent

Gradient descent

Let us now introduce early stopping into the fitting process. We will split our 50 points randomly into two separate sets: 40 points will be used to fit the model and 10 will form the early stopping validation set. Thus, technically, we now have two different objective functions to deal with:

    \[f_\mathrm{fit}(\mathrm{w}) := \sum_{i=1}^{40} \Vert \mathbf{x}_i - \mathbf{w}\Vert^2\]


    \[f_\mathrm{stop}(\mathrm{w}) := \sum_{i=41}^{50} \Vert \mathbf{x}_i - \mathbf{w}\Vert^2.\]

Each of those defines its own "paraboloid bowl", both slightly different from the original one (because those are different subsets of data):

Fitting and early stopping objectives

Fitting and early stopping objectives

As our algorithm descends towards the red point, we will be tracking the value of f_\mathrm{stop} at each step along the way:

Gradient descent with validation

Gradient descent with validation

With a bit of imagination you should see on the image above, how the validation error decreases as the yellow trajectory approaches the purple dot and then starts to increase after some point midway. The spot where the validation error achieves the minimum (and thus the result of the early stopping algorithm) is shown by the green dot on the figure below:

Early stopping

Early stopping

In a sense, the validation function now acts as a kind of a "guardian", preventing the optimization from converging towards the bottom of our main objective. The algorithm is forced to settle on a model, which is neither an optimum of f_\mathrm{fit} nor of f_\mathrm{stop}. Moreover, both f_\mathrm{fit} and f_\mathrm{stop} use less data than f_\mathrm{train}, and are thus inherently a worse representation of the problem altogether.

So, by applying early stopping we effectively reduced our training set size, used an even less reliable dataset to abort training, and settled on a solution which is not an optimum of anything at all. Sounds rather stupid, doesn't it?

Indeed, observe the distribution of the estimates found with (blue) and without (red) early stopping in repeated experiments (each time with a new random dataset):

Solutions found with and without early stopping

Solutions found with and without early stopping

As we see, early stopping greatly increases the variance of the estimate and adds a small bias towards our optimization starting point.

Finally, let us see how the quality of the fit depends on the size of the validation set:

Fit quality vs validation set size

Fit quality vs validation set size

Here the y axis shows the squared distance of the estimated point to the true value (0,0), smaller is better (the dashed line is the expected distance of a randomly picked point from the data).  The x axis shows all possible sizes of the validation set. We see that using no early stopping at all (x=0) results in the best expected fit. If we do decide to use early stopping, then for best results we should split the data approximately equally into training and validation sets. Interestingly, there do not seem to be much difference in whether we pick 30%, 50% or 70% of data for the validation set - the validation set seems to play just as much role in the final estimate as the training data.

Early Stopping with Non-convex Objectives

The experiment above seems to demonstrate that early stopping should be almost certainly useless (if not harmful) for fitting simple convex models. However, it is never used with such models in practice. Instead, it is most often applied to the training of multilayer neural networks. Could it be the case that the method somehow becomes useful when the objective is highly non-convex? Let us run a small experiment, measuring the benefits of early stopping for fitting a convolutional neural-network on the MNIST dataset. For simplicity, I took the standard example from the Keras codebase, and modified it slightly. Here is the result we get when training the the most basic model:

MNIST - Basic

MNIST - Basic

The y axis depicts log-loss on the 10k MNIST test set, the x axis shows the proportion of the 60k MNIST training set set aside for early stopping. Ignoring small random measurement noise, we may observe that using early stopping with about 10% of the training data does seem to convey a benefit. Thus, contrary to our previous primitive example, when the objective is complex, early stopping does work as a regularization method. Why and how does it work here? Here's one intuition I find believable (there are alternative possible explanations and measurements, none of which I find too convincing or clear, though): stopping the training early prevents the algorithm from walking too far away from the initial parameter values. This limits the overall space of models and is vaguely analogous to suppressing the norm of the parameter vector. In other words, early stopping resembles an ad-hoc version of \ell_p regularization.

Indeed, observe how the use of early stopping affects the results of fitting the same model with a small \ell_2-penalty added to the objective:



All of the benefits of early stopping are gone now, and the baseline (non-early-stopped, \ell_2-regularized) model is actually better overall than it was before. Let us now try an even more heavily regularized model by adding dropout (instead of the \ell_2 penalty), as is customary for deep neural networks. We can observe an even cleaner result:

MNIST - Dropout

MNIST - Dropout

Early stopping is again not useful at all, and the overall model is better than all of our previous attempts.

Conclusion: Do We Need Early Stopping?

Given the reasoning and the anecdotal experimental evidence above, I personally tend to think that beliefs in the usefulness of early stopping (in the context of neural network training) may be well overrated. Even if it may improve generalization for some nonlinear models, you would most probably achieve the same effect more reliably using other regularization techniques, such as dropout or a simple \ell_2 penalty.

Note, though, that there is a difference between early stopping in the context of neural networks and, say, boosting models. In the latter case early stopping is actually more explicitly limiting the complexity of the final model and, I suspect, might have a much more meaningful effect. At least we can't directly carry over the experimental examples and results in this blog post to that case.

Also note, that no matter whether early stopping helps or harms the generalization of the trained model, it is still a useful heuristic as to when to stop a lengthy training process automatically if we simply need results that are good enough.


October 13, 2017

TransferWise Tech BlogA Product Manager’s guide to planning for the team

One of the best things about working at TransferWise, is how seriously we take the autonomy of our teams. Each team can (and needs to) set its own Key Performance Indicators (KPIs), come up with a mission statement, and a vision for the team. Once those are worked out, a team then writes out its own plans — there’s no influence from the Leadership team. This process helps our teams do their best work.

As a new product manager to TransferWise, planning can seem daunting. You’ll have little input from the Leadership team or your manager on what to plan or how to do it. I wanted to share my experience planning in the Transfer Experience team, so you can you can confidently plan for your next quarter.

Why is planning so important?

We worked out that we spend about three weeks every quarter planning in the Transfer Experience team. Which feels like a lot. Why spend all that time planning, when we could be doing something else? Mostly because planning is about the process rather than the outcome. So while it’s great having high-quality plans, it’s how you arrive at those plans that matters.

Here’s what you need to get out of your team’s planning sessions:

  1. Get the team to commit. This one’s important. If your plan’s got a chance of succeeding, then it’s got to come organically from everyone on the team. Planning shouldn't just be engineers breaking tasks down into good old-fashioned man hours. It should also have them challenging the products you’re building, and the priorities of those products. Ultimately, everyone inside the team should own the plans for next quarter.

  2. Use it as a chance to gauge the team’s health. While part of planning is thinking about the future, it’s also a great opportunity to think about how things have been working so far. Have the team run retrospectives, check the team’s progress against your previous plans and try to get a feel for everyone’s happiness levels.

  3. Prioritize, prioritize, prioritize. As a product manager, it’s your job to create great products that solve customer problems. And prioritization is one of the ways that a product manager can help make that happen. Planning, by virtue of bringing together your team, is one of the best venues for prioritization.

How we organize planning

We tend to divide our team planning into four parts. They roughly fall into these categories:
Retrospectives. We usually run two kinds of retros: work-based and feelings-based ones. Both are equally important.

  • Work-based retrospectives try to answer the question “How did we do last quarter compared to our plans?” These start by tracking the team’s progress against the plans made last quarter, so having a higher-quality plan from last quarter will help you do better work-based retrospectives. This session usually leads to questions like:
    • How can we become more efficient as a team?
    • What did we spend most of our time on last quarter?
    • How can we do things faster?
    • Do we have too little/too much tech debt?
  • Feelings-based retrospectives aim to answer the question: “How happy are you in the team?” We like to get a feel for everyone’s happiness levels, because we care about creating a positive work environment. Because when people are happy where they are, they can give their all in their work. These kind of retros are hard to run — it’s not easy getting people to open up and share everything they’re thinking — but also the most rewarding. Usually the conversations revolve around these kinds of questions:
    • How well do we get along with each other?
    • Are we holding each other accountable?
    • Are people happy about their work-life balance?
    • How are we interacting with other teams in the company?
    • Do we get along with each other?

Mission & vision for the team. This one’s both fun and incredibly important. Every quarter, we review how we did against our KPIs, and challenge our mission and vision to see if they still fit us. We talk through how the products and features we worked on helped us get closer to achieving our mission and helping our customers. The team should come out of these sessions feeling 100% committed to their renewed mission and vision.

Internal organization & processes. This is normally a short discussion where we work out if we need to change how we do things as a team. Because we’re a distributed team of 15+ people, we usually only come together once a quarter. So when we do, it’s a great opportunity to talk through changing sensitive processes or the structure of the team.

Planning. Yes! Finally, we get down to actually planning. During these sessions, we mostly do the following (in this order):

  1. Review the products that we want to build, and prioritize each product. This should be a product-led session.
  2. Break down each product into tasks, and come up with estimates for how long it would take to build each product. This should be an engineering-led session.
  3. Review and reprioritise according to how long each product is estimated to take. We also try to bring up constrains such as cross-team dependencies, deadlines, upcoming vacations during this session. This session should be a product oriented session.

What improving planning looks like

The Transfer Experience team has been getting stronger and stronger at planning every quarter. Here are a couple ways we have seen we have been getting stronger at planning. If you see your team has improved in some of these aspects, it might be also because you have been improving the way you plan.

We spend less time wondering what to build and more time building. This is a great side effect of having great plans. you’ll spend less time discussing why/what and more time discussing how, because your team members should already be committed to the why. This time saving can be put into improving the quality of the execution, thus improving the outcome for your customers.

Better cross-team collaboration. Writing detailed and realistic plans has helped us with cross team collaboration. It makes it easier for people outside the team understand why and what we want to spend our time on. It also makes it easier for us to call cross-team dependencies, and inform them on the progress we are doing on several projects.

Better retrospectives. Since ideally everyone is committed to the plans, if plans don’t become reality, the team should feel responsible for that in the retrospective. Which means, team members should be pushing each other to understand how they can improve both execution and the plans.

Ultimately, we have been able to improve a lot as a team thanks to better planning, yet we are always looking for ways to improve how we do things here at TransferWise. What is your experience with planning? Leave your thoughts in the comment section below!

September 16, 2017

Raivo LaanemetsNow, 2017-09

This is an update on things related to this blog and my work.


Since the last update I have been returning to a more stable schedule of writing. I now usually write 1 or 2 articles per week and do most of the writing on weekends. I'm soon hitting the milestone of 200 articles and will review the blog in the similar way as I did for the previous milestone.


I have moved back to my home office. Construction work has practically ended and the noise is not an issue anymore. During the couple of noisy days I have been sitting in a local pub with my laptop and drinking coffee.

The last 2 months have gone into a single project. It is a system to replace lots of manual work of managing transportation orders. The client has been pleasure to work with. Their office has been very welcoming and they have really good coffee. Most of my work has gone into the user interface. The backend is built using Node.js and MySQL and the frontend is using server-side JSX to generate HTML and browser-side React for dynamic forms.

Open Source and side projects

Since the last update I have built one side project. It's an app to ask questions about a software project. I have had the idea to build it for a long time. The basic system for questions was inspired by Expert Systems and the Bethesda dialogue system. I built the initial prototype in Prolog but later rewrote it as a browser-side application. It uses a tree-like data structure (actually a directed acyclic graph) where nodes are questions or topics and branches are activated by positive answers.

I rewrote the UI of my feed reader app in React. I wanted to test out React and apply some changes to the reader app. The original UI was written in Knockout.js and a bit hackish. I wrote the original UI many years ago when I did not have much experience in structuring large browser-side applications. At the moment I'm very satisfied with the rewritten code.

September 14, 2017

TransferWise Tech BlogOn the Logging in Mobile Applications

Here I describe an approach from iOS point of view. An approach for Android platform is no different. I can recommend to use Timber framework as an equivalent for CocoaLumberjack.

Logs? Phew! Just write something like:

print("Expected `foo` to be 'bar', got: \(foo)")  

Later, when you debug an issue, the message in the console could give you a hint for the solution. Simple enough. But it’s good only for a “Hello World” application.


1st of all, logs should be available when you need them desperately. Having a large number of users is a guarantee that every possible crash will happen including those under really strange circumstances. Unable to reproduce the issue, with only a stack trace in your hands you would clearly benefit from an additional source of information — application logs. Is it possible to attach logs to a crash report? Yes. For example, Crashlytics Custom Logs can be used for that.

Let’s assume that in the 3rd party SDK, which you are using to collect crash reports there is a func cloudLog(_ msg: String) method (like CLSLogv method in Crashlytics). Say this method follows the Unix philosophy and the only thing it does is sending a provided message to the 3rd party no matter what the current build configuration is. But we would want to use this handy method only in production builds in order to avoid the noise from the 3rd party when debugging.

So it makes sense to introduce your own method like this:

func myLog(_ msg: String) {  

Hmm, didn’t we just start writing our own logging framework?

Our new myLog(_:) method works for most cases. But what if the application you are working on handles sensitive information? Banking, dating or health apps are good examples. If you are logging payloads of your network requests and responses (that would be handy, so probably you are) via previously introduced myLog method you might be sending sensitive information to the 3rd party. For some organisations that’s a huge no-no.

So what can we do about that? Something like this should work:

func myLog(_ msg: String, isSensitive: Bool = true) {  
    if !isSensitive {

Hmm, myLog keeps growing…

2ndly, sometimes a desire to log is in itself an indication of something unexpected happening and usually an additional action should be taken in such a case.

To be more specific imagine your application is talking to your beloved server and suddenly a terrible thing happens — incompatible JSON is served. Shouldn’t we do the following?

myLog("Error, look how terrible that JSON is: \(json)")`  

Not bad. But the problem won’t be noticed unless your application crashes right after. A crash is better than nothing in such a case, but the user might benefit from an offline data or remaining parts of the working functionality. A much better option would be to silently report the problem, receive the report, fix the API, meanwhile showing “Something went wrong. Sorry! We are already working on it.” message to the user. Is there a ready solution to report such a problem? Yes. For example, Crashlytics Caught Exceptions can be used for that. I will reference those as non-fatals — issues that are not crashing the application but should be reported.

Let’s assume that there is a func reportNonFatalError(_ error: Error) method to report problems silently (like recordError in Crashlytics).

We don’t want to report the issue in debug builds (to avoid the noise), but we still want to log to the console a message describing the problem. As for reporting, reusing this log message looks like a good idea. So the straightforward way to do that is like this:

let message = "Error, look how terrible that JSON is: \(json)"  
reportNonFatalError(makeError(withMessage: message))  

The above code looks too cumbersome. Much better is to rewrite our myLog method like this:

enum LogBehavior {  
    case writeToLog(isSensitive: Bool)
    case reportAsNonFatal
struct WrappedStringError: Error {  
    var value: String
    var localizedDescription: String {
        return value
func myLog(_ msg: String, behavior: LogBehavior = .writeToLog(isSensitive: true)) {  
    switch behavior {
    case let .writeToLog(isSensitive):
        if !isSensitive {
    case .reportAsNonFatal:
        reportNonFatalError(WrappedStringError(value: msg))

Now myLog‘s signature seems too complicated and its implementation is like a trash bin full of 3rd party specific logic. Is there a better way? Yes!

We better use some logging framework. Something like CocoaLumberjack. The idea behind it is really simple, but powerful. On the launch of the application you plug in some loggers (you can create your own or select from the predefined ones). When you log the message, it is broadcasted to every logger you plugged in. It’s up to a particular logger how to process the message. Given the lack of restriction on the number of loggers you can plug in you can untangle logging spaghetti using single responsibility principle.

But let’s give some examples.

The most simple but useful logger is a console logger which just prints a message to the console.

You might want to use file logger which persists logs on disk and rotates them. File logger is quite useful for feedback functionality — ask the user if he want to attach logs when composing email to customer support. A problem description with recent log is much more helpful than just the description. Of course you should not store any sensitive data in such logs.

Passing log messages to cloud log or reporting non-fatals is a good enough job for a separate logger as well.

Okay. Seems pretty straightforward. But how can we pass the information to a logger if the message is sensitive or if it should be reported as a non-fatal?

To answer that question let’s look at how we actually log. We do it the same way as we did before when using standard print, but now we have to flag the message as error, warning, info, debug or verbose one. Too many of them, isn’t it a complication? A bit it is, but a handy one. Of course those types impose specific semantics on messages and could be used to filter or color them appropriately in the console. But they can be also used to make a decision on what particular logger should do with a particular message. For example, the non-fatals logger should ignore all the messages except for of error type.

No more words, just look at the following logging policy scheme:

Build Console Non-fatal Cloud Log When To Use
Error Debug

Something bad has happened that should never happen. App is not crashing but a part of the functionality is not working. The situation should be reported.

E.g., a wrong JSON format, parsing errors when a source format is unsupported, some system service is not available but we expect it to be.

Warning Debug

Something bad has happened that should never happen. App is not crashing but a part of the functionality is not working. The situation should be reported.

E.g., a wrong JSON format, parsing errors when a source format is unsupported, some system service is not available but we expect it to be.

Info Debug

Something expected happened. Messages of that level are useful to understand the reason for a crash or an unclear bug.

E.g., a view controller is presented or dismissed, user interactions, some service finished an initialization, a network request succeeded, a DB is saved successfully.

Debug Debug Messages of that level are useful to log sensitive information such as a network response content if it’s really needed. Such log messages should not litter the console.
Verbose Debug Messages that are littering the console, should be manually turned on.

Important details:

  • log sensitive information only on debug and lower levels (that’s only verbose);

  • report only error messages as non-fatals;

That’s roughly the scheme we are using in TransferWise for our iOS application. Of course some amount of work is needed to setup such an approach for an existing application, but it will make the maintenance simpler. For applications that are suffocating from the tech debt it’s one of the first steps to do. But even if your application is pretty healthy, efforts made to keep a finger on the pulse are a thousand times worth it!

DDLogInfo(“Happy logging!")

September 10, 2017

Raivo LaanemetsWho is a Software Consultant

It does matter what you call yourself. It sends out a signal of what you do and how you can be useful. By the offers I have been getting recently I have come to a conclusion that my current title (software developer or full-stack freelancer) has not been descriptive enough. The new offers have not matched up with what I have been doing for the existing clients.

What and how I do

  • Solving problems by developing custom software.
  • Develop in collaboration with the client.
  • Providing services directly to the client.
  • Working with multiple clients at once.
  • Choosing my own clients and projects.
  • Managing the development process.
  • Working remotely or on-site.
  • Develop using popular platforms and libraries (mostly JavaScript).
  • It is a full-time job and not a side business.

I believe this is the best described as a "Software Consultant". Lots of other titles are approriate in some aspect but are not more descriptive as they are too generic, too specific, or have negative connotations in some communities.

More information:

September 05, 2017

Raivo LaanemetsCryptopocalypse in Estonia

A day ago (Tuesday, 6th September) Estonian Information System Authority was informed about a security vulnerability in the electronic chip of Estonian ID cards.

Use of electronic signatures is very common in Estonia. This includes all sorts of legal contracts. Estonia uses the same electronic cryptographic system for online electronic voting. An electronic signature from an ID card is fully binding. It has the same power as a manually written signature.

The vulnerability affects 750000 ID cards issued in the last 3 years. Estonian population is 1.4 million. Estonian ID card is mandatory for every citizen.

Henrik Roonemaa, a well-known Estonian journalist has put together an article from various sources of information. It is published in Estonian. The important points are:

  • The cost of breaking one card for signing is 40k EUR.
  • At least one card has been broken for demonstration purposes.

This shows the severity of the situation. The vulnerability is believed due to the process of generating private and public keys inside the card chip. The weakness allows to deduce the private key from the public key. The list of all public keys was publicity available until Tuesday when it was taken down. Full details of the vulnerability have not yet published.

You get the full power to forge a perfect signature once you gain access to the private key. The electronic signature would be extremely hard to disprove as such attack was deemed practically impossible by experts.

Estonian government says that the issue won't affect the Estonian state as a whole. They say that it would only happen when someone put 60 billion EUR into work to break most of the 750000 signatures. Estonia is worth way more than 60 billion EUR. The government has not yet addressed the issues of individual citizens getting targeted by attacks using this vulnerability.

New cards will have to be issued to fix the vulnerability.

Update 2017-09-10

It has not been confirmed that a card has been actually broken. The issue might then be a lot less severe than thought. The text has been removed from the article.

Update 2017-09-13

It takes about 10-12 hours to crack a card according to the former Estonian president Toomas Hendrik Ilves. Ilves says that this is based on facts. The information comes from an article published in the daily newspaper Postimees. The article also states that the vulnerability is between the chip hardware and the software directly communicating with the chip.

Update 2017-12-27

The current solution is to replace certificates on all vulnerable cards. I have done this to mine as it was vulnerable.

September 04, 2017

Raivo LaanemetsA solution to dreaded "Resolving Host" in Chrome

Sometimes Chrome hangs for a couple of seconds with the "Resolving Host" message on the status bar. The problem is very annoying as it delays page loading for multiple seconds. It happens randomly, not on every site, not on every page open.

I started to get this behavior in the last couple of months. Googling reveals that this could be solved by changing various settings inside Chrome. None of that worked for me. One of the most popular solutions is to use Google DNS servers. I thought it would have no effect on me because I was already using them.

For some time I was trying to debug my hardware. I was suspecting my router and cables and network card. I checked various machine logs every time I had the issue occur in the browser. I found nothing.

Finally I decided to apply strace to the DNS lookup. Strace is a debugging utility for Linux. I thought I might not be a first one to do this and found a blog post by Jeff Blaine. I applied his command on my machine. The command is:

strace -f getent hosts

The command spewed out a verbose log but otherwise completed fast. Then I ran it again and again until it started to hang. It can be seen that it connects to a Google DNS server and awaits a response from it. By the log it looks like sometimes the server won't respond. The third argument for the hanging call in the log, 5000, does seem to be a timeout in milliseconds. After the timeout expires, it tries to make a new request.

connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("")}, 16) = 0
poll([{fd=3, events=POLLOUT}], 1, 0)    = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "\223\254\1\0\0\1\0\0\0\0\0\0\3www\npuppetlabs
  \3com\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=3, events=POLLIN}], 1, 5000 <- HANGING HERE

The solution was to simply replace Google DNS servers with OpenDNS servers. They have no such behavior and the issue has now completely disappeared. You need consult your OS or distro manual or read online guides how to change DNS server settings for your computer. I currently have no idea what causes the problems with Google DNS.