Pinu planeet

January 15, 2018

Anton ArhipovThe essential tools for software developer

As a software developer I have used a ton of different tools for different purposes. However, it got me thinking, if I really need a lot of tools and what are the essential tools for software developer and what tools are rather “nice to haves”.


First, we better be able to write code. Hence, the editor is absolutely required. For Java, I’ve used different IDEs, but mostly IntelliJ IDEA. NetBeans IDE is nice. Eclipse-based IDEs shine in one way or another. I usually prefer to install JBoss Developer Studio as it provides a solid support for Java EE projects.

For one-off editing, I still use Vim and can’t get rid of it. I’m not even using the full power of the editor. I’ve tried other awesome editors at different times: Notepad++, Crimson, Atom, Sublime Text, VS Code (which I do have installed on my machine). Still, those editors haven’t got much use by me for some unknown reason.

The cool thing about the editors is that there’s so much choice!

Version Control System

In 2004 I started at one financial company as a Java developer. The first task I was assigned to was something likes this: “We have an application running in production and it has a few bugs. We need to fix those, but we don’t have the source code. Please do something?”. A disaster. They didn’t use source control properly and managed to lose it.

Today, the scenario above is very unlikely. However, I still hear stories how people store the source code in folders with suffixes _1, _2, _3 on the network drive. Argh!

Which version control system would you pick? IMO, today Git is the dominant one. However, the way people often use Git (or any other distributed VCS) by avoiding branching and playing with feature flags, they probably would be just fine by using Subversion instead :)

Issue Tracker

An issue tracker is absolutely needed for the team to plan their work, tasks, issues, etc. Now the question is in the implementation. I’ve seen tasks being tracked in MS Excel, text files, sticky notes, and other modern tools like Jira, Mingle, PivotalTracker, YouTrack, ForBugz, Bugzilla, Mantis, Trello… the list is infinite!

The thing with issue trackers is that they should support the approach you take for running the project. For a product company there are quite a few reasons to track issues. First, obviously, you need to track your work, i.e. tasks. From the project management perspective, it’s nice to have the ability to track the tasks, i.e. have the project management facilities in the task tracker. Lots of teams are happy to have it all in one so it’s easier to get an overview of the progress, especially if the tool provides Kanban style boards.

Second, good products usually have users who are eager to submit questions or bug reports. For that, most of the teams are using some kind of helpdesk software. HelpScout was the one I have used at ZeroTurnaround and can recommend. Plus, some teams make use of forum software that also serves as a channel for communication. Also, some teams expose a public issue tracker so that the users would directly submit issues there.

The above means that the issue tracker you choose is better have the required integrations available if that’s a requirement.

Continuous Integration

You write code in the editor, you test the code on your machine, you push it to Git. Someone pulls the code down, tries to run it, and it doesn’t work. Sounds familiar? Yeah, reminds me the famous “Works on my machine” excuse. Automation is absolutely required for a healthy software project. Automated builds, automated tests, automated code style checks, etc.

Previously at ZeroTurnaround, I’ve seen that the CI server was a very critical piece of project infrastructure. If a source control system want offline, it wasn’t an end of the world to them. But if the build server was down, for many developers the work pretty much stalled. Now at JetBrains, I also see that their own product, TeamCity, is a very important service internally, with many teams relying on it. Dogfooding at its best!

Continuous integration was brought to the masses by eXtrame Programming practices. Today, CI is absolutely essential to any healthy software project. Years ago it all started with automating the test execution, building the artifacts, and providing the feedback quickly. Remember CruiseControl? Or Hudson, when it appeared? CI servers have been evolving the in the past years as the notion of Continuous Delivery appeared. Better visualisation of the process is required to cope with growing complexity. Also scaling the CI server has become an important aspect of the process.

With the latest trends in CI, the build servers are eagerly implementing the notion of build pipelines (build chains in TeamCity) which provide a good overview of the process. And also the trend of 2018, I think, is running the CI tasks in a Kubernetes cluster.

Artifact repository

The build server produces the artifacts. Plenty of them. Either the artifact is a final software package, or a reusable component, or a test execution report, etc. Where would you store those artifacts with all the metadata associated with it? Today, there isn’t much choice, actually.

JFrog’s Artifactory is the dominant solution for storing binaries and managing them. Plus, the final artifacts could be promoted directly to Bintray for distribution. Sonatype’s Nexus was originally the go to solution for Java/Maven projects and added support for some other technologies as well in the recent years. Apache Archiva and ProGet are the other options but that’s pretty much it.

The security aspect of software development becoming more and more critical. I think, automated security checks will become an absolute requirement for any software as well. The trend that has been ongoing for years now and binary repositories, such as Artifactory and Nexus, are actually integrating with the services that provide such vulnerability checks. So, don’t be that guy, use the binary repository!


I have listed 4 categories of tools that I think are essential to any software project:

  • an editor,
  • a version control system,
  • CI server,
  • and an artifact repository
I didn’t mention build tools, compilers, static code verifiers, etc, but those are the small tools that will be used by developers anyway. And if not, some of the tools I’ve mentioned can replace or provide such functionality as a bundled feature.

January 06, 2018

Anton ArhipovSetting up JRebel for WebSphere AS in Docker environment

Getting any Java application server up and running in the development environment is usually a fairly simple task. You can just download the zip archive, and start the contain either from command line or via IDE integration. Configuring JRebel agent for the server is also quite straightforward. However, there are some exceptions to that. For instance, if you’d like to try JRebel on WebSphere AS and you are using MacOS, then you will have to take another route.

WebSphere Application Server is available for Linux and Windows platforms, but not for MacOS. The good news is that there is a WebSphere Docker image that you can use for development.

The developerWorks' article demonstrates it pretty clearly, what needs to be done in order to get WebSphere running in Docker and deploy a web application. With the help of the article I have assembled a demo project that deploys a Petclinic application on WebSphere running with JRebel in Docker container.

Let me explain some interesting bits of the outcome.

First of all, we need to derive from the base image, package the application archive into the new image, and make sure that WebSphere will deploy the application when it starts:

FROM ibmcom/websphere-traditional:profile
COPY target/petclinic.war /tmp/petclinic.war
RUN -lang jython -conntype NONE -c "AdminApp.install('/tmp/petclinic.war', \
'[ -appname petclinic -contextroot /petclinic -MapWebModToVH \
[[ petclinic petclinic.war,WEB-INF/web.xml default_host]]]')"

As you’ve noticed, it is not enough just to copy the application archive to some folder. You also need to invoke a script to actually deploy the application: call by providing it a snippet of Jython code.

Next, as we want to enable JRebel, we also need to package the agent binary into the image, and also we need to modify JVM arguments of the application server. Hence, the corresponding Dockerfile will get a little more complicated:

FROM ibmcom/websphere-traditional:profile
COPY ["jrebel-7.1.2","/tmp/jrebel"]
RUN -lang jython -conntype NONE -c "AdminConfig.modify(AdminConfig.list('JavaVirtualMachine', \
AdminConfig.list('Server')), [['genericJvmArguments', \
'-Xshareclasses:none -agentpath:/tmp/jrebel/lib/']])"
COPY target/petclinic.war /tmp/petclinic.war
RUN -lang jython -conntype NONE -c "AdminApp.install('/tmp/petclinic.war', \
'[ -appname petclinic -contextroot /petclinic -MapWebModToVH \
[[ petclinic petclinic.war,WEB-INF/web.xml default_host]]]')"

The Dockerfile above packages JRebel distribution into the image. That’s an easy part. The hard part was to figure out how to configure JVM arguments. In WebSphere, JVM arguments are set via server.xml configuration which is quite unusual. Normally, a developer would use an administrative user interface to modify the parameters, but in our case we need the arguments to be in the right place right at the start. Hence we need to do some Jython scripting via wsadmin again.

Now that the Dockerfile is ready, we can build and run the new image. In the terminal:

$ docker build -t waspet .
$ docker run -d -p 9043:9043 -p 9443:9443 -v `pwd`:/tmp/petclinic -v ~/.jrebel:/home/was/.jrebel waspet

The docker run command above also maps a few directories: a project folder and JRebel’s home folder. We map the project folder because JRebel agent could then see if any resource is updated. JRebel’s home folder (~/.jrebel) includes cached resources, so that if we would have to restart the Docker image then the application will start faster the next time.

Now it is possible to use JRebel to update the application instantly, without restarting the application server or redeploying the application. For the full list of instructions, see the file in GitHub repository.

January 04, 2018

TransferWise Tech BlogHibernate and Domain Model Design

Hibernate and Domain Model Design

When I first learned about Hibernate about 12 years ago (I guess just about when it was at its peak) I found it to be one of the coolest libraries ever created. Hibernate made it possible to use OOD in practice! Build complex domain model graphs that were not only data structures but also contained some actual domain logic. And then you could just persist all of that.

Unfortunately my views on Hibernate/JPA have changed since then. Where I once saw it as an enabler I now see it more as a constraint - a source of accidental complexity. So much so that I would not consider Hibernate (or any other ORM tool) as the first choice for persistence anymore.

I feel that nowadays it is popular to criticize Hibernate and JPA in general. Unfortunately it seems that very often these opinions are based on some superficial experience and emotions. I do believe that emotional aspect and overall popularity is important when evaluating some technology but it is also important to try to understand the underlying rational reasoning.

I do not consider myself a Hibernate expert but I hope my experience is good enough to represent the opinion of someone who has given it a good try. I have been part of a dozen or so projects using JPA/Hibernate. From small 4 dev 4 month ones to big enterprise systems built by many different teams.


As I said the main reason why I immediately liked Hibernate was its support for building complex object graphs. I used to believe that it is the OO way to always use references to other objects instead of holding only IDs of associations or limiting model traversability in general.

This all changed when thanks to Vaughn Vernon I finally understood what was the idea behind Aggregates in DDD. I realized that referencing objects outside of your aggregate by ID actually makes your design much better and helps to avoid this huge tangled mess where everything is somehow linked to everything. Also I understood that if you are relying on lazy-loading for something in your aggregate then you have probably modeled it incorrectly. Aggregate is all about keeping invariants - something that you don't seem to need if you can do lazy-loading.

So in short Hibernate's support for complex object graphs and lazy loading is not something we inevitably need for implementing rich domain models.

Design constraints

Hibernate supports persisting quite complex models. However, there are some cases where it is hard to find any good solution.

It gets hard to use Hibernate without making any sacrifices to the design as soon as we do not want to model our domain classes 1:1 based on our database schema. Once we start thinking how to make our domain entities smaller, introduce interfaces and multiple implementations for value objects, use unidirectional associations where it makes sense it will require a lot of effort to find the right blog post or stackoverflow answer. Yes, it is possible to use things like PrePersist, PostLoad hooks or force Hibernate to use AccessType.PROPERTY but these are all tricks that are only needed for the persistence framework and will not add any other value.

Violation of Single Responsibility Principle

The simplest way how this becomes obvious is when we start writing tests for the ORM managed classes.

Even though Hibernate takes care of all DB interaction we still need to verify that it understands our mapping the way we intended. If we have an entity MyEntity which uses annotations for mapping information we will need to write a test like MyEntityPersistenceSpec. However, we also need to test business logic so we will have something like MyEntityDomainLogicSpec as well. Now we have 2 specs for testing 2 different aspects of one production class. This kind of split in tests is often good sign that we should split the production code as well. One way how to achieve that is by moving all state into MyEntityState object as suggested by Vernon but this is again an example of particular tooling forcing design decisions on us.

Even if we go old-school and externalize mappings into XML we still have the problem that we cannot be fully sure if the way how we modify our entities does not somehow affect persistence. All the collections in MyEntity are not under our full control. They are implementations provided by Hibernate. For example, we must be careful when we create new collections vs when we use clear, addAll.

Learning curve

The reality is that often we as developers use tools that we don't fully understand. As long as we are willing to make compromises in our design - use bidirectional associations, avoid interfaces, keep model relatively simple - all is fine. There is a lot of good documentation available for JPA and Hibernate. However, if we want to build something more advanced it gets much harder to find help. There are simply too many variations for building domain models.

I still believe that Hibernate is an amazing technology. What better proof can there be when something is so widely used for more than a decade. It's just that when I used to believe that Hibernate is the enabler for building good domain models then now I believe it is more suitable when you are OK to keep your domain model relatively close to your DB schema.

I think quite many systems are built using anemic domain model where full power of OOD is not utilized. Depending on the essential complexity of given problem domain this can have little or significant effect on the overall cost of maintenance. So in that sense I don't believe that complex domain model is always a "must have" in which case using Hibernate might still be ok. At least when none of the reasons for not using Hibernate are applicable.

Used image from Ridley Scott's Alien

January 03, 2018

Targo tarkvaraInformaatika õppekava: good, bad and ugly

Infoaja kultuurist ja olustikust 

Detsembri keskel toimus gümnaasiumi informaatika ainekava ümarlaud. Kohal oli igasugust huvitavat rahvast: ministeeriumist, ettevõtetest, kesk- ja ülikoolidest, teemaspetsiifilistest asutustest, nagu HITSA ja Innove. Ka mind oli kutsutud, ilmselt seetõttu, et mul on üldiselt kombeks igale poole nina toppida ja erinevatel teemadel jaurata.

Olen viimastel aastatel töötanud peamiselt idufirmades, kus kõigile probleemidele lähenetakse ülipragmaatiliselt ja peamine eesmärk on kõike muud kõrvale jättes „teeme asja ära“. Haridusvalla seltskonda sattudes pidin esmalt kohanema hoopis teistsuguse emotsionaalse õhkkonnaga.

Näide: ettekandele, mis minu arvates oli huvitav, informatiivne ja täiesti neutraalne, järgnes teise asutuse esindaja reaktsioon „Miks te meie tööd maha teete?“. Sarnaseid suuremaid ja väiksemaid solvumisi ja neile järgnevaid silumisi oli päris mitu. Sain ka ise oma esimese sõnavõtu järel kergelt vastu näppe ja olin hiljem ettevaatlikum.

Seetõttu jäi mind kummitama õppekavaväline küsimus: tegu oli ju põhimõtteliselt üsna otsese ja mõneti isegi kuiva teemaga – miks osalised nii pinges on ja miks siin nii suurt umbusaldust ja kriitikahirmu tuntakse?

Lihtsaim selgitus oleks muidugi, et õpetajatel on liiga vähe raha ja see muudab inimesi närviliseks. Samas näen ma võrreldavaid agressiivses kaitseasendis reaktsioone igal tasemel, professorite, akadeemikute ja ministriteni välja. Mul on seetõttu hüpotees, et meie haridus põhineb fundamentaalselt rohkem laitusel kui kiitusel, seda juba lasteaiast alates (võiks kasvõi statistikat teha, kui palju öeldakse koolipäevas kiitvaid sõnu võrreldes laitvatega). Seetõttu on ka õpilased ning lapsevanemad võitlusvalmis, pelgavad seda, mis koolist tuleb, ning hammustavad teinekord juba ennetavalt vastu. Tagajärjena koguvad haridustöötajad endasse kõvasti allergiat ja kasvatavad okkaid.

Kui ma pärast üritust mainisin, et väga huvitavad teemad, refereeriksin neid kusagil, kuulsin ka reaktsiooni:  „Ära tee, inimesed võivad solvuda.“ Keeruline olukord tõesti, kuid leian, et antud teema on siiski väärt laiemalt kui mõnekümnele inimesele valgustamist. Praegu jäävad mitmed teemad selliste pingete pärast üldse arutamata ja lahendamata.

Eelnevat arvestades ütlen seega juba ette ära, et kui ma kellelegi varbale astun, siis see pole minu kavatsus. Eesmärgiks on ikka see, kuidas asju paremaks teha.

The Good 

Minu jaoks kõige positiivsem oli see, et kohal oli palju inimesi, kes tõesti soovivad elu edendada ja asju paremaks muuta. Eestis on mitmeid asutusi ja ettevõtmisi, mis vankrit sikutavad, kuigi nende struktuur ja võimalused on väga erinevad. Need, kelle jaoks üritusele kohaletulek polnud otsene tööülesanne, ongi muidugi definitsiooni kohaselt entusiastid, nii mõnigi neist korraldab ainealast ringi või kursust. Ma ise korraldan informaatikaolümpiaadi, mille üle olen ka päris uhke, aga kõige kõvema panuse on minu arvates andnud Eno Tõnisson, kelle juhitud internetikursused ( on õpetanud programmeerimist juba tuhandetele ja tuhandetele inimestele.

Eno tegi ka ühe põhiettekannetest, kus oma mass-internetikuruste (MOOCide) kogemusi jagas. Kokku on neil kursustel osaletud üle 15 000 korra, kuigi seal on ilmselt korduvusi, sest paljud on osalenud mitmel kursusel. Arv näitab, et ühiskondlik huvi programmeerimise vastu on kõrge, ületades suuresti kõigis Eesti vastavates erialakoolides õppijate koguarvu.

Eesti Informaatikaõpetuse ajaloost rääkis Urmas Heinaste. Siin saan õnne tänada, et mu enda keskkoolitee jäi selle ajaloo algusjärku, kui keskkoolides õpetatigi konkreetselt programmeerimist. See andis mulle omal ajal päris tugeva tõuke ja nii mõnedki praeguste kõvade IT-firmade juhid on samast perioodist. 1996. aastal eraldi programmeerimise õpe lõppes ning pärast seda on informaatika sildi all õpetatud mitmesuguseid erinevaid asju.

Erinevatest õppemudelitest rääkis Mart Laanpere, kelle ettekandest jäi mulle enim meelde järgmine joonis:

Skeemi mõte on selles, et on kolm peamist jõudu, mis informaatika õppekava erinevates suundades sikutavad. Ülemine neist on ”akadeemiline” suund, mis täidab eelkõige vastava eriala (antud juhul arvutiteadust õpetavate) kõrgkoolide ja ettevõtete tellimust. Parempoolne peaks ette valmistama nö laiatarbe IT-oskusi, selliseid mida nõutakse näiteks arstilt, taksojuhilt või müüjalt. Lõpuks alumine osa täidab eelkõige kooli enda eesmärke, arendades IKT-oskusi lõimitud kujul, näiteks läbi arvuti kasutamise teistes ainetundides.

Ettekandja oli teinud ära huvitava taustatöö, paigutades skeemile selle, kus mitmed riigid praegu asuvad ja kuhu nad liikuda plaanivad. Kui Inglismaa puhul oli määravaks erialaste firmade tellimus ja surve, siis näiteks Soome asus end liigutama kuulujutu tõttu, et Eestis õpivad kõikesimeste klasside lapsed programmeerimist, ega soovinud naabrist maha jääda.

Enamik ürituse diskussioonist keskenduski sellele, milline võiks olla Eesti liikumissuund (A tähistab praegust asukohta).

Headest asjadest edasi rääkides tahan tunnustada Eesti haridusasutusi, eriti Tartu Ülikooli, kus hoolimata spetsiifilise rahastuse puudumisest juba mitu aastat laia avalikkust on programmeerimise juurde toodud. Teiselt poolt lähevad minu tänusõnad paljudele ettevõtetele, kes mitmesuguseid haridusalaseid ettevõtmisi on toetanud, olgu siis tegu õppejõudude hankimise, oma töötajate ülikoolidesse õpetama saatmise või õpilasürituste sponsoreerimisega.

The Bad 

Kuigi üritus oli hästi läbi viidud ja ettekandjad valdasid selgelt teemat, jäid minu hinge kripeldama mitu asjaolu.

Esimene oli see, et vaatlusalune kolme tõmbesuunaga mudel keskendub sellele, mida mingid välised osalised täna soovivad. On selge, et arsti või taksojuhi kutsestandard on suure tõenäosusega koostatud vastava valdkonna juhtide ehk siis eelmise või üle-eelmise põlvkonna esindajate poolt. IKT ettevõtted on pisut paremad, kuid olles viibinud üritustel, kus arutatakse näiteks nende tellimust ülikoolidele, tundub ka seal olevat kahetsusväärselt palju lühiajalist mõtlemist. Kui täna on vaja Java-programmeerijaid, siis seda nõuamegi, kui teatud metoodikat tundvaid süsteemianalüütikuid, siis nõuame neid jne.

Ideaalses maailmas oleks joonisel esindatud ka konkreetselt tulevikku vaatav jõud, kes mõtleks, millistest oskustest on kasu nt 15-20 aasta pärast. Omaaegne keskkoolis programmeerimise õpetamine oli küll nõukogudeaegse kampaania vili ning sai koosviibimisel ka arvestatava sarkasmi osaliseks, kuid see oli siiski oluliselt tulevikku vaatav. Kui toona oleks lähtutud pigem vabrikudirektorite ja kolhoosijuhtide tellimusest, siis kas näiteks 90. aastate alguses keskkoolis käinud Skype’i tegijad oleks samaks ajaks sama kogemuse saanud?

Teiseks olid valikud suuresti kinni mitte sisus, vaid vormis. Kas õpetus peaks olema iseseisva ainena või teistega lõimitult? Kas rühmaprojekti või tunnipõhise kava alusel? Algama kümnendas või üheteistkümnendas klassis?

Nii palju kui sisust juttu oli, jäi mulle tunne, et see on küllaltki eklektiline: samal pulgal 3D-modelleerimine, geoinformaatika, mehhatroonika, programmeerimine, robootika. Programmeerimise kohta öeldi korduvalt, et see on spetsiifiliselt neile, kes seda erialaselt õppima ja hiljem vastavasse ettevõttesse tööle läheks.

Viimases punktis jäin ma eriarvamusele: minu seisukohalt pole programmeerimine sugugi ainult kitsalt erialainimeste pärusmaa, vaid väga praktiline üldoskus ja ka märgatavalt laiema kasutuspinnaga kui teised loetletud oskused.

Kui 90. aastate alguses olid arvutid veel vähelevinud ja programmeerimisoskust raskem rakendada, siis tänapäeval kasutatakse neid igal erialal. Arvuti eripära on aga selles, et ta on masinana universaalne – lisaks tööjuhendis vajalikele ülesannetele saab teda panna tegema ka igasuguseid muid asju. Ning kui sa suudad arvuti enda heaks tööle panna, on see sama võimas kui isiklik assistent, kes osa su tööst endale võtab – sinu efektiivsus ja ühiskonnale toodav lisaväärtus kasvavad hüppeliselt. Siin võivad näideteks olla nii lasteaiaõpetaja, kes kohalolijate märkimiseks lihtsa veebisaidi loob, ametiasutuse asjaajaja, kes skaneeritud dokumentide laialijagamiseks väikese skripti koostab, kui ka põllumees, kes optimaalse väetisekoguse leidmiseks Exceli makro teeb. Need kõik on elust võetud näited ja igaüks neist suurendab mingis pisikeses lõigus ühiskonna toimimise efektiivsust. Tulemusena saavutaksime aga samasuguse sotsiaalse hüppe nagu siis, kui kogu rahvas lugema ja kirjutama õppis.

Selliseid võimalusi panna masinaid tegema ära rutiinseid tegevusi on tuhandeid, kuid enamik neist ei jõua kunagi seda järge ära oodata, et „päris“ programmeerijad suure raha eest vastava töö ära teeks. Analoogiliselt olid kunagi olemas üksikud kirjaoskajad inimesed, kes teiste eest kirju kirjutasid ja neile raamatuid ette lugesid, ilmselgelt jäid siis enamik vajalikest kirjadest kirjutamata ning raamatud lugemata. Praegu kardaks enamik inimesi selliste ülesannete lahendamist isegi üritada, kuid need, kes on programmeerimisega natukegi kokku puutunud, saaks juba kasvõi veebist leitud instruktsioonide varal hakkama. Ka Tartu Ülikooli MOOCide populaarsus kinnitab vastavat vajadust.

Osalejaskond jagunes selles osas kahte lehte, oli nii neid, kes mind toetasid, kui ka neid, kes ütlesid, et koolis programmeerimise õpetamine on sama praktiline kui alpinismi õpetamine – ainult entusiastidele. Mis tegelikult saama hakkab, näitab aeg.

The Ugly 

Räägitud teemadest olulisem oli aga ruumis viibinud elevant, kelle vari kõigile teemadele langes, aga kelle olemasolu sellegipoolest keegi ei maininud. Tõenäoliselt sellepärast, et keegi ei osanud sellega midagi peale hakata, kuid igasugusele plaanile on ta takistuseks ikka.

Tegu on mõistagi informaatikaõpetajate palga teemaga. Kui informaatikaõpetajad saaks riigieelarvest rohkem palka kui teised õpetajad, põhjustaks see ilmselgelt kõvasti nurinat. Kui tal on aga olemas head IT-alased oskused (ja selleks, et lapsi õpetada, võiks need ju olla, lisaks ka pedagoogilised teadmised), aga palka saab vähem kui nooremspetsialist suvalises IT-firmas, on ka narr eeldada, et see eriala teab kui populaarne oleks. Ja tõepoolest, TLÜ sulges hiljuti vastava õppekava nõudluse puudumise tõttu.

Teiseks oleme langenud omaenda „kuvandi loomise“ kampaania ohvriks. Ühelt poolt, nagu eeltoodud Soome näitest näha, teevad naaberriigid ära asjad, millest Eesti puhul vaid pressiteated eksisteerivad. Teiselt poolt on ka kohalikud otsustajad uskuma jäänud, et IT-s ongi kõik juba suurepärane ja vastavasse haridusse pole tarvis rohkem panustada. Drastiline näide: Statistikaameti andmetel olid avaliku sektori kulutused teadus- ja arendustegevusele 2016. aastal väiksemad kui 2008. aastal! (

Silmaga nähtav väljund neile probleemidele ongi, et osalised on vastastikku umbusklikud (olen olnud korduvalt tunnistajaks näiteks olukorrale, kus üldhariduskoolid ja ülikoolid takerduvad koostööd arutades sellesse, kelle kaudu mingi raha peaks liikuma), tirivad õhukesevõitu tekki enda poole ja jagelevad saadaoleva ressursi pärast, kui tegelikult peaks tegelema sellega, kuidas laiem ja paksem tekk hankida, mis kõiki korralikult kataks.

Mis on lahendus?

Õppekava ürituse üks meeldejäänud joon oli veel erinevate osaliste arv – haridusasutused, riigiasutused, sihtasutused, ettevõtted, liidud, seltsid. Igaüks neist korraldab oma üritusi ja projekte, mõnel on raha, teisel on inimesi, mõnel õnnelikul ka mõlemat. Tegevused on aga tavaliselt suhteliselt piiratud, ebastabiilse rahastusega. Ma käin isegi igal sügisel erinevatelt firmadelt olümpiaadi korraldamiseks raha küsimas.

Rääkisin teemast TÜ Arvutiteaduse Instituudi juhi Jaak Viloga, kes kinnitas samuti: „Probleem on Eestis, et kõik käigu justkui õhina-tuhina korras ja projektikestega. Õppekava võimalusi nii ei saa arendada.“

Tõepoolest, entusiastide tegutsemine on väga hea, kuid küsimuse struktuurseks lahendamiseks peaks enim panustama need, kes IT ja programmeerimise õpetamisest ka kõige rohkem kasu saavad. Peamisteks kasusaajateks on ühelt poolt IT-ettevõtted, kelle praegust potentsiaalset käivet ja kasumit kvalifitseeritud tööjõu nappus otseselt piirab, ning teiselt poolt Eesti riik, kellele vastava võimekusega spetsialistid mitmekordse keskmise jagu makse maksavad ja samal ajal riigi üldist infrastruktuuri efektiivsemaks muudavad.

Kahjuks esineb siin aga palju näpuga näitamist: ettevõtjad süüdistavad riiki ja haridusasutusi, et nood ei tee piisavalt õigeid asju, aga riik uriseb vastu, et miks ettevõtted ise ei panusta. Lahendus võiks peituda koostöös.

Esimene koostöötase saaks aset leida eraettevõtete vahel. Selle asemel, et igaüks midagi eraldi teeb, saaks luua Eesti IT-hariduse konsortsiumi, millel on võimalik tuhandete eurode asemel käsutada miljoneid ning sellega midagi tõsisemat korda saata.

Teine koostöö võimalus on ettevõtete ja riigi vahel. Olgu meil vaja palgata ülikoolidesse paar kõva õppejõudu või meelitada hulk spetsialiste koolidesse programmeerimist õpetama – vastastikune hea tahte avaldus ja riske maandav käitumine oleks see, kui näiteks ettevõtete konsortsium panustaks veerandi jagu rahast lõiku, mis neid enim aitaks ja riik kataks vastavatest kuludest ülejäänud kolmveerandi (viimase skeemi soovituse sain samuti Jaak Vilolt).

Veel üks vastus peitub taas pikemaajalises mõtlemises ja tänasest kaugemale vaatamises. Praegu räägitakse palju spetsialistide sissetoomisest. Esiteks ei suuda me sellel turul iial konkureerida selliste industriaalse võimsusega ajuimejatega nagu Silicon Valley või ka London või Berliin, mistõttu Eestisse jõuavad vaid globaalse intellekti teise ešeloni esindajad. Teiseks on see nagu kolmanda maailma riikide majanduskäitumine – selle asemel, et investeerida omaenda tööstusse, ostetakse teiste riikide valmistoodangut.

Nii nagu Saksamaa ehitas 20. sajandi keskel uuesti nullist üles kõva tööstuse ja tagas endale sellega pikaajalise juhtpositsiooni, on ka spetsialistide osas pikas plaanis efektiivsem ostmise asemel „ehitada ajutööstus“ ehk panustada vastavalt haridussüsteemi.

Selleks tuleb aga ettevõtetel mõelda kaugemale järgmise paari aasta kasumiaruandest ning riigivalitsusel kaugemale järgmistest valimistest. Vastasel korral ei jää ei tiigrist ega kuvandist lõpuks midagi järele.

Viimane lahendus on rohkem unistuste vallast, kuid siiski realistlik. Palju on räägitud sellest, kuidas Skandinaavia maade elanikud ei pane pahaks kõrgeid makse maksta, sest nad saavad selle eest vastu häid teenuseid, maksude maksmine on rohkem au- kui häbiasi. Analoogselt võiks Eesti eripäraks olla uhkus tulevikku (eriti haridusse) panustamise üle. Nii riik, omavalitsused kui ettevõtted saaks kiidelda ja võistelda, millise protsendi keegi oma eelarvest/kasumist haridusse ja tulevikuprojektidesse paneb. Maailmas, kus pahatihti elatakse pigem tuleviku arvelt, oleks “IT-riigi” kuvandist veel kõvem sõna “tulevikku panustav riik ja ühiskond”.

December 27, 2017

Raivo LaanemetsMigration to FastMail

I have migrated my mail setup to FastMail. It removed the need to run my own mailservers with all the required infrastructure including webmail. I finished the migration a week before Christmas and should have not missed any mails.

Maintaining my own setup was a too high effort. Before FastMail I also tried G Suite (Gmail) but it does not have mail folders and would have required me to drastically change the way I read my mail.

The migration was done over IMAP interface, provided by the Dovecot-imap package. It took about an hour to transfer all 8000 messages. I'm using the Standard plan which is 50$ per year and which provides 25GB of space for mail.


I added a cron job to my backup server to backup the FastMail account automatically. It uses the offlineimap utility. Here is the article where I got the idea. The original article is about Gmail but it's easy to adopt to FastMail.


I have some servers that send out mail. I have configured them to send messages through my FastMail account SMTP. Here is an article how to set up Postfix to do it.

December 23, 2017

Targo tarkvaraProtected: Informaatika õppekava: good, bad and ugly

This post is password protected. To view it please enter your password below:

December 17, 2017

Kuido tehnokajamSQL SERVER CLR serialization assembly

Kui juhtub, et SQL SERVER-i veebiteenuse poole pöörduv CLR hakkab pilduma viga Cannot load dynamically generated serialization assembly. In some hosting environments assembly load functionality is restricted, consider using pre-generated serializer. Please see inner exception for more information.  System.IO.FileLoadException: LoadFrom(), LoadFile(), Load(byte[]) and LoadModule() have been

December 05, 2017

Four Years RemainingThe Mystery of Early Stopping

Early stopping is a technique that is very often used when training neural networks, as well as with some other iterative machine learning algorithms. The idea is quite intuitive - let us measure the performance of our model on a separate validation dataset during the training iterations. We may then observe that, despite constant score improvements on the training data, the model's performance on the validation dataset would only improve during the first stage of training, reach an optimum at some point and then turn to getting worse with further iterations.

The early stopping principle

The early stopping principle

It thus seems reasonable to stop training at the point when the minimal validation error is achieved. Training the model any further only leads to overfitting. Right? The reasoning sounds solid and, indeed, early stopping is often claimed to improve generalization in practice. Most people seem to take the benefit of the technique for granted. In this post I would like to introduce some skepticism into this view or at least illustrate that things are not necessarily as obvious as they may seem from the diagram with the two lines above.

How does Early Stopping Work?

To get a better feeling of what early stopping actually does, let us examine its application to a very simple "machine learning model" - the estimation of the mean. Namely, suppose we are given a sample of 50 points \mathbf{x}_i from a normal distribution with unit covariance and we need to estimate the mean \mathbf{w} of this distribution.



The maximum likelihood estimate of \mathbf{w} can be found as the point which has the smallest sum of squared distances to all the points in the sample. In other words, "model fitting" boils down to finding the minimum of the following objective function:

    \[f_\mathrm{train}(\mathrm{w}) := \sum_{i=1}^{50} \Vert \mathbf{x}_i - \mathbf{w}\Vert^2\]

As our estimate is based on a finite sample, it, of course, won't necessarily be exactly equal to the true mean of the distribution, which I chose in this particular example to be exactly (0,0):

Sample mean as a minimum of the objective function

Sample mean as a minimum of the objective function

The circles in the illustration above are the contours of the objective function, which, as you might guess, is a paraboloid bowl. The red dot marks its bottom and is thus the solution to our optimization problem, i.e. the estimate of the mean we are looking for. We may find this solution in various ways. For example, a natural closed-form analytical solution is simply the mean of the training set. For our purposes, however, we will be using the gradient descent iterative optimization algorithm. It is also quite straightforward: start with any point (we'll pick (-0.5, 0) for concreteness' sake) and descend in small steps downwards until we reach the bottom of the bowl:

Gradient descent

Gradient descent

Let us now introduce early stopping into the fitting process. We will split our 50 points randomly into two separate sets: 40 points will be used to fit the model and 10 will form the early stopping validation set. Thus, technically, we now have two different objective functions to deal with:

    \[f_\mathrm{fit}(\mathrm{w}) := \sum_{i=1}^{40} \Vert \mathbf{x}_i - \mathbf{w}\Vert^2\]


    \[f_\mathrm{stop}(\mathrm{w}) := \sum_{i=41}^{50} \Vert \mathbf{x}_i - \mathbf{w}\Vert^2.\]

Each of those defines its own "paraboloid bowl", both slightly different from the original one (because those are different subsets of data):

Fitting and early stopping objectives

Fitting and early stopping objectives

As our algorithm descends towards the red point, we will be tracking the value of f_\mathrm{stop} at each step along the way:

Gradient descent with validation

Gradient descent with validation

With a bit of imagination you should see on the image above, how the validation error decreases as the yellow trajectory approaches the purple dot and then starts to increase after some point midway. The spot where the validation error achieves the minimum (and thus the result of the early stopping algorithm) is shown by the green dot on the figure below:

Early stopping

Early stopping

In a sense, the validation function now acts as a kind of a "guardian", preventing the optimization from converging towards the bottom of our main objective. The algorithm is forced to settle on a model, which is neither an optimum of f_\mathrm{fit} nor of f_\mathrm{stop}. Moreover, both f_\mathrm{fit} and f_\mathrm{stop} use less data than f_\mathrm{train}, and are thus inherently a worse representation of the problem altogether.

So, by applying early stopping we effectively reduced our training set size, used an even less reliable dataset to abort training, and settled on a solution which is not an optimum of anything at all. Sounds rather stupid, doesn't it?

Indeed, observe the distribution of the estimates found with (blue) and without (red) early stopping in repeated experiments (each time with a new random dataset):

Solutions found with and without early stopping

Solutions found with and without early stopping

As we see, early stopping greatly increases the variance of the estimate and adds a small bias towards our optimization starting point.

Finally, let us see how the quality of the fit depends on the size of the validation set:

Fit quality vs validation set size

Fit quality vs validation set size

Here the y axis shows the squared distance of the estimated point to the true value (0,0), smaller is better (the dashed line is the expected distance of a randomly picked point from the data).  The x axis shows all possible sizes of the validation set. We see that using no early stopping at all (x=0) results in the best expected fit. If we do decide to use early stopping, then for best results we should split the data approximately equally into training and validation sets. Interestingly, there do not seem to be much difference in whether we pick 30%, 50% or 70% of data for the validation set - the validation set seems to play just as much role in the final estimate as the training data.

Early Stopping with Non-convex Objectives

The experiment above seems to demonstrate that early stopping should be almost certainly useless (if not harmful) for fitting simple convex models. However, it is never used with such models in practice. Instead, it is most often applied to the training of multilayer neural networks. Could it be the case that the method somehow becomes useful when the objective is highly non-convex? Let us run a small experiment, measuring the benefits of early stopping for fitting a convolutional neural-network on the MNIST dataset. For simplicity, I took the standard example from the Keras codebase, and modified it slightly. Here is the result we get when training the the most basic model:

MNIST - Basic

MNIST - Basic

The y axis depicts log-loss on the 10k MNIST test set, the x axis shows the proportion of the 60k MNIST training set set aside for early stopping. Ignoring small random measurement noise, we may observe that using early stopping with about 10% of the training data does seem to convey a benefit. Thus, contrary to our previous primitive example, when the objective is complex, early stopping does work as a regularization method. Why and how does it work here? Here's one intuition I find believable (there are alternative possible explanations and measurements, none of which I find too convincing or clear, though): stopping the training early prevents the algorithm from walking too far away from the initial parameter values. This limits the overall space of models and is vaguely analogous to suppressing the norm of the parameter vector. In other words, early stopping resembles an ad-hoc version of \ell_p regularization.

Indeed, observe how the use of early stopping affects the results of fitting the same model with a small \ell_2-penalty added to the objective:



All of the benefits of early stopping are gone now, and the baseline (non-early-stopped, \ell_2-regularized) model is actually better overall than it was before. Let us now try an even more heavily regularized model by adding dropout (instead of the \ell_2 penalty), as is customary for deep neural networks. We can observe an even cleaner result:

MNIST - Dropout

MNIST - Dropout

Early stopping is again not useful at all, and the overall model is better than all of our previous attempts.

Conclusion: Do We Need Early Stopping?

Given the reasoning and the anecdotal experimental evidence above, I personally tend to think that beliefs in the usefulness of early stopping (in the context of neural network training) may be well overrated. Even if it may improve generalization for some nonlinear models, you would most probably achieve the same effect more reliably using other regularization techniques, such as dropout or a simple \ell_2 penalty.

Note, though, that there is a difference between early stopping in the context of neural networks and, say, boosting models. In the latter case early stopping is actually more explicitly limiting the complexity of the final model and, I suspect, might have a much more meaningful effect. At least we can't directly carry over the experimental examples and results in this blog post to that case.

Also note, that no matter whether early stopping helps or harms the generalization of the trained model, it is still a useful heuristic as to when to stop a lengthy training process automatically if we simply need results that are good enough.


October 13, 2017

TransferWise Tech BlogA Product Manager’s guide to planning for the team

One of the best things about working at TransferWise, is how seriously we take the autonomy of our teams. Each team can (and needs to) set its own Key Performance Indicators (KPIs), come up with a mission statement, and a vision for the team. Once those are worked out, a team then writes out its own plans — there’s no influence from the Leadership team. This process helps our teams do their best work.

As a new product manager to TransferWise, planning can seem daunting. You’ll have little input from the Leadership team or your manager on what to plan or how to do it. I wanted to share my experience planning in the Transfer Experience team, so you can you can confidently plan for your next quarter.

Why is planning so important?

We worked out that we spend about three weeks every quarter planning in the Transfer Experience team. Which feels like a lot. Why spend all that time planning, when we could be doing something else? Mostly because planning is about the process rather than the outcome. So while it’s great having high-quality plans, it’s how you arrive at those plans that matters.

Here’s what you need to get out of your team’s planning sessions:

  1. Get the team to commit. This one’s important. If your plan’s got a chance of succeeding, then it’s got to come organically from everyone on the team. Planning shouldn't just be engineers breaking tasks down into good old-fashioned man hours. It should also have them challenging the products you’re building, and the priorities of those products. Ultimately, everyone inside the team should own the plans for next quarter.

  2. Use it as a chance to gauge the team’s health. While part of planning is thinking about the future, it’s also a great opportunity to think about how things have been working so far. Have the team run retrospectives, check the team’s progress against your previous plans and try to get a feel for everyone’s happiness levels.

  3. Prioritize, prioritize, prioritize. As a product manager, it’s your job to create great products that solve customer problems. And prioritization is one of the ways that a product manager can help make that happen. Planning, by virtue of bringing together your team, is one of the best venues for prioritization.

How we organize planning

We tend to divide our team planning into four parts. They roughly fall into these categories:
Retrospectives. We usually run two kinds of retros: work-based and feelings-based ones. Both are equally important.

  • Work-based retrospectives try to answer the question “How did we do last quarter compared to our plans?” These start by tracking the team’s progress against the plans made last quarter, so having a higher-quality plan from last quarter will help you do better work-based retrospectives. This session usually leads to questions like:
    • How can we become more efficient as a team?
    • What did we spend most of our time on last quarter?
    • How can we do things faster?
    • Do we have too little/too much tech debt?
  • Feelings-based retrospectives aim to answer the question: “How happy are you in the team?” We like to get a feel for everyone’s happiness levels, because we care about creating a positive work environment. Because when people are happy where they are, they can give their all in their work. These kind of retros are hard to run — it’s not easy getting people to open up and share everything they’re thinking — but also the most rewarding. Usually the conversations revolve around these kinds of questions:
    • How well do we get along with each other?
    • Are we holding each other accountable?
    • Are people happy about their work-life balance?
    • How are we interacting with other teams in the company?
    • Do we get along with each other?

Mission & vision for the team. This one’s both fun and incredibly important. Every quarter, we review how we did against our KPIs, and challenge our mission and vision to see if they still fit us. We talk through how the products and features we worked on helped us get closer to achieving our mission and helping our customers. The team should come out of these sessions feeling 100% committed to their renewed mission and vision.

Internal organization & processes. This is normally a short discussion where we work out if we need to change how we do things as a team. Because we’re a distributed team of 15+ people, we usually only come together once a quarter. So when we do, it’s a great opportunity to talk through changing sensitive processes or the structure of the team.

Planning. Yes! Finally, we get down to actually planning. During these sessions, we mostly do the following (in this order):

  1. Review the products that we want to build, and prioritize each product. This should be a product-led session.
  2. Break down each product into tasks, and come up with estimates for how long it would take to build each product. This should be an engineering-led session.
  3. Review and reprioritise according to how long each product is estimated to take. We also try to bring up constrains such as cross-team dependencies, deadlines, upcoming vacations during this session. This session should be a product oriented session.

What improving planning looks like

The Transfer Experience team has been getting stronger and stronger at planning every quarter. Here are a couple ways we have seen we have been getting stronger at planning. If you see your team has improved in some of these aspects, it might be also because you have been improving the way you plan.

We spend less time wondering what to build and more time building. This is a great side effect of having great plans. you’ll spend less time discussing why/what and more time discussing how, because your team members should already be committed to the why. This time saving can be put into improving the quality of the execution, thus improving the outcome for your customers.

Better cross-team collaboration. Writing detailed and realistic plans has helped us with cross team collaboration. It makes it easier for people outside the team understand why and what we want to spend our time on. It also makes it easier for us to call cross-team dependencies, and inform them on the progress we are doing on several projects.

Better retrospectives. Since ideally everyone is committed to the plans, if plans don’t become reality, the team should feel responsible for that in the retrospective. Which means, team members should be pushing each other to understand how they can improve both execution and the plans.

Ultimately, we have been able to improve a lot as a team thanks to better planning, yet we are always looking for ways to improve how we do things here at TransferWise. What is your experience with planning? Leave your thoughts in the comment section below!

September 16, 2017

Raivo LaanemetsNow, 2017-09

This is an update on things related to this blog and my work.


Since the last update I have been returning to a more stable schedule of writing. I now usually write 1 or 2 articles per week and do most of the writing on weekends. I'm soon hitting the milestone of 200 articles and will review the blog in the similar way as I did for the previous milestone.


I have moved back to my home office. Construction work has practically ended and the noise is not an issue anymore. During the couple of noisy days I have been sitting in a local pub with my laptop and drinking coffee.

The last 2 months have gone into a single project. It is a system to replace lots of manual work of managing transportation orders. The client has been pleasure to work with. Their office has been very welcoming and they have really good coffee. Most of my work has gone into the user interface. The backend is built using Node.js and MySQL and the frontend is using server-side JSX to generate HTML and browser-side React for dynamic forms.

Open Source and side projects

Since the last update I have built one side project. It's an app to ask questions about a software project. I have had the idea to build it for a long time. The basic system for questions was inspired by Expert Systems and the Bethesda dialogue system. I built the initial prototype in Prolog but later rewrote it as a browser-side application. It uses a tree-like data structure (actually a directed acyclic graph) where nodes are questions or topics and branches are activated by positive answers.

I rewrote the UI of my feed reader app in React. I wanted to test out React and apply some changes to the reader app. The original UI was written in Knockout.js and a bit hackish. I wrote the original UI many years ago when I did not have much experience in structuring large browser-side applications. At the moment I'm very satisfied with the rewritten code.

September 14, 2017

TransferWise Tech BlogOn the Logging in Mobile Applications

Here I describe an approach from iOS point of view. An approach for Android platform is no different. I can recommend to use Timber framework as an equivalent for CocoaLumberjack.

Logs? Phew! Just write something like:

print("Expected `foo` to be 'bar', got: \(foo)")  

Later, when you debug an issue, the message in the console could give you a hint for the solution. Simple enough. But it’s good only for a “Hello World” application.


1st of all, logs should be available when you need them desperately. Having a large number of users is a guarantee that every possible crash will happen including those under really strange circumstances. Unable to reproduce the issue, with only a stack trace in your hands you would clearly benefit from an additional source of information — application logs. Is it possible to attach logs to a crash report? Yes. For example, Crashlytics Custom Logs can be used for that.

Let’s assume that in the 3rd party SDK, which you are using to collect crash reports there is a func cloudLog(_ msg: String) method (like CLSLogv method in Crashlytics). Say this method follows the Unix philosophy and the only thing it does is sending a provided message to the 3rd party no matter what the current build configuration is. But we would want to use this handy method only in production builds in order to avoid the noise from the 3rd party when debugging.

So it makes sense to introduce your own method like this:

func myLog(_ msg: String) {  

Hmm, didn’t we just start writing our own logging framework?

Our new myLog(_:) method works for most cases. But what if the application you are working on handles sensitive information? Banking, dating or health apps are good examples. If you are logging payloads of your network requests and responses (that would be handy, so probably you are) via previously introduced myLog method you might be sending sensitive information to the 3rd party. For some organisations that’s a huge no-no.

So what can we do about that? Something like this should work:

func myLog(_ msg: String, isSensitive: Bool = true) {  
    if !isSensitive {

Hmm, myLog keeps growing…

2ndly, sometimes a desire to log is in itself an indication of something unexpected happening and usually an additional action should be taken in such a case.

To be more specific imagine your application is talking to your beloved server and suddenly a terrible thing happens — incompatible JSON is served. Shouldn’t we do the following?

myLog("Error, look how terrible that JSON is: \(json)")`  

Not bad. But the problem won’t be noticed unless your application crashes right after. A crash is better than nothing in such a case, but the user might benefit from an offline data or remaining parts of the working functionality. A much better option would be to silently report the problem, receive the report, fix the API, meanwhile showing “Something went wrong. Sorry! We are already working on it.” message to the user. Is there a ready solution to report such a problem? Yes. For example, Crashlytics Caught Exceptions can be used for that. I will reference those as non-fatals — issues that are not crashing the application but should be reported.

Let’s assume that there is a func reportNonFatalError(_ error: Error) method to report problems silently (like recordError in Crashlytics).

We don’t want to report the issue in debug builds (to avoid the noise), but we still want to log to the console a message describing the problem. As for reporting, reusing this log message looks like a good idea. So the straightforward way to do that is like this:

let message = "Error, look how terrible that JSON is: \(json)"  
reportNonFatalError(makeError(withMessage: message))  

The above code looks too cumbersome. Much better is to rewrite our myLog method like this:

enum LogBehavior {  
    case writeToLog(isSensitive: Bool)
    case reportAsNonFatal
struct WrappedStringError: Error {  
    var value: String
    var localizedDescription: String {
        return value
func myLog(_ msg: String, behavior: LogBehavior = .writeToLog(isSensitive: true)) {  
    switch behavior {
    case let .writeToLog(isSensitive):
        if !isSensitive {
    case .reportAsNonFatal:
        reportNonFatalError(WrappedStringError(value: msg))

Now myLog‘s signature seems too complicated and its implementation is like a trash bin full of 3rd party specific logic. Is there a better way? Yes!

We better use some logging framework. Something like CocoaLumberjack. The idea behind it is really simple, but powerful. On the launch of the application you plug in some loggers (you can create your own or select from the predefined ones). When you log the message, it is broadcasted to every logger you plugged in. It’s up to a particular logger how to process the message. Given the lack of restriction on the number of loggers you can plug in you can untangle logging spaghetti using single responsibility principle.

But let’s give some examples.

The most simple but useful logger is a console logger which just prints a message to the console.

You might want to use file logger which persists logs on disk and rotates them. File logger is quite useful for feedback functionality — ask the user if he want to attach logs when composing email to customer support. A problem description with recent log is much more helpful than just the description. Of course you should not store any sensitive data in such logs.

Passing log messages to cloud log or reporting non-fatals is a good enough job for a separate logger as well.

Okay. Seems pretty straightforward. But how can we pass the information to a logger if the message is sensitive or if it should be reported as a non-fatal?

To answer that question let’s look at how we actually log. We do it the same way as we did before when using standard print, but now we have to flag the message as error, warning, info, debug or verbose one. Too many of them, isn’t it a complication? A bit it is, but a handy one. Of course those types impose specific semantics on messages and could be used to filter or color them appropriately in the console. But they can be also used to make a decision on what particular logger should do with a particular message. For example, the non-fatals logger should ignore all the messages except for of error type.

No more words, just look at the following logging policy scheme:

Build Console Non-fatal Cloud Log When To Use
Error Debug

Something bad has happened that should never happen. App is not crashing but a part of the functionality is not working. The situation should be reported.

E.g., a wrong JSON format, parsing errors when a source format is unsupported, some system service is not available but we expect it to be.

Warning Debug

Something bad has happened that should never happen. App is not crashing but a part of the functionality is not working. The situation should be reported.

E.g., a wrong JSON format, parsing errors when a source format is unsupported, some system service is not available but we expect it to be.

Info Debug

Something expected happened. Messages of that level are useful to understand the reason for a crash or an unclear bug.

E.g., a view controller is presented or dismissed, user interactions, some service finished an initialization, a network request succeeded, a DB is saved successfully.

Debug Debug Messages of that level are useful to log sensitive information such as a network response content if it’s really needed. Such log messages should not litter the console.
Verbose Debug Messages that are littering the console, should be manually turned on.

Important details:

  • log sensitive information only on debug and lower levels (that’s only verbose);

  • report only error messages as non-fatals;

That’s roughly the scheme we are using in TransferWise for our iOS application. Of course some amount of work is needed to setup such an approach for an existing application, but it will make the maintenance simpler. For applications that are suffocating from the tech debt it’s one of the first steps to do. But even if your application is pretty healthy, efforts made to keep a finger on the pulse are a thousand times worth it!

DDLogInfo(“Happy logging!")

September 10, 2017

Raivo LaanemetsWho is a Software Consultant

It does matter what you call yourself. It sends out a signal of what you do and how you can be useful. By the offers I have been getting recently I have come to a conclusion that my current title (software developer or full-stack freelancer) has not been descriptive enough. The new offers have not matched up with what I have been doing for the existing clients.

What and how I do

  • Solving problems by developing custom software.
  • Develop in collaboration with the client.
  • Providing services directly to the client.
  • Working with multiple clients at once.
  • Choosing my own clients and projects.
  • Managing the development process.
  • Working remotely or on-site.
  • Develop using popular platforms and libraries (mostly JavaScript).
  • It is a full-time job and not a side business.

I believe this is the best described as a "Software Consultant". Lots of other titles are approriate in some aspect but are not more descriptive as they are too generic, too specific, or have negative connotations in some communities.

More information:

September 05, 2017

Raivo LaanemetsCryptopocalypse in Estonia

A day ago (Tuesday, 6th September) Estonian Information System Authority was informed about a security vulnerability in the electronic chip of Estonian ID cards.

Use of electronic signatures is very common in Estonia. This includes all sorts of legal contracts. Estonia uses the same electronic cryptographic system for online electronic voting. An electronic signature from an ID card is fully binding. It has the same power as a manually written signature.

The vulnerability affects 750000 ID cards issued in the last 3 years. Estonian population is 1.4 million. Estonian ID card is mandatory for every citizen.

Henrik Roonemaa, a well-known Estonian journalist has put together an article from various sources of information. It is published in Estonian. The important points are:

  • The cost of breaking one card for signing is 40k EUR.
  • At least one card has been broken for demonstration purposes.

This shows the severity of the situation. The vulnerability is believed due to the process of generating private and public keys inside the card chip. The weakness allows to deduce the private key from the public key. The list of all public keys was publicity available until Tuesday when it was taken down. Full details of the vulnerability have not yet published.

You get the full power to forge a perfect signature once you gain access to the private key. The electronic signature would be extremely hard to disprove as such attack was deemed practically impossible by experts.

Estonian government says that the issue won't affect the Estonian state as a whole. They say that it would only happen when someone put 60 billion EUR into work to break most of the 750000 signatures. Estonia is worth way more than 60 billion EUR. The government has not yet addressed the issues of individual citizens getting targeted by attacks using this vulnerability.

New cards will have to be issued to fix the vulnerability.

Update 2017-09-10

It has not been confirmed that a card has been actually broken. The issue might then be a lot less severe than thought. The text has been removed from the article.

Update 2017-09-13

It takes about 10-12 hours to crack a card according to the former Estonian president Toomas Hendrik Ilves. Ilves says that this is based on facts. The information comes from an article published in the daily newspaper Postimees. The article also states that the vulnerability is between the chip hardware and the software directly communicating with the chip.

Update 2017-12-27

The current solution is to replace certificates on all vulnerable cards. I have done this to mine as it was vulnerable.

September 04, 2017

Raivo LaanemetsA solution to dreaded "Resolving Host" in Chrome

Sometimes Chrome hangs for a couple of seconds with the "Resolving Host" message on the status bar. The problem is very annoying as it delays page loading for multiple seconds. It happens randomly, not on every site, not on every page open.

I started to get this behavior in the last couple of months. Googling reveals that this could be solved by changing various settings inside Chrome. None of that worked for me. One of the most popular solutions is to use Google DNS servers. I thought it would have no effect on me because I was already using them.

For some time I was trying to debug my hardware. I was suspecting my router and cables and network card. I checked various machine logs every time I had the issue occur in the browser. I found nothing.

Finally I decided to apply strace to the DNS lookup. Strace is a debugging utility for Linux. I thought I might not be a first one to do this and found a blog post by Jeff Blaine. I applied his command on my machine. The command is:

strace -f getent hosts

The command spewed out a verbose log but otherwise completed fast. Then I ran it again and again until it started to hang. It can be seen that it connects to a Google DNS server and awaits a response from it. By the log it looks like sometimes the server won't respond. The third argument for the hanging call in the log, 5000, does seem to be a timeout in milliseconds. After the timeout expires, it tries to make a new request.

connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("")}, 16) = 0
poll([{fd=3, events=POLLOUT}], 1, 0)    = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "\223\254\1\0\0\1\0\0\0\0\0\0\3www\npuppetlabs
  \3com\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=3, events=POLLIN}], 1, 5000 <- HANGING HERE

The solution was to simply replace Google DNS servers with OpenDNS servers. They have no such behavior and the issue has now completely disappeared. You need consult your OS or distro manual or read online guides how to change DNS server settings for your computer. I currently have no idea what causes the problems with Google DNS.

September 03, 2017

Raivo LaanemetsChrome 59/60+ font rendering in Linux

Chrome 59 on Linux renders fonts differently than the rest of the system. This happens when you use subpixel hinting on your system. Chrome 59+ seems to ignore your settings. It can be fixed by tweaking the FreeType configuration. FreeType is a font rendering system used by most Linux distributions.

The issue can be fixed by setting an environment variable before launching Chrome:

export FREETYPE_PROPERTIES="truetype:interpreter-version=35"

I'm using an Application Launcher in MATE desktop to start Chrome. To automatically set the environment variable, I created a wrapper script in my home directory and edited the launcher settings to invoke the wrapper script instead of google-chrome directly. The full contents of the script:

#!/usr/bin/env bash
export FREETYPE_PROPERTIES="truetype:interpreter-version=35"
/opt/google/chrome/google-chrome &

With medium or full hinting this brings back skinny font rendering in the browser. Without the fix, the text looks similar to text on Windows 10.

More information:

August 17, 2017

TransferWise Tech Blog4 fallacious reasons why we estimate

‘Estimate’ as defined in this article is: A guess about the future regarding the time or effort that a task(s) will take to be completed. In other words, anything that can be used to foresee when tasks will be done.

I have to confess, I’m not very good at estimating. Because of that, for the past 3 or 4 years, I’ve been trying to use different approaches other than estimates. Although I lost my faith in estimation long ago, as a software engineer, I still receive requests for my opinion. Regardless if I end up giving an estimate or not, I always ask the person why they want one.

In this post, I'll share some of the given reasons the team decided weren’t worth spending the energy to create estimates. However, keep in mind this is merely my own point of view and my own particular circumstances; this may not apply to your scenario.

1. I need an estimate because… I need to measure productivity

Although I understand measuring productivity could work well for repeatable activities, it's hard to believe it works well for abstract and, ultimately, non-repeatable tasks like software development. For non-repeatable activities - meaning it’s never been done before - no one really knows how much time a task should take. Thus, the common approach to "measure productivity" is to compare the estimates against what, in fact, happened. This can cause a number of problems, including the fact that once developers realize they are being judged by their own estimates, they start to be very careful about the process of creating those same estimates, usually spending a lot of precious time completing an activity which doesn’t accrue additional value to the product that they are building.

From my experience, seeking to measure productivity is usually more intense when the side requiring the estimate doesn't have a profound trust that the other side will work very hard. That's why I strive to create an environment with mutual trust between the engineers and the different stakeholders. One of the most powerful techniques in this sense is to have a genuine interest in understanding the reasons behind the requirements for estimates. We engineers should always keep in mind that software development is usually one part of many others inside the company. After having worked for outsourcing, consultancy and product companies I believe that creating a place where people really trust each other is easier when software engineers and stakeholders are both part of the same organization. That's one of the reasons why more and more companies are now considering software development as a core business - thus creating really strong software engineer teams.

2. I need an estimate because… someone else is asking me to provide one

It's incredibly common to hear this reason in environment with traditional hierarchy where decisions are made top-down. I strongly believe that "agile transformations are not about transforming IT, they are about transforming organizations".

I remember once working for a product where our project manager asked us for an upfront estimate regarding a large feature that would be a game changer in the product. We tried to make it clear it was just a guess, and then provided a 3-month estimate.

Without us realizing it, the marketing team then created a beautiful campaign and started giving hints for the customer that the feature would be launched soon. Unfortunately, due to a variety of reasons, the feature was released with a huge delay. At this point, the marketing team came and asked: "Why did you tell us it would take 3 months? If you didn't know upfront, why didn't you just say you didn't know?"

Cases like this helped me learn not to provide estimates unless someone specifically tells you the reason. In this case, had we realized that marketing would take our estimate as something solid to use to begin promoting to customers, we would have acted quite differently.

The interesting thing is we are so ingrained to the culture of giving estimates that we end up giving them even when no one is asking us! Having in mind how dangerous a reckless estimate can be, I would not provide an estimate unless someone is explicitly asking you one. You can still have a general time frame in your mind, but it may be better to keep it to yourself, thus avoiding problems like the anchoring effect.

3. I need an estimate because… having one motivates us to work harder

This is a dangerous statement. Although it may encourage the team to stay in the office for longer hours, it’s not a healthy long-term approach. I've seen many teams working very hard to finish tasks before the estimated date. Many have ended up forgetting about other important things in the process, like code quality and a focus on simplicity. The cost of this cannot be underestimated. It will really come to the surface later when the team realizes that, as a result, they have accrued significant technical debts.

4. I need an estimate because… I need one to prioritize my backlog

One of the most important and complex questions in software development is, “What should we work on next?” Although I believe that in order to prioritize we should take into consideration the size and effort of the tasks, I'm not convinced we really need estimates to do that. This statement may seem contradictory, but I’ll show you how I've been prioritizing without using estimates.

First - backlog prioritization is essentially a sorting process. As software developers, we know the only requirement to order a list of items is to be able to compare elements. In other words, we don't need to know the absolute value of an item, but we need to know how to compare them. The trick as it relates to estimates is that although we aren’t very good at determining absolute numbers, we are better comparing things of the same nature.

To illustrate this principle, let's consider these three images:


Would you be able to tell me how many people we have in each picture? More than that, would you bet your money on your estimates? On the other hand, if I asked you to sort the pictures according to the number of people in each one, I'm sure you’d be able to do it. I've been using the same principle to prioritize my backlog.

How are estimates used in TransferWise?

Before answering this question, it's important to understand how teams are organized in TransferWise. As Nilan, our VP of growth, explained, "We hire smart people and trust them”. Teams are autonomous – no one can tell them what to do.

For example, we have a currencies team. It decides each quarter which currencies it's going to launch. No one tells the team what currencies to launch. Anyone can challenge the team on what it's doing and the team should have a rationale for explaining why it's focussing where it is.

As a result, it's completely up to team members to decide whether they want to use estimates as a tool to help them or not. In my current team, there's no planning poker session or any other form of estimates for individual tasks. The closest process we have to the traditional way of creating estimates is that we have a quarterly planning where we prioritize our backlog. After that, we decide the topics we’ll detail a bit more so that other people in the company would be able to understand where we’re headed and why we came to that conclusion.

After all, is there a good reason why I should use estimates?

To be honest, I don't know the answer to this question. The first thing I try to keep in mind is, "estimations are valuable when it helps you make a significant decision". Thus, the question I usually ask myself is, “What is the decision we need to make? Can we possibly make this decision without an estimate?”

Even more important than answering these questions is to open our minds to new ideas. In this way, I’d like to ask you, "What are the scenarios where you’re using estimates and they aren’t actually required?"

I'd love to hear from you.

August 13, 2017

Kiibi lühiuudiste blogiDragon viib kosmosesse superarvuti

Rahvusvahelise kosmosejaama arvutusvõimsus pole just suurem asi – astronaudid askeldavad seal vahel sülearvutitega, kuid teadusandmed saadetakse kohe edasi Maale ja arvutada on mõistlikum arvutuskeskustes. Kuid nüüd saadab NASA eksprimendi käigus kosmosesse HPE Apollo 40 klassil põhineva vedelikjahutusega arvuti. Uue arvuti teadusprojekti pikkuseks on üks aasta, mis on ka umbes see aeg, millega kaugemas tulevikus kosmoselaevad Marsile lendama võivad hakata. Kuna side Marsiga orbiidi vastaskülgedel võtab aega umbes 20 minutit (ehk edasi-tagasi 40 minutit), siis võib selliste pikamaa lendude jaoks arvutusvõimsusest kasu olla, et eriolukordades õigeid otsuseid teha saaks. Kosmosesse peaks SpaceX Dragon startima juba saabuval esmaspäeval, 14. augustil.

Põhjalikumalt pajatab Neowin ja HPE.

August 07, 2017

Anton ArhipovXRebel for standalone apps with embedded Jetty

XRebel was designed to work specifically with Java web applications. Currently, it relies on Servlet API to serve its UI. It has been tested with a range of application servers (Apache Tomcat, WildFly, WebSphere, and others) and some full stack frameworks as well, including Spring Boot and JavaSpark.

Sometimes however, you might want to use XRebel with a standalone Java process that is not using Servlet API. Hence, XRebel cannot display its embedded UI to render the data it collected from the application. What can we do about this? Well, XRebel is tested on Jetty and it works quite well there. Jetty is also often used as an embedded HTTP server. So why don’t we use this trick and embed Jetty into our standalone app to serve XRebel UI.

Here’s our example application:

<script src=""></script>

Just for the demonstration purposes, it executes an HTTP request every 3 seconds and reads the response. Quick and dirty.

To embed Jetty server we need is to add a dependency to Jetty container and initialize the server when the application starts. The required dependencies are jetty-server and jetty-servlet:

<script src=""></script>

Then we can start Jetty by adding the following code snippet into the static initializer of the class:

<script src=""></script>

To enable XRebel agent, you need to start the application with -javaagent VM argument pointing to the location of xrebel.jar. For instance:

-javaagent:/Users/anton/tools/xrebel/xrebel.jar -Dxrebel.traces.all=true

I also added -Dxrebel.traces.all=true VM property to enable tracing of non-HTTP activities by XRebel. Tracing of non-HTTP activities in XRebel is disabled by default, hence I need to add this parameter in order to see profiling data for the periodic tasks (if I wish).

Once I launch the application, it will boot up the embedded Jetty instance on port 8080. The standalone XRebel UI is deployed by the agent to /xrebel context. Hence, if we open http://localhost:8080/xrebel in the browser we will see the following:

As you can see, it is quite easy to use XRebel with the standalone apps with this little trick. Just start an embedded Jetty instance on some port and you will be able to see what is your application doing. Perhaps, you can spot a sneaky bug with it before it gets to production! :)

If you want to play with the example application, I have it at GitHub. Download XRebel and start the application with the appropriate VM arguments to enable the agent. It will be fun! :)

August 04, 2017

Raivo LaanemetsA set of questions

I'm often approached by potential clients wanting to build something. Usually they have a vague vision of the final result but not a concrete execution plan to reach it. My job is to reach the successful result but there are always constraints out of my control.

I do not want to drive the potential clients away but sometimes the overhead of arranging and attending meetings is not worth it once I discover that I won't be able to meet some of the project requirements or I won't fit into the existing team. This is a wasted time for both my potential client and myself.

The most common reasons for turning down a project have been:

  1. Timeline is unsuitable.
  2. The budget is way underestimated.
  3. Timezone makes communication hard.
  4. Type of technology (desktop, mobile, embedded, etc) is not my expertise.
  5. Gut feeling.


The most common reason is a timeline. A project should have a start and an end. It is often helpful to divide the project into shorter periods each with a definite length.

I always have a queue of projects. These projects pass many of the constraints I point out here and often have negotiated deadlines. A single project taking longer than planned is much worse than a project ending prematurely. A project running longer requires re-negotiation of every other project in the queue and forces me to break promises I have made to my clients. My clients will suffer and I do not want that to happen.


Estonia is not the most expensive country to live in but the difference with Western Europe countries is not 2-3 times, it's more likely 20-30%. The employment taxes are relatively high. On top of the personal income tax is the hefty 33% social tax, increasing the employee cost. Experienced developers are in high demand. Estonia is not a country to find cheap developers. Clients with a certain budget range are preferred. A budget is nearly always underestimated but knowing the budget helps to set expectations. Sometimes low-quality work is expected. This is not something I would like to do.


A software project cannot be carried out without effective communication. This is especially important in a remote project. Some time ago I had a remote project with a client in US West Coast. I shifted my workday to late hours and she shifted hers to early hours. This was very exchausting and took a heavy toll on the life outside the project. Eventually we gave up on this schedule but important questions had to often wait an answer a day which slowed down the project.


The choice of the technology is often more important in short projects. There is enough time in a 3-year project to learn whatever language or platform but you can't realistically create a mobile app in a week if you haven't built any before.

Couple of years ago I worked on a desktop application. We chose Qt as the application framework because it was cross-platform and had a suitable free license. Qt uses C++ and C++ is a complete beast to learn. I had no experience with C++. I was able to get the initial prototype out in 3 days by copy/pasting together source code from examples. In the following months I picked up proper C++ and added all those const qualifiers I had missed before.

One of my clients needed an Android app recently to be done in a week. I thought I give it a chance. I spent 3 days trying to install Android development tools. I always got some weird errors. Once I got it running the emulator turned out to be extremely slow and useless. The time pressure did not help either. I had to admit defeat. The project would have been doable in the timeframe by someone knowing the tools already.

The choice of the technology is also important for longer projects to ensure project continuity. Example of a such negative choice would be Flash which is currently being removed from browsers and is completely dead in 1-2 years.

Gut feeling

I must be able to envision myself using the product or service, getting a great value out of it. Is it actually possible to accomplish it given all the hard constraints? That is the ultimate indicator for me.

Throughout the career I have had only a single client who gave me too much information. Every day I received 3-4 page essays containing bug reports, weekly plans, long term future plans, what-if scenarios and dreams all mixed together. In every other case it has been the opposite.

Business ideas

I'm careful about discussing business ideas. Ideas are considered worthless but frequently the client has some existing business tied to the idea. The existing business can contain crucial details that are important for the project's financial success. These details are trade secrets and the client should not be put into a situation where these are to be accidentally revealed.

A not-go decision for a project does not mean that you should reject the client. I have had clients with multiple projects and I have proceeded with some and turned down some others.

Simple questions

Some time ago I was studying a frontent framework and wanted to make an app with it. I decided to make an app asking questions about a potential project. It is available here. Using the app does not mean you have to work with me. It contains simple questions with yes/no answers and does not ask anything about your business idea. You get a shareable link with the answers and a downloadable PDF file.

July 27, 2017

Raivo LaanemetsStack Review 2017

In 2017 the typical setup of my project stack has been:


  • Node.js 8+ with the available ES6+ features.
  • Express 4 with either EJS or JSX-based templates.
  • MySQL database accessed through the mysql package.

The applications are based on a common project template. I do not maintain the template as a separate project. It is improved through project to project and taylored to specific needs. The template covers:

  • Directory structure.
  • A set of always needed dependencies.
  • Better static file serving with cache busting and CDN support.
  • Trivial async/await wrapper for Express.
  • jsend-style API response generator.
  • Useful defaults and helpers for templates (default app title, date formatting, etc).
  • Sessions with signed cookies.
  • MySQL transactional async/await wrapper for queries.


  • Bootstrap 3.
  • Vue.js 2+ or React 15+.
  • Webpack 3+.

The Bootstrap framework provides the common approach to styling. Most web designers are familiar with it, even if not, the documentation is very good and accessible.

Both Vue.js and React are decent view libraries. I have used them both. Vue.js uses HTML as the templating language and is more suitable for projects where the majority of markup is coming from the backend and it needs interactivity. React wants to generate and fully control the markup itself.

I use Webpack for building browser-side JavaScript bundles. It has iterative compilation, and integrates really well with Babel.

Other things

I generally avoid non-web application projects as these are not my expertise. Web scrapping is one of the related topics. I have mostly used PhantomJS so far.

There are some problem domains that benefit from relational and logic programming with Prolog. For example, industrial planning is a such domain. Besides that, I have successfully used SWI-Prolog for developing web applications, including this blog. I currently maintain two Node.js/SWI-Prolog bridge interfaces.

My opinion on the development of mobile applications is to get them done by the platform expert. The Android development tools are extremely complex, iOS tools are a bit better but still require specific knowledge. Recently I have been playing around with DroidScript. DroidScript is a JavaScript-based development system for Android but I have not used it in a project much larger than a Hello World.

For a desktop application project I would pick Electron. Electron is a fusion of Node.js and Chromium runtimes. So far I have developed one project with it. The process is very similar to creating a web application and that is something I know very well.