Pinu planeet

August 22, 2016

Raivo LaanemetsWindows Git user SSH key location

I rarely develop on Windows but recently I had to generate a Windows installer for a Node.js console application. I used Git on Windows to access the project's codebase.

I could not find any documentation where to put my SSH private key. The repository did not support username/password authentication. There seems to be some confusion over the key location. Besides, it seems to depend on which git implementation you use and how you use it. I used it directly from the CMD terminal. At the end I figured out that you can just put the key file id_rsa or similar into the folder:

C:\Users\Account\.ssh

where Account is your user name.

This approach worked on Windows 7 and Git version 2.9.0.windows.1.

August 15, 2016

Raivo LaanemetsEvolus Pencil 3

Evolus Pencil 3 was released recently. It is a GUI prototyping tool, suitable for mobile, desktop and web applications. I have used it on some projects to quickly put together screen views. Evolus Pencil is Open Source and is built onto the Electron platform.

Main features

  • Page trees, subpages (good for reusing common background)
  • Page links (also work in the exported HTML output)
  • Flowchart symbols built in (with page links it's great to visualize user flow)
  • Lots of primitve elements (form controls, tables, and so on) in different styles
  • PDF and HTML exports
  • Works on Windows, Linux and Mac
Screenshot of Evolus Pencil 3

Download here.

August 09, 2016

Raivo LaanemetsWorking with ES5 and ES6 projects at the same time

ECMAScript 6 (ES6) was officially published a year ago but there is no native runtime that fully supports it. A year ago I wrote a small article on it but I have not yet started using it on most my projects over various tooling and runtime compatibility issues.

Some time ago I started working on a client project which had ES6 as one of the techical requirements. Our target runtime is Node.js version 6 but we still need to use the Babel transpiler to get support for ES6 modules. The modules are in a sad state as only the import/export syntax was specified in the standard but not the module loading process (the loader spec). The implementation details for Node.js and npm are currently open.

The setup has worked quite well for the project but has created an additional issue for me: language context switching. ES5 and ES6 are both JavaScript but are rather different if you want to utilise ES6 to its fullest (let, const, classes, destruction, arrow functions, modules). I still have to maintain a number of projects targeting older runtimes, sometimes in cooperation with other developers who do not want to hear anything about transpilers or anything else that breaks the simple edit/refresh workflow.

This has made us to use ESLint to avoid the accidental use of ES6 on older projects and to enforce ES6 where it can be used (prefer const/let etc.). ESLint is quite configurable. It now replaces jshint, a similar tool I used previously. ESLint is configured using a project-local .eslintrc file (see the full documentation). The example file for ES5 support is:

{
  "parserOptions": {
    "ecmaVersion": 5,
    "sourceType": "script"
  },
  // ... other entries
}

This will prevent the accidental usage of the new syntax in ES5 projects. For ES6, the ecmaVersion value has to be changed to 6 and the sourceType value to module (it is also possible to use ES6 without modules). In the actual ES6 project above we used additional check rules for ESLint. Most of them can be found in the canonical configuration project.

I still have some mixed feelings about ES6 due to lack of runtime support and I avoid ES6 syntax on projects whenever possible. I have been using Promises (a standardized object in ES6, shims available in older versions, the best productivity boost in ES6 in my opinion) a long time and I see that the language constructs like const and let help to communicate the intent of code. However, I do not see transpilers like Babel a good long-term solution and hope that the need for them goes away one day.

August 05, 2016

Raivo LaanemetsServer upgrade to RAID-1

Some days ago I took my home server offline to upgrade it. The server was running on a single hard drive so far. It has data backups but full restore from a backup would take too much downtime for some services.

The server was originally running as my home NAS (mostly for my main desktop backups) and for hosting the test environments of my clients projects. Recently the server started to run some partially critical automation apps.

I decided to set up RAID-1 with two disks to reduce possible downtime. When a disk fails in RAID-1, it can be quickly replaced and synced with the remaining disk. No OS and services would have to be reinstalled.

The upgrade process started a week ago. I made the list of running services and documented the whole system. I also tried out the RAID setup with a VirtualBox machine. I chose 2x1TB Western Digital Red disk drives for the hardware. The current 1TB Western Digital Black disk was left for backups.

I reinstalled the OS to get Debian Jessie, the old one was Wheezy. I used Legacy Boot and DOS partition tables for the new disks. I was not sure how well Grub handles RAID with GPT partition tables. Googling showed me some warnings. DOS partition tables are supposed work well for disks smaller than 2TB.

Largest chunk of the downtime was spent on copying the old data. I had about 570GB of files on the old disk. cp -p preserved all permissions and modification times (required for my backup system). I had set up all apps and services into the directory /files which meant I did not have to install and setup every one of them again. I just had to add specific include directives to supervisor, Nginx and Apache configuration files and reinstall Node.js and SWI-Prolog runtimes.

There were some things I did not fully think of before upgrading but were still necessary:

  • Notify ALL people about the upgrade. They can get worried when stuff is not running.
  • Install NTP to keep clock correct: apt-get install ntp.
  • Use the UTC timezone: dpkg-reconfigure tzdata.
  • Use nameserver 8.8.8.8 in /etc/resolv.conf to avoid local censoring.
  • Adjust /etc/ssh/sshd_config to keep keys consistent over upgrades (other tools will freak out otherwise when contacting the machine).
  • Make sure you re-add all custom cron entries after reinstall.
  • Realtek ethernet complains about missing firmware but still works. Later it screws up other machines connection for some reason. Actually installing the firmware fixes it: apt-get install firmware-realtek.
  • Use cp -p to preserve timestamps, users and permissions when copying the old files.

Everything (about 20 apps and services) worked again after 4 hours of downtime. The server has been rock stable for almost a week now.

Andrei SolntsevПочему разработчики должны быть хорошими тестировщиками

Это мой ответ на статью о том, почему разработчики не могут быть хорошими тестировщиками.

вы можете научить собаку нескольким трюкам, но вы уж точно не сможете научить ее летать.

Всё это полная ерунда! Собаку нельзя научить летать, потому что она физически по-другому устроена. А разработчики и тестировщики устроены ОДИНАКОВО!

По каждому приведённому пункту: если это могут тестировщики, то смогут и разработчики.

1. “Родительские чувства” к коду

Разработчики привязаны к тому, что они пишут. Может быть это прозвучит глупо, но очень сложно оценивать объективно то, что ты создаешь.

Вы могли обратить внимание на то, что нормальный родитель относится к своим детям наиболее требовательно. Я хочу, чтобы код - моё детище - хорошо работало, и чтобы ни у кого не было ни малейшей причины его критиковать. Поэтому я очень хочу найти в нём баги.

2. Акцентирование на позитивных моментах

Работа разработчика основывается на позитивных сценариях и их реализации в продукте. Чтобы работать тестировщиком нужно иметь диаметрально противоположный склад ума, так как тестировщики всегда акцентируются на негативных сторонах продукта нежели чем на позитивных. Следовательно разработчику нужно поменять склад ума, чтобы стать тестировщиком, а это не кажется столь реальным на мой взгляд.

Это правда, в период “созидания” я концентрируюсь на том, как СДЕЛАТЬ. Но я никогда на этом не заканчиваю. Потом наступает следующий период, когда я пересматриваю и улучшаю код, иногда по много раз. Перепроверяю и дописываю тесты. Критически думаю, а что там может СЛОМАТЬСЯ. Ведь я не хочу, чтобы моё детище критиковали.

Если разработчик после первого этапа считает дело сделанным, то это не разработчик, а ребёнок. Не надо по ним судить.

3. Работа на основе принципа упрощения сложных вещей

Рассматривание сложных сценариев с целью нахождения багов является одной из составляющих процесса тестирования. В двух словах мы берем простую вещь и придумываем как ее можно усложнить. Разработчики поступают ровно наоборот. Они берут сложный процесс или проект и разбивают его на более мелкие компоненты с целью найти решение проблемы.

Ровно то же самое, что п. 2. Вначале я упрощаю, потом усложняю.

4. Неспособность видеть мелкие детали на большой картине

Одним из основных качеств любого тестировщика является умение определить что не так на этой картине. Разработчики говорят, что у тестировщиков нюх на баги. Возможно это утверждение не так далеко от истины.

Это правда, такая проблема часто есть у разработчиков. Но если это могут сделать тестировщики, то могут и разработчики. У них точно такой же мозг.

Да не просто могут, а обязаны! Как иначе без видения общей картины они смогут хорошо спроектировать программу?

Если они сделают программу без учёта всего этого, а потом придут тестировщики и скажут, что не годится - это будет СЛИШКОМ ПОЗДНО! Придётся переписывать и потратить вдвое больше времени.

5. Недостаток опыта работы с типичными багами и программными ошибками

Все приходит с опытом, и знание типичных багов и программных ошибок не исключение. Опытный тестировщик видит форму и сразу начинает думать какие типичные баги там могут скрываться и что следовательно может быть протестировано в первую очередь.

Да с какой это стати недостаток? Почему? За всю свою карьеру я сделал такое множество багов, что опыта работы с ними хоть отбавляй.

Я думаю, разработчики всё это прекрасно могут, но обычно не хотят. Тупо лень. А лень потому, что они знают, что они не последний рубеж. За ними есть ещё обезьянки-тестировщики, которые налетят и за них их работу сделают.

В итоге всем хуже: разработчики сильно не напрягаются, не очень хорошо продумывают дизайн программы, не продумывают все крайние случаи, не пишут юнит-тесты. Проблемы растут лавинообразно, и тестировщики не могут их всех сдержать.

  • Тестировщики не могут обнаружить все неучтённые кейсы. Лишь некоторые.
  • Тестировщики не могут найти все баги. Лишь некоторые. Вы знаете, некоторые баги невозможно повторить.
  • Тестировщики-автоматизаторы не могут покрыть интеграционными тестами всё, что должно было быть покрыто юнит-тестами. Вы знаете, некоторые баги вообще невозможно покрыть интеграционными тестами, только юнит.

Мир катится в пропасть.

Андрей Солнцев

asolntsev.github.io

July 28, 2016

Raivo LaanemetsBeware of HTML5 drag-and-drop API

HTML5 drag-and-drop API does not work well sometimes and can have hard to debug behavior. Last week I worked on an HTML5 application for optimizing truck load plans. The app represents two sides of the truck and packages that can be dragged into the sides.

At first, I chose to use the drag-and-drop API. I had successfully used it before to re-order uploaded images. The API itself is quite complex and has lots of events to handle. The app ran very well on Chrome on my computer. However, the feedback from the client was not good. The movable part (also called the ghost image, moves together with the mouse) was blurred and unusable on her computers (both running Chrome 52 on Windows). The result can be seen on the screenshot:

HTML5 blurry ghost image

I was not able to replicate the issue on my computers. Chrome 52 on Windows 7 and Windows 8.1, and Chrome 50 on Linux rendered the ghost image a lot better. Googling for the issue revealed nothing. I finally found this article ("The HTML5 drag and drop disaster") which led me to abandon the native API altogether and implement the drag-and-drop functionality by other means.

The alternative implementation uses now a position: fixed element that changes its location with the mousemove event. The code became also a lot cleaner as there were only 2 events to handle: mousemove and click. The new code allows to move elements without having to hold down the left mouse button, reducing strain on hand and improving user experience. The new code also works on Firefox and IE11. A screenshot of the alterative solution (yellow box as the ghost image):

Drag-and-drop alternative solution

I'm still not exactly sure what went wrong in our application with the native API. Anyway, I would use it again only when I need integration with external services (like file uploads) or need to implement drag-and-drop between different windows or tabs. Otherwise the plain DOM and the mousemove event seem to work better.

July 27, 2016

Four Years RemainingSlide-a-maze

While writing the previous post I was thinking of coming up with a small fun illustration for Aframe. I first heard about AFrame at the recent European Innovation Academy - a team-project-based entrepreneurship summer school. The team called MemVee was aiming to develop an AFrame-based site which would allow students to design and view interactive "Memory Palaces" - three-dimensional spaces with various objects related to their current study topics, organized in a way that simplifies remembering things for visual learners. Although I have never viewed a "Memory Palace" as something beyond a fantastic concept from a Sherlock Holmes TV episode, I am a visual learner myself and understand the importance of such illustrations. In fact, I try to use and preach graphical references in my teaching practice whenever I find the time and opportunity:

  • In this lecture the concept of a desk is used as a visual metaphor of "structuring the information" as well as to provide an outline to the talk.
  • Here and here an imaginary geographical map is used in a similar context.
  • For the computer graphics course I had to develop some "slides" as small OpenGL apps for visualizing the concepts during the lecture. This has been later taken to extreme heights in the practical materials designed by Raimond-Hendrik, who went on to give this course (alongside with a seminar) in the following years. Unfortunately the materials are closed for non-participants (yet I still hope they will be opened some day, do you read this, Raimond?), but the point is that every single notion has a tiny WebGL applet made to illustrate it interactively.
  • Once I tried to make a short talk about computer graphics, where the slides would be positioned on the walls of a 3D maze, so that to show them I'd have to "walk through the maze", like in a tiny first-person shooter game. Although this kind of visualization was not at all useful as a learning aid (as it did not structure anything at all), it none the less looked cool and was very well appreciated by the younger audience of the talk, to whom it was aimed at.

I have lost the sources of that last presentation to a computer error and decided to recreate a similar "maze with slides" with AFrame. The night was long and I got sucked into the process to the point of making an automated tool. You upload your slides, and it generates a random maze with your slides hanging on the walls. It is utterly useless, but the domain name "slideamaze.com" was free and I could not resist the pun.

Check it out. If you are into programming-related procrastination, try saving the "mazes" generated by the tool on your computer and editing the A-frame code to, say, add monsters or other fun educational tools into the maze.

July 25, 2016

Four Years RemainingThe 3D Web

The web started off with the simple idea of adding hyperlinks to words within text documents. The hyperlinks would let the reader to easily "jump" from one document or page to another, thus undermining the need to read the text sequentially, page by page, like a book. Suddenly, the information available to a computer user expanded from single documents into a whole network of disparate, interconnected pages. The old-school process of reading the pages in a fixed order became the process of browsing this network.

Hyperlinks could be added to the corresponding words by annotating these words with appropriate mark-up tags. For example, "this sentence" would become "this <a href="other_document">sentence</a>". This looked ugly on a text-only screen, hence browsers were soon born - applications, which could render such mark-up in a nicer way. And once you view your text through a special application which knows how to interpret mark-up (a serious advance back in the 90s), you do not have to limit yourself to only tagging your text with hyperlinks - text appearance (such as whether it should be bold, italic, large or small) could just as well be specified using the same kind of mark-up.

Fast-forward 25 years, most documents on the web consist nearly entirely of mark-up - some have nearly no text at all. The browsers now are extremely complex systems whose job goes way beyond turning hyperlinks into underlined words. They run code, display graphics and show videos. The web experience becomes more graphical and interactive with each year.

There is one area, however, that the web has not yet fully embraced - 3D graphics. First attempts to enrich the web experience with 3D go back as far as 94, when VRML was born. It picked some traction in the scientific community - projects were made which used VRML to, say, visualize molecules. Unfortunately, the common web developers of the time mostly regarded VRML as an arcane technology irrelevant to their tasks, and the layman user would not care to install a heavyweight VRML plug-in in order to view a molecule in the browser. Now, if it were possible to make, say, an addictive 3D shooter game with VRML, things would probably be different - a critical mass of users would be tempted to install the plug-in to play the game, and the developers would become tempted to learn the arcane tech to develop a selling product for that critical mass. Apparently, no selling games were created with VRML.

It took about fifteen years for the browsers to develop native support for 3D rendering technology. It is now indeed possible to create addictive 3D games, which run in the browser. And although a whole market for in-browser 3D applications has been born, the common web developer of our time still regards 3D as an arcane technology irrelevant to their tasks. It requires writing code in an unfamiliar API and it does not seem to interoperate with the rest of your webpage well. The successors of VRML still look like rather niche products for the specialized audience.

I have recently discovered the A-Frame project and I have a feeling that this might finally bring 3D into the web as a truly common primitive. It interoperates smoothly with HTML and Javascript, it works out of the box on most browsers, it supports 3D virtual reality headsets, and it relies on an extremely intuitive Entity-Component approach to modeling (just like the Unity game engine, if you know what that means). Let me show you by example what I mean:

<a-scene>
    <a-cube color="#d22" rotation="0 13 0">
        <a-animation attribute="position"
                     dur="1000"
                     easing="ease-in-out-quad"
                     direction="alternate"
                     to="0 2 0"
                     repeat="indefinite">
         </a-animation>
     </a-cube>
</a-scene>

<iframe frameborder="0" height="150" src="http://feeds.feedburner.com/wp-content/uploads/aframe.html" style="margin: 0 auto; display: block;" width="300"></iframe>

This piece of markup can be included anywhere within your HTML page, you can use your favourite Javascript framework to add interactivity or even create the scene dynamically. You are not limited to any fixed set of entities or behaviours - adding new components is quite straightforward. The result seems to work on most browsers, even the mobile ones.

Of course, things are not perfect, A-Frame's version number is 0.2.0 at the moment, and there are some rough edges to be polished and components to be developed. None the less, the next time you need to include a visualization on your webpage, try using D3 with A-frame, for example. It's quite enjoyable and feels way more natural than any of the 3D-web technologies I've tried so far.

Ingmar TammeväliSelf made tuulegeneka staatori ja rootori “upgrade”

Seekord pikka teksti ei kirjuta, varasem kirjutis on siin http://www.stiigo.com/ideed/generaator/generaator_osa1.htm

Vahel ikka hea igapäevase programmeerimise kõrvalt midagi muud teha, sest siis saab aju ka 99% tööle panna😛

Rootoris: N52 klassiga magnetid, üldiselt tehke mida tahate, aga ärge neid magneteid omavahel kokku laske, sest siis võivad peened killud näkku lennata. Staatoris 1.6mm mähisetraat ja EPOt kasutatud. Rootoris kasutatud polüestervaiku, millesse lisatud talki, et tugevust juurde anda, samuti et kuivamisel ei kuumeneks üle. Talgiga see asi, et selle võiks enne atsetoonis ära lahustada.

IMG_0031

IMG_0033

IMG_0034

IMG_0039

IMG_0040

IMG_0042

IMG_0045

IMG_0081

IMG_0082

IMG_0084


July 23, 2016

Raivo LaanemetsClient IP in Express behind Nginx

Last week I worked on a web application that needed to restrict some pages and routes to specific IP addresses. This is considered security through obscurity but still works for simpler non-critical cases.

The application is reverse proxied through Nginx and does not see client's real IP addresses directly (it sees the proxy's). The actual client IP address is passed through the header X-Real-IP by using the proxy_set_header directive in the reverse proxy configuration:

location / {
    proxy_pass http://127.0.0.1:9091;
    proxy_set_header X-Real-IP $remote_addr;
    # ... rest of configuration
}

The app runs without a proxy in some configurations (development, test) and that requires IP check inside the app itself. Otherwise the check could have been implemented on Nginx as well.

At the beginning I started to look at the existing Node.js packages to obtain the client's IP. One of such packages was ipware. It seems to be a port from some other platform's similar library. It did not work for me. There seems to be some confusion over HTTP header normalization in Node.

Other such libraries seem to be quite complex as well and contain too much magic. I ended up using my own middleware that sets req.clientIP from the header when it's given. The IP address is taken from the client socket when the header does not exist. This is done with a middleware:

module.exports = function() {
    return function(req, res, next) {
        var realIP = req.get('x-real-ip');
        if (realIP) {
            req.clientIP = realIP;
        } else {
            req.clientIP = req.socket.remoteAddress;
        }
        next();
    };
};

The HTTP header normalization in Node is documented in the API reference:

Header names are lower-cased.

There is no transform that replaces dashes with underscores or does some other fancy stuff.

Security warning

In the current application I also added proper password-based authentication and authorization. The approach with IP aadress inside an header has a potential security hole: when the app is not behind Nginx then anyone can set their "IP address" by setting the X-Real-IP request header!

July 13, 2016

Raivo LaanemetsAndroid-based SMS gateway

Recently I worked on an automation project where we needed to receive SMS messages. This was not my first time to deal with messages. In one of the older projects we needed a functionality to send out messages ourself.

In both of these cases we used an Android app named very appropriately: SMS Gateway. It is more developer-friendly than many other similar applications as it allows to use HTTP requests directly instead of more consumer-oriented methods.

In the case of receiving messages, the app will make an HTTP request to the configured URL with the sender number and message as the URL query parameters. When configured to send out messages, it acts as a web server that you can query from the backend of your system.

The gateway also needs a phone to run on. In the last project we used an Alcatel Pixi 3. It is a cheap Android phone, costing near 50-60 EUR, but runs a relatively recent Android 4.x version. It is currently running 24/7 on the charger. It's also power effient as the charger does not heat up. The latter might be a problem with some other phones. The network connection is done over WiFi. So far the setup has ran for two weeks and has been rock stable.

July 11, 2016

Raivo LaanemetsEnding (online) fraudulent agreements

Online scams are quite common. Nigerian scam mails rarely work nowadays because everyone knows about them but there are other types of frauds that can trick some people and companies into harmful legal agreements. One of such cases is the European Business Number that I blogged about some time ago. The EBN letter is sent to companies. This avoids spam-related laws in many countries. For example, in Estonia it's OK to send spam to company public email address while it's not OK to send to an individual person address.

The EBN letter is sent by snail mail and looks like something from an official EU institution. It is not. It is even stated in the small print. The letter consists of two pieces. The first one talks about updating some registry entries which is supposed to be free. The second part has the actual form to enter the company data. It also contains the fine print with the actual terms.

I almost filled the form myself. I had a starting company selling IT services and I liked to get as much exposure as possible. A registry of companies sounded good. Thankfully, I read the fine print. 677 or something euros per year did not look very good. I have had my data and entries in some free catalogs and registries before and for my type of company it has not turned out to be very effcient, especially compared to other approaches like blogging and networking.

Unfortunately, some companies have responded to the letter and are now stuck with an useless agreement that only benefits one side. I have got some feedback to the original EBN article asking how to get rid of the agreement and whether you get sued if you do not pay them. I'm not a lawyer but I did some online research on it.

Estonian Chamber of Commerce and Industry (ECCI) has an article on these types of scams. The article is in Estonian and therefore not very useful for the most of my readers. It written by a lawyer and strongly advices to ignore such letters. It also tells what to do when you have already responded and started the agreement: you should contact the EBN registry and let them know that you end the agreement on the basis that the agreement is a fraud. Once you have done that, you should ignore all threats and invoices sent by the EBN fraudstrers.

There should be an equivalent to ECCI in your country. You should contact them if you want further information. They are likely to give the same suggestion to end the agreement. Or contact them anyway. We might get enough international pressure to end EBN once and for all. Paying EBN just gives fraudsters more resources to send the letter to more companies and trick them into something they do not want.

July 10, 2016

Raivo LaanemetsSlackware 14.2 released

Slackware Linux version 14.2 was recently released. I have been a Slackware user since 2003. The new release brings lots of updated libraries. For me it means a rock stable base system to run the latest versions of software that I need every day: a text editor, browsers, and some other common stuff. The Slackware team has done lots of good work and the release has been beta-tested for the last 5 months.

I have tried the new release in VirtualBox. It seems to work well, at least the installation with XFCE. I have used KDE so far but I'm considering XFCE if it proves to be stable enough. XFCE is supposed to have fewer moving parts than KDE but only the actual usage will tell whether it turns out to be stable enough for me.

The only minor issue I noticed was XFCE asking me whether I want to start with the default setup or with 1 panel. I did not understand the question at first, chose the latter, and it made the desktop a bit useless. I guess the choice makes sense if you are an existing XFCE user but the option should not offered to the new users. I got this fixed by deleting and recreating the user account. This is probably not Slackware's fault but an issue in XFCE itself.

I'm currently waiting for less busy times. The update might mean some downtime for me. At the moment I'm running 14.1 with a custom kernel on a very recent hardware. The system has lots of patches and tweaks applied and getting the new version running so well might take some time. Linux desktops are a bit rough by default anyway although the simplicity of Slackware allows me to tweak it to the perfect level.

July 09, 2016

Andrei SolntsevНастоящие пэдж-объекты

Недавно я писал о том, что на самом деле нельзя хардкодить.

Давайте рассмотрим этот принцип на примере пэдж объектов.

Классический пэдж обжект

У тестировщиков-автоматизаторов очень популярен паттерн “Пэдж Обжект”. Собственно, это самый популярный в мире объект поклонения после статуи Будды в Лэшане и Шестого Айфона.

Суть его в том, что все селекторы, по которым ищутся элементы на веб-страничке тестируемого приложения, выносятся… точно, в константы:

public class RegistrationPage {
  @FindBy(name = "firstName")
  public WebElement firstName;

  @FindBy(name = "lastName")
  public WebElement lastName;

  @FindBy(name = "birthday")
  public WebElement birthday;
}

Красиво, да? Такие пэдж объекты хороши тем, что один и тот же локатор не приходится дублировать в сотне разных тестов. Когда локатор меняется, его нужно поменять всего в одном месте.

А тесты не нужно часто менять, потому что они используют пэдж объект:

  @Test
  public void userCanRegister() {
    RegistrationPage page = new RegistrationPage(webdriver);
    firstName.sendKeys("Bruce");
    lastName.sendKeys("Willis");
    birthday.sendKeys("19.03.1955");
  }

Такой подход стал особенно популярен с лёгкой руки Мартина Фаулера, который выдумал Золотое Правило Автоматизаторов:

В тесте не должно быть ни одного локатора. Локаторы должны быть спрятаны в пэдж объектах. Если в тесте виден локатор - это негодный тест.

В чём же подвох?

Мы забываем, что меняются не только локаторы. Как я рассказывал в изначальной статье, меняется логика работы с элементами. Допустим, есть у вас сотня тестов, в которых есть такая строчка:

birthday.sendKeys("19.03.1955");

Да, типа хорошо, если меняется локатор - сотня тестов останется нетронутой.

Но что если этот элемент из <input> вдруг превратиться в контрол типа “календарь”? Знаете, в котором нужно сначала кликнуть иконку календаря, потом выбрать год, потом месяц, и только потом день.

Упс!

Теперь вам придётся перелопатить всю сотню тестов. Да ещё и разобраться хорошенько, в каких нужно выбирать год, а в каких не нужно. В каких нужно менять месяц, а в каких не нужно. И как эти тесты будут работать в последний день месяца или года.

Упс! Всё пропало, шеф!

Что же делать?

Вот тут и вступает в силу правило: выносить нужно не константы, а логику. В пэдж объекте должно быть не поле birthday, а метод

public class RegistrationPage {
  public void enterBirthday(String birthday) {
  
    $(By.name("birthday")).sendKeys(birthday);
  
    // а потом - вся логика календаря здесь
  }
}

Вот теперь действительно, если поменяется логика работы с элементом “birthday” (в том число локатор!), ни один тест не поменяется - только пэдж объект.

Понятное дело, в этом случае поля пэдж объекта должны быть приватными, чтобы никто не мог использовать их напрямую, а только через методы пэдж объекта.

Ну а в случае Selenide поля так и вовсе становятся ненужными, ведь гораздо легче использовать доллар (как в примере выше).

Вот это труёвый пэдж объект!

Контрольный в мозг

Давайте проверим, что мы понимаем друг друга правильно. Если вы согласны с приведёнными выше доводами, то вы тоже считаете, что аннотации @FindBy не нужны, PageFactory не нужна. И инифиализировать пэдж-обжект нужно только с помощью конструктора:

MyPage page = new MyPage();

Никаких фабрик, аннотаций и другой чёртовой магии. Она точно добавляет сложности и точно не решает никаких проблем.

А если уж по чесноку, я думаю, что пэдж обжекты вообще не нужны. Но об этом отдельная статья.

Автоматизируйте с умом, други, и не забывайте главное правило из книжки “Design patterns”:

Объект - это не данные и операции с ними (как учат в школе).
Объект - это поведение.

Андрей Солнцев

asolntsev.github.io

July 08, 2016

Andrei SolntsevЧто на самом деле нельзя хардкодить

Хардкод

Это страшное слово - хардкод. Все боятся это сделать, но иногда каждый из нас это делает.

Но я утверждаю, что хардкод в привычном нам понимании вовсе не так уж страшен, и на самом деле гораздо страшнее, когда в коде прописывают кое-что иное.

Так что же на самом деле нельзя хардкодить?

Классический хардкод

Все обычно считают, что

  • Нельзя хардкодить числа в коде! Надо вынести в константу.
  • Нельзя хардкодить настройки в коде! Надо вынести в файл с настройками (а у некоторых и в базу данных).

То есть если джуниор девелопер напишет в коде if (age >= 23), ему за это надо дать по рукам. Так обычно считается. Чтобы сберечь руки, он должен срочно вынести “23” в константу типа MINIMUM_LOAN_AGE.

Давайте разбираться в причинах

А почему плохо прописать в коде “23”?

Обычно называют две причины. Их втирают нам в сознание ещё с университетской скамьи.

  1. Когда нужно будет поменять “23” на “24”, её придётся поменять во многих файлах - трудоёмко.

  2. Само по себе “23” плохо читается. Что означает “23” - возраст, длину волос, объём бензобака? Почему именно 23, а не 22 или 24?

Почему эти причины не катят?

Эти причины настолько нам привычны, что мы даже не задумываемся, насколько они актуальны в наше время. Вы удивитесь, но не очень-то актуальны. Прямо скажем, они устарели. Смотрите сами.

  1. Во всех современных IDE очень легко поменять “23” на “24”. Одной кнопкой. Ctrl+R -> Enter. Всё. Хоть у тебя в проекте три файла, хоть три миллиона.

  2. Да, “23” плохо читается. Но часто при вынесении в константу оно не становится более читаемым. Да, название константы MINIMUM_LOAN_AGE говорит о том, что это минимальный возраст, с которого можно брать кредит. Но и выражение if (age >= 23) в методе canRequestLoan() говорит ровно о том же ничуть не хуже.

А почему именно 23, почему не 22 или 24 - это всё равно непонятно. Чтобы это узнать, в наше время легче заглянуть в историю изменений (git -> annotate) или в тесты (Ctrl+Shift+T) - с нашими IDE это легко.

Ладно, ладно

Я знаю, вас переполняют эмоции. Вы хотели бы вбить мне в грудь осиновый кол за такую ересь. Но потерпите, сейчас мы дойдём до главного.

Конечно, всё-таки выносить такие штуки в константы иногда полезно.

НО

Я хотел сказать, что самый страшный хардкод - это вовсе не константы.

А что же - самый страшный хардкод?

Вглядитесь внимательно в это выражение. Все обычно думают, что самый страшный хардкод - это вот: Hardcode

Но вглядитесь, неужели это действительно самое страшное место? Оглядитесь вокруг, не притаилось ли рядом что-то более опасное? На самом деле самая страшная часть - это вот: Hardcode

Потому, что вот её-то поменять во всём коде на порядок сложнее. Когда однажды выяснится, что для получения кредита нужно стать старше 23 лет, да ещё и найти работу, нам придётся найти в коде все места, где прописано if (age >= 23) и поменять их на if (age > 23 && employed). Но как найти все знаки все знаки >=? Их же тысячи! Вот это ручная работа на столетия!

Но самое страшное, что в коде могут быть и выражения вида

  • if (!(age < 23)), и
  • if (23 > age),

и даже такие места, которые совсем нереально обнаружить:

if (age < 23) {
  // 100500 строк кода
}
else {
  // можно получить кредит
}

Что же делать?

Вот почему важно выносить не константы, а логику. Важно следить, чтобы любое знание в коде было прописано ровно в одном месте. В данном случае - знание о том, в каких случаях клиент может взять кредит (то самое >= 23) должно быть вынесено в отдельный метод.

Например, так:

class User {
  public boolean canRequestLoan() {
    return age >= 23;
  }
}

И все остальные места должны использовать этот метод. Кажется тривиальным? О нет. Если это знание действительно в одном месте, зачем вы так рьяно хотите вынести “23” в константу?

Упростим

Всё ещё кажется тривиальным? Ок, давайте упростим пример. Забудьте 23. Пусть будет 0.

Я уверен, в вашем коде миллион таких мест:

  if (account.balance > 0) {
    // могу сделать платёж
  }

И я таких видел миллион. if balance > 0 прописан и на странице платежей, и на странице кредитов, и депозитов, и т.д.

Но однажды приходит новое требование: клиент не может сделать платёж, если на его счёт наложен арест. Нам приходится добавить условие типа такого:

  if (account.balance > 0 && !account.arrested) {
    // могу сделать платёж
  }

Но тут… опачки. Оказывается, что в десяти местах прописано if (balance > 0), в ещё двадцати - if (balance <= 0), а в грёбаном яваскрипте и вовсе if (account.balance).

И вот тут-то начинаются проблемы. Все эти места нужно анализировать отдельно. В некоторые из них нужно добавить && !arrested, а в некоторые не нужно - ведь там речь идёт не о платежах.

Я не придумываю, это абсолютно реальный пример.

Юнит-тесты

Очевидный плюс вынесения логики в методы - её легко тестировать.

Поначалу этот тест кажется даже избыточным и даже бесполезным:

  public class AccountTest {
    @Test public void canMakePaymentIfHasMoney() {
      assertTrue(new Account(1).canMakePayment());
      assertFalse(new Account(-1).canMakePayment());
    }
  }

Но всё меняется, как только добавляются ньвые требования:

  public class AccountTest {
    @Test public void canMakePayment_ifHasMoney_and_notArrested() {
      assertTrue(new Account(1, false).canMakePayment());
    }
    @Test public void cannotMakePaymentIfHasNoMoney() {
      assertFalse(new Account(-1, false).canMakePayment());
    }
    @Test public void cannotMakePaymentIfArrested() {
      assertFalse(new Account(1, true).canMakePayment());
    }
  }

Пэдж обжекты

Всё ещё кажется, что для разумных людей это очевидные вещи?

Тогда посмотрите на пэдж обжекты - воплощение константного антипаттерна во всей красе! Миллионы людей выносят локаторы в константы и даже не задумываются, что что-то здесь не так…

Андрей Солнцев

asolntsev.github.io

Andrei SolntsevНастоящие пэдж-объекты

Недавно я писал о том, что на самом деле нельзя хардкодить.

Давайте рассмотрим этот принцип на примере пэдж объектов.

Классический пэдж обжект

У тестировщиков-автоматизаторов очень популярен паттерн “Пэдж Обжект”. Собственно, это самый популярный в мире объект поклонения после статуи Будды в Лэшане и Шестого Айфона.

Суть его в том, что все селекторы, по которым ищутся элементы на веб-страничке тестируемого приложения, выносятся… точно, в константы:

public class RegistrationPage {
  @FindBy(name = "firstName")
  public WebElement firstName;

  @FindBy(name = "lastName")
  public WebElement lastName;

  @FindBy(name = "birthday")
  public WebElement birthday;
}

Красиво, да? Такие пэдж объекты хороши тем, что один и тот же локатор не приходится дублировать в сотне разных тестов. Когда локатор меняется, его нужно поменять всего в одном месте.

А тесты не нужно часто менять, потому что они используют пэдж объект:

  @Test
  public void userCanRegister() {
    RegistrationPage page = new RegistrationPage(webdriver);
    firstName.sendKeys("Bruce");
    lastName.sendKeys("Willis");
    birthday.sendKeys("19.03.1955");
  }

Такой подход стал особенно популярен с лёгкой руки Мартина Фаулера, который выдумал Золотое Правило Автоматизаторов:

В тесте не должно быть ни одного локатора. Локаторы должны быть спрятаны в пэдж объектах. Если в тесте виден локатор - это негодный тест.

В чём же подвох?

Мы забываем, что меняются не только локаторы. Как я рассказывал в изначальной статье, меняется логика работы с элементами. Допустим, есть у вас сотня тестов, в которых есть такая строчка:

birthday.sendKeys("19.03.1955");

Да, типа хорошо, если меняется локатор - сотня тестов останется нетронутой.

Но что если этот элемент из <input> вдруг превратиться в контрол типа “календарь”? Знаете, в котором нужно сначала кликнуть иконку календаря, потом выбрать год, потом месяц, и только потом день.

Упс!

Теперь вам придётся перелопатить всю сотню тестов. Да ещё и разобраться хорошенько, в каких нужно выбирать год, а в каких не нужно. В каких нужно менять месяц, а в каких не нужно. И как эти тесты будут работать в последний день месяца или года.

Упс! Всё пропало, шеф!

Что же делать?

Вот тут и вступает в силу правило: выносить нужно не константы, а логику. В пэдж объекте должно быть не поле birthday, а метод

public class RegistrationPage {
  public void enterBirthday(String birthday) {
  
    $(By.name("birthday")).sendKeys(birthday);
  
    // а потом - вся логика календаря здесь
  }
}

Вот теперь действительно, если поменяется логика работы с элементом “birthday” (в том число локатор!), ни один тест не поменяется - только пэдж объект.

Понятное дело, в этом случае поля пэдж объекта должны быть приватными, чтобы никто не мог использовать их напрямую, а только через методы пэдж объекта.

Ну а в случае Selenide поля так и вовсе становятся ненужными, ведь гораздо легче использовать доллар (как в примере выше).

Вот это труёвый пэдж объект!

Автоматизируйте с умом, други, и не забывайте главное правило из книжки “Design patterns”:

Объект - это не данные и операции с ними (как учат в школе).
Объект - это поведение.

Андрей Солнцев

asolntsev.github.io

July 04, 2016

Raivo LaanemetsMake npm 3 play nice with Watch

The npm package manager has a progress bar that sometimes confuses other tools. Many of my recent projects use npm as a build tool. Watch is a periodic execution utility and I have used it with Makefiles to rebuild bundles during development. When I attempted to use it with npm, the output started to drift away, line by line. Debugging took me to npm's GitHub issues list where I found that this could be caused by the progress bar. The bar itself was not visible for me. The bar can be disabled but it is not well documented how to do it.

The solution is linked from this issue and is simply (works only in a recent npm 3 version!):

npm config set progress false

That will disable the progress bar and makes the Watch output not drift away anymore.

June 25, 2016

Raivo LaanemetsA new logging infrastructure with rsyslog

Last week I have been migrating logging from Papertrails to my new setup. Papertrail was nice for some time but is a bit costy for larger amount of logs and is especially costy for longer-term log storage.

Monitoring

My main use case of logging has been error and intrusion monitoring. The minor use cases have been visitor analytics (usually handled by Google Analytics), performance monitoring (call logs with execution times) and audit logs (whether an action was performed or not).

Once a good logging infrastructure is in place, specific monitors can be built onto it. For example, I have a couple of shell scripts, executed periodically by cron, that check various services and write PASS/FAIL result to a log file (and now directly into rsyslog). The logs are watched for specific patterns (containing a string "FAIL" in this case) and alerts are automatically sent to my mail address.

Debugging

I have found it very effective to debug web applications by filtering and correlating different logs (web server access log + application stderr log). Doing this with basic text tools (grep etc.) is not easy and not even possible when one of the log files does not contain timestamps.

Centralizing

A one way to solve logging and monitoring is to log and monitor at each individual machine. There are tools like logwatch for this. I did not find this effective. While alerts work, you still need to grep and match multiple logs separatedly to debug anything.

Possible solutions

I looked through various self-hosted and cloud alternatives to Papertrail:

  • AWS CloudWatch (cannot add non-AWS sources, useless to me)
  • Loggly (cloud, too expensive)
  • Logentries (cloud, too expensive, used them before Papertrail)
  • ELK stack (self-hosted, looks too complex to work reliably, written in Java)
  • Fluentd (self-hosted, looks like syslog reimplementation, no UI for browsing logs)
  • Graylog (self-hosted, looks too complex to work reliably, written in Java)
  • Splunk (self-hosted but licenses might cost a lot)

Centralized logging requires a common protocol. All the solutions above, excluding CloudWatch, including Papertrail, support the syslog protocol. Papertrail even maintains a custom syslog-based log forwarder. This made it easy to test out the solutions (no need to install lots of fancy daemons!).

Almost every Linux installation has a syslog-compatible log agent already installed. There is a high chance that it is rsyslog. It comes by default on Debian, Ubuntu, RedHat, etc. Playing around with rsyslog while configuring it to forward logs to those services above gave me an idea that rsyslog can be used itself to collect logs.

rsyslog

The rsyslog description is quite broad and does not describe a particular use case. It is described as

swiss army knife of logging

It has lots of features. Grasping everything took lots of reading. It is able to do the following (and a lot more but this is related to our use case):

  • Forward logs to remote machines (securely, over TLS).
  • Watch log files (and not break with log rotation).
  • Assign custom tags to log files.
  • Store logs in MySQL (my choice for storing logs).
  • Do all of this in very performant manner.

What it cannot do:

  • Stateful alerts (i.e on 2 events in last hour).
  • Browse logs (UI).
  • Remove old logs (depends on storage).

I have built a custom analytics application before and knew exactly what to do once I have logs in the database. So the plan was to get all machines to forward logs to the central rsyslog, make it save logs into the MySQL database, and then throw together a small app that implements the alerts, log browser and custom log retention.

Configuration issues

Rsyslog has one huge downside: its configuration format depends on the version. There is the legacy format (documented somewhere at all?) and the new RainerScript syntax. Many of my machines are running Debian Wheezy that has rsyslog 5.x and supports legacy syntax mostly (this depends on the specific rsyslog plugins too!). I have stuck with the legacy syntax for now for the sake of consistency. I will use the new format for new machines, running Debian Jessy at least. The documentation for the legacy format seems to come mostly from the documention of specific rsyslog modules.

Secure forwarding (client)

The secure log forwarding uses transmission encryption through TLS. It is documented here. The TLS setup uses self-generated CA (root) certificate. The official guide generates separate certificate to each machine and verifies individual machines. It is also possible to setup a single certificate and make server/clients verify only the certificate (as used here). This still provides encryption for log transmission.

I also made these decisions in the client configuration:

  • Only forward local7 facility (my own set of files).
  • Discard forwarded messages locally (fewer files with the same data).

I also added the $LocalHostName directive at the top of the configuration file. This makes sure that the forwarded messages contain the actual hostname, not something generic like localhost.

Receiving messages (server)

The server TLS configuration is similar to the client configuration. The MySQL configuration is automatically set up by installing rsyslog-mysql on Debian. ALL logs from rsyslog (from both local and remote sources) will be stored in the database now. As I did not want to log everything, I made a change to rsyslog MySQL configuration to only store messages from the local7 facility.

Watching files

Rsyslog contains a module to watch log files. Warning! It is tricky to get the configuration (you need legacy directives on Debian Wheezy!) right if you have many files to watch. My busiest server has 42. In order to avoid errors I wrote a small Node.js script to validate the configuration. It checks for each file block:

  • File's facility is set to local7.
  • File's custom tag is set and is unique.
  • File's state file is stat-<tag>.
  • File actually exists.
  • File's severity is either info or error.
  • The file monitor start directive exists.

Errors like non-unique state files cause missing logs.

What I watch

For each app I watch:

  • Each processes stdout and stderr log.
  • Audit log (if it has).
  • Web server access log (if web app).
  • Web server error log (if web app).

All logs for a the app are grouped by the tag prefix <myapp>-.

Besides app logs I also watch:

  • Machine's supervisord (I use it as a process manager) log.

And globally:

  • Machine ping logs (machine uptime log).
  • Logged status codes of a list of URLs (app uptime log).

I currently do not watch/forward SSH login failures, mail arrival, etc. I have no idea what exactly to do with these. I do not have password login for SSH so that brute force attach is not going to get someone into the system.

Alerts

I built my own application to implement alerts. It works by chunking/stream processing the log table by primary key (autoincremented - ordered by time!). This makes it very efficient. No log line is polled twice by a single alert. I have not yet released the application but I plan to do it sometimes. I have currently implemented pattern alerts with mails (with custom throttle time for sending mails) but I also want to implement rate (X lines per sec) and idle (no line received in certain period) alerts.

Performance

I tested the system with message rate of 5k per second (I usually get 10-100 per second). MySQL showed some high iowait with 5k rate. It went away after switching to periodic flush (from flush after commit). The logs table uses InnoDB engine. This can be configured by setting

innodb_flush_log_at_trx_commit = 0

This also means that 1 second of transactions can be lost on server crash. That was fine for me. I also looked at switching to MyISAM table engine but with alert reads and heavy concurrent inserts it might perform worse.

I have added two indexes to the table. One on SysLogTag and the other on FromHost. The average row size for 1.7M rows is 290 bytes. The average message length is 125 bytes. This makes overhead 2.32 (it is actually better as this does not consider metadata such as tag, timestamp and host).

Log browsing

I have built a simple Node.js app for log browsing. It started from live tail view. Live tail and search is implemented by a single simple SQL query. Live browsing is very fast, a typical request from web app to display live tail for "Everything" takes < 10ms. For search it depends how many logs it actually has to scan. Good performance came as a suprise and I'm very happy with it.

The app groups different logs by tag prefix. The currently displayed or searched group is selected by a two-level dropdown. The first level is machine name and the second level is the app name. I currently have 8 machines and about 30 apps.

I plan to release the log browser, rsyslog conf checker, and the alert system as Open Source at some point of time. I will write a longer post on it. Until that time comes, the system will be extensively used by me and my clients for monitoring various projects.

June 22, 2016

Raivo LaanemetsALT Linux text-based live USB for backups

Last week I needed to make a backup dump from my main desktop disk. The hardware is pretty new and not stable with many live CD/USB images. The Skylake CPU has built-in GPU and everything seems to crash when trying to initialize it. I use my Nvidia PCI-E graphics card daily with proprietary binary drivers on Slackware and it's very stable. Unfortunately, live CD/USB images do not contain these drivers, try to initialize the built-in GPU and crash. I think it's possible to initialize and use the built-in graphics but it requires very very recent Xorg drivers and libraries. These are missing, too.

This led me to look for a text-based live system that I can use solely for the backups. The system would need to have the following features:

  • Bootable from USB (I haven't had optical drive for many years).
  • Able to initialize network connection.
  • Text-based boot that does not require hard-to-remember long boot options.
  • Supports NVMe disks.
  • Comes with the standard set of disk tools (dd, tar, etc).

This rules out Debian Live (does not boot, USB image uses DOS partition table, MSI boot selector only recognizes GPT USB disks), Ubuntu Live (cannot select text mode, graphical boot is unstable), Slackware Live (cannot select text mode, graphical boot comes up with a blank screen).

I also noticed that many Live systems come with an older kernel (pre 4.x), including SystemRescueCd. After looking through these and many other live systems, I found an usable system provided by ALT Linux.

ALT Linux

ALT Linux is a Linux distribution with multiple live and install images. It also has a nice text-based rescue image. It boots straight into the text mode and has a very recent kernel (4.4.x). It does not crash and lacks unneeded bullshit. All the standard tools are there.

I use the following commands to create my disk backups.

Select the Estonian keyboard layout:

loadkeys et

Initialize the network connection (dhcpcd forks to background):

dhcpcd eth0

Create mountpoint for NAS:

mkdir /mnt/backups

Mount NAS over NFS:

mount 192.168.1.200:/backups /mnt/backups

Copy the whole disk:

dd if=/dev/nvme0n1 of=/mnt/backups/desktop-`date -I`.img bs=128K conv=noerror,sync

Then, to watch the progress on the other terminal (switch with Alt+F2):

watch stat /mnt/backups/desktop-`date -I`.img

This will show the dump file size periodically.

I usually take these whole disk dumps once per 2-3 months or on special occasions before and after large changes in the system.

ALT Linux worked much better for this than any other Live CD based solution. GUI-based live systems provide no value here as the commands above are specific enough. Text-based system is just better to work with in critical tasks and has a smaller number of moving parts to break and crash.

June 18, 2016

Raivo LaanemetsHaskell Platform on Slackware 14.1

I needed to use ShellCheck and it requires Haskell/Cabal to be installed. After installing the Haskell Platform I received an error:

$ cabal install shellcheck

cabal: The program 'ghc' version >=6.4 is required but
the version of /usr/local/bin/ghc could not be determined.

Trying to execute /usr/local/bin/ghc:

$ /usr/local/bin/ghc

/usr/local/haskell/ghc-8.0.1-x86_64/lib/ghc-8.0.1/bin/ghc:
error while loading shared libraries: libtinfo.so.5: cannot
open shared object file: No such file or directory

Which is interesting, considering it's not even a dynamic executable:

$ ldd /usr/local/bin/ghc

not a dynamic executable

Anyway, the error can be fixed by symlinking ncurses library to the missing library (do as root):

ln -s /usr/lib64/libncurses.so.5 /usr/lib64/libtinfo.so.5

The command is for the 64-bit version. The 32-bit Slackware should have paths starting with /usr/lib. Cabal started working after this fix and allowed to install ShellCheck successfully.