Jekyll2022-09-02T10:11:16+00:00https://epic.blog/feed.xmlThe epic.blogThinking about tech/problems and solutions.
Becoming S/W Engineer2022-09-02T09:00:00+00:002022-09-02T09:00:00+00:00https://epic.blog/programming/2022/09/02/becoming-software-engineer<p>I was invited to speak to an auditorium of young high school students not so long ago. A few hundred teens, full of energy and enthusiasm and with a plan to become software engineers. Well. A few of them had a plan. Some were there as they had some time to spare before imminent time for lunch.</p>
<p>Without further ado, I took upon the challenge of giving them an overview of what it takes to become a software engineer and what traits they should develop to be successful players in the industry, given that it’s now more than 25 years since my first lines of code. I thought about my experience and the world I was in when I started building and learning about stuff and things.</p>
<p>There was no point in explaining how the internet was a new and shiny (+shitty) yet magical thing back then and how there was no StackOverflow, no GitHub, no Torrents, no broadband and phones were all wired up and only for calling. I only focused on a few key competencies, traits and findings that I would consider consistent and most valuable throughout my career. Also, I wanted to identify traits I cherish when looking for new colleagues or team members.</p>
<h1 id="curiosity">Curiosity</h1>
<p>This one is the most obvious and likely also the most overlooked. Especially in the early days, be curious! Explore. Don’t be afraid to try many different things. When you master something, don’t just stay there. Learn the tools that surround you; learn the tools that you use. Learn about the tools that your friends use. However, some might say that that’s not the right path to specialisation. I agree - regardless, especially early in your career, you need to broaden your understanding of the ecosystem. Start with basics, continue with frameworks and systems, toolsets and networking.</p>
<p>I would also argue that you need to embrace and understand that everybody was at the point in their career where they didn’t know much about anything. Not knowing what you don’t know can be considered a considerable challenge. To battle that - be brave. Be brave to ask questions, no matter how silly they might be. Put effort into doing your research and practice formulating your questions in a way that people can help you navigate. <a href="https://dontasktoask.com/">Don’t ask to ask. Just ask!</a></p>
<p>Find the pleasure in discovering new things, and don’t stop there. Go deeper and never stop! Try to understand how things work. And if you can’t. Try to break them. Figure out what those black boxes do. Don’t be afraid to open them up and blow them into pieces. Have you ever crashed a Linux kernel by loading your own custom faulty kernel module? I have. Do you know what happens when you do that? I do. How do you build a Roblox game? I have no idea. Yet.</p>
<p>Oh, and modern education has this beautiful way of overloading you with a bunch of stuff that might be complete garbage. Don’t allow that to put you off or temper your curiosity.</p>
<h1 id="focus">Focus</h1>
<p>Although this one might be a bit contradictory to the curiosity. I believe that focus is one of the toughest ones to master today. The world we live in now is skewed, so everybody is racing for your attention, time and peace of mind. If you remove all distractions that bombard you daily and hourly, you might get that precious, calm time to be curious and do the legwork. Leg work is needed to achieve anything. I advise you to set goals in your life and for your projects. Long-term, short term and something in between. Then evaluate yourself against them. Be critical and try to be objective about your ability to achieve the goals that you’ve set for yourself.</p>
<p>One good piece of advice to focus more on yourself is to separate time so that you have time for consuming and reacting to stuff and then another part. Time to produce, create, build. Establish clear separation. Good division of these two is usually physical.</p>
<p>There are a lot of techniques, tools and ways that you can use to learn how to be more focused. One that helps me is introducing some routine with physical activity in your life. Running, cycling, walking, MMA, playing the drums, whatever… and make sure it’s fun! Nothing sucks more than doing sports that aren’t fun.</p>
<h1 id="perseverance">Perseverance</h1>
<p>Software engineering, or engineering in general, is all about solving problems. Yes. Building things means you will have to solve problems, a lot of them. Your long-term value will be determined by your ability and experience when solving problems effectively and if you are lucky with technology. But there is also math, physics, philosophy, art and sometimes silly things such as paper, pencil or hours of Zoom calls with people who don’t speak your language.</p>
<p>You will fail! A lot! Most of the time, you will fail. Do not allow yourself to be crushed by the emotions that come from that. It is part of the job. Embrace it. The only way you can become a good problem solver is to solve many problems; that means you’ll have to expose yourself to many of them.</p>
<p>Hey, you’ll start looking for problems to entertain yourself and learn new technologies at some point. The industry calls these “side projects” and “toy projects”; some might even go as far as saying that they are “side hustles”. One thing is also sure about these. Your next boss and your colleagues will appreciate them, especially as it will allow them to look into your interests, skills and ability to solve challenges.</p>
<h1 id="community">Community</h1>
<p>The world is a vast place, and it has a lot of people in it. They don’t often speak your language, share your values or live near you. However, technology has connected all those people. Think of the world as this giant connected network of beings. Then imagine that these connections are tools for creating teams, groups, communities, and tribes. You can belong to multiple tribes. You can find tribes. You can even create your tribes.</p>
<p>When you hit on a problem, when you need support or when you need inspiration. Look into these tribes. There are millions of people who are just as eager as you to learn things, master things and explore things you. You are not alone. Talk to them.</p>
<p>If it’s a good community, they will accept you. They will help you, and they will welcome you. Being a good and respectful tribe member will pay off. I promise.</p>
<h1 id="passion">Passion</h1>
<p>This one is often oversold, and especially if you are a young person, it can be very frustrating and soul-cruising to be burdened with a quest to find your passion. If you find your passion, your calling, especially early on. Good for you!</p>
<p>However, I would consider that to be a very rare thing. I believe that to value something and become good in something or even master something, you need to spend time and effort on it. You will need to evolve discipline and do the shitty things repeatedly before you become good and before things are fun.
I would advise you to combine the abovementioned things and create a feedback loop that will give you a sense of accomplishment, satisfaction and progress. Rather than doing tremendous long and huge things. Do a few smaller ones repeatedly. Find excitement and joy in accomplishing smaller tasks. Completed tasks accumulate and lead to immense success.</p>
<h1 id="summary">Summary</h1>
<p>This is internet advice. Written by an old guy. I wouldn’t have read it when I was young.</p>
<p>However - I sure would like someone to have told me… now. Sometimes you need to discover things for yourself. Hopefully, you’ll have a lovely journey and fun along the way. I sure did.</p>
<p>P.s.: If you’ve liked this thing, please share it and let me know what I’ve missed in the comments section below.</p>
<p>Good luck!</p>
<p><br /></p>
<blockquote>
<p>This article was also published on my LinkedIn profile as an <a href="https://www.linkedin.com/pulse/becoming-sw-engineer-oto-brglez/">“Becoming S/W Engineer”</a></p>
</blockquote>I was invited to speak to an auditorium of young high school students not so long ago. A few hundred teens, full of energy and enthusiasm and with a plan to become software engineers. Well. A few of them had a plan. Some were there as they had some time to spare before imminent time for lunch.The Compression Puzzle Challenge2022-03-04T09:00:00+00:002022-03-04T09:00:00+00:00https://epic.blog/programming/2022/03/04/compression-puzzle<p>Life is short, exciting and full of small events. Small events can change your daily life and sometimes inspire you beyond your wildest imagination. This is one of those stories when a small and neat idea took off and challenged people beyond my or everyone’s imagination…</p>
<p>My dear friend <a href="https://www.linkedin.com/in/davidlicen/">David</a> brought this small and exciting coding challenge/puzzle to our hackerspace a few weeks ago. A challenge that is, in essence, brutally simple.</p>
<blockquote>
<p>Write a program that will take a string “AAABBAAC” and compress it into its compressed form “3A2B2A1C”.</p>
</blockquote>
<p><img src="/assets/2022-03-04-compression-puzzle/compression-puzzle.png" alt="The Compression Puzzle" /></p>
<p>Now. If you’ve ever been to a place where talent is in abundance, where people are creative and where nothing is impossible, you’ll correctly conclude that we had three solutions in the next 3 minutes and that in the next hour, we had five more.</p>
<p>Ten solid solutions in an hour… more than enough, some might say. Done. Nah,… <strong>We can do better.</strong></p>
<h2 id="level-up">Level up</h2>
<p>One of the most appreciated and head to thought virtue to teach a creative person is for sure the “curiosity”, and this little trait that some people have and some don’t is hard to cherish. Make no mistake. It’s worth nurturing. It is also one trait that makes some of the best engineers best at what they do.</p>
<p>As soon as we put our solutions together, the discussions started. Looking at one’s solutions to understand what’s the approach became exciting. And to build on this momentum, people started replicating methods in more languages and began looking for inspiration outside of our loop.</p>
<p>The next logical step was to put it on <a href="https://github.com/otobrglez/compression-puzzle">GitHub</a>, define some <a href="https://github.com/otobrglez/compression-puzzle#basic-rules">basic rules</a> and slowly scale up the operation and the challenge to more people.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/basic_rules.png" alt="4 basic rules of "the compression puzzle"" /></p>
<p>This could be opened up to the “whole universe”. But perhaps opening to everyone would also challenge no one. So I decided to pick the very best engineers from my network that I know and have worked with in the past to give this challenge a go. My “selection” was primarily based on languages that might be appealing to wider audiences and might have mechanisms and capabilities that would make solving the given task easier…</p>
<h2 id="basic-approaches-and-beauty">Basic approaches and beauty</h2>
<p>At this point, I could break down a couple of solutions and compare the approaches that the engineers took and compare these approaches across programming languages. I could also benchmark each solution against the other. I could benchmark solutions within languages, benchmark on small data sets, or find the solution with the minimum memory footprint but the best FP approach. That would all for sure be an exciting comparison. However, I would like to emphasise the elegance of solutions and the readability of the implementation. As I’m pretty sure by now that you, dear reader, already have a solution of your own in your mind.</p>
<p>Let’s start by appreciating three <strong>Python</strong> solutions submitted by <a href="http://github.com/brodul">Andraž Brodnik</a> and <a href="https://github.com/lknix">Luka Kacil</a>. First, one uses <a href="https://book.pythontips.com/en/latest/mutation.html">mutations</a> <a href="https://realpython.com/python-recursion/">basic recursion</a> and is one of few submissions where the author took time to <a href="https://www.simplilearn.com/tutorials/python-tutorial/comments-in-python#what_are_the_advantages_of_using_comments_in_python">comment</a> on the code. The second one uses Python’s powerful <a href="https://docs.python.org/3/library/itertools.html">intertools</a> library for “efficient looping” and its <a href="https://docs.python.org/3/library/itertools.html?highlight=groupby#itertools.groupby">groupby</a> function. Although there are many more Python solutions now in the collection, I’m highlighting Luka’s “<a href="https://chrispenner.ca/posts/python-tail-recursion">tail-recursion</a>” solution as a third example. It also highlights the author’s consideration of challenges associated when dealing with recursion.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/python_round.png" alt="Python solutions by Andraž Brodnik and Luka Kacil." /></p>
<p>Looking into the abyss with JavaScript should always be on the table. So it makes also sense to see how we approached one of the most popular languages on the planet. In the first JS example below, you’ll see David’s solution that splits the sequence of input characters and does so-called reduction or folding with <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce">reduce()</a> function (a.k.a. <a href="https://en.wikipedia.org/wiki/Fold_(higher-order_function)">fold higher-order function</a>). The author then increments counters accordingly. We can then follow the same theme in the second example submitted by <a href="https://github.com/kkogovsek">Klemen Kogovšek</a>, where the author also cleverly uses <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax">spread syntax (…)</a> to convert string to a sequence of characters and <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Conditional_Operator">conditional (ternary) operator</a> as an elegant replacement for if…else statement. A similar approach was then also put to “extreme” within Goran Kodrun’s TypeScript solution, which can be <a href="https://github.com/otobrglez/compression-puzzle/blob/master/src/ts/compress_go2.ts">found here</a>. The third example in this group comes from me. It uses a regular expression to break together string into groups; for that, it uses the so-called <a href="https://www.regular-expressions.info/lookaround.html">negative lookahead (lookbehind)</a>. After these groups are made, it computes the length and picks the character representing the group, following with the nice <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/flatMap">flatMap</a> to flatten everything into a sequence. After some discussion with other authors, we made a so-called “gentlemen’s agreement” that regular expressions should be considered as cheating and are not encouraged as wanted. :)</p>
<p><img src="/assets/2022-03-04-compression-puzzle/javascript_round.png" alt="JavaScript solutions by Klemen Kogovšek, David Ličen and Oto Brglez." /></p>
<p>At this point, you get the gist. It’s time to look into other popular languages.</p>
<p>For the sake of elegance. Let’s compare modern <strong>C++</strong>, <strong>C#</strong> and <strong>Go</strong> next. The C++ opens with <a href="https://www.cplusplus.com/reference/string/string/reserve/">string::reserve</a> and clever usage of mutation with <a href="https://www.cplusplus.com/reference/string/string/push_back/">string::push_back</a>. The following C# submission pops with <a href="https://www.programiz.com/csharp-programming/do-while-loop">while loop</a> and Go nails with its powerful <a href="https://golangdocs.com/rune-in-golang">rune</a> function. It was submitted by Nejc Ilenič, Peter A. Pirc and Tit Petrič respectfully. However, the “rune-less” Go solution was also <a href="https://github.com/otobrglez/compression-puzzle/blob/master/src/go/compress_mitja.go">submitted</a> by <a href="https://linkedin.com/in/mitja-živković-367206">Mitja Živković</a>.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/cpp_cs_go.png" alt="C++, C# and Go" /></p>
<p>Now that we sifted through the OOP and not-so-oop solutions, it’s time to uncover more powerful FP solutions and solutions somewhere between those two. So we have a neat comparison of <strong>F#</strong>, <strong>Clojure</strong> and <strong>Scala</strong>. Examples submitted by <a href="https://github.com/pkese">Peter Keše</a>, <a href="https://github.com/sbelak">Simon Belak</a> and myself. In F# and Scala, you’ll first notice the usage of <a href="https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching">pattern matching</a> to deconstruct strings and their “<a href="https://hestia.typepad.com/flatlander/2010/07/f-pattern-matching-for-beginners-part-4-lists-and-recursion.html">tails</a>”. In Simon’s Clojure example, however, you should also see the use of <a href="https://clojuredocs.org/clojure.core/partition-by">partition-by</a> that applies f to each value in the collection, splitting it each time f returns a new value. Clojure also nicely shows <a href="https://clojuredocs.org/clojure.core/transduce">transduce</a> and <a href="https://clojuredocs.org/clojure.core/mapcat">mapcat</a> on top of collections.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/scala_fp_pm.png" alt="Scala, F# and Clojure." /></p>
<p>Now that we understand that splitting the string when the sequence changes from one letter to another could be helpful, we can start getting the grip of the following three examples submitted by <a href="https://github.com/szlend">Simon Žlender</a>. First two in <strong>Elixir</strong> and the last one in <strong>Rust</strong>. In Elixir, we can see more precise usage of pattern matching and <a href="https://hexdocs.pm/elixir/1.12/Enum.html#chunk_while/4">Enum#chunk_while</a>, and in Rust, we can get some power from its <a href="https://doc.rust-lang.org/nightly/std/iter/struct.Peekable.html">Peekable - next_if_eq</a>.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/elixir_rust.png" alt="Elixir and Rust" /></p>
<p>Hold on, Oto! Did you get any <a href="https://www.investopedia.com/terms/f/faang-stocks.asp">FAANG</a> engineers to do this? Sure.</p>
<p>These two Kotlin solutions were respectfully submitted by <a href="https://github.com/izacus">Jernej Virag</a> (Google) and <a href="https://github.com/mihanovak1024">Miha Novak</a> (ex-Outfit7). Although the solutions both follow similar patterns to other patterns above with their usage of the <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/fold.html">fold</a>, <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/reduce.html">reduce</a> and <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/map.html">map</a>, we can also see Kotlin’s <a href="https://agrawalsuneet.github.io/blogs/kotlin-let-function/">let function</a>. A standard-library extension function to the Template class takes a lambda function as a parameter, apply a contract on it and ultimately return the execution of the lambda we passed as a parameter to it.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/kotlin.png" alt="Kotlin" /></p>
<h2 id="exotic-beauty">Exotic beauty</h2>
<p>Now, before we continue, I would like to emphasise that I’m a firm believer that beauty is, as they say, in the eye of the beholder. And that beauty comes in so many shapes and forms, and perhaps language itself may not be your cup of tea. It might be about other things that make it unique.</p>
<p>Martek nicely mixed pattern matching, folding (foldr), and monadic binding/composition operators (=«). The first round of such solutions should be given to <a href="https://github.com/turboMaCk">Marek Fajkus</a>’s <strong>Haskell</strong> solution and <a href="https://github.com/kkogovsek">Klemen Kogovšek</a>’s <strong>ReScript</strong> variation. Klemen shows us that modern superscripts such as ReScript can bring new capabilities even to JavaScript so we can see <a href="https://rescript-lang.org/docs/manual/latest/pattern-matching-destructuring">pattern matching / destructing</a> something that is “impossible” on common terms.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/haskell.png" alt="Haskell and ReScript" /></p>
<p>Blushing a bit. One of the languages I had never seen before that blew me away doing this exercise was a language called <a href="https://en.wikipedia.org/wiki/Red_(programming_language)">Red</a>. <strong>Red</strong> is a programming language designed to overcome the limitations of the programming language <a href="http://www.rebol.com/">Rebol</a>. It is a so-called successor, designed to replace his older brother. Below you’ll find three solutions, written by <a href="http://github.com/rebolek">Boleslav “Rebolek” Březovský</a>, <a href="https://github.com/greggirwin">Gregg Irwin</a> and <a href="https://github.com/hiiamboris">Boris</a>. One of the themes that make Rebol, Red and these solutions “unique” is that they are all “syntax free”. The theory being as you become fluent, you end up writing code in sentences, somewhat similar to human languages. Because our brains are well optimised for sequences of words, Rebol sentences feel natural, and you become more productive.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/red.png" alt="Red" /></p>
<p>Now that you’re familiar with Rebol and its clean syntax, I also invite you to take a peek at <a href="https://github.com/MrChriss">Krištof Črnivec</a>’s <strong>Ruby</strong> solutions. In both of his solutions, you’ll see the usage of modern Ruby’s <a href="https://medium.com/@baweaver/ruby-2-7-numbered-parameters-3f5c06a55fe4">numerated parameters</a>, conditional slicing with <a href="https://rubydoc.info/stdlib/core/Enumerable:slice_when">Enumerable#slice_when</a>, <a href="https://rubydoc.info/stdlib/core/Enumerable:chunk_while">Enumerable#chunk_while</a> and <a href="https://medium.com/@baweaver/ruby-2-7-enumerable-tally-a706a5fb11ea">tally</a> for counting.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/ruby.png" alt="Ruby" /></p>
<h2 id="exotic-exotic-beauty">Exotic exotic beauty</h2>
<p>Although the solutions above are all awe-inspiring on their own, they are still written in languages that people use and know. But there is also this submission that blew my mind. My friend <a href="https://github.com/refaktor">Janko Metelko</a> has been working on his new programming language for the last couple of years. If you know anything about programming, you’ll appreciate how very hard it is to design and build a helpful programming language. I’m incredibly privileged to be able to invite Janko to give his new programming language as a spin. Please enjoy the compression puzzle implemented in a new programming language named <a href="https://github.com/refaktor/rye">Rye</a>.</p>
<p>Janko reimplemented the solutions seen in Python, Go and finished in style by adding support for higher-order functions to the Rye language so that he could submit the third solution.</p>
<p><img src="/assets/2022-03-04-compression-puzzle/rye.png" alt="Elixir and Rust" /></p>
<h2 id="impressive-stats">Impressive. Stats?</h2>
<p>Today - 4th of March 2022 - we’ve collected 45 solutions spanning 19 programming languages written by 23 authors. Most of the authors were invited to participate, and many authors submitted several solutions in several languages.</p>
<h2 id="the-next-step">The next step</h2>
<p>Now that we established basic rules, procedures and GitHub actions with the <a href="https://github.com/otobrglez/compression-puzzle/blob/master/.github/workflows/test.yml">CI pipeline</a> (via <a href="https://nix.dev/">Nix</a> and <a href="https://www.docker.com/">Docker</a>), the next step is to <strong>invite you</strong>, dear reader. What have we missed? What deserves to be among the best? It’s an easy challenge, after all. :)</p>
<p>P.s: yes, there is also <a href="https://github.com/otobrglez/compression-puzzle/blob/master/src/sqlite/compress_rec.sql">SQLite solutions in the repo</a>. But PHP, Java, Brain<em>**</em> and Assembler are missing…</p>
<h2 id="summary">Summary</h2>
<p>I’m deeply thankful to everyone who took the time and gave their best shot. This is already a good story to tell! Thank you!</p>
<p><strong>Share if you care,</strong> <a href="https://github.com/otobrglez/compression-puzzle"><strong>join if you dare</strong></a><strong>.</strong></p>
<p><br /></p>
<blockquote>
<p>This article was also published on my LinkedIn profile as an <a href="https://www.linkedin.com/pulse/compression-puzzle-challenge-oto-brglez">“The Compressin Puzzle Challange.”</a></p>
</blockquote>Life is short, exciting and full of small events. Small events can change your daily life and sometimes inspire you beyond your wildest imagination. This is one of those stories when a small and neat idea took off and challenged people beyond my or everyone’s imagination…Cracking the WORDLE game2022-01-26T21:00:00+00:002022-01-26T21:00:00+00:00https://epic.blog/hacking/2022/01/26/cracking-the-wordle-game<p>If you’ve recently opened any social media app, be it <a href="https://www.facebook.com/search/posts/?q=wordle">Facebook</a>, <a href="https://twitter.com/search?q=wordle">Twitter</a>, or if you are a vivid reader of <a href="https://www.forbes.com/sites/erikkain/2022/01/24/todays-wordle-word-of-the-day-answer-219-monday-january-24th/?sh=6213935f65b8">Forbes</a> or <a href="https://www.theguardian.com/games/2022/jan/11/wordle-creator-overwhelmed-by-global-success-of-hit-puzzle">Guardian</a>, you’ve must have seen the new viral craze called <strong>WORDLE</strong>. This lovely, neat and simple game where the user must guess the five-letter word of the day within five steps. If you are unfamiliar with the game, please do <a href="https://www.powerlanguage.co.uk/wordle/">give it a try</a> here and come back to this article. I’ll wait… Done?</p>
<h2 id="basic-investigation-and-hypothesis">Basic investigation and hypothesis</h2>
<p>WORDLE is a so-called single-page web application without any kind-of backend. The only external resource or interaction it ever makes with the outside world is when Google Analytics is connected for primary tracking purposes. If you play a game, nothing will be sent to the server; all the validation is done in the client itself. Meaning that the application must carry all the possible words embedded inside the JavaScript “bundle”. It also means that the algorithm for choosing the words and validation must be baked into it. However, the state of the game is tied to your browser and to store the state in the browser the game must use either some <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/cookies">cookies</a> or <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage">LocalStorage</a>.</p>
<p>Looking into how the application behaves and monitoring traffic with <a href="https://developer.chrome.com/docs/devtools/">Chrome DevTools</a> soon confirmed this hypothesis. No network calls and local storage has serialized <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON">JSON</a> object that represents the current state of the game, theme and statistics.</p>
<p><img src="/assets/2022-01-26-cracking-the-wordle-game/w1.png" alt="Inspecting WORDLE Web Application with Chrome Devtools" /></p>
<h2 id="the-sandbox">The sandbox</h2>
<p>If the game is so neatly packed and if everything is bundled, then it is feasible to mirror the site, indemnify the JavaScript bundle and fiddle with the code. We start by creating a <a href="https://www.gnu.org/software/wget/manual/wget.html#Very-Advanced-Usage">local mirror</a> of the web application with the help of wget tool.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>wget <span class="nt">--mirror</span> <span class="nt">--convert-links</span> <span class="nt">--adjust-extension</span> <span class="se">\</span>
<span class="nt">--page-requisites</span> <span class="nt">--no-parent</span> https://www.powerlanguage.co.uk/wordle/
</code></pre></div></div>
<p>Then we need to “deminify” or decompress the JavaScript. This is the process that reverses the process of minifying. Minifying, or minification, is a process where you remove unnecessary characters from your code, whether they might be whitespace (such as indentation), code that isn’t ever used, comments in your code or long names for variables that can be replaced with something shorter.</p>
<p>Minification of your code usually results in it taking up less space, making it faster to send from a server to a client, as well as using less bandwidth in doing so. This improves the user experience on your site as it can load much faster. It also adds an element of security; since it becomes tough to figure out what the code does if it is all mingled and shortened. In this step, however, we use the same code and compile it back to “original” as much as that is possible. In this case, <a href="http://js-beautify/">js-beautify</a> was used.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npx js-beautify www.powerlanguage.co.uk/wordle/main.<span class="k">*</span>.js <span class="o">></span> <span class="se">\</span>
www.powerlanguage.co.uk/wordle/main.js
</code></pre></div></div>
<p>After this step, we can boot up any web-server (in my case Node.js <a href="https://www.npmjs.com/package/http-server">http-server</a>) and we can start serving the site from our local machine.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npx http-server www.powerlanguage.co.uk/wordle <span class="nt">-p</span> 8000 <span class="nt">-c-1</span>
</code></pre></div></div>
<h2 id="jumping-into-the-hole-">Jumping into the hole 🕳🐇</h2>
<p>Now that we have a local “copy” of the web application running on our machine, we can open the site in the browser. Then it’s a wise idea to trace how user interactions map from UI to actual business logic code. Using <a href="https://developer.chrome.com/docs/devtools/">Chrome DevTools</a> you can click on “keyboard” (<code class="language-plaintext highlighter-rouge">game-keyboard</code> element), and inside the Event Listeners tab, you’ll see the code listening to these virtual keyboard interactions. As seen in the screenshot below, it all starts at line 1269 of our “deminifed” main.js.</p>
<p><img src="/assets/2022-01-26-cracking-the-wordle-game/w2.png" alt="Inspecting WORDLE Web Application with Chrome Devtools" /></p>
<p>From this point on we start exploring where the “checks” happen where the <code class="language-plaintext highlighter-rouge">Array</code> of words lives and how check or validation is connected to it. Some more investigation leads us to <code class="language-plaintext highlighter-rouge">La</code> and <code class="language-plaintext highlighter-rouge">Ta</code> arrays. Remember that these must have more descriptive names in original implementation and that the minification process must have shortened them. <strong>Still good</strong>.</p>
<p><img src="/assets/2022-01-26-cracking-the-wordle-game/w3.png" alt="The evaluateRow function" /></p>
<p>Now we have two important pieces; we know where the words live where the check happens, and we know that “solution” must be somehow pre-computed before the check can actually happen. So, let’s find this assignment next…</p>
<p><img src="/assets/2022-01-26-cracking-the-wordle-game/w4.png" alt="The "solution"" /></p>
<p>Ok. We now know that there is some function with the name <code class="language-plaintext highlighter-rouge">Da</code> that takes “today” (probably <code class="language-plaintext highlighter-rouge">Date</code>), and it computes word for the day. Let’s explore how this <code class="language-plaintext highlighter-rouge">Da</code> function looks and what it does.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">Ha</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="mi">2021</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">19</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="kd">function</span> <span class="nx">Na</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span> <span class="nx">a</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">s</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="nx">e</span><span class="p">),</span>
<span class="nx">t</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="nx">a</span><span class="p">).</span><span class="nx">setHours</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">-</span> <span class="nx">s</span><span class="p">.</span><span class="nx">setHours</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">return</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">round</span><span class="p">(</span><span class="nx">t</span> <span class="o">/</span> <span class="mi">864</span><span class="nx">e5</span><span class="p">)</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">Da</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">a</span><span class="p">,</span> <span class="nx">s</span> <span class="o">=</span> <span class="nx">Ga</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">a</span> <span class="o">=</span> <span class="nx">s</span> <span class="o">%</span> <span class="nx">La</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="nx">La</span><span class="p">[</span><span class="nx">a</span><span class="p">]</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">Ga</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">Na</span><span class="p">(</span><span class="nx">Ha</span><span class="p">,</span> <span class="nx">e</span><span class="p">)</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">Ba</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">abcdefghijklmnopqrstuvwxyz</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">Fa</span> <span class="o">=</span> <span class="p">[].</span><span class="nx">concat</span><span class="p">(</span><span class="nx">g</span><span class="p">(</span><span class="nx">Ba</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">slice</span><span class="p">(</span><span class="mi">13</span><span class="p">)),</span> <span class="nx">g</span><span class="p">(</span><span class="nx">Ba</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">""</span><span class="p">).</span><span class="nx">slice</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">13</span><span class="p">)));</span>
<span class="kd">function</span> <span class="nx">Wa</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">a</span> <span class="o">=</span> <span class="dl">""</span><span class="p">,</span> <span class="nx">s</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">s</span> <span class="o"><</span> <span class="nx">e</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">s</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">t</span> <span class="o">=</span> <span class="nx">Ba</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="nx">e</span><span class="p">[</span><span class="nx">s</span><span class="p">]);</span>
<span class="nx">a</span> <span class="o">+=</span> <span class="nx">t</span> <span class="o">>=</span> <span class="mi">0</span> <span class="p">?</span> <span class="nx">Fa</span><span class="p">[</span><span class="nx">t</span><span class="p">]</span> <span class="p">:</span> <span class="dl">"</span><span class="s2">_</span><span class="dl">"</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">a</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Impressive. I guess.</p>
<p>Now, let us rewrite it and this time, let’s use some more meaningful function names and remove “minification” hacks introduced when minification occurred. We should also use modern JavaScript. To ensure that our code works, let’s also add some assertions against the known words.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">assert</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">assert</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">normalWords</span> <span class="o">=</span> <span class="p">[</span><span class="dl">"</span><span class="s2">cigar</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">rebut</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">...omitted</span><span class="dl">"</span><span class="p">];</span>
<span class="kd">const</span> <span class="nx">gameBeginning</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="dl">'</span><span class="s1">19 June 2021</span><span class="dl">'</span><span class="p">).</span><span class="nx">setHours</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">dateIndex</span> <span class="o">=</span> <span class="p">(</span><span class="nx">beginning</span><span class="p">,</span> <span class="nx">date</span><span class="p">)</span> <span class="o">=></span>
<span class="nb">Math</span><span class="p">.</span><span class="nx">round</span><span class="p">((</span><span class="nx">date</span><span class="p">.</span><span class="nx">setHours</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">-</span> <span class="nx">beginning</span><span class="p">)</span> <span class="o">/</span> <span class="mi">864</span><span class="nx">e5</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">wordleForDate</span> <span class="o">=</span> <span class="p">(</span><span class="nx">date</span><span class="p">)</span> <span class="o">=></span>
<span class="nx">normalWords</span><span class="p">[</span><span class="nx">dateIndex</span><span class="p">(</span><span class="nx">gameBeginning</span><span class="p">,</span> <span class="nx">date</span><span class="p">)</span> <span class="o">%</span> <span class="nx">normalWords</span><span class="p">.</span><span class="nx">length</span><span class="p">];</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="nx">wordleForDate</span><span class="p">(</span><span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="dl">'</span><span class="s1">26 Jan 2022</span><span class="dl">'</span><span class="p">)),</span> <span class="dl">'</span><span class="s1">whack</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="nx">wordleForDate</span><span class="p">(</span><span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="dl">'</span><span class="s1">25 Jan 2022</span><span class="dl">'</span><span class="p">)),</span> <span class="dl">'</span><span class="s1">sugar</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="nx">wordleForDate</span><span class="p">(</span><span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="dl">'</span><span class="s1">06 Aug 2022</span><span class="dl">'</span><span class="p">)),</span> <span class="dl">'</span><span class="s1">aphid</span><span class="dl">'</span><span class="p">);</span>
</code></pre></div></div>
<h2 id="the-algorithm">The Algorithm</h2>
<p>The core algorithm of the WORDLE game is time or rather date based. First, it computes the difference between the current date and the 19th of June 2021 (set by author). In JavaScript, the <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date">Date object</a> is represented by a number of milliseconds since 1 January 1970, 00:00:00 UTC, with leap seconds ignored. The second part of getting this number is a freaky division by the number 864e5 and rounding of it. 864e5 is the same as 86400000 or 1000<em>60</em>60*24 and represents 24 hours or a day.</p>
<p>After this number is computed, we do an array lookup. In this lookup, we take the “seed”, and we calculate <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Remainder">reminder</a> with the length of a given array. A simplified version would then read:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">array</span> <span class="o">=</span> <span class="p">[</span><span class="dl">'</span><span class="s1">A</span><span class="dl">'</span><span class="p">,</span><span class="dl">'</span><span class="s1">B</span><span class="dl">'</span><span class="p">,</span><span class="dl">'</span><span class="s1">C</span><span class="dl">'</span><span class="p">,</span><span class="dl">'</span><span class="s1">D</span><span class="dl">'</span><span class="p">,</span><span class="dl">'</span><span class="s1">E</span><span class="dl">'</span><span class="p">]</span>
<span class="kd">const</span> <span class="nx">seed</span> <span class="o">=</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">round</span><span class="p">(</span><span class="mf">41.2</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">index</span> <span class="o">=</span> <span class="nx">seed</span> <span class="o">%</span> <span class="nx">array</span><span class="p">.</span><span class="nx">length</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">array</span><span class="p">[</span><span class="nx">index</span><span class="p">])</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Although this is now an extremely popular game and it is played by millions worldwide, I believe that it is still important to point out that so-called “client validation” where all the business logic or in this case game logic is validated on users device or client is extremely dangerous and it should be used with caution. If you are building new applications, please make sure that you validate user input on the “server-side” and that your core algorithm is never leaked.</p>
<p>Have fun!</p>
<p><img src="/assets/2022-01-26-cracking-the-wordle-game/w5.png" alt="Bunch of WORDLEs" /></p>
<p>P.s.: If you are struggling with WORDLE’s and would like to get WORDLE for any day, my DMs are open. Oh, and I don’ take crypto, but I LOVE coffee. 😘</p>
<p><br />
<em>This post was also published as an <a href="https://www.linkedin.com/pulse/cracking-wordle-game-oto-brglez/">article on my LinkedIn profile</a>.</em></p>If you’ve recently opened any social media app, be it Facebook, Twitter, or if you are a vivid reader of Forbes or Guardian, you’ve must have seen the new viral craze called WORDLE. This lovely, neat and simple game where the user must guess the five-letter word of the day within five steps. If you are unfamiliar with the game, please do give it a try here and come back to this article. I’ll wait… Done?Cracking the SECDIM S/W Challenge2022-01-12T07:00:00+00:002022-01-12T07:00:00+00:00https://epic.blog/engineering/2022/01/12/cracking-the-secdim-challenge<p>My good security expert friend has recently shared this lovely puzzle that <a href="https://secdim.com/">SECDIM</a> uses for filtering out potential candidates. I couldn’t help myself but give it a shot. So, here it goes.</p>
<h1 id="intro">Intro</h1>
<p>On their <a href="https://secdim.com/careers/">careers page</a>, you’ll find these lovely, yet simple instructions.</p>
<p><img src="/assets/2022-01-12-cracking-the-secdim-challenge/jd.png" alt="jd" /></p>
<h1 id="exploration">Exploration</h1>
<p>Ok. So first step is to pull down the Docker image, run it and then investigate the payload.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> secdim/interview:latest
</code></pre></div></div>
<p>That gives this lovely output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[i] Please send your application to
Traceback (most recent call last):
File "/usr/src/app/main.py", line 22, in <module>
print(unhide(key, email))
File "/usr/src/app/main.py", line 16, in unhide
raise NotImplementedError
NotImplementedError
</code></pre></div></div>
<p>The exception that we get back hints that it is apparently a Python-based Docker image and exception hints that some kind of exception has occurred. Giving us the first clue <code class="language-plaintext highlighter-rouge">NotImplementedError</code> - probably implying that a solution will be here. 🤔</p>
<p>Next; let’s override the entry point to the docker image and let’s take a peek into its content. Some experts might also hint that “unpackaging” the image would be a good idea. But since we are adventurous lets just go in…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="nt">--entrypoint</span> /bin/sh secdim/interview
/usr/src/app <span class="nv">$ </span><span class="nb">ls</span> <span class="nt">-l</span>
<span class="nt">-rw-rw-rw-</span> 1 root root 569 Jun 27 2021 main.py
</code></pre></div></div>
<p>Inside the container, we’ll find this lovely <code class="language-plaintext highlighter-rouge">main.py</code> Python script:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">base64</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">getenv</span>
<span class="k">def</span> <span class="nf">hide</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="n">encoded_chars</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">string</span><span class="p">)):</span>
<span class="n">key_c</span> <span class="o">=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)]</span>
<span class="n">encoded_c</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">string</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">+</span> <span class="nb">ord</span><span class="p">(</span><span class="n">key_c</span><span class="p">)</span> <span class="o">%</span> <span class="mi">256</span><span class="p">)</span>
<span class="n">encoded_chars</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">encoded_c</span><span class="p">)</span>
<span class="n">encoded_string</span> <span class="o">=</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">encoded_chars</span><span class="p">)</span>
<span class="k">return</span> <span class="n">base64</span><span class="p">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">encoded_string</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">unhide</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="k">raise</span> <span class="nb">NotImplementedError</span>
<span class="n">email</span> <span class="o">=</span> <span class="sa">b</span><span class="s">"w4vCl8OSwp/CpMKNZGtnbMKhw5jCmMKVwpTCmcKdwpPCnMKkwqE="</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">getenv</span><span class="p">(</span><span class="s">"KEY"</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"[i] Please send your application to "</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">unhide</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">email</span><span class="p">))</span>
</code></pre></div></div>
<p><strong>Nice.</strong>
So apparently we have a script that has “encoded” email and some kind of <code class="language-plaintext highlighter-rouge">KEY</code> is needed for ncoding and decoding. Script pulls the key-value out of the environment with the help of Python’s stdin function <a href="https://docs.python.org/3/library/os.html"><code class="language-plaintext highlighter-rouge">getenv</code></a>. Let’s see if that exists in the container…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="nt">--entrypoint</span> /usr/bin/env secdim/interview |grep <span class="nt">-i</span> key
<span class="nv">KEY</span><span class="o">=</span>c2f35b3819ae32000e9541172c7e2216
</code></pre></div></div>
<p>And it does…</p>
<h1 id="coding-time">Coding time!</h1>
<p>Apparently the email that you need to solve this is encoded in the <code class="language-plaintext highlighter-rouge">email</code> variable, and to encode/unhide it you need a “key”. We now have both. The last peace that is now missing is the implementation of the unhide function. To understand unhiding we should first look at the hiding function. There are 3 pieces to it - <a href="https://en.wikipedia.org/wiki/Base64">base64 encoding/decoding</a>, loops and <a href="https://en.wikipedia.org/wiki/Modulo_operation">modulo</a> 256 mathematical operations. The code loops are pretty basic and if you ever took any algebra or basic crypto class you’ll soon recognise the famous “<a href="https://en.wikipedia.org/wiki/Caesar_cipher">Caesar cipher</a>” (or <a href="https://en.wikipedia.org/wiki/Vigenère_cipher">Vigenère cipher</a>). The decrypt / unhide for it is very <em>well known</em>.</p>
<p>With this function, source email and key we can now write the full solution like so:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">base64</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">getenv</span>
<span class="k">def</span> <span class="nf">hide</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="n">encoded_chars</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">string</span><span class="p">)):</span>
<span class="n">key_c</span> <span class="o">=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)]</span>
<span class="n">encoded_c</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="nb">ord</span><span class="p">(</span><span class="n">string</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="o">+</span> <span class="nb">ord</span><span class="p">(</span><span class="n">key_c</span><span class="p">)</span> <span class="o">%</span> <span class="mi">256</span><span class="p">)</span>
<span class="n">encoded_chars</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">encoded_c</span><span class="p">)</span>
<span class="n">encoded_string</span> <span class="o">=</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">encoded_chars</span><span class="p">)</span>
<span class="k">return</span> <span class="n">base64</span><span class="p">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">encoded_string</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">unhide</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">string</span><span class="p">):</span>
<span class="n">chars</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">encoded_string</span> <span class="o">=</span> <span class="n">base64</span><span class="p">.</span><span class="n">urlsafe_b64decode</span><span class="p">(</span><span class="n">string</span><span class="p">).</span><span class="n">decode</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">e</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">encoded_string</span><span class="p">):</span>
<span class="n">key_c</span> <span class="o">=</span> <span class="n">key</span><span class="p">[</span><span class="n">i</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">key</span><span class="p">)]</span>
<span class="n">chrc</span> <span class="o">=</span> <span class="nb">ord</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="o">-</span> <span class="nb">ord</span><span class="p">(</span><span class="n">key_c</span><span class="p">)</span> <span class="k">if</span> <span class="nb">ord</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="o">-</span> <span class="nb">ord</span><span class="p">(</span><span class="n">key_c</span><span class="p">)</span> <span class="o">>=</span> <span class="mi">0</span> <span class="k">else</span> <span class="nb">ord</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="o">-</span> <span class="nb">ord</span><span class="p">(</span><span class="n">key_c</span><span class="p">)</span> <span class="o">+</span> <span class="mi">256</span>
<span class="n">c</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="n">chrc</span><span class="p">)</span>
<span class="n">chars</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="k">return</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">chars</span><span class="p">)</span>
<span class="c1"># key = getenv("KEY")
</span><span class="n">key</span><span class="o">=</span><span class="s">"c2f35b3819ae32000e9541172c7e2216"</span>
<span class="n">email</span> <span class="o">=</span> <span class="sa">b</span><span class="s">"w4vCl8OSwp/CpMKNZGtnbMKhw5jCmMKVwpTCmcKdwpPCnMKkwqE="</span>
<span class="k">print</span><span class="p">(</span><span class="s">"[i] Please send your application to "</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">unhide</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">email</span><span class="p">))</span>
</code></pre></div></div>
<p>Then you send a friendly email…</p>
<p>Done. ✅</p>My good security expert friend has recently shared this lovely puzzle that SECDIM uses for filtering out potential candidates. I couldn’t help myself but give it a shot. So, here it goes.My hacks, experiments and products of 20212022-01-05T07:00:00+00:002022-01-05T07:00:00+00:00https://epic.blog/engineering/2022/01/05/hacks-experiments-products-of-2021<p>As the year 2021 has finally ended and we open the new chapter of 2022 - I’ve taken a couple of minutes and composed my list of projects, side-projects and highlights that dictated my year.</p>
<h1 id="the-pollpass-pie-poof-of-concept">The Pollpass Pie (Poof-of-concept)</h1>
<p>I started 2021 with a proof-of-concept called Pollpass PIE 🍰 - real-time collaborative market research “scripting” tool. The idea was to explore tools and services that could be used to make the design of surveys easier, faster, less error-prone and more user-friendly. The hypothesis or the question was - can we make the design of surveys as easy as writing Google Docs?</p>
<div style="text-align:center">
<iframe style="width:100%; height:500px; margin:20px" src="https://www.youtube.com/embed/JEKv-YjwiI8?controls=0" title="GWI / Pollpass PIE" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
<p>In the series of experiments, I evaluated Google’s Firebase or to be precise <a href="https://firebase.google.com/docs/database">Firebase Realtime Database</a>. I was looking into technologies that could be used for building modern “<a href="https://en.wikipedia.org/wiki/Serverless_computing">serverless</a>” web applications without the usage of traditional “storage” stacks and APIs. The PoC proved that the tech is “ready” for reasonably complex definitions and surveys; however, it gets very tricks to “migrate” documents and assure consistency. The “support” for very complex “validations” that are part of real-life scenarios is quite challenging to achieve. The conclusion at the time was to discontinue this line of experiments and focus on other things. However, the area of real-time editors and the power that they bring to modern web apps are on the rise. <strong>Pay attention to this area of innovation!</strong></p>
<h2 id="the-oracle-peak">The Oracle Peak</h2>
<p>One of the highlights from my public activity was surely The Oracle Peak project; a project that I did with <a href="https://www.linkedin.com/in/andrazsraka/">Andraž Sraka</a> at the end of 2020 and was presented at <a href="https://www.confluent.io/events/kafka-summit-europe-2021/how-to-over-engineer-things-and-have-fun/">Kafka Summit Europe 2021 (talk)</a>. The Oracle Peak is another PoC where I was exploring the usage of <strong>real-time streaming data</strong> on top of modern data processing stacks. With the help of <a href="https://akka.io/">Akka</a>, <a href="https://doc.akka.io/docs/akka/current/stream/index.html">Akka Streams</a>, <a href="https://kafka.apache.org/">Apache Kafka</a>, <a href="https://ksqldb.io/">ksqlDB</a> and <a href="https://www.influxdata.com/">InfluxDB</a> we proved that it is as easy as ever to “spy” on your neighbour’s Wi-Fi as ever. The final system setup has consisted of several custom “IoT devices” and several locations… to learn more hear me speak at the summit tune in here:</p>
<div style="text-align:center">
<iframe style="width:100%; height:500px; margin:20px" src="https://www.youtube.com/embed/QJTZG7JgUoI?controls=0" title="GWI / Pollpass PIE" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
<p>The <a href="https://github.com/pinkstack/oracle-peak">code for Oracle Peak</a> was open-sourced and published on <a href="https://github.com/pinkstack/oracle-peak">GitHub</a> for your pleasure and satisfaction.</p>
<h2 id="the-pollpass-voice-proof-of-concept">The Pollpass Voice (Proof-of-concept)</h2>
<p>In the second half of the year, most of my focus went towards the exploration of emerging “voice technologies” and their potential and challenges specifically within the market research industry. With the series of PoCs and experimentations, I looked into a fusion of <a href="https://cloud.google.com/speech-to-text">Speech-to-Text</a> and <a href="https://cloud.google.com/text-to-speech">Text-to-speech</a> technologies and how they can be used on the web to replace traditional means of classical surveying. Hypothesis or the question asked was - can we build a tool that would make interviewing as easy as talking to someone? It seems plausible; if its scalable is yet to be seen. :)</p>
<div style="text-align:center">
<iframe style="width:100%; height:500px; margin:20px" src="https://www.youtube.com/embed/P47LvjUnQnE?controls=0" title="GWI / Pollpass PIE" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</div>
<h2 id="bitmaps-support-for-redis4cats">Bitmaps support for redis4cats</h2>
<p>I’ve spent a lot of time in 2021 learning the wisdom of modern <a href="https://en.wikipedia.org/wiki/Functional_programming">functional programming</a>, particularly with <a href="https://www.scala-lang.org/">Scala</a> and the ecosystem around it. I dug deeper into Scala’s <a href="https://typelevel.org/cats/">Cats</a> and the <a href="https://typelevel.org/">Typelevel</a> ecosystem around it. <a href="https://http4s.org/">http4s</a>, <a href="https://fs2.io/">fs2</a>, <a href="https://typelevel.org/cats-effect/">Cats Effect</a> and more. Although community and adoption of <a href="https://zio.dev/">ZIO</a> are growing I have yet to also dip my toes into that pool. Perhaps in 22.</p>
<p>My peak of this teaching was my contribution to redis4cats - <a href="https://redis.io/">Redis</a> client built on top of Cats Effect, Fs2 and <a href="https://lettuce.io/">Lettuce</a>. An open-source library that can be used to nicely and neatly work with Redis. My contribution to the lib and the ecosystem was support for <a href="https://scalegrid.io/blog/introduction-to-redis-data-structure-bitmaps/">Redis Bitmaps</a>. Bitmaps (also called bit arrays, bit vectors) is the data structure that immediately pops in your head when the need is to map boolean information for a huge domain into a compact and fast representation. It is a very popular data structure whenever memory space is at a premium: OS kernels (memory pages, inodes etc), digital imaging etc. The structure can be used for fast and advanced analytics as well. I.e. imagine computing user-funnels, drop-offs, logins etc as fast as possible.</p>
<p>At this point, I should also thank <a href="https://www.linkedin.com/in/gmvolpe/">Gabriel Volpe</a> and all the amazing people on Typelevel Discord that helped and supported my efforts.</p>
<h2 id="bicikelj-client-implemented-in-bash">BicikeLJ client implemented in Bash</h2>
<p>What started out as a mere joke has turned into an interesting social experiment (that got me some new friends 👋). I wrote a command-line Bash application that tells you where is the nearest - available - public bike station. Yes; <a href="https://en.wikipedia.org/wiki/Haversine_formula">haversine</a> is implemented in Bash. If you live in Ljubljana or if you just fancy Bash, <a href="https://github.com/otobrglez/bicikelj-sh/blob/master/bicikelj.sh">check it out</a>.</p>
<h2 id="the-ballon-project">The Ballon Project</h2>
<p>In the very last days of the year, I spent some time with tech that I’m least familiar with. I decided to dig my toes into Apple’s Swift, <a href="https://developer.apple.com/xcode/swiftui/">SwiftUI</a> and iOS development. Something that I haven’t done in a while. I’ve built a simple app that shows the real-time situations (location, speed, route) with buses in Ljubljana (Obviously; I also built a custom WebSocket backend on top of public data - but that is another <a href="https://github.com/pinkstack/tocen-live">open-source side-project</a>)</p>
<p><img src="/assets/2022-01-05-hacks-experiments-products-of-2021/ballon.png" alt="ballon" /></p>
<p>Data used to get this prototype going was provided by <a href="https://tocen.si/">tocen.si</a> and <a href="https://api.ontime.si/">OnTime API</a> - spearheaded by <a href="https://www.linkedin.com/in/matevzp/">Dr. Matevž Pesek</a> and his team. Advanced tutoring and help were also provided by my dear friend <a href="https://www.linkedin.com/in/jan-haložan-bb74a895/">Jan Haložan</a>, an iOS legend. Thank you both! :)</p>
<h2 id="talks-conferences-and-hackathons">Talks, conferences and hackathons</h2>
<p>As it was still a bit hard to travel in 2021 I haven’t really been “anywhere” so I did some Zoom talks and conferences, hackathons and “inspirational speeches”.</p>
<ul>
<li><strong>Kafka Summit Europe 2021 - How to over-engineer things and have fun?</strong> I spoke about Oracle Peak (mentioned above) a custom build real-time Wi-Fi spying system that I’ve to build on top of Kafka and ksqlDB. The talk was recorded and slides are <a href="https://www.confluent.io/events/kafka-summit-europe-2021/how-to-over-engineer-things-and-have-fun/">available here</a>.</li>
<li><strong>Gentle Introduction to Akka with Scala</strong> - <a href="https://www.meetup.com/Backend-developers-of-Ljubljana/">Backend Meetup Ljubljana</a>, June 2021. I was invited to speak about Akka and its ecosystem with the intention to inspire and introduce new engineers to the magical world of Akka. To this day <a href="https://www.youtube.com/watch?v=RfKxoW8HQUY">my most viewed YouTube video</a>.</li>
<li><strong>The Greenhack 2021 - Pavla</strong> - Slovenian open-data hackathon. As a vivid lover of hackathons, I participated in now “traditional” Slovenian open-data hackathon where I’ve worked on a service called Pavla. Pavla should help people navigate their cities in a more sustainable way. The idea made it to finals and it was <a href="https://www.youtube.com/watch?v=UZBWNqPXZqs&t=2157s">publically presented (recording)</a>. Although the thing is now “cold” it still helped me make some new connections and friends. The <a href="https://bit.ly/p-case">Business case</a>, <a href="https://bit.ly/pafla-fin">slides</a> and code are all publically available.</li>
<li><strong>Becoming S/W Engineer</strong> - The last public out-reach was my talk at my old high school (<a href="https://ers.scv.si/en/">ERŠ, Velenje</a>) where I spoke to a new generation of software engineers about the skills and sufferings of today’s developers and what it takes to become “great”. I hope that I’ve managed to convert at least one student to follow this road that I love so much. <a href="https://www.slideshare.net/Zver/becoming-sw-engineer">Slides</a>.</li>
</ul>
<h2 id="conclusion-and-22">Conclusion and 22</h2>
<p>I would consider 2021 to be a very successful year. I’ve learned a lot, expanded my network and deepened my knowledge - the “social presence” was noticeable. So for 22 - I hope to continue walking this path and jumping over the hoops and challenges as they come.</p>
<p><strong>Happy New Year and hope yours will be as successful as mine! 😋</strong></p>
<p><br />
<em>This post was also published as an <a href="https://www.linkedin.com/pulse/my-hacks-experiments-products-2021-oto-brglez/">article on my LinkedIn profile</a>.</em></p>As the year 2021 has finally ended and we open the new chapter of 2022 - I’ve taken a couple of minutes and composed my list of projects, side-projects and highlights that dictated my year.The four dusty pillars of software engineering2021-11-19T07:00:00+00:002021-11-19T07:00:00+00:00https://epic.blog/engineering/2021/11/19/four-dusty-pillars-software-engineering<p>As organisations are finishing the last quarter of 2021 and as we put all our heads together and try to compose the plans, roadmaps and strategies for an upcoming year. We must remind ourselves not just what we want to achieve; but how will we get there. As planning is important and needed, so are discussions around the essentials for growth and continuous delivery of business value.</p>
<p>With that intro, I’ll try to illustrate what I consider to be a few key topics that often get overseen, forgotten or intentionally left out of conversations as it is tough to justify “investment” in those areas as they don’t immediately show up on your fancy <a href="https://www.investopedia.com/terms/r/returnonassets.asp">ROA</a>/<a href="https://en.wikipedia.org/wiki/Performance_indicator">KPI</a>/<a href="https://en.wikipedia.org/wiki/OKR">OKR</a> dashboards. Especially compared to excellent and shiny new features that “customers would really like”.</p>
<h1 id="complexity">Complexity</h1>
<p>One of the first “pillars” that often start shaking, crumbling and cracking is the “complexity”. It is this tiny side-effect that happens when you engineer for the future that never happened. That little library you added does just one thing, yet it pulls in the universe of other dependencies that do god-knows-what. This combo of 100+ micro-services that you really really needed as you had big <a href="https://en.wikipedia.org/wiki/Fear_of_missing_out">FOMO</a> when you’ve read brilliant HackerNews PR. But now you have to deploy them, version them, evolve them, fuse them and retire them (Hello, Systems Development Life Cycle)… Then there is this organisational complexity; as your product, team, and company grow, so does the number of people, number of “papers”, number of stakeholders - tables are bigger and harder it is to move swiftly and nimbly.</p>
<p>Another scary thought that should cross your mind when thinking about complexity should be this. Every day that passes every engineering hour spent, it’s spent on building “something”. Is this “something” the right thing? Is this the right thing for “now”, or is this the right thing 2, 3, 5, 10+ years from now? The longer the answer, the more scared you should be. Fight the complexity with good problem definitions, clear goals and continuing “simplification” across the board.</p>
<p>When talking and reasoning about complexity, arm yourself with “time measurements”; log or count hours wasted on the support of features, count hours lost because X needs to happen or be done before Y can get to “production”. Don’t forget to also account for things like “how long it takes for me to find information about Z”… In general, also make sure that people understand what is needed and why.</p>
<h1 id="tech-debt">Tech-debt</h1>
<p>That brings us to the second pillar that will destroy morale, plans, deadlines and, in not-so-few cases, bring down the whole organisation. Every serious project has it, every organisation has. It is this fella that few like to talk about it, or if they talk about it, often gets blamed for a lot of delays and “that’s just freaking impossible!” - excuses.</p>
<p>Sure, you can build another patch. Then another patch. Then a hack on top. Then you can tweak the UI so that the progress bar stays on the screen for just a few seconds longer. You can also extend the timeouts to minutes on your load balancer so that the service can “digest” the payload. You can also wrap the whole service with another service and then treat it like a plague.</p>
<p>All these improper solutions are indeed buying you time, and in many cases, that is an essential commodity, especially young products.</p>
<p>However, make sure that you keep track of all these “hacks”; treat them as “loans” because the time will come when you will have to settle the debt. And if you don’t. It will cripple you and your organisation. Tech debt is a risk, and especially if you are in “managerial” position, you need to keep track of it.</p>
<p>Discuss it. Document it. Explain to other engineers why certain things are done in “shady ways”. Talk to management about it - in terms that will be familiar to them - loss of revenue, downtime, long development cycles, errors, bugs and fire.</p>
<h1 id="resources">Resources</h1>
<p>One of those pillars that are kind-a obvious and could also have different names - “assets”, “people”, “infrastructure”, “things needed to build things”… And we often plan or discuss this in terms of the future - we need to hire 50+ more engineers, we need to hire 20+ data-scientist, we need to expand our cloud budget, buy new SaaS service subscriptions etc.</p>
<p>Let’s spend more time thinking about how we will retain existing people, leverage existing services, and utilise our resources in more efficient ways.</p>
<p>From the perspective of risk, it is likely less risky to enrich existing experts who know your domain with additional knowledge than getting new ones - especially in current times. Equality with technology - new technology means you’ll have another “dragon” in your stack that you likely don’t know how to tame.</p>
<p>I suggest spending some serious thinking time on your current resources and elevating their value and performance.</p>
<h1 id="experimentation">Experimentation</h1>
<p>I love this pillar. It has a particular smell, a specific scent some might even say magical appeal. Yet I’ve seen so many examples where organisations forget to do it - or be more explicit; they restrict experimentation and innovation so much that it becomes “business as usual”, some might even say forced. Sure, you’ve been on those “we must invent something” meetings? Right?</p>
<p>I want to encourage you to provide a framework and genuine support for plainly “doing things that have not been done before” and individually guiding people’s goals towards solving the problems that resonate with your organisational vision and challenges.</p>
<p>As an engineer, challenge your management with questions on how the company plans on innovating and competing? As managers think about how much time and resources you can give for such activities, how can those tiny ideas turn into more prominent solutions?</p>
<h1 id="conclusion">Conclusion</h1>
<p>There is more than just four pillars to every organisation, project or product. There are also things such as priorities, budgets, sales and many many more… However, I think that keeping these in good shape will likely help your engineering stay on top…</p>
<p>Tell me, how healthy are your tech pillars? 🏛</p>
<blockquote>
<p>This post was originally published as an <a href="https://www.linkedin.com/pulse/four-dusty-pillars-software-engineering-oto-brglez/">article on my LinkedIn profile</a>.</p>
</blockquote>As organisations are finishing the last quarter of 2021 and as we put all our heads together and try to compose the plans, roadmaps and strategies for an upcoming year. We must remind ourselves not just what we want to achieve; but how will we get there. As planning is important and needed, so are discussions around the essentials for growth and continuous delivery of business value.Attending a hackathon with Scala and Akka (Streams) on Microsoft Azure2020-10-05T07:00:00+00:002020-10-05T07:00:00+00:00https://epic.blog/akka/2020/10/05/hackathon-with-scala-akka-on-azure<p>While browsing through the unlimited supply of sh** posts on Facebook, I’ve stumbled upon an exciting invite to an “environmental hackathon” - named “<strong>Living with environmental changes</strong> / <strong>Življenje s podnebnimi spremembami</strong>” (<a href="https://transformation-lighthouse.com/hackathon-zivljenje-s-podnebnimi-spremembami/">event</a>).</p>
<p>Online hackathon organised by a private company - <a href="https://transformation-lighthouse.com/">Transformation Lighthouse</a> - with collaboration with <a href="https://www.arso.gov.si/">Slovenian Environment Agency - ARSO</a> and <a href="https://www.gov.si/en/state-authorities/ministries/ministry-of-the-environment-and-spatial-planning/">Ministry of the Environment and Spatial Planning [MOP]</a>, plus few initiatives like Open Data Slovenia <a href="https://podatki.gov.si/">OPSI</a> and business partners like Microsoft Slovenia, Oracle Slovenia and others.</p>
<p><strong>At this point, you’re very likely already snoozed.</strong> 😴 And that is completely fine; so was I when I first read about the event.</p>
<p>It got me thinking, though,… Why would someone like to speed 48 hours working for the “government” during the weekend and lose a lot of sleep? 🤔 Well; my rationale is simple. By joining the event, <strong>I’ll get a rare glimpse into how government agencies - that usually sit on tons of incredibly valuable data - think about product development, the resources, data and insights that they can get out of it.</strong> It is the perfect environment to get the data about the environment - at the source (<em>yeah I know I could just build my own weather station instead 🌤</em> ). In my current seek for interesting big-data and real-time data; could be the perfect quest.</p>
<p>Sold. I attended. ✅</p>
<h2 id="day-zero-the-research">Day zero: The research.</h2>
<p>The day before the event I started browsing through all the available material that the organisers have compiled together while also looking for anything that is or could be turned into an API or data feed on all of the government sites. I found this lovely golden nugget on one of the pages.</p>
<p><img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/arso-podatki-postaje-map.png" alt="ARSO Water Stations Map" /></p>
<p>It is a “widget” that shows water levels and the flow of water in the country. You can see “buckets” that simulate if the flow is low, medium, or high. And after some more browsing, I found out that this data is available in XML format, and after that, I also found out that there is also historical data, in some cases 30+ years back.</p>
<p>And that got me thinking. <strong>What if I extract this “current” data and merge it with “historical” data; would then - in theory - be possible to predict the future with some degree of certainty? And that’s the idea. Bingo!💡</strong></p>
<p><img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/UI.png" alt="Voda on Azure - UI" /></p>
<h2 id="day-1-idea-validation-poc-and-application">Day 1: Idea validation, PoC and Application</h2>
<p>The following morning; the event took off. It was a traditional intro/marketing/ideas/make-world-a-better-place combo. The organisers and the agencies expressed their fields where they think the attendees could chip in. And provided a beneficial set of experts from various fields,…</p>
<p>We were given the following three subjects to chose from</p>
<ol>
<li>Impact of environmental changes</li>
<li>Our relationship to natural resources - especially to water and drought</li>
<li>Strategic game related to environmental changes</li>
</ol>
<p>I’ve chosen the 2nd one - as I thought that the idea of understanding the water sources might somehow fit into that.</p>
<p>Problem numer one. With a bit of bad luck, I was put in a team where nobody showed up. Oops. 😅 I thought that this might ruin part of the fun but on second thought… 💡 If I work alone, I can move faster,… and I can focus on the core of my exploration. How to get the data, quickly and how to get to insight asap?</p>
<p>I spoke with organisers and asked them if it would be possible to work alone. To my surprise, they agreed, and I was ready. 🚀</p>
<h3 id="the-proof-of-concept">The proof of concept</h3>
<p>My idea was conceptually simple. Fetch the water data from government sites as quickly as possible, few requests per second with excellent parallelism… and turn that data into time-series. Make everything except the collection part “real-time”. And for the second part - archive - also use similar piping to fetch and process the historical data,… and hopefully, also ingest that into time-series or some other format.</p>
<p>My original idea was to scrape with Akka, and emit JSON to two separate <a href="https://kafka.apache.org/">Apache Kafka</a> topics and then combine these two topics with <a href="https://ksqldb.io/">ksqlDB</a> query. After that, send the results of the streaming question to <a href="https://www.influxdata.com/">InfluxDB</a> and visualise everything in <a href="https://grafana.com/">Grafana</a> of Grafana/<a href="https://www.elastic.co/">ElasticSearch</a> combo.</p>
<p>But then,… Like all great ideas. It evolved. Since one of the organisers of the event was Microsoft, I thought that I might look into Microsoft’s offering on Azure. So, what is managed alternative for Kafka / ksqlDB / InfluxDB on Azure?</p>
<p>Let’s ask the expert in the house. 💡</p>
<p>After a beautiful and insightful session with one of Microsoft’s Senior Cloud Architects, I got all the answers that I needed for this part. I just replaced the original components with Azure’s similar offering. And this is the result.</p>
<p><img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/architecture_overview.png" alt="Voda on Azure" /></p>
<p>For those curious; what is used.</p>
<ul>
<li><a href="https://azure.microsoft.com/">Microsoft Azure Cloud</a> as a magical platform,…</li>
<li><a href="https://azure.microsoft.com/en-us/services/event-hubs/">Microsoft Azure <strong>Event Hubs</strong></a>, fully managed real-time ingestion service. That integrates seamlessly with existing <a href="https://kafka.apache.org/">Apache Kafka</a> clients. Capable of streaming millions of events per second. Fun fact. It also comes with <a href="https://azure.microsoft.com/es-es/blog/schema-validation-with-event-hubs/">Schema Registry</a>.</li>
<li><a href="https://azure.microsoft.com/en-us/services/kubernetes-service/">Microsoft Azure <strong>Kubernetes</strong> Service</a> (AKS) - Highly available, secure, and fully managed Kubernetes service. The commonplace to deploy and run your containerised application.</li>
<li><a href="https://azure.microsoft.com/en-us/services/stream-analytics">Microsoft Azure <strong>Stream Analytics</strong></a> - Serverless real-time analytics, from the cloud to the edge. Think of this as fully managed “streaming analytics platform.”</li>
<li><a href="https://azure.microsoft.com/en-us/services/container-registry/">Microsoft Azure <strong>Container Registry</strong></a> - Azure’s answer to Docker Hub or GitHub container registry or … AWS’s or Google GCP’s. A place for storing your container images.</li>
<li><a href="https://azure.microsoft.com/en-us/services/time-series-insights/">Microsoft Time <strong>Series Insights</strong></a> (Environment and Explorer) - End-to-end IoT analytics platform to monitor, analyse, and visualise your industrial IoT data at scale. Think of it as your go-to IoT solution on Azure.</li>
<li>Few other bits here and there,… but that’s mostly it.</li>
</ul>
<h3 id="the-scala--akka-application">The Scala / Akka application</h3>
<p>Conceptually the application is split into two components; one long-running process that fetches water data every 5 seconds, transforms XML and pushes that into Azure Event Hub and another that is manually triggered - per demand - and fetches the whole archive in CSV and again pushes to the hub.</p>
<h4 id="collection-of-current-measurements">Collection of current measurements</h4>
<p>Let’s look in the core or main one first,… First attempt at the collection,…</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">val</span> <span class="nv">toEventData</span><span class="k">:</span> <span class="kt">StationReadingCurrent</span> <span class="o">=></span> <span class="nc">EventData</span> <span class="k">=</span>
<span class="n">m</span> <span class="k">=></span> <span class="k">new</span> <span class="nc">EventData</span><span class="o">(</span><span class="nv">m</span><span class="o">.</span><span class="py">asJson</span><span class="o">.</span><span class="py">noSpaces</span><span class="o">)</span>
<span class="k">val</span> <span class="nv">hub</span> <span class="k">=</span> <span class="nv">AzureEventBus</span><span class="o">.</span><span class="py">currentMeasurementsProducer</span>
<span class="k">val</span> <span class="nv">collection</span> <span class="k">=</span> <span class="nv">Source</span><span class="o">.</span><span class="py">tick</span><span class="o">(</span><span class="mf">0.</span><span class="n">seconds</span><span class="o">,</span> <span class="mf">5.</span><span class="n">seconds</span><span class="o">,</span> <span class="nv">Model</span><span class="o">.</span><span class="py">Tick</span><span class="o">)</span>
<span class="c1">// flow for requesting and parsing</span>
<span class="o">.</span><span class="py">via</span><span class="o">(</span><span class="nv">HydroData</span><span class="o">.</span><span class="py">currentFlow</span><span class="o">)</span>
<span class="c1">// Emitting the events to Azure Event Hub</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">measurement</span> <span class="k">=></span> <span class="nv">hub</span><span class="o">.</span><span class="py">send</span><span class="o">(</span><span class="n">measurement</span><span class="o">))</span>
<span class="o">.</span><span class="py">runWith</span><span class="o">(</span><span class="nv">Sink</span><span class="o">.</span><span class="py">ignore</span><span class="o">)</span>
</code></pre></div></div>
<p>And parsing,…</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">currentFlow</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">URL</span><span class="o">)</span>
<span class="o">(</span><span class="k">implicit</span> <span class="n">system</span><span class="k">:</span> <span class="kt">ActorSystem</span><span class="o">,</span>
<span class="n">config</span><span class="k">:</span> <span class="kt">Configuration.Config</span><span class="o">)</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">system.dispatcher</span>
<span class="c1">// Transformation of XML to "list" of "readings"</span>
<span class="k">val</span> <span class="nv">toStationSeq</span><span class="k">:</span> <span class="kt">NodeSeq</span> <span class="o">=></span> <span class="nc">Seq</span><span class="o">[</span><span class="kt">StationReadingCurrent</span><span class="o">]</span> <span class="k">=</span> <span class="n">xml</span> <span class="k">=></span>
<span class="o">(</span><span class="n">xml</span> <span class="o">\\</span> <span class="s">"postaja"</span><span class="o">).</span><span class="py">map</span> <span class="o">{</span> <span class="n">x</span> <span class="k">=></span>
<span class="nc">StationReadingCurrent</span><span class="o">(</span>
<span class="n">sifra</span> <span class="k">=</span> <span class="n">x</span> <span class="o">\@</span> <span class="s">"sifra"</span><span class="o">,</span> <span class="c1">// Station key</span>
<span class="n">reka</span> <span class="k">=</span> <span class="o">(</span><span class="n">x</span> <span class="o">\</span> <span class="s">"reka"</span><span class="o">).</span><span class="py">text</span><span class="o">,</span> <span class="c1">// River name</span>
<span class="n">tempVode</span> <span class="k">=</span> <span class="o">(</span><span class="n">x</span> <span class="o">\</span> <span class="s">"temp_vode"</span><span class="o">).</span><span class="py">text</span><span class="o">.</span><span class="py">toDoubleOption</span> <span class="c1">// Temperature</span>
<span class="c1">// Others are emitted...</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="c1">// Part that fetches and invokes transforamtion</span>
<span class="k">def</span> <span class="nf">fetch</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">Source</span><span class="o">[</span><span class="kt">StationReadingCurrent</span>, <span class="kt">NotUsed</span><span class="o">]]</span> <span class="k">=</span>
<span class="nc">Http</span><span class="o">().</span><span class="py">singleRequest</span><span class="o">(</span><span class="nc">HttpRequest</span><span class="o">(</span><span class="n">uri</span> <span class="k">=</span> <span class="n">url</span><span class="o">))</span>
<span class="o">.</span><span class="py">flatMap</span><span class="o">(</span><span class="n">r</span> <span class="k">=></span> <span class="nc">Unmarshal</span><span class="o">(</span><span class="n">r</span><span class="o">).</span><span class="py">to</span><span class="o">[</span><span class="kt">NodeSeq</span><span class="o">])</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">toStationSeq</span><span class="o">)</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nc">Source</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
<span class="c1">// Flow of ticks, that generates requests, ... </span>
<span class="nc">Flow</span><span class="o">[</span><span class="kt">Model.Tick</span><span class="o">]</span>
<span class="o">.</span><span class="py">mapAsyncUnordered</span><span class="o">(</span><span class="mi">1</span><span class="o">)(</span><span class="k">_</span> <span class="k">=></span> <span class="n">fetch</span><span class="o">)</span>
<span class="o">.</span><span class="py">flatMapConcat</span><span class="o">(</span><span class="n">identity</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now; since we live in a world where services go down, and in a world where it is irrational to bombard them with too much data….</p>
<p>We can make our collection a bit more resilient by wrapping the collection flow with <a href="https://doc.akka.io/docs/akka/current/stream/operators/RestartFlow/onFailuresWithBackoff.html">RestartFlow.onFailuresWithBackoff</a>. By doing this; we now made sure that if targeted service is down, our app will back-off, wait a bit and retry. If that fails 10 times; the whole app will come to a stop and terminate. Then ultimately - in our case - be restarted by Kubernetes.</p>
<p>The first enhancements look like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">RestartFlow</span><span class="o">.</span><span class="py">onFailuresWithBackoff</span><span class="o">(</span><span class="mf">5.</span><span class="n">seconds</span><span class="o">,</span> <span class="mf">30.</span><span class="n">seconds</span><span class="o">,</span> <span class="mf">0.3</span><span class="o">,</span> <span class="mi">10</span><span class="o">)</span> <span class="o">{</span> <span class="o">()</span> <span class="k">=></span>
<span class="nc">Flow</span><span class="o">[</span><span class="kt">Model.Tick</span><span class="o">]</span>
<span class="o">.</span><span class="py">mapAsyncUnordered</span><span class="o">(</span><span class="mi">1</span><span class="o">)(</span><span class="k">_</span> <span class="k">=></span> <span class="n">fetch</span><span class="o">)</span>
<span class="o">.</span><span class="py">flatMapConcat</span><span class="o">(</span><span class="n">identity</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And the second enhancement, since we use Akka Streams and Azure Hubs here, is to bulk these measurements together and emit them in batches. This can easily be achieved with the introduction of <a href="https://doc.akka.io/docs/akka/current/stream/operators/Source-or-Flow/groupedWithin.html">groupedWithin</a>.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="nv">collection</span> <span class="k">=</span> <span class="nv">Source</span><span class="o">.</span><span class="py">tick</span><span class="o">(</span><span class="mf">0.</span><span class="n">seconds</span><span class="o">,</span> <span class="mf">5.</span><span class="n">seconds</span><span class="o">,</span> <span class="nv">Model</span><span class="o">.</span><span class="py">Tick</span><span class="o">)</span>
<span class="o">.</span><span class="py">via</span><span class="o">(</span><span class="nv">HydroData</span><span class="o">.</span><span class="py">currentFlow</span><span class="o">)</span>
<span class="c1">// This creates neet Seq of measurement batches</span>
<span class="o">.</span><span class="py">groupedWithin</span><span class="o">(</span><span class="mi">100</span><span class="o">,</span> <span class="mf">100.</span><span class="n">milliseconds</span><span class="o">)</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">measurements</span> <span class="k">=></span>
<span class="c1">// New "batch" is created and EventData "list" is attached to it. </span>
<span class="nv">hub</span><span class="o">.</span><span class="py">send</span><span class="o">(</span><span class="nv">measurements</span><span class="o">.</span><span class="py">foldLeft</span><span class="o">(</span><span class="nv">hub</span><span class="o">.</span><span class="py">createBatch</span><span class="o">())</span> <span class="o">{</span> <span class="o">(</span><span class="n">batch</span><span class="o">,</span> <span class="n">m</span><span class="o">)</span> <span class="k">=></span>
<span class="nv">batch</span><span class="o">.</span><span class="py">tryAdd</span><span class="o">(</span><span class="n">m</span><span class="o">)</span>
<span class="n">batch</span>
<span class="o">})</span>
<span class="o">).</span><span class="py">runWith</span><span class="o">(</span><span class="nv">Sink</span><span class="o">.</span><span class="py">ignore</span><span class="o">)</span>
</code></pre></div></div>
<p>And since we don’t want to rely on the timestamp of the original data; we also override the “date”, with the date that the measurement was actually pushed to the hub with simple “copy”, like so</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">hub</span><span class="o">.</span><span class="py">send</span><span class="o">(</span><span class="nv">measurements</span><span class="o">.</span><span class="py">foldLeft</span><span class="o">(</span><span class="nv">hub</span><span class="o">.</span><span class="py">createBatch</span><span class="o">())</span> <span class="o">{</span> <span class="o">(</span><span class="n">batch</span><span class="o">,</span> <span class="n">m</span><span class="o">)</span> <span class="k">=></span>
<span class="c1">// Override of "date" with current date</span>
<span class="nv">batch</span><span class="o">.</span><span class="py">tryAdd</span><span class="o">(</span><span class="nv">m</span><span class="o">.</span><span class="py">copy</span><span class="o">(</span><span class="n">datum</span> <span class="k">=</span> <span class="nv">LocalDateTime</span><span class="o">.</span><span class="py">now</span><span class="o">))</span>
<span class="n">batch</span>
<span class="o">})</span>
</code></pre></div></div>
<h3 id="collection-of-the-archive-data">Collection of the archive data</h3>
<p>The central part is very similar to the first application; just without the <a href="https://doc.akka.io/docs/akka/current/stream/operators/Source/tick.html">Source.tick</a>.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">val</span> <span class="nv">toEventData</span><span class="k">:</span> <span class="kt">StationReadingHistorical</span> <span class="o">=></span> <span class="nc">EventData</span> <span class="k">=</span>
<span class="n">m</span> <span class="k">=></span> <span class="k">new</span> <span class="nc">EventData</span><span class="o">(</span><span class="nv">m</span><span class="o">.</span><span class="py">asJson</span><span class="o">.</span><span class="py">noSpaces</span><span class="o">)</span>
<span class="k">val</span> <span class="nv">hub</span> <span class="k">=</span> <span class="nv">AzureEventBus</span><span class="o">.</span><span class="py">historicalMeasurementsProducer</span>
<span class="k">val</span> <span class="nv">f</span> <span class="k">=</span> <span class="nv">VodaArchive</span><span class="o">.</span><span class="py">stationsSource</span>
<span class="o">.</span><span class="py">via</span><span class="o">(</span><span class="nv">VodaArchive</span><span class="o">.</span><span class="py">collectMetrics</span><span class="o">)</span>
<span class="c1">// Bigger batches here</span>
<span class="o">.</span><span class="py">groupedWithin</span><span class="o">(</span><span class="mi">5000</span><span class="o">,</span> <span class="mf">1.</span><span class="n">second</span><span class="o">)</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">measurements</span> <span class="k">=></span>
<span class="nv">hub</span><span class="o">.</span><span class="py">send</span><span class="o">(</span><span class="nv">measurements</span><span class="o">.</span><span class="py">foldLeft</span><span class="o">(</span><span class="nv">hub</span><span class="o">.</span><span class="py">createBatch</span><span class="o">())</span> <span class="o">{</span> <span class="o">(</span><span class="n">batch</span><span class="o">,</span> <span class="n">m</span><span class="o">)</span> <span class="k">=></span>
<span class="nv">batch</span><span class="o">.</span><span class="py">tryAdd</span><span class="o">(</span><span class="n">m</span><span class="o">)</span>
<span class="n">batch</span>
<span class="o">})</span>
<span class="o">).</span><span class="py">runWith</span><span class="o">(</span><span class="nv">Sink</span><span class="o">.</span><span class="py">ignore</span><span class="o">)</span>
<span class="c1">// Note: I could just write custom Sink for AzureEventBus, would be cleaner.</span>
</code></pre></div></div>
<p>Since there is way more requests here that are emitted to fetch the data for every station for every year for last 20 years; <a href="https://doc.akka.io/docs/akka/current/stream/operators/Source-or-Flow/throttle.html">throttling</a> is introduced.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">VodaArchive</span><span class="o">.</span><span class="py">buildRequests</span>
<span class="c1">// Throttling of requests before they are sent out.</span>
<span class="o">.</span><span class="py">throttle</span><span class="o">(</span><span class="mi">60</span><span class="o">,</span> <span class="mf">1.</span><span class="n">second</span><span class="o">,</span> <span class="mi">40</span><span class="o">,</span> <span class="nv">ThrottleMode</span><span class="o">.</span><span class="py">Shaping</span><span class="o">)</span>
<span class="o">.</span><span class="py">mapAsync</span><span class="o">(</span><span class="mi">4</span><span class="o">)</span> <span class="o">{</span> <span class="nf">case</span> <span class="o">(</span><span class="n">r</span><span class="o">,</span> <span class="n">ctx</span><span class="o">)</span> <span class="k">=></span>
<span class="nc">Http</span><span class="o">().</span><span class="py">singleRequest</span><span class="o">(</span><span class="n">r</span><span class="o">)</span>
<span class="o">.</span><span class="py">flatMap</span><span class="o">(</span><span class="nc">Unmarshal</span><span class="o">(</span><span class="k">_</span><span class="o">).</span><span class="py">to</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="n">body</span> <span class="k">=></span> <span class="nf">parseCSV</span><span class="o">(</span><span class="n">body</span><span class="o">)(</span><span class="n">r</span><span class="o">))</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nf">concat</span><span class="o">(</span><span class="k">_</span><span class="o">)(</span><span class="n">ctx</span><span class="o">))))</span>
<span class="o">}</span>
<span class="o">.</span><span class="py">filterNot</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">isEmpty</span><span class="o">)</span>
<span class="o">.</span><span class="py">mapConcat</span><span class="o">(</span><span class="n">identity</span><span class="o">)</span> <span class="c1">// Oops,...</span>
<span class="o">.</span><span class="py">filterNot</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">isEmpty</span><span class="o">)</span>
<span class="o">.</span><span class="py">map</span><span class="o">(</span><span class="nv">_</span><span class="o">.</span><span class="py">getOrElse</span><span class="o">(</span><span class="k">throw</span> <span class="k">new</span> <span class="nc">Exception</span><span class="o">(</span><span class="s">"Only Row pass this point."</span><span class="o">)))</span>
<span class="c1">// Note: I could juse use FlowWithContext / SourceWithContext.fromTuples</span>
</code></pre></div></div>
<p>Oh; and the Azure Event Hub producers look like this</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">AzureEventBus</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">currentMeasurementsProducer</span><span class="o">(</span><span class="k">implicit</span> <span class="n">config</span><span class="k">:</span> <span class="kt">Configuration.Config</span><span class="o">)</span> <span class="k">=</span>
<span class="k">new</span> <span class="nc">EventHubClientBuilder</span><span class="o">()</span>
<span class="o">.</span><span class="py">connectionString</span><span class="o">(</span><span class="nv">config</span><span class="o">.</span><span class="py">collecting</span><span class="o">.</span><span class="py">currentMeasurements</span><span class="o">.</span><span class="py">connectionString</span><span class="o">)</span>
<span class="o">.</span><span class="py">buildProducerClient</span><span class="o">()</span>
<span class="k">def</span> <span class="nf">historicalMeasurementsProducer</span><span class="o">(</span><span class="k">implicit</span> <span class="n">config</span><span class="k">:</span> <span class="kt">Configuration.Config</span><span class="o">)</span> <span class="k">=</span>
<span class="k">new</span> <span class="nc">EventHubClientBuilder</span><span class="o">()</span>
<span class="o">.</span><span class="py">connectionString</span><span class="o">(</span><span class="nv">config</span><span class="o">.</span><span class="py">collecting</span><span class="o">.</span><span class="py">historicalMeasurements</span><span class="o">.</span><span class="py">connectionString</span><span class="o">)</span>
<span class="o">.</span><span class="py">buildProducerClient</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And thats the core,… 😊</p>
<h3 id="docker-helm-and-deployment">Docker, HELM and deployment</h3>
<p>The two apps were bundled together into one application, and packeged together into Docker Image with help of <a href="https://github.com/sbt/sbt-native-packager">sbt-native-packager</a>. The Docker image was then pushed to Azure Container Registry (ACR).</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Boot</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">main</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">Array</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nv">Array</span><span class="o">.</span><span class="py">empty</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span>
<span class="nf">if</span> <span class="o">(</span><span class="nv">args</span><span class="o">.</span><span class="py">contains</span><span class="o">(</span><span class="s">"--archive"</span><span class="o">))</span>
<span class="nv">ArchiveMain</span><span class="o">.</span><span class="py">main</span><span class="o">(</span><span class="n">args</span><span class="o">)</span>
<span class="k">else</span>
<span class="nv">FetchMain</span><span class="o">.</span><span class="py">main</span><span class="o">(</span><span class="n">args</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Some Bash used for deployment,…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Docker Image is built and pushed to Azure Container Registry.</span>
<span class="c"># Then HELM is invoked to upgrade the thing.</span>
sbt docker:publish <span class="o">&&</span> helm upgrade one chart/voda <span class="se">\</span>
<span class="nt">-f</span> chart/voda/values.yaml <span class="nt">--set</span> image.tag<span class="o">=</span>0.1.3
</code></pre></div></div>
<p>The thing is now deployed to Kubernetes,… <strong>… and collection runs smoothly. Win! 🏆</strong></p>
<h4 id="sidenotes-and-tips">Sidenotes and tips:</h4>
<ul>
<li>
<p><em>Frinds don’t let friends use <code class="language-plaintext highlighter-rouge">kubectl</code>!</em> Use something like <a href="https://k9scli.io/">k9s</a> for inspecting he state of your application on Kubernetes.
<img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/k9s.png" alt="k9s" /></p>
</li>
<li>
<p>At this point. I should also state that using <a href="https://skaffold.dev">skaffold</a> also came to my mind.
However, this lovely error massage just made me not wanna spend more time on it. Perhaps in some other case…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>skaffold init
Projects <span class="nb">set </span>up to deploy with helm must be manually configured.
See https://skaffold.dev/docs/pipeline-stages/deployers/helm/ <span class="k">for
</span>a detailed guide on setting your project up with skaffold.
</code></pre></div> </div>
<p>Althoug I didn’t use Skaffold here. I strongly suggest that you use it for rapid development; especially its <code class="language-plaintext highlighter-rouge">skaffold dev</code> command</p>
</li>
<li>
<p><a href="https://www.telepresence.io/">Telepresence</a> - this is also one of those tools that you just have to use if you wanna move quickly around Kubernetes “ecosystem”. What it does is simple, it maps kuberntes ports and “hosts” to your local machine; and with that you can then your run your local app as it would be deployed inside Kubernetes. Very useful for debugging or just plain development.</p>
</li>
</ul>
<h2 id="day-2-wrap-up-finish-and-the-results">Day 2. Wrap-up, finish and the results</h2>
<p>After all the magical things were disigned, put together, deployed and once I’ve explored and configured everything that is sketched above… It was the second and final day of the hackathon. On this day I focused myself into wrapping up what I’ve started. I’ve polished the code a bit, made few changes here and there and made sure that collection and visualisation was working as expected.</p>
<p>Here, dear reader.</p>
<p>I give you a lovely real-time chart. On the left side you’ll see three measurement stations that are located on river <a href="https://en.wikipedia.org/wiki/Ljubljanica">Ljubljanica</a>. Each of these stations measures temperature, flow and watter level.</p>
<p><img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/s1.png" alt="Visualisation in Time Series Insights I." /></p>
<p><img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/s2.png" alt="Visualisation in Time Series Insights II." /></p>
<p><img src="/assets/2020-10-05-hackathon-with-scala-akka-on-azure/s3.png" alt="Visualisation in Time Series Insights III." /></p>
<p>And some more stuff,..</p>
<p>I played around some more with streaming of events and transformation with Streaming Queries. So this also worked nicely; here a snippet if you want to transform JSON to “table rows” (SQL table) via Azure Streaming Analytics:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span>
<span class="o">*</span><span class="p">,</span>
<span class="c1">-- Extraction of date "parts" from timestamps</span>
<span class="n">DATEPART</span> <span class="p">(</span><span class="nb">year</span><span class="p">,</span> <span class="n">stream</span><span class="p">.</span><span class="n">datum</span><span class="p">)</span> <span class="k">as</span> <span class="n">c_year</span><span class="p">,</span>
<span class="n">DATEPART</span> <span class="p">(</span><span class="n">dayofyear</span><span class="p">,</span> <span class="n">stream</span><span class="p">.</span><span class="n">datum</span><span class="p">)</span> <span class="k">as</span> <span class="n">c_dayofyear</span><span class="p">,</span>
<span class="n">DATEPART</span> <span class="p">(</span><span class="n">quarter</span><span class="p">,</span> <span class="n">stream</span><span class="p">.</span><span class="n">datum</span><span class="p">)</span> <span class="k">as</span> <span class="n">c_quarter</span><span class="p">,</span>
<span class="c1">-- Creating a Geo-Point out of two floats (Coordinates)</span>
<span class="n">CreatePoint</span><span class="p">(</span><span class="n">stream</span><span class="p">.</span><span class="n">geSirina</span><span class="p">,</span> <span class="n">stream</span><span class="p">.</span><span class="n">geDolzina</span><span class="p">)</span> <span class="k">as</span> <span class="k">location</span>
<span class="k">INTO</span>
<span class="c1">-- Destination Azure Event Hub (or Table)</span>
<span class="p">[</span><span class="n">historicalmeasurements</span><span class="p">]</span>
<span class="k">FROM</span>
<span class="c1">-- Azure Event Hub</span>
<span class="p">[</span><span class="n">eh</span><span class="o">-</span><span class="n">archive</span><span class="p">]</span> <span class="k">AS</span> <span class="n">stream</span>
</code></pre></div></div>
<h3 id="finish-line-">Finish-line 🏁</h3>
<p>I then filled out all the needed forms that organisers needed and wanted. Described my idea, solution and uniqness of it in great detail. Submitted everything… And waited… Waited some more… Fell asleep…</p>
<p>The results came in.</p>
<p><strong>I didn’t make it. Sry. 😞</strong> (🎻 <- this violin plays just for me)</p>
<h2 id="conclusion-and-summary">Conclusion and summary</h2>
<ol>
<li>You should attend a hackathon. Its great fun and you can meet a lot of great people. It will also force you to think about problems more, and it will also make you think more about what “can” you do within time constraints and what is out of reach or scope.</li>
<li>Don’t expect to win just because you use fancy tech. Business people don’t care about it! They love shiny slide decks and magical videos that speak their language.</li>
<li>Akka is incredible; the more you master it, the more will another tech around you look old and inadequate.</li>
<li>Socialise. Although I went intentionally 100% solo in this endeavour and had support from organisers and partners, these things are social events. Perhaps the final idea might be as good if not better If I would collaborate with people more.</li>
<li>Perhaps just draw a sketch or fake the data. (Yeah I hate that, just not me, I hate cheating!)</li>
<li>Hackathons are mostly not what you can build; they are more about how and what you present.</li>
<li>Research who is the target audience, what problem are you solving and if the jury appreciates the solution</li>
<li>HALM, K8, etc… are over-kill unless you want to have fun and you don’t care.</li>
<li>Data. Companies “hord” vast amount of incredibly valuable and unique data and don’t have the experts, tools or knowledge on how to monetise that data.</li>
<li><strong>If this kind of challenge is also your challenge; please let me know. I want to know more about it!</strong> ⚠️</li>
<li>You are never too old to have fun! Life is short. Live it.</li>
<li>We only have one planet, take care of it! 🌍</li>
</ol>
<h2 id="ps-whats-up-with-azure-dude">P.s.: What’s up with Azure, dude?</h2>
<p>This is the first time I’ve used Microsoft Azure. I’ve been running and building things on AWS, GCP, DigitialOcean, Hetzner, Heroku, … own metal, etc. for ages now. But I never gave Microsoft a chance. Perhaps I’m one of the “old guys” now that has been around when Microsoft was waaaaay less open if not hostile to Open Source movement. So I always take Microsoft with caution…</p>
<p>I was very very impressed by Azure. Things felt very enterprise-ready, very coherent; the UI was clean; I felt “home”. I did struggle a bit with “resource groups” - I still think that they are entirely unnecessary, and I did have some challenges with finding the right logs, and reporting tools inside Azure.
I was impressed by how well Streaming Analytics was packaged inside the SQL Server setup. As in - “look, just set it up here… its easy”. That was some nice up-sell from the Azure team.</p>
<p>Another plus; I’ve discovered this after the hackathon was; <a href="https://azure.microsoft.com/en-us/services/devops/">Azure DevOps Services</a>. You can quite trivially setup <a href="https://docs.github.com/en/free-pro-team@latest/actions">GitHub Actions</a>, and CI/CD pipeline with DevOps services. Making the deployment even more accessible and even more friendly.
Sidenote. Microsoft didn’t put a cap on accounts that we used for the event. That gave me free hands to explore “everything” that I could get my hands on. I need to admit that perhaps if I had a bit more time, I would explore more of their IoT offering and machine learning suite. That s*** also looks extremely well-integrated and “easy”.</p>
<p>Fin.</p>
<p>P.s.s.: Give me some feedback if this kind or articles are interesting to you… or not? Below. 👇</p>
<p>P.s.s.s.: All the code from this hackathon are hosted on <a href="https://github.com/pinkstack/voda">GitHub the project name is “voda”</a>.</p>
<p>P.s.s.s.s: “Voda” stands for water in <a href="https://en.wikipedia.org/wiki/Slovenia">Slovenian 🇸🇮</a>.</p>While browsing through the unlimited supply of sh** posts on Facebook, I’ve stumbled upon an exciting invite to an “environmental hackathon” - named “Living with environmental changes / Življenje s podnebnimi spremembami” (event).Reverse engineering an Android Application2020-07-27T07:00:00+00:002020-07-27T07:00:00+00:00https://epic.blog/reverse-engineering/2020/07/27/reverse-engineering-android-app<p>Hello, a good reader of my <a href="https://epic.blog">epic.blog</a>!</p>
<p>On this lovely and streaming hot July day, I wish to take you on a journey of my reverse engineering trip across the mountains of Android and the valleys of decompilers. Pour yourself a cold drink and enjoy this voyage.</p>
<p>So; I wanted to demonstrate how to reverse engineer an Android Application and what tools you can use to achieve this - even without owning an Android Phone.</p>
<p>The app used for this demonstration is called <a href="https://play.google.com/store/apps/details?id=hr.molekula.bikekrk">Krk Bike</a>, and it is a mobile application that you can download from <a href="https://play.google.com/store/apps/details?id=hr.molekula.bikekrk">Google Play store</a>, and it will show you many of bike trails on <a href="http://www.krk.hr/en/">Croatia’s island of Krk</a>. <em>I wanted to see all of this bike trails in a single, un-cluttered map.</em> How could we get such data out of this app?</p>
<p>When you open up this app on your phone, it looks something like this:</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/krk-bike-in-play-store.png" alt="krk-bike-in-play-store" /></p>
<p>The app itself has detailed trails with paths and pictures plus contacts, and it even comes with small “navigation module” that will guide you through the track while while you are on the route. Pretty decent and slick app. Obviously - dear reader - you’ll choose your own target at some point, but this one is interesting to me. It is definitely a good first step to be familiar with the app as much as possible.</p>
<h2 id="how-are-android-apps-built">How are Android apps built?</h2>
<p>Well. First, you need to understand how Android apps are <strong>built</strong> and <strong>distributed</strong>. Usually, Android developers develop their apps with the help of <a href="https://developer.android.com/studio">Android Studio</a>, compile them, sign them and upload the “apks” to Google Play Store. The more detailed explanation about the process of building Android apps for beginners and getting them to end-users can be found on <a href="https://developer.android.com/training/basics/firstapp/creating-project">Android documentation page (Create an Android project)</a>.</p>
<p>Hold on, mister! ✋How are apps compiled, and how do they even run?</p>
<p>Now that you’ve asked. In a nutshell. Usually, Android apps are compiled in a few stages. The first stage; depending on your source code gets compiled either with Java compiler or Kotlin compiler to Java bytecode. These compilers spit-out <code class="language-plaintext highlighter-rouge">*.class</code> files. The <code class="language-plaintext highlighter-rouge">*.class</code> files are then fed into DX (<a href="https://developer.android.com/studio/command-line/d8">DEX compiler - <code class="language-plaintext highlighter-rouge">d8</code></a>). DEX compiler spits out DEX bytecode that runs on Android devices, and it allows you to use Java 8 language features in your app’s code. DEX bytecode is then something that <a href="https://en.wikipedia.org/wiki/Dalvik_(software)">Dalvik Virtual Machine (DVM for short)</a> actually runs your app.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/android-app/compilation.png" alt="compilation" /></p>
<p>Now; if you know anything about Android. You’re likely to say: “Sorry, Oto! Old news! DVM is deprecated since KitKat.” And you would be right! Dalvik has been replaced with something much more magical - called Android Runtime (ART for short). But few core concepts are the same, just better.</p>
<blockquote>
<p>The successor of Dalvik is <a href="https://en.wikipedia.org/wiki/Android_Runtime">Android Runtime</a> (ART), which uses the same bytecode and .dex files (but not .odex files), with the succession aiming at performance improvements transparent to the end users. The new runtime environment was included for the first time in Android 4.4 “KitKat” as a technology preview, and replaced Dalvik entirely in later versions; Android 5.0 “Lollipop” is the first version in which ART is the only included runtime.</p>
</blockquote>
<p>Ok; so the next step in the building process for any Android app is to package it into something that is called <a href="https://developer.android.com/google/play/expansion-files">APK</a>. An <a href="https://developer.android.com/google/play/expansion-files">Android Package Kit (APK for short)</a> is the package file format used by the Android operating system for distribution and installation of mobile apps. Think of it as a package with some more meta-information attached, sophisticated Java <code class="language-plaintext highlighter-rouge">*.jar</code> or Debian’s <code class="language-plaintext highlighter-rouge">*.deb</code> package.</p>
<p>After developers successfully build these APK’s they “push” them to devices or to Google Play Store. If its development process, that’s usually done via <a href="https://developer.android.com/studio/command-line/adb"><code class="language-plaintext highlighter-rouge">adb</code></a> in command-line (or Android Studio will use something similar in the background for you). If its production release, people sometimes also <a href="https://developer.android.com/studio/publish/app-signing">sign these things</a> with their keys either with <a href="https://developer.android.com/studio">Android Studio</a> or with <a href="https://developer.android.com/studio/command-line/apksigner"><code class="language-plaintext highlighter-rouge">apksigner</code></a> via CLI directly. The Android system uses the certificate as a means of identifying the author of an application and establishing trust relationships between applications.</p>
<p>Ok, now that you know how apps are built, packaged and pushed to store,…</p>
<h2 id="how-can-i-get-apk-of-an-app-from-google-play-store">How can I get APK of an app from Google Play Store?</h2>
<p>There are multiple ways you can get APK of any given Android App. The most trivial is to use one of multiple “mirror” sites that collect these APKs. Sometimes they automate the process and sometimes - as I found out via my uber Googling skills - they just have people fetch populat APKs manually. To name a few of these <a href="https://apkpure.com/">apkpure.com</a>, <a href="https://apps.evozi.com/apk-downloader/">evozi APK Downlowder</a> and <a href="https://www.apkmirror.com/">apkmirror.com</a>. These sites have few disadvantages; the foremost being that not “all” apk’s that are on Google Play Store is available, and that this is a “mirror” site and there is no way of assuring that these apk’s and Apps that these belong to are untempered. Although hard, it is possible to inject malicious code and spread it via these or similar sites.</p>
<h3 id="-but-they-dont-have-my-app-there">… but they don’t have my app there!</h3>
<p>Oh, yeah. In that case, you can do what I did. You can</p>
<ol>
<li>
<p>Install <a href="https://www.virtualbox.org/">Oracle VirtualBox</a> on your machine.</p>
</li>
<li>
<p>Get Android-x86 OS image from <a href="https://www.osboxes.org/android-x86/">OSBoxes.org (follow this link)</a>.</p>
</li>
<li>
<p>Bootup the Android image and then go to Google Play Store,…</p>
</li>
<li>
<p>then install the app you wanna look into.</p>
</li>
<li>
<p>after app installs, download an app called <a href="https://play.google.com/store/apps/details?id=com.yschi.MyAppSharer">MyAppSharer</a>.</p>
<p>I stored the APK of targeted app into <code class="language-plaintext highlighter-rouge">/storage/emulated/0/</code></p>
</li>
<li>
<p>at this point, you can either install Android tools on your machine and then use adb to pull apk to host machine with following command. <strong>Or</strong> you can creativly just email yourself apk from MyAppSherer app in the emulator. (lol)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb pull /storage/emulated/0/hr.molekula.bikekrk.apk
</code></pre></div> </div>
</li>
</ol>
<p>At this point, I wish to clarify a few things. Although for a given example this works; Android-x86 is an open-source effort to port Android to x86 architecture. An experiment that has now also extended beyond the primary x86 platform. Please make sure you visit their site and <a href="https://www.android-x86.org/donate.html">buy them a coffee</a> if you like the project. Second. There are also other ways where you could pull apk from an Android phone or emulator, but I found all of them to be hard or unreliable, so that’s why I’m sharing this app here.</p>
<h2 id="i-have-the-apk--now-what-">I have the APK! ✅ Now what? 🤔</h2>
<p>After you obtain the APK of an app that you wish to peek into. The next step is to decompile <code class="language-plaintext highlighter-rouge">*.apk</code> into something useful. i.e. <code class="language-plaintext highlighter-rouge">*.java</code> source code. The decompilation process of Android source code is usually a two-step process. First, you unpack raw APK back to “files”, then those dalvik codes (<code class="language-plaintext highlighter-rouge">*.smil</code>) needs to be converted back to Java classes. Note at this stage. Compilation usually means trimming some characters, names and also introducing some optimisations, so by no means expect that the decompiled code will match back to original source code. But in general, it is good enough that an expert will be able to figure out what is going on.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/android-app/decompilation.png" alt="decompilation" /></p>
<p>In other to do that there are several open-source tools available that you can use. For this experiment, I’ve used these. Some of them are pretty basic, and very Android specific, others however, like NSA’s Ghidra are more general purpose reverse engineering tools.</p>
<h3 id="apktool">Apktool</h3>
<p>This reverse engineering tool is the most probably most sophisticated and Android specific from this list and it can disassemble resources nearly to the original; including XMLs, images and other assets.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> apktool_2.4.1.jar d hr.molekula.bikekrk.apk
</code></pre></div></div>
<p>If you run this tool it will spit out tree-structure of the original app that looks something like this; please note that it does not convert *.smil files back to Java source code. But it does an incredible job with other resources.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/apktool-files.png" alt="apktool-files" /></p>
<h3 id="java-decompiler">Java Decompiler</h3>
<p>This decompiler can be used practically on any Java application. If comes in tree flaveours. JD-GUI, the variation with user interface, JD-Eclipse as an plugin for Eclipse IDE and JD-Core as an lib.</p>
<p>The small gotcha with <a href="http://java-decompiler.github.io/">Java Decompiler</a> is that you need to convert your APK to JAR before it can be decompiled and inspected by it. To do that you need to geta tool called <a href="https://github.com/pxb1988/dex2jar">dex2jar</a> that will help you construct JAR.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./dex2jar-2.0/d2j-dex2jar.sh <span class="nt">-f</span> hr.molekula.bikekrk.apk <span class="c"># for DEX -> JAR</span>
java <span class="nt">-jar</span> jd-gui-1.6.6.jar <span class="c"># to bootup Java Decompiler GUI</span>
</code></pre></div></div>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/jd-gui.png" alt="jd-gui" /></p>
<h3 id="jadx">JADX</h3>
<p>This tool is very likely the most “for Android” that I’ve found and managed to use. It can be used with GUI or purely from CLI. It can also run directly on APK / DEX combo without the extra step like <a href="http://java-decompiler.github.io/">Java Decompiler</a> mentioned early. It has also magical ability to deal with deobfuscation and from what I can tell the best full text search, decleartion jumping and usage lookup. When ran; it looks like this:</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/jadx-gui.png" alt="jadx-gui" /></p>
<h3 id="ghidra">Ghidra</h3>
<p>Although I originaly explored tools that are very Android or Java specific. I also get a recommendation from a friend to look into usage of <a href="https://ghidra-sre.org/">Ghidra</a>, a more general purpose software reverse engineering (SRE) suite of tools developed by NSA’s Research Directorate in support of the Cybersecurity mission.</p>
<p>Initially if feels a bit cluncky, and although it should work directly on *.APK’s I had to covert APK to JAR with <a href="https://github.com/pxb1988/dex2jar">dex2jar</a> the same way as I did with <a href="https://github.com/skylot/jadx">JADX</a> before I could use it. There are other ways you could also just load <code class="language-plaintext highlighter-rouge">classes.dex</code> via “file system” import that I later found to also works.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/ghidra.png" alt="ghidra" /></p>
<p>Ghidra is <strong>definitly next level</strong> when it comes to reverse engineering and the toolbox seems to have everything and even more than a human might need for a simple experiment like the one I’m trying to demonstrate here.</p>
<h3 id="javadecompilerscom">javadecompilers.com</h3>
<p>In some cases you even might not need to install any of these tools and frameworks and you can just use the online solution like <a href="http://www.javadecompilers.com/">javadecompilers.com</a>. Alghouth bit cumbersone; it can still be used to look into simple Java / Android apps.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/javadecompilers.png" alt="javadecompilers" /></p>
<h2 id="i-have-the-source--now-what-">I have the source. ✅ Now what?! 💡</h2>
<p>After you’ve successfully obtained the source code with decompilation process the next step is to probably <strong>limit the scope of your investigation</strong> and start reading the code. In my case I was looking for two things; how does the app connect to the web service that exposes “tracks” and for the coordinates of the tracks themselfs. And secondly what security measures are used on server and client side to bridge the communication gap.</p>
<p>For my case, the easist thing was to search though the whole codebase for strings “<code class="language-plaintext highlighter-rouge">http</code>” and “<code class="language-plaintext highlighter-rouge">https</code>” and then start the search from there. With the help of JADX I’ve quickly found following sweet breadcrumb:</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/jadx-http.png" alt="jadx-http" /></p>
<p>That led me to this lovely class named <code class="language-plaintext highlighter-rouge">Url</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">hr.molekula.bikekrk.net</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Url</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">String</span> <span class="n">base</span> <span class="o">=</span> <span class="s">"http://krk.molekula.net/"</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">String</span> <span class="n">jsonBase</span> <span class="o">=</span> <span class="o">(</span><span class="n">base</span> <span class="o">+</span> <span class="s">"api/v1/"</span><span class="o">);</span>
<span class="c1">// few lines omitted here</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="nc">String</span> <span class="n">tracks</span> <span class="o">=</span> <span class="o">(</span><span class="n">jsonBase</span> <span class="o">+</span> <span class="s">"tracks"</span><span class="o">);</span>
<span class="c1">// and a few here,...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>So with simple logical deduction we can deduct that tracks are fetched by HTTP request via following path</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET http://krk.molekula.net/api/v1/tracks
</code></pre></div></div>
<p>Quick test with <a href="https://httpie.org/">HTTPie</a>,… 🍰</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http <span class="nt">--follow</span> get http://krk.molekula.net/api/v1/tracks
</code></pre></div></div>
<p>Spits out few more breadcrumbs… but, no luck! Looks like we need the second peace; API key?</p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">HTTP</span><span class="o">/</span><span class="m">1.1</span> <span class="m">401</span> <span class="ne">Unauthorized</span>
<span class="na">Content-Type</span><span class="p">:</span> <span class="s">application/json</span>
<span class="s"># few headers omitted.</span>
<span class="na">expires</span><span class="p">:</span> <span class="s">-1</span>
<span class="na">pragma</span><span class="p">:</span> <span class="s">no-cache</span>
<span class="p">{</span><span class="w">
</span><span class="nl">"error"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Unauthorized. Wrong or missing API key."</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>At this point it might be a good exercise to look deeper into how the app itself builds HTTP requests before they are dispatched to end service. Lets see what classes use <code class="language-plaintext highlighter-rouge">hr.molekula.bikekrk.net.Url</code> and what do they do with it and follow the trail from there… and that quickly leads us to a wrapper <code class="language-plaintext highlighter-rouge">BikeClient</code> that looks like this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">hr.molekula.bikekrk.net</span><span class="o">;</span>
<span class="c1">// ... few imports</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">BikeClient</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">AsyncHttpClient</span> <span class="n">client</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">AsyncHttpClient</span><span class="o">();</span>
<span class="kd">public</span> <span class="nf">BikeClient</span><span class="o">(</span><span class="nc">String</span> <span class="n">lang</span><span class="o">,</span> <span class="nc">String</span> <span class="n">apiKey</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">client</span><span class="o">.</span><span class="na">addHeader</span><span class="o">(</span><span class="s">"Language"</span><span class="o">,</span> <span class="n">lang</span><span class="o">);</span>
<span class="k">this</span><span class="o">.</span><span class="na">client</span><span class="o">.</span><span class="na">addHeader</span><span class="o">(</span><span class="s">"API-Key"</span><span class="o">,</span> <span class="n">apiKey</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">getJsonContent</span><span class="o">(</span><span class="nc">String</span> <span class="n">url</span><span class="o">,</span> <span class="nc">JsonHttpResponseHandler</span> <span class="n">responseHandler</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">client</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="k">new</span> <span class="nc">RequestParams</span><span class="o">(),</span> <span class="n">responseHandler</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Meaning that before request is build and dispatched it also gets two additional HTTP headers: <code class="language-plaintext highlighter-rouge">Language</code> and <code class="language-plaintext highlighter-rouge">API-Key</code>. Nice find. Now; where is this <code class="language-plaintext highlighter-rouge">API-Key</code> stored and what is the <code class="language-plaintext highlighter-rouge">Language</code>. Again; few clicks via usage finder in JADX will point us to class that - in most cases sets language to what users device uses - for demonstration purposes we’ll just default it to “en” and now we only need <code class="language-plaintext highlighter-rouge">api_key</code>. With text search we can now look around and we find this:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">hr.molekula.bikekrk</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">R</span> <span class="o">{</span>
<span class="c1">// A lot of stuff omitted ...</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">string</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">abc_action_bar_home_description</span> <span class="o">=</span> <span class="mi">2131230720</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">action_settings</span> <span class="o">=</span> <span class="mi">2131230768</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">api_key</span> <span class="o">=</span> <span class="mi">2131230840</span><span class="o">;</span> <span class="c1">// <- this, maybe?</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">app_name</span> <span class="o">=</span> <span class="mi">2131230841</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">app_name_short</span> <span class="o">=</span> <span class="mi">2131230842</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The R class in Android projects is a special class. <code class="language-plaintext highlighter-rouge">R.java</code> is the dynamically generated class, created during build process to dynamically identify all assets (from strings to Android widgets to layouts), for usage in java classes in Android app. And that means that <code class="language-plaintext highlighter-rouge">api_key</code> is not <code class="language-plaintext highlighter-rouge">2131230840</code> but rahter something else that is defined in XML.</p>
<p>At this point we know that there is an API key somewhere in the source/apk, but it is hidden somewhere else. A good time to go back to source code - in my case I went back to <a href="https://ibotpeaches.github.io/Apktool/">apktool</a> - and grep the whole decompiled code with resources included and I found the key itself…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">grep</span> <span class="nt">-nw</span> <span class="s2">"api_key"</span> <span class="nt">-r</span> <span class="nb">.</span>
./res/values/public.xml:941: <public <span class="nb">type</span><span class="o">=</span><span class="s2">"string"</span> <span class="nv">name</span><span class="o">=</span><span class="s2">"api_key"</span> <span class="nb">id</span><span class="o">=</span><span class="s2">"0x7f080078"</span> />
./res/values/strings.xml:124: <string <span class="nv">name</span><span class="o">=</span><span class="s2">"api_key"</span><span class="o">>[</span> API KEY HERE <span class="o">]</span></string>
./smali/hr/molekula/bikekrk/R<span class="nv">$string</span>.smali:80:.field public static final api_key:I <span class="o">=</span> 0x7f080078
</code></pre></div></div>
<p>And now we can reconstruct the full HTTP request with all the right headers like so with HTTPie:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http get https://krk.molekula.net/api/v1/tracks Language:en API-Key:[API KEY HERE]
</code></pre></div></div>
<p>And bingo… we get JSON response of all tracks with URLs to GPX files with coordinates for all paths.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/tracks-json.png" alt="tracks-json" /></p>
<h2 id="reconstruction-of-the-map-">Reconstruction of the map 🗺</h2>
<p>Now that we know where data is and how to get it; we can fetch that and re-use it in some way. For example; this is a simple BASH script that fetches all that GPX track files.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env bash</span>
<span class="nb">set</span> <span class="nt">-ex</span>
<span class="nb">export </span><span class="nv">LANGUAGE</span><span class="o">=</span>en
<span class="nb">export </span><span class="nv">API_KEY</span><span class="o">=</span>API_KEY_COULD_BE_HERE
<span class="nb">rm</span> <span class="nt">-rf</span> ./<span class="k">*</span>.gpx<span class="k">*</span> <span class="o">&&</span> <span class="se">\</span>
http get https://krk.molekula.net/api/v1/tracks <span class="se">\</span>
Language:<span class="k">${</span><span class="nv">LANGUAGE</span><span class="k">}</span> API-Key:<span class="k">${</span><span class="nv">API_KEY</span><span class="k">}</span> <span class="se">\</span>
| jq <span class="nt">-r</span> <span class="s2">".data[].gpx"</span> | parallel <span class="nt">--gnu</span> <span class="s2">"wget {}"</span>
</code></pre></div></div>
<p>And the final result - when all the GPX tracks are visualised together is pretty lovely.</p>
<p><img src="/assets/2020-07-27-reverse-engineering-android-app/map-full.png" alt="map-full" /></p>
<h2 id="summary">Summary</h2>
<p>Purpose of this post was to take you, dear reader, on a yourney of decompilation of Android app; for the sake of demonstration I took one app that looked interesting to me. The procedure is a bit more sophisticated then traditional old school “view source” in “web” but few concepts are similar.</p>
<p>Please note that I’m everything but Android engineer and a lot of details in this post are written by studiying various resources and migh be wrong or incomplete. Please, if you do feel that there is something off in my post, say it! I always like a good feedback.</p>
<p>Cheers!</p>
<p>P.s.: I wish to thank deeply to following friends and colegues that have helped with ammendments, recommendations and comments so that I’m not spreading nonsens. :) Kudos <a href="https://twitter.com/solarb">@solarb</a>, <a href="https://twitter.com/damjancvetko">@damjancvetko</a>, <a href="https://twitter.com/lowk3y">@lowk3y</a>, <a href="https://twitter.com/milangabor">@milangabor</a> and <a href="https://github.com/lknix">@lknix</a> for your help.</p>
<p>P.s.s.: I also received a brilliant recommendation to look into <a href="https://github.com/MobSF/Mobile-Security-Framework-MobSF">MobSF</a>. You should definity look into if you wanna look deeper in the subject of reverse engineering of mobile apps, pen-testing and malware analysis.</p>Hello, a good reader of my epic.blog!The content is king 👑2020-07-10T07:00:00+00:002020-07-10T07:00:00+00:00https://epic.blog/content/thinking/2020/07/10/content-is-king<p>I was thinking for a while if I should attempt to reboot
my idea of blogging and sharing some (tech) thoughts with the internet.</p>
<p>But.🤔 This opens so many questions.</p>
<p>What should I share? Where do I start? Wasn’t everything already written down? Can you write? So many questions,…</p>
<p>Could it be right? Well, let’s break it down a bit.</p>
<h2 id="what-should-i-share">What should I share?</h2>
<p>I think my peers, colleagues and friends are in general, very tech-savvy individuals who are keen to learn something new, something that could help them be better, build better products,… Quicker ways of pushing radical ideas from paper to production.</p>
<p>OK. That’s doable. ✅ I’m gonna explore this angle and see if my audience fits into this framework.</p>
<h2 id="where-do-i-start">Where do I start?</h2>
<p>Well. The first step was to find a domain that is short and would be easy to remember.
Took me about 15 minutes before I thought of <a href="https://epic.blog">epic.blog</a>. After quick lookup I’ve realised that it’s for sale and that one of the domain resellers is offering it for… … “premium” price. I did some negotiation on the amount and - 🥁 - here it is.</p>
<p>The second step was trivial. Get <a href="https://pages.github.com/">GitHub Pages</a> up and runnin’ with <a href="https://jekyllrb.com/">jekyllrb</a> and push everything to <a href="https://github.com">GitHub Repository</a>.</p>
<p>The third step; configure the thing and write <a href="/about">about section</a>. Done. ✅</p>
<h2 id="wasnt-everything-already-written-down">Wasn’t everything already written down?</h2>
<p>Yeah, it was. But you haven’t read this stuff before, not like this and not from me. ✅</p>
<h2 id="can-you-write">Can you write?</h2>
<p>I’ll do my best and hope <a href="http://grammarly.com">Grammarly</a> will help me a bit. ✅</p>
<p>Bottom line.</p>
<p>I’ll start with some tech stories, show some code, share some legends and hopefully you’ll stick around.</p>
<p>Cheers and welcome.</p>I was thinking for a while if I should attempt to reboot my idea of blogging and sharing some (tech) thoughts with the internet.[Slovenian] Petrol / ivo - osebni AI asistent, ki je skoraj zmagal na Petrolovem Poslovnem Hackathonu.2017-01-17T07:00:00+00:002017-01-17T07:00:00+00:00https://epic.blog/engineering/2017/01/17/petrol-ivo<p>Pretekli vikend sem sodeloval na II. Petrolovem Poslovnem Hackathonu, v organizaciji <a href="https://www.facebook.com/ABCaccelerator/">ABC Accelerator</a>-ja. Napisano je povzetek priprav, razvoja, lekcij, in rezultata ki smo jih z ekipo dobili pri sodelovanju na omenjenem dogodku.</p>
<h1 id="kaj-je-hackathon">Kaj je hackathon?</h1>
<p>Hackathon je organiziran dogodek pri katerem se zbere množica ljudi; ki se glede na različne interese, ozadja in ideje združijo v male ekipe. Ekipe, ki potem v omejenem času - 36 ur - izdelajo poslovni načrt / idejo / predstavitev / rešitev za izbrani problem? Problemi, ki se na teh dogodkih rešujejo so ponavadi izpostavljeni s strani organizatorja oziroma partnerja - v tem primeru - <a href="https://www.facebook.com/petrol.si/">Petrol Slovenija</a>. Glede na aktivno / pasivno vlogo organizatorja oziroma partnerja so ideje oziroma cilji lahko zelo različni. Kar pomeni, se lahko zgodi karkoli, spekter je izjemno širok in spremenljivk, ki bi zagotovile rezultat je nepredvidljivo veliko.</p>
<h1 id="pred-ideja">Pred-ideja</h1>
<p>Kot “sveže pečeni” oče in z visoko urno postavko sem se odločil, da na tem dogodku ne bom samo sodeloval. Če grem. Želim zmagati. Vse ostalo je potrata mojega dragocenega časa. V tem času se želim nekaj novega naučiti, nekaj novega preizkusiti, hkrati pa želim spoznati nove ljudi in izpeljat celovit produkt.</p>
<p>Kaj trenutno počnem? Za svojo trenutno stranko <a href="https://gwi.com">GlobalWebIndex</a> razvijam PollPass. Pollpass je “web based chat bot” za masovno anketiranje ljudi. Pollpass je R&D produkt ter GWIjeva strategija kako v prihodnosti povečati število podatkovnih točk, in število anketirance iz 18m na 50m in več. Eden tistih projektov, ki bo prepolovil trenutni čas globalne raziskave trga iz treh mesecev na realni čas. Skratka; delam na realnem problemu, ki ga rešujemo z praktično rešitvijo na ogromnem dosegu.</p>
<p>Razvoj takega produkta zahteva poglobljeno znanje številnih tehnologij, ogromno eksperimentiranja in res veliko vztrajnosti. Ogromno dela z nepreverjenimi tehnologijami in ogromno učenja na lastnih napakah.</p>
<p>Ob branju opisa Petrolovega Hackathona tako nisem potreboval veliko, da sem prišel do zelo jasne vizije in produkta, ki ga podjetje ki se ukvarja z toliko storitvami potrebuje. Petrol potrebuje “orodje” s katerim bo bolj dostopen svojim uporabnikom, potrebuje “orodje” s katerim bo lažje prodal, predvsem pa potrebuje nekaj kar ga bo dolgoročno odtrgalo od odvisnosti od “goriv”. Petrol ima več kot 487 poslovalnic in menda v srednji Sloveniji vsakih 5-10 minut pelješ mimo poslovalnice. Več kot 4 tisoč zaposlenih in malo več kot 4 milijarde čistih prihodkov od prodaje. (Vir: <a href="https://www.petrol.si/files/attachment/letno_porocilo_petrol_2015.pdf?fbclid=IwAR3T8WuTsvVVHRg7t1ddeYxJWT0kurri2XJttO-tQOlpxWZo6Tn5VdkOL3A">Petrol Letno poročilo 2015</a>)</p>
<h1 id="ideja">Ideja</h1>
<p>Produkt, ki je 24 ur na dan, 7 dni v tednu vedno v povezan s tabo. Na poti, doma, vseeno. Kadar nekaj potrebuješ ko si na poti, je ob tebi. Ko bo ledeno vreme te bo opomnil, da lahko doliješ Vitrex, ko bo vroče te bo povabil na Petrolovo kavo. Opomnil te bo, da že dolgo nisi menjal olja, usmeril te bo na najbližji Petrol, ki ima trenutno odprta vrata in lahko pošlješ pošto,… Skratka, kanal, entiteta, AI, robot, stroj, ki bo s tabo vedno.</p>
<h1 id="adijo-aplikacije">Adijo aplikacije!</h1>
<p>Dejstvo je, da večino top aplikacij, ki jih ljudje uporabljajo opravljata dva velikana. Facebook in <a href="https://www.facebook.com/Google/">Google</a>. Market je res velik, vendar je nasičen in te aplikacije pokrijejo večino “potreb”, ki jih ima povprečen človek. Dejstvo.</p>
<p>Uporabnik mora najprej za tvojo aplikacijo vedeti - se pravi jo mora najti, oziroma mu mora biti “prodana”. Aplikacijo mora potem namestiti. Po namestitvi je aplikacija na sistemu samo še ena “pasivna” ikona. Nekaj kar zaseda prostor, “kuri batarijo”. V kolikor ni dovolj pametna in nima “feeda” oziroma ne pošilja “push notification”-ov jo pozabimo. Omenjeno združimo še z dejstvom da večina aplikacij, ki so na Play storu dobi samo 1 update. Potem pa zginejo v pozabo. Seveda pa prilijmo še dejstvo, da imajo vsi veliki ponudniki sistemov za telefone vgrajene iskalnike, ki vam v večini primerov že sami dajo dovolj podatkov, aplikacije niti niso več potrebne. <strong>Dobrodošli v moj svet.</strong></p>
<h1 id="alternativa">Alternativa</h1>
<p>Digitalni uporabnik, oziroma njegovo obnašanje se je spremenilo. Top aplikacije so “messengerji” - “čvekalniki”. <a href="https://www.facebook.com/WhatsApp/">WhatsApp</a>, Facebook Messanger, SnapChat, Signal, Google Hangouts in par “korporativni” - <a href="https://www.facebook.com/slackhq/">Slack</a> in <a href="https://www.facebook.com/Skype/">Skype</a>… Ljudje čvekamo. 1-na-1. V Skupinah. Uporabljamo jezik. Uporabljamo besede. Uporabljamo stavke. Pošiljamo vsebine; slike, videje, povezave in glasbo… V stiku smo nepretrgoma. Vsi velikani opazujejo te trende in obnašanje; zato smo odprli nove kanale za oglaševalce; boti. Boti; okrajšava za “robot” - so pametni - pa tudi ne-tako-zelo-pametni “programi”, ki živijo v vaših “čvekalnikih”. Ob pogovoru se odzovejo na vaše pisanje, ali pa vas opomnejo, ko se jim zdi primerno, da vas nekaj opomnejo. … Ljudje čvekamo. 1-na-1. V Skupinah. Uporabljamo jezik. Uporabljamo besede. Uporabljamo stavke. Pošiljamo vsebine; slike, videje, povezave in glasbo… V stiku smo nepretrgoma. Vsi velikani opazujejo te trende in obnašanje; zato smo odprli nove kanale za oglaševalce; boti. Boti; okrajšava za “robot” - so pametni - pa tudi ne-tako-zelo-pametni “programi”, ki živijo v vaših “čvekalnikih”. Ob pogovoru se odzovejo na vaše pisanje, ali pa vas opomnejo, ko se jim zdi primerno, da vas nekaj opomnejo. V kolikor me še do sedaj niste zgubili predlagam preizkus <a href="https://www.facebook.com/troljo/">Troljo</a>,…</p>
<h1 id="analiza-priprava-in-ekipa">Analiza, priprava in ekipa</h1>
<p>Pred hackathonom sem se zelo sistematično pripravil. Analiziral sem Petrolovo poslovno poročilo, globalne trende, konkurenco, <a href="http://petrol.si">Petrolovo spletno stran</a>, Petrolovo spletno trgovino - eShop, Petrolovo organizacijsko shemo, cene goriv, OMV-jevo spletno stran, Petrolovo aplikacijo, OMVjevo aplikacijo. Aplikacije za promet - Slovenian Traffic (by <a href="https://www.facebook.com/izacus">Jernej Virag</a>) Slovenske, tuje. Chat-bote, Slovenske in tuje. Storitve in ogrodja za NLP, NLG, OCR in ML…</p>
<p>* Spoiler. Pripravil sem tudi API ozrima “feed” kjer zbiram vse bencinske črpalke, odpiralne čase bencinskih črpalk, GPS koordinate, ter vse storitve na črpalkah. ~> <a href="https://github.com/bencinmonitor">github.com/bencinmonitor</a></p>
<p>* Spoiler. Cene na Petrolovih straneh so v slikah; <a href="https://github.com/bencinmonitor/collect/blob/master/explore/exploring-images-v2.ipynb">zato sem razvil svoj OCR</a>, ki iz slike prebere ceno bencina. S tem korakom tako v realnem času spremljam cene goriva za celo slovenijo.</p>
<p><img src="/assets/2017-01-17-petrol-ivo/01.jpg" alt="/assets/2017-01-17-petrol-ivo/01.jpg" /></p>
<p>Ekipa. V ekipo sem povabil 3 prijatelje:</p>
<ul>
<li><a href="https://www.facebook.com/blaz.solar">Blaž Šolar</a> - Head of Android development at Kamino - Blaž je eden izmed avtorjev Petrolovega app-a za Android.</li>
<li><a href="https://www.facebook.com/jozko.skrablin">Jožko Škrablin</a> - Ops / System Engineer at IBM - Avtomatizacija, skaliranje, monitoring</li>
<li><a href="https://www.facebook.com/xzarex">Marko Vuletič</a> - UI/UX oblikovalec - Specialist za mobile UI; <a href="http://markovuletic.com">markovuletic.com</a></li>
</ul>
<p>Vse sem izbral glede na njihove izvrstne sposobnosti, izdelke, ki so jih v prihodnje že razvili, glede na “mindset” ter glede na njihove interese do tega projekta.</p>
<p>* Na našo željo se nam je na samem dogodku pridružila še <a href="https://www.facebook.com/eva.urbanc">Eva Urbanc</a>; iz Petrola, področje marketing / maloprodaja.</p>
<h1 id="razvoj-produkta-na-dogodku">Razvoj produkta na dogodku</h1>
<p>Na samem dogodku vedno zmaga “pitch”. Ne glede na izdelek, zmaga vedno ekipa z najlepše predstavljeno idejo. Pomemben je poslovni načrt, pomembna je finančna plat produkta, pomemben je “brain-storiming” pomembno je kako in kdo predstavla. Pomembna je sinergija in pomembna je enotnost ekipe. V kolikor vsa ta kemija deluje, potem je prototip samo še poslastica.</p>
<p><img src="/assets/2017-01-17-petrol-ivo/02.jpg" alt="/assets/2017-01-17-petrol-ivo/02.jpg" /></p>
<blockquote>
<p>Nabor podatkov s katerimi ivo razpolaga. Potencialnimi in realnimi.</p>
</blockquote>
<p>Na dogodku smo najprej v globino predebatirali vse espekte asistentov. Pomankljivosti, prednosti. Analizirali smo pričakovanja, analizirali smo možnosti, analizirali smo finančno plat. Analizirali smo učinke, izračunali smo vrednost investicije in izračunali smo kakšen bi bil finančni izpen implementacije asistenta…</p>
<p>Spisali smo jasno vizijo, cilje in popisali vse espekte naše rešitve. Spisali smo dober pitch, ki bil podkrepljen z realnimi številjami. Menim da vse to zadnje brez <a href="https://www.facebook.com/eva.urbanc">Eva</a>, nebi bilo niti približno tako dobro kot je bilo ob koncu!</p>
<p><img src="/assets/2017-01-17-petrol-ivo/03.jpg" alt="/assets/2017-01-17-petrol-ivo/3.jpg" /></p>
<blockquote>
<p>Ivo for dummies.</p>
</blockquote>
<p>Preko noči sem v miru tišini in malo pomoči Spotify “rock” kolekcije pripravil prototip. V jutru je <a href="https://www.facebook.com/xzarex">Marko</a> pripravil izvrsten UI. Čist, učinkovit, modern in svež.</p>
<p><img src="/assets/2017-01-17-petrol-ivo/04.jpg" alt="/assets/2017-01-17-petrol-ivo/04.jpg" /></p>
<blockquote>
<p>Landing page - cc Marko Vuletič</p>
</blockquote>
<p><img src="/assets/2017-01-17-petrol-ivo/05.jpg" alt="/assets/2017-01-17-petrol-ivo/05.jpg" /></p>
<blockquote>
<p>Kontekstualni pogovor - cc Marko Vuletič</p>
</blockquote>
<p>Hkrati pa tudi enostaven za implementacijo. Ob manjših popravkih, dopolnitvah itn… Smo se odločli da je prototip dovolj dober za predstavitev. Hkrati pa tudi enostaven za implementacijo. Ob manjših popravkih, dopolnitvah itn… Smo se odločli da je prototip dovolj dober za predstavitev. Ob 14.00 smo ga objavili na <code class="language-plaintext highlighter-rouge">https://ivo.si</code>. P.s.: <a href="https://www.facebook.com/jozko.skrablin">Jožko</a> nas je še zadnjič rešil z SSL-om (Opomba: pridobivanje lokacije iz uporabnikovega brskalnika je mogoče samo, če spletna stran uporablja HTTPS!)</p>
<p><img src="/assets/2017-01-17-petrol-ivo/06.jpg" alt="/assets/2017-01-17-petrol-ivo/06.jpg" /></p>
<blockquote>
<p>Ivo, kot je trenutno viden na ivo.si</p>
</blockquote>
<h1 id="predstavitev--pitch">Predstavitev / Pitch</h1>
<p>Na pitchu, ki je trajal 5 minut smo sodelovali “vsi”. Predstavili smo problem; rešitev, prednosti in bežno prototip. Po predstavitvi je sledil krajši Q&A kjer je bilo nekaj zanimivih vprašanj,… Komisijo je zanimalo:</p>
<ol>
<li>Koliko bi razvoj omenjene rešitve koštal?</li>
<li>Kako težko je “razumeti” jezik?</li>
</ol>
<p><img src="/assets/2017-01-17-petrol-ivo/07.jpg" alt="/assets/2017-01-17-petrol-ivo/07.jpg" /></p>
<blockquote>
<p>Q&A po pitchu.</p>
</blockquote>
<h1 id="rezultat">Rezultat</h1>
<p><strong>Na dogodku smo dosegli 2. mesto.</strong> Premagala nas je ekipa, ki je razvila produkt, ki ga boste videli med točenjem goriva. Enostavno prikazovanje “oglasa” med točenjem. Simple, efektivno in nekaj kar lahko <a href="https://www.facebook.com/petrol.si/">Petrol Slovenija</a> imlementira v parih tednih. :) Čestitke zmagovalcem!</p>
<p><img src="/assets/2017-01-17-petrol-ivo/08.jpg" alt="/assets/2017-01-17-petrol-ivo/08.jpg" /></p>
<blockquote>
<p>Push komunikacija začeta s strani Iva.</p>
</blockquote>
<h1 id="postmortem---aka-zakaj-nismo-zmagali">Postmortem - a.k.a. zakaj nismo zmagali?</h1>
<ol>
<li>
<p>UX fail. Uporabniški vmestnik nima nobenega “onboardinga” nobene točke kjer bi lahko uporabnika “naučil” kaj zna. Uporabnik tako ne ve, kaj sploh lahko vpraša iva?</p>
</li>
<li>
<p>Cena. Estimirali smo, da razvoj takega produkta košta nekje 500k EUR in da bi v letu Petrol dobil nekje 1m EUR čistega dobička (pesimistična ocena!). Kar je vseeno predstavljalo preveliko tveganje. Na vprašanje “koliko košta”, bi torej moral odgovoriti. “Za Petrol je ivo zastonj; mi imamo znanje in tehnologijo za razvoj produkta. Razvijemo ga na lastno pest; poberemo pa % od vsake transakcije, ki je posledica interakcije z Ivom. Tako mi prevzamemo rizik in ne Petrol.” Seveda je odgovor bolj pogumen kot ta, ki smo ga dali na dejanskem Q&A-ju, vendar bi pokazal da imamo jajca in da znamo!</p>
</li>
<li>
<p>Ocenjujem tudi, da Petrol trenutno ni pripravljen tvegati in je raje izbral bolj varno in realno rešitev. Kar je po svoje pametno; vendar dvomim, da bo zares rešilo podjetje na dolgi rok.</p>
</li>
</ol>
<p><img src="/assets/2017-01-17-petrol-ivo/09.jpg" alt="/assets/2017-01-17-petrol-ivo/09.jpg" /></p>
<blockquote>
<p>Leve proti desni; Oto, Marko, Jožko, Eva. Manjka Blaž Šolar.</p>
</blockquote>
<h1 id="primeri-vprašanj-na-katera-ivo-zna-odgovorit">Primeri vprašanj na katera ivo zna odgovorit?</h1>
<ol>
<li>Kje je najbližji odprti Petrol?</li>
<li>Kje lahko zamenjam olje in pijem kavo?</li>
<li>Kje lahko pošljempošto in pojem sendvič?</li>
<li>Kje lahko podaljšam urbano?</li>
<li>Kje lahko kupim vinjeto in pojem pico?</li>
<li>Cena nafte in bencina?</li>
<li>Kje lahko operem avto…</li>
<li>Kje lahko grem na wc, pijem kavo, menjam olje in jem sendvič?
… preverite na `ivo.si.</li>
</ol>
<p>* Ivo upošteva lokacijo, ter odpiralni čas poslovalnic.</p>
<p>* Opomba. Vprašanje mu lahko potavimo tekstovno ali z govornjenjem. Za potrebe PoC glasovnega vnosa nisem implementiral. Pa bi ga morda lahko,…</p>Pretekli vikend sem sodeloval na II. Petrolovem Poslovnem Hackathonu, v organizaciji ABC Accelerator-ja. Napisano je povzetek priprav, razvoja, lekcij, in rezultata ki smo jih z ekipo dobili pri sodelovanju na omenjenem dogodku.