Search

Codingbunny

A blog about coding, run by a bunny

Fear of Change

Is going to be the end of your application…

There, I’ve said it and I stick to that opinion. I do not want to generalize, but I’ve noticed this to be more prevalent in big corporations than in startups, although the problem exists there as well.

How do you know there’s a fear of change when dealing with software? Rather easy. When you’re proposing a change, you will have people saying “yes, but….” or outright refusing your idea. The more classical reply might also be “If it ain’t broken, don’t fix it” and my favorite: “It’s been working like that for x years”.

And I get it. I really do. You’ve worked on this code base for so many years, you’ve nurtured it, made it grow. It’s like your own child you’ve raised. And you don’t want to see this get thrown out, you don’t want to see your work undone. But do you know what this also is? Sunken-cost fallacy.

You’re making rational decisions based on the future value of the software code base. The misconception however is that these decisions are influenced by the emotional points I’ve written above. You’re effectively stopping yourself from growing, learning and adapting to a constantly changing market out there if you do not keep your software relevant, up to date and in line with the standards of the technology.

This blog post by Jeff Atwood, is a great example on how you should be treating code. I’m fully supporting the notion that you need to treat your code as disposable, and that it’s okay to break things. Not in your production environment, but go nuts in your development environment and learn new things. Understand things and see how you can improve them.

My situation

To illustrate more clearly what I am dealing with. I’m not calling out names as I don’t want to create trouble with the people involved, but the situation does illustrate the above points.

We have an old API written in Spring. It’s claimed to be a REST API, yet it’s not. It’s slow and actually nobody really understands how it works, where the data is coming from and in my opinion it even violates some design standards by directly exposing internal data structures.

So I’ve been writing a document with suggestions, on properly representing the data in JSON and XML models, decoupled from the database representation, creating the required end-points to export, import and create new representations, allowing the front-end client to be more thin and have the server handle the brute work. It will speed up the development of the client, as there’s less hacks needed and the server actually does what the API contract is promising without having to use parameters in URLS to overwrite the behavior (very un-rest like).

Yet the answers I’m received are:

  • “We’re not changing the API, it will break things” – What about our unit-tests? What about them being “internal” usage only then?
  • “It’s been working like this just fine” – Sure, but they’re slow, and nobody knows what the correct payload is. The questions in my ticket are still unanswered
  • “You’re just wrapping existing functionality” – I’m not. These are new endpoints with their own representation and functionality. I will use the underlying business-logic however, as a good n-tier application should.
  • “You should find something else to work on” – No I don’t. I actually care about the progress my team is making, and it can be so much better/faster if we actually take time for this API.

How to deal with this?

Honestly, I’m still out on that one. At the moment I’m talking with my manager and team lead, and they are on the same side. Willing to invest the extra work needed to make these APIs more flexible, stable, properly tested and compliant to the standards they are claiming to represent.

Will keep you posted on how this evolves….

Advertisements

Pet-peeves as Developer/Engineer

Okay,

This post is probably going to be a venting of frustration than anything sensible, but bear with me on this one. I’m betting that this will apply to you as well in a certain way.

Let me start by posing a question: Am I unreasonable to expect people to own up for a mistake made? Be it a bad commit, be it breaking the build or implementing something wrong?

I’m not saying you can’t make mistakes. Everyone makes mistakes during his professional career. I am asking that when you make such a mistake, own up to it. Speak up and say: “Yes, this is my fault, I will fix it or can you help me fix it?”

And this does not apply to just making mistakes. Too often I experience that people are wasting hours trying to solve an issue that could have been explained by someone in 10 minutes. An example is the environment simulation tool we use for our PaaS. Locally we run a simulator to intercept calls as if the software was running on the PaaS. And yes this thing has it’s problems, and spits errors at you. But why waste three hours trying to resolve an error? Why don’t people ask their Seniors/Architects on how to deal with the problem? Take 30 minutes, take even an hour to try and figure out what is going wrong, and when you can’t solve it, ask! You save so much time and frustration from both sides by simply asking for help.

Nobody is going to look down upon you for asking help. Nobody knows everything, and there is no shame in admitting that you don’t know something or don’t understand the problem. We’ve all been there, and we’ve all learned to rely on our co-workers, so why can’t you?

However the things that annoys me the most is not taking up responsibility. I get you’re ambitious and you want to fix this tool, upgrade a library and use the new stuff. I know I want to myself, but why didn’t you test this? Why didn’t you run this on the build server with a production template? Why was this a one-man-show?

And the problem is not that you broke it. The problem is that you remained quiet about it while others have a broken build and are stopped from working. Now 10 people got impeded and only found out because the Jenkins became red.

To all developers and software engineers out there: Trust your co-workers, and ask for help when something is unclear. Don’t run a one-man-show and remain quiet about problems. It gets you nothing but frustration.

Backlog Grooming Sessions

Agile

Pretty sure everyone is familiar with this buzzword these days. Our company and team is no different in this regard. We’re lucky that management is open for this and actually encourages us to become more agile in order to keep up with a fast changing market and its requirements.

Like most teams we are suffering from the dreaded low-quality user stories in our backlog. These stories are often written in bad grammar, one-liners or simply referencing materials and documents that do not exist or are out of date. A rampant problem in almost any backlog, and one that can lead to a team performing poorly because too much time is lost deciphering these kind of stories.

Taking Action

Today, as a Scrum Master, I organized the first backlog grooming session with just the team. I opted for not inviting the stakeholders and work only with the local product owner and team members in order to identify the pain points as well as show them examples on good stories and let them speak up about what they thought was important.

The meeting itself was pretty straightforward. We had a single laptop with project and started at the top of the backlog, working our way down through all JIRA tickets during the one hour long session.

We agreed that all stories should be following the standard template, and have dedicated sections. So we came up with the following JIRA template that we filled in for every story:

h2. User Story
As <persona>,
I want <something>,
in order to <reason>,


h2. Acceptance Criteria

h2. Technical Clarification

h2. Open Questions

Using this template we started to work over the list. Turning one-liners into properly written stories, adding acceptance criteria and questions to them. Some stories didn’t seem to make sense to us, where other turned out to be obsolete for being fixed already or no longer being relevant as request.

After one hour we salvaged roughly 15 tickets into proper stories.
The best thing coming out of this however is that the team agreed to run a weekly session of one hour to keep pruning the backlog of bad stories and work on improving the quality of existing ones to make planning meetings easier.

Here’s to my team and their journey on the road to good Scrum!

Round Robin : Thread-Safe Cycling list

Hello!

It’s been a while since I’ve written anything on the blog, but today I have an interesting article to share. Recently started a new position as Sr. Software Engineer in the Java programming world. Probably a weird turn considering my background, but I’m actually enjoying it quite a bit!

I was asked to come up with a Round Robin solution for storing and distributing credentials to use when communicating with the Google Cloud services. The basic idea is that we wanted to distribute the credentials used to spread out the load on them, avoiding calling limitations etc that might be imposed on customers.

Naturally there’s no out-of-the-box solution in Java for this, and I had to get creative. Keeping the main goal in mind, I thought of a way to iterate over a list and keep starting from the beginning every time the last element was reached. Sounds simple in theory, but it also needed to be done in a thread-safe manner, since we’re running a multi-threaded application.

Getting Started

So the first step. How do you get started?
I created a class called RoundRobinList that would represent the cycling list for me. Internally it would use a List<T> type to track whatever it was supposed to be tracking. Since we were dealing with a generic, the entire class could be a generic, and the initial representation looked like this:

public class RoundRobinList<T> {
  private List<T> list;
 
  public RoundRobinList(List<T> list) {
    this.list = list;
  }
}

So we didn’t really care about the type of List we are working with, as long as it adheres to the contract, not breaking our functionality.

Getting an Element

So the next step, actually getting an element from the list…
While this was pretty straightforward, we wanted to make sure the interface of our class was easy to understand. So the following function got added:

public T get(int index) {
  return this.list.get(index);
}

There was however a problem with this. We were not cycling through the contents and actually just wrapped the list. Not what we were aiming for. Our goal is to get a new element every time we call get. So no index specifying. How can we achieve this? With an iterator! So our class and method had to change a little bit:

package com.company.project.util;

import java.util.List;
import java.util.ListIterator;

public class RoundRobinList<T> {
  private ListIterator<T> iterator;
  private List<T> list;
 
  public RoundRobinList(List<T> list) {
    this.list = list;
    this.iterator = list.listIterator();
  }
 
  public T get() {
    if(!iterator.hasNext()) {
      iterator = list.listIterator(); 
    }

    return iterator.hasNext() ? iterator.next() : null;
  }
}

So at this point, we can create a new instance of the RoundRobinList with any given List implementation, ask for it to keep returning elements every time we call get() , and we’ve covered the case where the list is empty.

Testing the behavior

Of course, all good needs to be tested. So how can we be sure our code actually implements the behavior we wanted? With the following unit-test of course:

package com.company.project.util;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import java.util.ArrayList;
import java.util.List;
import org.junit.Before;
import org.junit.Test;

public class RoundRobinTest {
  private RoundRobinList<String> subject;
  private List<String> testCollection;
 
  @Before
  public void setUp() {
    testCollection = new ArrayList<String>();
 
    testCollection.add("A");
    testCollection.add("B");
    testCollection.add("C");
 
    subject = new RoundRobinList<String>(testCollection);
 }
 
  @Test
  public void getCyclesProperly() {
    assertEquals("A", subject.get());
    assertEquals("B", subject.get());
    assertEquals("C", subject.get());
    assertEquals("A", subject.get());
  }
}

One test that properly checks if we get every element in the sequence defined, and whether the list actually restarts when we’ve reached the end!

Adding and Removing

Now obviously a List were you cannot add or remove elements was getting a bit impractical for our usage. Because we required the information to be updated at regular points, so we wanted to be able to add and remove elements from the list. Mind you, our class is still not thread-safe at this point, we’ll tackle that problem later. For now, let’s add the functions needed to add/remove elements, and refresh our iterator.

public boolean add(T object) {
  if(list.add(object)) {
     iterator = list.listIterator(iterator.previousIndex() + 1);
     return true;
  } else {
    return false;
  }
}
public boolean remove(T object) {
  if(list.remove(object)) { 
    iterator = list.listIterator(list.size() > 0 ? iterator.previousIndex() + 1 : 0); // To cover empty lists
    return true;
  } else {
    return false;
  }
}

Now you might be wondering, why did I use the iterator.previousIndex() + 1 in the code, and not just start at the previous index?
The reason is that when we remove or add an element, the iterator itself might be invalid depending on which element gets removed/add at which position. It’s a fairly safe bet to assume the previous index existed, and we can jump to the next element in line.

Remember, our get has already the checks in place to handle the situations for empty lists, out of bound index etc.

Testing the added code

Like before, we want to make sure the new code is actually performing how we want it perform. So for the add and remove we created some additional test cases:

@Test
public void addProperlyAppends() {
  subject.add("D");
 
  assertEquals("A", subject.get());
  assertEquals("B", subject.get());
  assertEquals("C", subject.get());
  assertEquals("D", subject.get());
  assertEquals("A", subject.get());
}
 
@Test
public void addProperlyInjects() {
  assertEquals("A", subject.get());
  assertEquals("B", subject.get());
  assertEquals("C", subject.get());
 
  subject.add("D");

  assertEquals("D", subject.get());
  assertEquals("A", subject.get());
}

public void removeProperlyRemoves() {
  subject.add("D");
  subject.remove("D");
 
  assertEquals("A", subject.get());
  assertEquals("B", subject.get());
  assertEquals("C", subject.get());
  assertEquals("A", subject.get());
}
 
@Test
public void removeProperlyDeletes() {
  subject.add("D");
 
  assertEquals("A", subject.get());
  assertEquals("B", subject.get());
  assertEquals("C", subject.get());
 
  subject.remove("D");
 
  assertEquals("A", subject.get());
}

Making it all thread-safe

Now the hardest part, making our code work in a multi-threaded environment. The problem when multiple threads are accessing the internal list, we might run into problem when elements are being added or removed from the list by one thread, and another thread is reading the same list. Our iterator is shared so we might run into unpredictable behavior.

Luckily, Java comes with a concurrent library that allows us to create read and write locks around code, controlling the access of the threads depending on the operation they want to perform. Our goal is now the following:

  • We want every method inside a read-lock. Reading is non-destructive, and can be allowed by multiple threads at the same time, as the iterator is going to jump ahead on every sequential call.
  • The steps however where we update the data and the iterator need to be inside a write-lock. Write-locks only allow a single thread access at any time and prevent read operations as well, allowing us to safely make the data structure modifications without breaking any threads.

So how does this look in our code?

public class RoundRobinList<T> {
  private ListIterator<T> iterator;
  private List<T> list;
  private static final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
 
  public RoundRobinList(List<T> list) {
    this.list = list;
    this.iterator = list.listIterator();
  }
 
  public T get() {
    final Lock readLock = lock.readLock();
    readLock.lock();
 
    try { 
      readLock.unlock();
      final Lock writeLock = lock.writeLock();

      try {
        writeLock.lock(); 
          if(!iterator.hasNext()) { iterator = list.listIterator(); }
      } finally {
        writeLock.unlock();
        readLock.lock();
      } 
 
      return iterator.hasNext() ? iterator.next() : null;
    } finally {
      readLock.unlock();
    }
  }
 
  public boolean add(T object) {
    final Lock writeLock = lock.writeLock();
    writeLock.lock();
 
    try {
      if(list.add(object)) {
        iterator = list.listIterator(iterator.previousIndex() + 1);
        return true;
      } else {
        return false;
      }
    } finally {
      writeLock.unlock();
    }
  }
 
  public boolean remove(T object) {
    final Lock writeLock = lock.writeLock();
    writeLock.lock();
 
    try {
      if(list.remove(object)) { 
        iterator = list.listIterator(list.size() > 0 ? iterator.previousIndex() + 1 : 0); // To cover empty lists
        return true;
      } else {
        return false;
      }
    } finally {
      writeLock.unlock();
    }
  }
}

At this point we have surrounded our calls with read-lock objects, which will only block when the write-lock is claimed by a thread. Which is what we want. No read operations are allowed while we’re modifying the data.

The only part I will need to tweak a bit is the switch between the write and read lock in the get method. There’s a bit too much overhead there switching the locks back and forth, but at the moment I know of no better way yet to handle this and check whether we actually need to swap at that point or not.

But that’s an improvement I can write about once I’ve implemented it properly.

Windows Desktop Drivers

The Problem

Most people have a Windows computer at home. And this thing comes with hardware. And hardware requires drivers in order to function properly. But how many people keep their drivers up to date? How many people actually install the correct drivers instead of the generic Microsoft ones?

Sure, your system will work with the Microsoft drivers, but when you’re into gaming, you want to have the correct drivers to ensure that your system is running with the best possible configuration.

Driver Easy

I’m normally not someone to promote tools or things, but Driver Easy is someone that caught my attention and is something I use regularly now to find that one missing driver in my system that causes a component to not work at the best possible performance. Having a high-end gaming system, this means often updating drivers to make sure everyone performs at peak-performance.

With the guide here, I want to demonstrate how you can keep your system in great working condition, using the free version and the step by step guide on updating every driver.

Step By Step Guide

When you open the application, you will see the screen below. Press the big “Scan Now” button to start the driver scanning process of the application. Depending on your computer’s speed and the hardware installed this can take a few seconds or a few minutes.

 

main page

Once the scanning has been completed, you will see the result below. There’s a list with devices that can be updated with new drivers and a list of devices that’s already up to date. We’re interested in the list that needs to be updated.

Click on the Update button to start downloading. If the application asks for a restore point, just tell it that you already did this. Don’t use the Update All feature, that’s only in the paid version.

scan results

This is the screen when downloading the drivers. This might take some time for some of the packages as the download speed is pretty capped.

downloading

When the download is completed, you want to pick the “Manual Installation” mode. Automated installations are again only supported in the Pro version, but adjusting your drivers is easy enough if you stick to this guide.

manual installation

As soon as you press the Continue button, a Windows Explorer will pop up. Click inside the address bar at the top, and copy the entire path using CTRL + C. Once you’ve copied the path, you can close the explorer window.

copy path

The next step is opening the Device Manager in Windows. This can be done really quick by pressing the windows + X button on your keyboard and selecting the “Device Manager from the popup window.

window x

The goal is now to find the correct device. In most cases the name is going to match what you downloaded, but on a fresh system the names might be extremely generic. So if you cannot find the device immediately, there’s a nice trick. Press the grey information button next to the device for which you downloaded the drivers in the Driver Easy app, this opens a small overlay:

info

The overlay shows the new driver info, but when you click on “Device Information”, you will receive the key info for finding your hardware in the Device Manager:

info 2

These Hardware IDs are what uniquely identifies a piece of hardware in your system. There’s not a single device with the same ID in your system, so now that we know what we are looking for, we can start our search in the Device Manager. Open the properties of the hardware you think matches what you want to update:

device manager

Go to the Details tab, and select the Hardware IDs from the dropdown menu.

hardware id

Compare the list to what the Driver Easy application shows you. If they match, then you found the piece for which you have just downloaded the drivers.

device manager match

Now that we compared the data, we know this is our device. So go to the driver tab, and press the “Update Driver…” button to continue the process.

update driver

You want to browse for the files you downloaded, so don’t select the automated update. This will in most cases just tell you that your device is up to date, which it isn’t. So just press “Browse my computer for driver software”.

browse driver

In the next screen you want to replace the path being displayed by the one we copied from the Windows Explorer folder. Because that’s where we stored the drivers. So select all the text inside the input field, delete it and paste inside the path we copied at the beginning.

path driver

Now finalize the process by pressing the Next button.
Repeat this for all the drivers needing an update, and before you know it, your system is running with the latest version.

Just make sure to reboot the system once you’re done with installing ALL drivers. No need to reboot in between installations.

Load-balancer requests in the application

Load-balancers.

Almost every platform uses them these days to distribute the incoming requests between applications, to make sure one node or cluster does not get overloaded and that all users experience an acceptable performance.

The downside of these load-balancers however is that they need to send out some kind of keep-alive messages to the back-end, in order to verify whether the various endpoints they are balancing are still reachable. In our case, this was done by triggering a specific URL and check if we get an HTTP 200 response.

Now without disclosing too much of the internals, we track every web-request, rely on sessions and don’t want to pollute our database with information that’s not relevant to the system. So we originally had the idea to filter out all load-balancer requests from the controllers. Sounds perfectly okay so far, right? Except code wise this was a nightmare. We had a bunch of filters, an API to support and sessions to clean up. This resulted in the ApplicationController becoming a mess of hooks, cleanup code and basically undoing everything Rails did when a new request came in.

When another change-request came in to filter out session data from Redis for the load-balancer, I started asking myself the question: Do we even need to hit the Rails stack in order to know our application is up and running? The answer to that, is no. You don’t need to hit Rails to know whether your application is up and running.

Rails has this feature called Rack Middleware. Creating a custom middleware that responds simply to the load-balancer requests, we created a simple solution that returns an empty HTTP 200 response for the load-balancer, but processes all other requests further up the stack.

  • The Rails stack is never called. This means no cleaning of sessions, data or implementing dozen’s of hooks and if-statements to filter out load-balancer requests.
  • The middleware runs before everything else, so we don’t even need to load all code or triggers
  • The load is removed from the database and Redis system as the calls are never made
  • A lot of the code is reduced to a single module.
module Middleware
  class LoadBalancer
    def initialize(app)
      @app = app
    end

    def call(env)
      params = { url: env['REQUEST_URI'], ip: env['REMOTE_ADDR'] }
      return blank_response if LoadBalancer.skip_web_request?(params)
      @app.call(env)
    end

    def blank_response
      [200, { 'Content-Type' => 'text/html', 'Content-Type' => 0 }, ['']]
    end

    class << self
      def skip_web_request?(url:, ip:)
        skip_web_ips.include?(ip) && skip_web_urls.include?(url)
      end

      def skip_web_ips
        Settings.proxy.trusted_ips
      end

      def skip_web_urls
        ['http://our_fancy_balance_url']
      end
    end
  end
end

So basically what this class does it return an empty HTTP 200 response if the request matches our Load-Balancer request or delegate it up the stack in all other cases.

To get the load-balancer to work before everything else in your stack, you need to create an initializer for Rails and add the following code:

Rails.application.config.middleware.insert(0, ::Middleware::LoadBalancer)

 

Extracting functionality to a Service

Aether,

The name I’ve given to my new service that I am developing at work. The goal is a complete data and logic isolation of one of our features, extracting it from the monolithic web-application we currently have running for our customers.

And while the idea in itself is amazing, the actual implementation is not so easy. The goal is to have a service that crawls the web for data, transforms it and stores it inside our system using the correct configuration and structure. The problems however are:

  • The core is so tightly coupled, that I’m not even sure I can start on this without some preliminary refactoring.
  • There is a data-dependency in both systems, where I do not see an easy separation.
  • There’s a constant need for synchronising data between the systems, which makes me wonder if there’s really a need to extract this.
  • Once we start on this, we need to drive it through completely.

Our Product Owner sees the issues and supports that we need to clean up our technical dept regarding this feature. The business however has requirements that need to be fulfilled as well. Doing both at the same time might be possible I suppose, but do we want that?

  1. We could say that all feature development for this topic is on hold, and the new service needs to be finished first. (We all know how “finished” software is)
  2. We port over the established components and keep developing features in the app, porting them later  (refactor hell)
  3. We put the service on hold and finish the new features first (like that’s ever going to happen)

All three approaches have their benefits and downsides. The biggest problem I have right now is that I see no clean technical solution at the moment on the sync need between the both of them, and how we can proceed with this in an orderly manner.

I’ve tossed the problem in front of the team, and I’m waiting on feedback at the moment, but my mind is wandering, and a little voice says to keep everything in the monolithic application right now and go for code-isolation first, and data-isolation later on.

Lint/ScriptPermissions for Rakefile?

Today I wanted to activate this rule. The cop for this is actually straight forward, you don’t want to add/write script permissions inside your scripts. We have a Rakefile in our application, as any good standard Ruby on Rails application has. And at the top of the file, this is written:  #!/usr/bin/env rake

There is a problem with adding this, namely that your system will use whatever rake is defined, and not necessarily the one you had in mind or configured through your Gemfile or Bundler. That’s why it’s better to NOT include these lines in your script, and make sure the script is actually executable when it needs to be, or you’re using the right commands to do so like bundle exec rake my_script.

Now why doesn’t it make sense to add this as well? The cop kind of gives it away. The error message is that the file doesn’t have any execute permissions, which honestly it should not have in the first place. Rake is running and needs to be executable, but your Rakefile is the script being parsed by Rake. At least that’s how I understand it. I’ve never had to set execute permissions on this file before.

The current Ruby on Rails master branch doesn’t have this entry in it either, perhaps older versions used to have this, but seeing as this is not required anymore, I opted to activate this rule AND remove the line from our Rakefile​.

Binary Gap

So,

Today I was given a coding challenge. Or rather I took one voluntarily to prepare myself for the future.  I know there are companies that require candidates to perform a technical test when applying, and there’s that don’t.

I understand the requirement. A good interview includes a coding test, because you want to see your candidate think and what kind of code he writes to solve a particular problem. I personally don’t care if a candidate did not get the correct solution during the test. You simply cannot know everything, and I’m usually more interested in how someone approaches a problem and comes to a solution.

But back to the test: I was asked to figure out the binary gap for any given number. The binary gap is the largest repeating amount of 0 in the binary notation of that given number. So for example, the binary gap for 15 is 0, because 15 in binary is 1111. For 16 however it’s 4, as 16 in binary is 10000. I hope you understand the question.

# Author: Arne De Herdt
#
# Calculates the binary gap for any given number.
# The binary gap is the largest repeating amount of 0
# in the binary notation of a given number.
#
# For example:
# n = 1041=10000010001_2 => 5
# n = 16 = 10000 => 4
#
def binary_gap(n)
  highest_count = 0
  binary_notation = "%b" % n
  counter = 0

  binary_notation.each_char do |character|
    if character == "1"
      highest_count = counter if counter >= highest_count
      counter = 0
    else
      counter += 1
    end
  end

  return highest_count
end

Is this code covering all cases? Yes, it does. It scored 100% on the Codility website, where I got it from. Is this code I want to be running in a production environment? Most likely not. This code is slow and consumes memory. Classic O(n) space problem where the memory depends on the input.

Can I write production code? Probably. Given a few iterations and improvements I can come up with some magical code in Ruby that does it faster and better.  But is that something I want to show off with on these tests? Rather not. This code is way easier to explain for a thinking process, and gives me a chance to also address the issues with it, showing my understanding of programming patterns and designs.

Feel free to leave comments on how you’d approach this, or if you disagree with me or not.

Blog at WordPress.com.

Up ↑