Archive for March, 2014


The Library is my Coffee Shop

Library

I recall the first time I ever really used the library for anything significant. During my 3rd year of university I needed to escape the computer science side of campus so I could sit down and focus on my assignments distraction free. For a while it worked, but I was also involved in student politics so I knew people all over. Even the other end of campus wasn't distraction free. I needed to get somewhere I could camp out with my head down and get some work done. It was around then that I figured it was time to step into the library. They're supposed to be quiet and intended for study, so I gave it a shot. That was where I discovered the study desks reserved for graduate students on the top floors. I had finally found my Oasis!

Fast forward a few years and I was working for real from coffee shops and co-working spaces. These were good spots to work from but they are loaded with distractions; something I don't cope with very well. With a coffee shop people are constantly entering or leaving, people are having conversations, the internet expires every hour or is of disturbing quality, the list goes on. Co-working spaces sometimes suffer from the same problem plus you now have technically minded people talking about stuff too!

So here I am hiding in a public library in my study desk blocking out as many distractions as possible while hacking on some code. It's liberating to be able to stare forward and not accidentally make eye contact with a stranger or get distracted by someones erratic hand motions in my periphery. As an added bonus I don't need to worry about being kicked out since it's a public space for anyone to use for however long they want.

Perhaps next time you're looking for a place outside of the office or home to work, head over to your local public library and give it a try.


Playing around with the Philips Hue SDK for Android

Rainbows
After having seen a friend's Hue installation I decided that I'd be interested in some of these little devices myself. The idea of being able to schedule when my lights turn on or perhaps get even more advanced with my configuration would be definitely interesting.

The developers at Philips have taken an awesome approach and published the API for these lights, which means we have the opportunity to do whatever we want with these lights. Aside from just publishing how the API works they also have an iOS and Android SDK.

Eager to work on an Android app again I grabbed all the relevant pieces and started working on new project. If you are a native to Android development you might find the SDK a little bit weird to work with! The SDK in question has definitely been a direct port from the iOS SDK, which makes some things look a little weird. The API is also very callback heavy, which can make learning it at first a little bit weird.

public class BridgeSearcher implements PHSDKListener {
    private PHHueSDK sdk;
    public BridgeSearcher() {
        sdk = PHHueSDK.create();
        sdk.setDeviceName("Bridge Searcher");
        sdk.getNotificationManager()
            .registerSDKListener(this);
        PHBridgeSearchManager manager =
            (PHBridgeSearchManager)
            sdk.getSDKService(PHHueSDK.SEARCH_BRIDGE);
        // Perform uPnP search
        manager.search(true, true);
    }
}

Now I haven't shown the callbacks yet, but this is what you need to do in order to detect all the bridges in your area. In a home you'll probably only ever have one, so you can safely assume that when you successfully detect the bridge you can grab the first (and only) element from the List.

// Still in BridgeSearcher.java -- One of the Callbacks
@Override
public void onAccessPointsFound(List<PHAccessPoint> accessPoints) {
    if(accessPoints == null || accessPoints.size() == 0){
        // Android Logging
        Log.d(
            "BridgeSearcher",
            "For whatever reason we found APs but got null or an empty collection!"
        );
        return;
    }
    PHAccessPoint ap = accessPoints.get(0);
    // Now that we have an access point lets 
    // connect to the bridge!
    ap.setUsername("BridgeSearcher");
    sdk.connect(ap);
}

So we've grabbed our detected access point and set the username. The username is required since that's how authentication works on Hue. It's not really strong or anything, so it's easy to masquerade as other apps. Though this really isn't an issue since there is a step that requires direct user input. Sending that connect attempt will actually fail because my user isn't configured yet. We will find out about that because we'll get a bunch of error callbacks as well as an authentication required callback. Note: Your username needs to be sufficiently long otherwise the Hue SDK will raise an Exception

// Still in BridgeSearcher.java -- Another callback
@Override
public void onAuthenticationRequired(PHAccessPoint ap) {
    sdk.startPushLinkAuthentication(ap);
}

With this you are now waiting for the Hue button on the bridge to be pressed. Once that button is pressed you will have a user account and can change the colours of your lamps and do a bunch of other things too.

// Again still in BridgeSearcher.java -- Final callback of this article!
@Override
public void onBridgeConnected(final PHBridge bridge) {
    // You don't need all the details in order
    // to actually modify a light. Just it's ID.
    // From my observations light IDs start at 1
    final PHLight light = new PHLight("", "1", "", "");
    Thread thread = new Thread(new Runnable(){
        boolean on = true;
        PHLightState state = new PHLightState();
        // Setting to 255 causes an exception to raise :(
        state.setBrightness(254);
        state.setHue(46920); // Make light blue
        state.setOn(true);
        bridge.updateLightState(light, state);
    });    
}

And there you have it, you've found a bridge, gone through the authorization flow and updated one of the lamps. It's a little bit weird, but one of the nice things about this is it's all asynchronous so your code doesn't need to worry about every blocking on the main thread and getting the ANR of death. Though don't forget that these all occur in a separate thread, so be sure to run any UI updating code on the main thread!


Shop Product Sink: Simplify your Shopify Product Synchronization

Down the Drain

A little while ago I mentioned that I was working on a rails engine to simplify keeping products in sync with Shopify. I'll admit that the post was pretty sparse with details and I should really clarify what I meant. Currently the tools available to work with Shopify are somewhat few and far between and mainly cover the most basic requirements. There are many aspects of the product if you want to do anything substantial still require a bunch of boilerplate code.

Gotta Fetch 'em All

A major aspect of many applications is keeping products in sync with what is available on Shopify. Let's say you are building something that catches order creation webhooks. Let's also assume that you don't want to deal with the implementation of synchronization and you'll just pull in the product details as you need it from the API.

# webhooks_controller.rb
def create
  order = ShopifyAPI::Order.new(webhook_data)
  products = order.line_items.map do |line|
    ShopifyAPI::Product.find(line.product_id)
  end
  process_order_with_products(order, products)
end

With the newly released leaky bucket API you are at a higher risk of running out of API calls far more frequently. The refresh rate is about 2 requests every second and let's say your "best customer" has orders with an average cart size of 2 products coming in every second. You shouldn't really have to worry about anything because you'll rarely get close to the limit of 40 requests.

Now let's just say that your average cart size were to increase by 1 on average. Now, after processing each request you are 1 request in deficit. After about 40 seconds of constant orders you'll have burnt through your API requests. Now you have an extremely frustrating edge case because now your application doesn't work when you need it to. Either you'll have to handle for this or add a caching layer for products.

Cache, Cache, Cache, Cache, Cache, Cache. Everbody!

While handling the edge cases might seem like a solution, it's a hack and doesn't really result in easy to understand code. Handling an edge case means adding a layer of indirection that makes your code murky. Using a caching layer allows you to cleanly abstract that part out and keeps things simple, primarily because you shouldn't experience those edge cases as frequently.

# webhooks_controller.rb
def create
  order = ShopifyAPI::Order.new(order_params)
  products = order.line_items.map do |line|
    ProductCache.find(line.product_id)
  end
end

# product_cache.rb
def find(id)
  if product = fetch(id)
    return product
  else
    # Note: This is a blocking operation
    remote = ShopifyAPI::Product.find(id)
    store(remote)
  end
end

def fetch(id)
  # CachedProduct is an ActiveRecord
  CachedProduct.find(id)
end

def store(api_product)
  CachedProduct.create(api_product.attributes)
end

Now this isn't even the best approach here. This will cover the majority of the popular products which is pretty decent but you still run the risk of having some kind of sale go on where you get hammered by a bunch of different orders for different products. Another thing to keep in mind is this implementation does run the risk of going stale. This requires either running jobs to fetch the products and update them or to drain your cache and rebuild it again.

Call me when something interesting happens. Here's my number

So instead of trying to handle the fetching of products during the request which runs the risk of timing out let's move this around a bit. We really want that cache because it keeps things simple for us but we also want it to use as few requests as possible and keep it as fresh as possible. You might have noticed that I mentioned we were handling order webhooks; and well awesomely we can also handle product webhooks too!

There is one thing that sucks about this though, now you are responsible for building out an entire infrastructure to handle those products and the events related to them. We all have more important things to do and this is where a tool I've been working on, shopProductSink comes in. This is a Rails Engine (sorry Sinatrans) that you can easily glue into your rails app to get full product synchronization working. I'll apologize that there is a bit of configuration involved, but I couldn't think of a better way around it :(

Configuration!?!?!

Yes, indeed there is a bit of work that needs to be done before the engine can work it's magic. The engine doesn't have any way of knowing where your application is hosted, so you'll need to setup your default_url_options otherwise when registering webhooks we'll fail since we don't know what host to report. Say you were to only set this in config/production.rb then it would look something like this:

# Somewhere in config/production.rb
app_routes = Rails.application.routes
app_routes.default_url_options[:host] = 'your.domain.tld'

Until further notice another thing you need to ensure is that your application secret is set in the environment for user that will be running your app in production. This is required in order for the webhooks controller to validate any incoming requests. Let's say you were deploying then all you'd need to do is the following:

heroku config:add SHOPIFY_API_SECRET=abracadabra

Jacked up and Good to go

After you are all configured and you're running all you need to do is fire a few commands and you'll have an active product cache as well as have your application setup for receiving webhooks for your shop.

def post_shop_install
  installation_params = {
    shop: newly_installed_shop,
    token: newly_installed_token
  }
  import_products(installation_params)
  setup_webhooks(installation_params)

Let's look at our import_products method in detail and explain what's going on here.

def import_products(options)
  importer = ShopProductsSink::Importers::Product.new(options)
  importer.import
end

Here what we are doing is grabbing the Product import engine that is available in the shopProductSink engine and getting it to import the shops products (provided in the options/installation_params). Keep in mind this depending on the shop this can be a long-running task. So it's probably in your best interest to run this in the background using a tool such as delayed_job or resque.

Now let's look into our setup_webhooks method!

def setup_webhooks(options)
  ShopProductSink::WebhookRegistration.register(
    installation_params
  )
end

I took a slightly different approach to how this works and you can see it in action on github. The tool tells Shopify that it wants to be informed about product creation, deletion and updates in order to keep the local cache in sync. Now after a user of your application modifies anything to do with products you'll receive a message from Shopify about it. The engine comes with a default controller that will be used that will take care of validating all incoming webhooks.

Caveats

The importer is rather aggressive and doesn't handle the edge cases quite yet, though I am hoping it's slow enough and grabbing large enough pages for this to not be an issue. The default is 250 products per page which should be sufficient, but you never know, there's some shops out there with crazy numbers of products in them.

There are a few cleanup tasks I need to do for the engine, but it's ready for general usage by the public and I look forward to your feedback!

Some of you might have noticed that I didn't mention sidekiq as a background worker. This was primarily because the Shopify API gem wasn't thread-safe, though there has been some work done to use a fork of ActiveResource that makes the gem thread-safe which you should be able to sub in.

Special Thanks

I'd like to thank Stephan Hagemann for helping review this article.