Posts Tagged With ‘ruby&8217


Asynchronous Programming with Goroutines in Ruby

pipeline

I’ve been programming on and off in Go for a little while now, and I came across a gem that makes it possible to use channels and threaded functions for sequential processing. The gem is called “agent” which was originally written Ilya Grigorik. It provides a number of extensions to the Kernel such as go! and channel!.

Overview

What is a channel?

Channels aren’t really a new idea, they came out of a research paper1 called Communicating Sequential Processes. They became really popular when the Go programming language came out in which channels are a core construct of the language and really the way most of your work is done.

Channels are able to hold units of data, called messages which allow other relevant parties to listen on. A channel has two “ports” one for pushing data into and one for pulling data out of. They are also atomic, so there is no need to worry about a read or write resulting in something inconsistent, such as getting the same message twice.

In agent a channel can optionally be buffered. Using a buffered channel allows you to write a message to it without having to worry about immediately blocking. Otherwise a channel has a buffer size of zero, which means that writing to the channel blocks until someone decides to read from it. Knowing how your channels work is extremely important, which will be covered in “Ugh Deadlock“.

require 'agent'

# Create an unbuffered channel
unbuff = channel!(String)

# Create a buffered channel
buff = channel!(String, 1)

# Write to the channels
buff << "Hello Unblocking World"
# Execution blocks until someone reads from it
unbuff << "Hello Blocking World"

What does a goroutine do?

In the Go programming language a goroutine is “the execution of a function call as an independent concurrent thread of control, […], within the same address space”. You can think of them as threads but don’t need to worry too much about the details. Also if the function provided is a lambda, you get the benefits of closures.

In Ruby the benefits are very much the same, we are passing in a block of code to the go! method call. This method takes care of all the gross bits of threading for us, though if you grab onto the result of the go! call you’ll see that we aren’t working with any magic language construct, just a thread.

goroutine = go! { sleep(5); }
puts goroutine.inspect
# #<Thread:0x007fbf66ed5508 sleep>

In practice though, you don’t need to worry about this. You’ll invoke your goroutine and carry on your merry way.

How can the two be used together?

Goroutines and channels become very powerful because they allow us to communicate asynchronously. There are lots of ways to handle work that might be slow, such as jobs that create other jobs in a system like Sidekiq. While this approach works, it results in systems that are difficult to understand because of the separation.

Instead we can use channels as a tool to let other aspects of our pipeline know when they can start the next stage of processing. Since the channel becomes the method of communication, we can write our code in a way that makes it look sequential even though when we actually execute it, it will be running asynchronously. This takes a huge load off of our brains, which get really fuzzy when dealing with asynchronicity.

Consuming a channel

Let’s say we are building a system that makes strawberry rhubarb pies. If we were to make a pie on our own it would look something like this:

  1. Cut together flour and shortening to make the dough
  2. Clean strawberries and rhubarb removing all dirt
  3. Quarter strawberries and chop rhubarb into cubes
  4. Roll out dough and place enough to cover the bottom of your pie tin. Trim the edges
  5. Measure out 2 cups of strawberries, 2 cups of rhubarb and 1 cup of white sugar
  6. Mix measured strawberries, rhubarb and sugar together until even
  7. Pour mixture on top of dough
  8. Cover pie with other piece of dough and make some cuts to let steam out
  9. Place in oven

For a family friendly baking day, doing this synchronously is no big deal. You have a good time, make some pie and all is fabulous. But if we are looking to crank out pies as quickly as possible, this just won’t do. We need to split this up into a bunch of separate pieces. Looking at the process we can make a few optimizations:

  1. A person makes the dough, and prepares the tins as well as the pie cover
  2. A person cleans, cuts and mixes the fruits into the required portions. Let’s say they place them in bowls that someone can just grab.
  3. A person that prepares the pies and places them in the ovens. For simplicity we are going to assume that we have unlimited space in our ovens.
PreparedTin = Struct.new(:bottom, :top)
MixedFruits = Struct.new(:mixture)
RawPie      = Struct.new(:bottom, :mixture, :top)

class Pie
  def initialize(slices=6)
    @slices = slices
  end

  def take_slice
    if slices?
      @slices -= 1
      "slice o pie"
    end
  end

  def slices?
    @slices > 0
  end
end

class Oven
  def initialize(chan)
    @completed = chan
  end

  def bake(pie)
    go! do
      sleep(10)
      @completed << Pie.new
    end
  end
end

prepared_tins  = channel!(PreparedTin)
mixed_fruits   = channel!(MixedFruits)
completed_pies = channel!(Pie)
oven           = Oven.new(completed_pies)

# Prepare our Pie Tins
go! do
  loop do
    dough = mix_flour_and_shortening
    dough = roll_out_dough(dough)
    top, bottom = cut_dough(dough)
    prepared_tins << PreparedTin(top, bottom)
  end
end

# Prepare our Mixed Fruits
go! do
  loop do
    strawberries   = clean(Strawberry.new)
    rhubarb        = clean(Rhubarb.new)
    sliced_berries = quarter(strawberries)
    cubed_rhubarb  = cube(rhubarb)
    sugar = measure(Sugar.new)
    mixture = mix(sliced_berries, cubed_rhubarb, sugar)
    mixed_fruits << MixedFruits.new(mixture)
  end
end

# Bake our pies
go! do
  loop do
    # Receiving from an agent channel returns a pair
    # so if we were to receive and not call first
    # we'd get something like [#PreparedTin, true]
    #
    tin = prepared_tins.receive.first
    fruits = mixed_fruits.receive.first
    raw_pie = RawPie.new(tin.bottom, fruits.mixtures, tin.top)
    oven.bake(raw_pie)
  end
end

# Eat our pies
go! do
  pie = nil
  loop do
    pie ||= completed_pies.receive.first
    if pie.slices?
      puts "Om nom nom a #{pie.take_slice}"
    else
      # Throw out the tin, this pie is dead to us
      pie = nil
    end
  end
end

So here we’ve gone and created 4 workers. One prepares the pie tins, another prepares our pies contents, another who puts the pies together and bakes them and finally, someone to consume all those pies. The code can be read from top to bottom, and we really don’t need to worry about concurrency all that much.

Ugh Deadlock

We need to be very careful when working with channels because we run the risk of deadlocking our system. Luckily the library has built in protections that will raise exceptions when a deadlock is detected.

f = channel!(String)
go! { f << "Hello World" }
f.receive # ["Hello World", true]
f.receive # Raises Exception
# #<Class:0x007fefc38dc768>: No live threads left. Deadlock?

Sometimes these can catch us off guard but do not fret, there are plenty of ways to get around this. We can use select! which is a based off a similar function for the Go language. select acts very similar to a case statement, though the big difference is that they are used for communication operations and this allows us to conditionally act on messages sent to different channels.

In agent we are also able to use a timeout which allows our select! to stop blocking on channels. For example, we can use select! with a timeout that will ensure we don’t deadlock.

select! do |s|
  s.case(f, :receive) { |result| puts result }
  s.timeout(0.1) { puts "Exceeded 100ms timeout!" }
end

Caveats

In the Go programming language you can put whatever you want on a channel. Because the system is statically typed it knows how big each message is going to be. In agent this isn’t the case and there appears to be some marshalling going on under the covers. I came across this problem while trying to send objects that wrapped Nokogiri on some channels.

require 'agent'
require 'nokogiri'
require 'rest-client'

pages = ["http://github.com",
    "http://bitbucket.org",
    "http://sourceforge.net"]

class Result
  def initialize(data)
    @doc = Nokogiri::HTML(data)
  end

  def title
    @doc.css('title').first
  end
end

results = channel!(Result, 3)

pages.each do |page|
  response = RestClient.get(page)
  results << Result.new(response.body) # Raises error
  # agent/push.rb:8:in `dump': no _dump_data is defined for class Nokogiri::HTML::Document (TypeError)
end

processing = true
while processing do
  select! do |s|
    s.case(results, :receive) { |r| puts r }
    s.timeout(5) { processing = false }
  end
end
puts "done"

After looking around a bit I figured that it was just the objects cannot be marshalled. I was caught off guard by the issue, but I was able to come up with a work around by simply using a Hash instead. I haven’t tried doing anything with value objects yet, but I don’t see why those would cause any problems either.

In Conclusion

The agent library was an excellent way to approach the asynchronous programming problem in Ruby. It’s got some rough edges that might cut you, but with the right understanding of the limitations you can easily work around them. Other than that it’s a fabulous way to learn about this alternative approach to programming. I’d even go as far to say this could be a gateway into the Go programming language since it helps you establish the basics of using goroutines and channels to communicate.


Shop Product Sink: Simplify your Shopify Product Synchronization

Down the Drain

A little while ago I mentioned that I was working on a rails engine to simplify keeping products in sync with Shopify. I'll admit that the post was pretty sparse with details and I should really clarify what I meant. Currently the tools available to work with Shopify are somewhat few and far between and mainly cover the most basic requirements. There are many aspects of the product if you want to do anything substantial still require a bunch of boilerplate code.

Gotta Fetch 'em All

A major aspect of many applications is keeping products in sync with what is available on Shopify. Let's say you are building something that catches order creation webhooks. Let's also assume that you don't want to deal with the implementation of synchronization and you'll just pull in the product details as you need it from the API.

# webhooks_controller.rb
def create
  order = ShopifyAPI::Order.new(webhook_data)
  products = order.line_items.map do |line|
    ShopifyAPI::Product.find(line.product_id)
  end
  process_order_with_products(order, products)
end

With the newly released leaky bucket API you are at a higher risk of running out of API calls far more frequently. The refresh rate is about 2 requests every second and let's say your "best customer" has orders with an average cart size of 2 products coming in every second. You shouldn't really have to worry about anything because you'll rarely get close to the limit of 40 requests.

Now let's just say that your average cart size were to increase by 1 on average. Now, after processing each request you are 1 request in deficit. After about 40 seconds of constant orders you'll have burnt through your API requests. Now you have an extremely frustrating edge case because now your application doesn't work when you need it to. Either you'll have to handle for this or add a caching layer for products.

Cache, Cache, Cache, Cache, Cache, Cache. Everbody!

While handling the edge cases might seem like a solution, it's a hack and doesn't really result in easy to understand code. Handling an edge case means adding a layer of indirection that makes your code murky. Using a caching layer allows you to cleanly abstract that part out and keeps things simple, primarily because you shouldn't experience those edge cases as frequently.

# webhooks_controller.rb
def create
  order = ShopifyAPI::Order.new(order_params)
  products = order.line_items.map do |line|
    ProductCache.find(line.product_id)
  end
end

# product_cache.rb
def find(id)
  if product = fetch(id)
    return product
  else
    # Note: This is a blocking operation
    remote = ShopifyAPI::Product.find(id)
    store(remote)
  end
end

def fetch(id)
  # CachedProduct is an ActiveRecord
  CachedProduct.find(id)
end

def store(api_product)
  CachedProduct.create(api_product.attributes)
end

Now this isn't even the best approach here. This will cover the majority of the popular products which is pretty decent but you still run the risk of having some kind of sale go on where you get hammered by a bunch of different orders for different products. Another thing to keep in mind is this implementation does run the risk of going stale. This requires either running jobs to fetch the products and update them or to drain your cache and rebuild it again.

Call me when something interesting happens. Here's my number

So instead of trying to handle the fetching of products during the request which runs the risk of timing out let's move this around a bit. We really want that cache because it keeps things simple for us but we also want it to use as few requests as possible and keep it as fresh as possible. You might have noticed that I mentioned we were handling order webhooks; and well awesomely we can also handle product webhooks too!

There is one thing that sucks about this though, now you are responsible for building out an entire infrastructure to handle those products and the events related to them. We all have more important things to do and this is where a tool I've been working on, shopProductSink comes in. This is a Rails Engine (sorry Sinatrans) that you can easily glue into your rails app to get full product synchronization working. I'll apologize that there is a bit of configuration involved, but I couldn't think of a better way around it :(

Configuration!?!?!

Yes, indeed there is a bit of work that needs to be done before the engine can work it's magic. The engine doesn't have any way of knowing where your application is hosted, so you'll need to setup your default_url_options otherwise when registering webhooks we'll fail since we don't know what host to report. Say you were to only set this in config/production.rb then it would look something like this:

# Somewhere in config/production.rb
app_routes = Rails.application.routes
app_routes.default_url_options[:host] = 'your.domain.tld'

Until further notice another thing you need to ensure is that your application secret is set in the environment for user that will be running your app in production. This is required in order for the webhooks controller to validate any incoming requests. Let's say you were deploying then all you'd need to do is the following:

heroku config:add SHOPIFY_API_SECRET=abracadabra

Jacked up and Good to go

After you are all configured and you're running all you need to do is fire a few commands and you'll have an active product cache as well as have your application setup for receiving webhooks for your shop.

def post_shop_install
  installation_params = {
    shop: newly_installed_shop,
    token: newly_installed_token
  }
  import_products(installation_params)
  setup_webhooks(installation_params)

Let's look at our import_products method in detail and explain what's going on here.

def import_products(options)
  importer = ShopProductsSink::Importers::Product.new(options)
  importer.import
end

Here what we are doing is grabbing the Product import engine that is available in the shopProductSink engine and getting it to import the shops products (provided in the options/installation_params). Keep in mind this depending on the shop this can be a long-running task. So it's probably in your best interest to run this in the background using a tool such as delayed_job or resque.

Now let's look into our setup_webhooks method!

def setup_webhooks(options)
  ShopProductSink::WebhookRegistration.register(
    installation_params
  )
end

I took a slightly different approach to how this works and you can see it in action on github. The tool tells Shopify that it wants to be informed about product creation, deletion and updates in order to keep the local cache in sync. Now after a user of your application modifies anything to do with products you'll receive a message from Shopify about it. The engine comes with a default controller that will be used that will take care of validating all incoming webhooks.

Caveats

The importer is rather aggressive and doesn't handle the edge cases quite yet, though I am hoping it's slow enough and grabbing large enough pages for this to not be an issue. The default is 250 products per page which should be sufficient, but you never know, there's some shops out there with crazy numbers of products in them.

There are a few cleanup tasks I need to do for the engine, but it's ready for general usage by the public and I look forward to your feedback!

Some of you might have noticed that I didn't mention sidekiq as a background worker. This was primarily because the Shopify API gem wasn't thread-safe, though there has been some work done to use a fork of ActiveResource that makes the gem thread-safe which you should be able to sub in.

Special Thanks

I'd like to thank Stephan Hagemann for helping review this article.


An Approach to Token Based Key Distribution for Mobile Clients

Do not duplicate

Back during Rails Rumble 2013 the team I was on was building an application for use by peoples phones. We were deliberating on how we could easily get the application data from the server to the phone without having to resort to poor solutions like username/password combinations or some convoluted OAuth flow that requires embedding application secrets in the binary.

Token-based authentication is an approach I've heard of before and is used in popular applications such as Campfire. The one problem though is the key distribution isn't the nicest thing in the world. With Campfire the user is required to find their token, copy it then paste it in the correct location of their API consumer in order to interface with the product. While this might work alright for a desktop client, once you venture into mobile data entry can sometimes be a little tedious.

Our assumption for the application was that a user would be using the app from their desktop first. This would mean all their session data isn't available on their mobile device so in order to go somewhere to get the data they need would require logging in via their phones web browser and going through the same steps as described previously.

I wanted an easy way to get the information to our mobile clients that required very little in terms of technology. Our approach was to provide a link embedded in a QR Code that could be scanned by your phones barcode scanning application. Once you've scanned the code you now know where to go in order to grab the token that you need in order to get API access.

This Message will Self-Destruct in 5 Seconds

inspector gadget and stuff

There were a few things that needed to be taken care of when creating these URLs. Firstly, we couldn't lock the endpoints behind sessions since that would make it impossible to fetch the key. Secondly, it needed to be very hard for anyone to arbitrarily guess what the URL to get access to a token would be. Finally, we needed to be able to prevent "replay attacks" to the resource that contains the API token information.

In order to prevent people from guessing what the URL would be to get a token, we simply used UUIDs as the resource location. We had an object that held a reference to a token and would provide the token details when requested via JSON. This meant that whenever we needed to provide access to a token we could just create one off objects and link off to it's UUID identifier. Another added benefit is that if we wanted to we could regenerate another object that would reference the same token.

When it came to replay attacks, the main concern here was having someone watching traffic as it went by and clueing in that an endpoint is how keys were distributed. So what could possibly happen is I request my token, while this evil third party (let's call him Carçon) who is watching my requests also makes a request to get the same token. If we aren't careful we could accidentally give the API token to Carçon as well! The way we took care of this was by making it such that by looking at the token you destroy the external reference to it. In this case it would mean that when the attacker makes the request, instead of getting an API token they'd get a 404 instead since the object providing access was destroyed at the end of the initial request!

While I'm certain there are still some problems that might exist with this approach it felt like a novel way of solving the problem of key distribution. It also made much simpler for the developer writing our Android app since there they didn't need to store user credentials or do the OAuth dance.


Testing request-based modules and understanding Rails/Rack header munging

bike rack

I'm working on a tool to help make it easier to keep a local cache of product data from Shopify and one part of that means keeping up to date with changes. Thankfully the API provides an easy way to subscribe to webhooks which is quite useful. Though, I'm looking into the problem and know that just building another webhook controller would really only help me in the short term. It'll help me build out another part of the project but it won't really be usable outside of the project.

One big part of this project is I want to be able to tell someone "yeah, take this and add it to your project" with confidence. I don't really feel confident letting anyone use my code unless I've tested it at least a enough that I know the important parts work the way they should. When working with models and such it's pretty simple, though when you start working with requests things get a bit more complicated. The last thing we want our test suite to do is make external calls, they're slow and slow tests make us reluctant to run them.

Using the documentation I started building out some really simple tests to ensure that when a request comes in with the X-Shopify-Hmac-SHA256 header and I want to verify that I'm reading that in and calculating the HMAC correctly. I looked around for some examples of mocking request objects and only found some stuff talking about using tools like mocha or rspec to mock out methods that are called. I've been burned too many times by being stupid and not understanding APIs fully so I figured it would be better to use something. The most obvious is just work with the Rails request object since that's what I'm going to be using anyway. It's Rails, so as far as I'm concerned it's stable; not Standard library stable, but still pretty darn stable.

After giving the initializer a quick glance over and realizing I needed to work with a Rack env I dug a bit deeper into figuring out how to easily create the environment. This brought me to Rack::MockRequest which gave me exactly what I needed in order to build up that request environment.

With my test setup everything should be peachy and I'll be able to pull out that header and have me some passing tests.

class MyTest < ActiveSupport::TestCase
    test "authenticating a request" do
      controller('abracadabra')
      hmac = "u8aZR7htZKE6uWRg6M7+hTZJXZpcRmh5P4syND1EM24="
      controller.request = request('a message from shopify', hmac)
      assert controller.valid_webhook?
    end

    def request(data, hmac=nil)
      options = {
        :method => 'POST',
        :input => data,
        'X-Shopify-Hmac-SHA256' => hmac
      }
      ActionDispatch::Request.new(Rack::MockRequest.env_for("", options))
    end
end

I'd fire up my test and… it failed. I was getting a little frustrated and confused since this should work right? After a bit of digging around I got into ActionDispatch::Header#env_name which was doing some kinda weird stuff. Whenever you are dealing with a header that isn't your "normal" header (Content Type, Accept, etc.) it get's munged a bit. I'm not fully sure where, I think Rack might be involved in this mystery somewhere but I'm probably wrong. Anyway, let's say that along with your request to a Rails app you include the header Needs-More-Taters, well it'll go through your middlewares and all that business and by the time you actually end up looking at it from your controller it's now HTTP_NEEDS_MORE_TATERS!

What does this mean for my tests? Well my module isn't actually going through the whole dance for controller tests, my environment isn't getting setup correctly. Instead what I need to do is pass in the environment in the "proper" state. In my case it means in my options hash I needed to pass in the fully qualified names that Rails would expect to be there.

def request(data, hmac=nil)
      options = {
        :method => 'POST',
        :input => data,
-        'X-Shopify-Hmac-SHA256' => hmac
+        'HTTP_X_SHOPIFY_HMAC_SHA256' => hmac
      }

With this now discovered all my tests are passing! There was a bit of a yak shave involved (I'm notoriously good at falling into them), but it's cool because there was also a chance to learn a bit more about the innards of Rails and how I can just use plain Rack for testing.


Reducing the suck with Rails Engines

train engine

Mid last year I worked on adding a feature to the forum software at Shopify because I wanted to have a bit of context before clicking on a link in my emails. Since adding that feature I feel like I've become far more active on our developer forums and have helped answer a number of questions. It's also given me a lot of insight into the kinds of things that make software sucks.

One thing I'd often see are people asking about getting products along with order information. The problem is if an order would come in with 4 line items, they'd need to make another 4 API calls before they could do anything since the line items only contain product IDs. This is pretty shitty, and my response would typically end up being "you'll need to cache that locally and keep it up to date". Then I started thinking about the problem a bit more and realized that I'd be pretty pissed off if someone told me that.

On top of solving the business problem, I now need to do some boring bullshit in order to save on some API calls. Of course, if I want to passively keep the data up to date I need to add Webhook support and now this minor detail has exploded in scope if I want to be able to scale this at all.

Looking into it a bit more I realized that this is a problem that only needs to be solved once (per language/framework). Being a Rails developer and hearing so much about these engines that are all the rage I figured this is the perfect use case for them. The goal is to give Rails developers a drop in module that makes it super easy to get product detail caching working in your projects with little to no effort.

Currently the engine is in a state that with a little bit of code you can get a bunch of products imported into your application. For example, I have a controller with an import action that does the following:

def import
  api_products = ShopifyAPI::Product.find(:all)
  ShopProductSink::Product.create_from_resources(api_products)
  redirect_to root_path
end

And here is accessing that route in action:

Invoking /import for a logged in user

So there's a bit of work still on the developers side that requires a bit of insight and the API is still a little bit in flux. I'm currently not planning on doing any magic (I'm not smart enough for that yet), though I figure once I get it out there and get feedback I'll be able to make it even better.

If you're interested you can keep track of the project on github


Rails Rumble Recap: Continuous Integration and Deployment

110909-N-ZZ999-001

 

Last weekend marked my second year participating in the Rails Rumble programming competition. After my experiences last year I decided that we were going to be done a little bit differently.  Last year was a bit of a rush and we also partially changed our system half-way through. Another big part of the system was a lack of any real monitoring of what we were building whatsoever. No CI, no error reporting, no system health monitoring so when we deployed and were waiting for judging to happen we had no way of knowing what the health of our system was. This is going to be a part of a multi-part series where I breakdown the system and discuss various tools we used to make our application shippable and also give us the confidence in it’s stability.

To start us off I’ll be discussing our Continuous Integration and Deployment setup that we had. One thing I didn’t want to have us worry about was remembering to deploy the system after a merge into our master branch. We also wanted to know as quickly as possible when a part of our system broke so we could quickly react to the failure and get the system back to a healthy state. For this I looked around for some deployment and continuous integration software. I initially looked into Deploy for deploying our systems and Travis CI for running keeping track of the health of our project but wasn’t really interested in them. I personally found that for our project Travis would be far too expensive because we are planning on continuing the project once the judging is over. Deploy was tempting but it still required the press of a button in order to cause a deployment; also it didn’t provide CI services. Having fewer services to keep track of during 48 hours is always a nice plus. That being said, Deploy does look like a good tool if you are more in the camp of wanting explicit control over when code is deployed.

While googling for CI platforms I came across one called Codeship.io that really caught my eye. It provides CI services that are very similar to Travis or Circle CI as well as a continuous deployment system. That was a very attractive aspect of the product especially considering how we’d be pushing out a lot of code over the next 48 hours. I also found the pricing palatable after the trial period, which helped with the decision as well.

Getting the system integrated with our deploy scripts was pretty simple as well. Our production system was effectively turn key because I didn’t want to spend a lot of time configuring a server the way I wanted it. I currently don’t have a deployment system tool belt such as chef or ansible yet so I needed a system up and running fast. I decided that the easiest would be to just go with the stack scripts that were provided by the Rails Rumble team. Because of this everything was good to go in about half an hour and all that was left was finalizing the codeship integration. I want to say that their pitch of “integrates in 3 minutes” is true, because that’s how it felt.

Getting a project up and deployable on codeship mainly just requires picking a repository from your code hosting platform such as bitbucket or github, followed by giving it some instructions on how tests are run (which you can choose from a bunch of pre-baked configurations such as rails or python) and how to deploy. Once that is all set you just needed to drop an SSH key into your known hosts for your deploy user and you’re off to the races. On the next push to master Codeship will run the tests and if they are green will proceed to go through the deployment process. After having the first build deployed I was pretty happy.

The pricing of the product is a little bit weird though. For the smallest paid plan you are limited to 50 builds a month, which if you have any velocity whatsoever will result in burning through those in no time. The main reason is because Codeship runs against commits to every branch you have up on your code hosting platform. There is a way to get around it, which includes adding –no-ci to your commits though I’m pretty forgetful and wouldn’t be surprised if I constantly forgot to include that in my git commits. An alternative I haven’t really tried yet would be to have 2 repos, one for your production version or code that’s ready for pull requests so that builds are only triggered when you actually want them. Also having another repo that isn’t your default would prevent you from triggering builds by mistake.

Aside from that one personal problem (which when you are in bigger teams on unlimited plans wouldn’t be an issue at all) I was extremely pleased with the platform. It was also my first experience in setting up integration services to help give myself and the team confidence in the code we were pushing. So if you’re looking for some Continuous Integration and Deployment services, I’d definitely suggest you check out Codeship!


Thoughts on the Community’s Recent Black Mark

toxicity

As some of you may know, a major event happened within the Ruby community that is all too familiar within the greater tech community. Another one of our lady Rubyists was sexually assaulted, and bravely decided to share her experiences with the world at large.

As a cis-gendered, white male I have a difficult time even writing this since I’m aware of my privilege and have never experienced violation at the level this woman did. I do though consider myself to be an ally and feminist, and even if I am not the most educated in the subject, I try.

From what I have observed through my windows into the community and watching the fallout, there seems to be three main themes: aggressors and how they should be dealt with as a result of their actions; the community itself and how it reacts; and, finally, the conference scene. I do not feel like there is anything to add for the assaulted, since none of this was their fault.

I want to point out that I’ve been informed on the opinions of the following paragraph. I’m keeping it here for historical reasons, but the following article explains why sharing this information is important for the community

I feel like the actions that were brought against the aggressor were not constructive. The anonymity of the Internet offers the possibility of arming everyone with pitchforks, and facilitating mob mentality. I understand wanting to name and shame the guilty party so that others in the community are not put at risk, but I also feel that this type of community reaction will force them into hiding, and possibly into other communities where they might reoffend. Instead of trying to work on getting this person to the counselling they so clearly need, we push them away with no plans on rehabilitation. This is the same approach that has been used in general via the penal system. The underlying issues are never actually dealt with.

I believe that as a community it is our responsibility to make sure that everyone feels safe within it. While it would be nice to see community leaders standing up in record time, we do need to remember that at their level it becomes political and they need to tread very carefully with what they say. Meanwhile, we as individuals are able to stand up and call out what isn’t acceptable. Members at large are also more numerous than the leaders, and we can help sculpt the community as much as they can, if not more. By calling out the bad behaviour we let the leaders know what we won’t accept and set a positive example for those just getting into the community.

Encouraging the community can be done by praising good behaviour. I am not a fan of shit-lists as they draw attention to the bad behaviour and reinforce by negative example. It’s easier to build up a list of good examples than it is for the negative ones, especially since the negative example set is theoretically infinite. When you see good behaviour, share it via your social circles. Show those in your community what you find acceptable and want to encourage. If you are lucky, a prominent figure in the community will share it, further showing the community what is awesome.

A prominent member of the community suggested that women should avoid attending Ruby conferences; I disagree. By avoiding the conferences it doesn’t challenge the status quo. If anything, I’d love to see more women attend the conferences as a result of this. It might sound strange, but having more women makes those who are attending feel less alienated and creates a community within the community to which they can identify and feel safe. This may result in having what would appear to be cliques, and may make the group unapproachable, but I feel that we as a community still need to mature and grow to a point where this isn’t necessary. It doesn’t mean that the groups aren’t approachable, but it does take a lot of courage to walk up to them and start conversation and you may get shot down. Though, this might be what it currently feels like to be a lady at a tech conference.

Most evening events at conferences tend to be alcohol driven, which can make some people uncomfortable. Aside from things such as lowering the tabs or perhaps issuing out beer tickets, which would deem some companies as uncool I guess, what else could we do? Some conferences have started having alcohol-free alt events such as board game nights. A nice benefit of this is it gives minors something they can go to and rub elbows with other members of the community. Since they are booze free it’s a much safer environment for everyone.

I do not want to see the bar scene go away though. They do provide a more relaxed environment to talk to other people, and I do enjoy pints. I do want to see that aspect of conferences made more safe and inclusive. Having the events in more conducive environments would be a start in my opinion. I went to a conference in San Francisco last year and it seemed like the official outing locations were all clubs with extremely loud music, which made conversation all but impossible. Another thing that would be interesting would be placing a limit on what people can select when choosing their beverages. Limiting people to singles and half-pints should help with the getting excessively drunk bits. I’d honestly love to see members who are pounding back drinks–since it’s a free tab–kicked out. I’ve also found it very difficult to have a conversation with someone whose words are slurred or incomprehensible as a result of drinking too much.

It’s our job as a community to make it a safer place for everyone, from providing alternative safe-space events to encouraging positive behaviour in the community


Hacking Homebrewing — Part 2

Brewwatcher Web in Action
Last week I told you about the Arduino side of my homebrew hardware project. Now, I'd like to talk about the Raspberry Pi, Ruby and JavaScript side of things.

The Pi was really just a deployment platform that would make it easy to get a low power Linux box running that I could easily configure. I wanted it to be headless, so I chose to go the Arch Linux route since that would allow me to have full control over the system and what parts were installed.

As I mentioned in the previous post, I'm using the serialport gem to do all the interaction with my Arduino. The gem takes care of all the business that is required in properly connecting to the device, and helps me reach my goal – to read and write data. The only awkward part of the library is establishing a connection to the device, after that it's just IO.

# Assuming bundler
require 'serialport'

baud      = 9600
data_bits = 8
stop_bits = 1
arduino   = '/dev/arduino.tty' # totally incorrect!
serial    = SerialPort.new(arduino, baud, data_bits, stop_bits)

serial.puts "help"
puts serial.gets # blocking call!!

The biggest lesson I learned was that gets calls to the device are blocking, which threw a wrench in my programs for a bit. One approach I took was to build up a message queue and just read from that, but for whatever reason I messed up my Mutex and the Thread responsible for stuffing message into the actual queue could never get priority. It also made things a bit complicated because of the reboot on connection issue I mentioned in the last post.

As for packages I really only needed a handful: Ruby 2.0, SQLite + SQLite development dependencies and Nginx. I took the approach I've seen in several other places and just have a thin server running my web app, and use Nginx as a reverse proxy. The solution works out fairly well and required little additional setup since I'm using plain old Nginx. I thought of using passenger, but lack of experience and no desire to compile things on the Pi made me change my mind.

I didn't want to have to rely on having my web server running in order to collect data from my Arduino, so I set up a Cron job that would run my data collection ruby script. The script did the fewest things possible in order to connect to the Arduino, grab some data and store it into my SQLite database. I'm using SQLite and there is a bit of a race-condition that if I'm writing Arduino data, I can cause my Sinatra app to fail since it cannot establish a connection to the database. For now, I'm not going to worry about it since my scripts only run every hour and I don't see the contention being an issue for myself.

I had my base system running and collecting data, but without any way of interacting or seeing the data it's just a bunch of numbers in a table. This is really where Sinatra came in. Its primary goal was to make it easy to do some basic CRUD such as: creating new brews, setting them as active, configuring the TTY to connect to and so on. Another thing I used it for was to provide a very simple JSON API for getting access to a brew's readings during fermentation.

Charting out the results of a test run
The reason I needed a JSON endpoint is because I'm using the d3 graphing library for visualization and I based most of my charting code off of Mike Bostock's line chart example that was linked from the d3 gallery page. I added a few minor changes, such as some zones to easily see whether a beer is within its normal fermentation range or not. I'll admit that the charts are far from pretty, but they met my requirements and once I go back to it I'll look into making them a bit sexier. One really nice aspect of d3 is it's just building SVGs which can have CSS classes attached to them, so they are fully styleable.

The Arduino device testing/selection screen
A somewhat tricky part of the application was coming up with a way of making it easy to change what device I was connected to without having to change code. The one thing I hate always doing is having configuration in my code that makes it complicated to change things – especially when I'll be deploying to a different or unpredictable environment. I got around this by making a few slight changes to the system. I assumed that the user the application would be running as would have read and write access to the tty group on the system. With that assumption made, all I had to do was issue a directory listing query to /dev and then let the user choose which device to connect to.

Now, if we run a ls /dev on any UNIX-like platform, we're going to see a boatload of items, and this wasn't going to be acceptable. Based on a few observations I was able to slim down that list to a handful of devices. Because of prior knowledge I somewhat knew which device to connect to, but I'm forgetful so I had to add some tools to help with that. This was solved by adding a little "test connection" button in my Sinatra app that would try to open the connection and if it was successful know that I'm probably connected to the right device. I am using sort of bad habits since the exceptions kind of control my program flow, but I'm still a little green when it comes to the right way to do these kinds of things in UNIX.

Two weeks ago I had brewed my first batch of beer in the new apartment and the system was ready to roll! I was pretty excited to have my little hardware project actually be put to the test, and set everything up – while being extremely sanitary and ensuring everything was sprayed or soaked in Starsan! Though, I noticed after a day or two that my airlock wasn't bubbling, despite obviously active fermentation and got a little bit concerned. After looking around a bit I noticed that the wire from the temperature probe was preventing my bung from getting a perfect seal. Defeated, I removed the sensor and turned everything off before returning the bung back to it's home. About fourteen hours later, the airlock was bubbling like a champ. With that in mind, I'm planning on taking one of my rubber corks and drilling another hole in it that'll be large enough to get the probe through, but will keep my beers environment safe and tasty.


My charting code on github — it's gross! You've been warned
Brewwatcher on Github