Friday, March 8, 2013

ArangoDB Driver for Python: added support of Python 3.3 and ArangoDB 1.2

At least added support of Python 3.3 to the ArangoDB Python driver v 0.1.6 (github). Move forward with pip install arango. By default in Python 3.3 pycurl client will not work, so it's better to specify client by hands using: Also, added support of ArangoDB 1.2.

Monday, February 18, 2013

arango: ArangoDB Driver for Python (updated Feb 24, 2013)

Past year I've heard about new database - ArangoDB. I was not able to test it using my daily tools - there was no driver for Python. So, I've decided to create it. I've tried to create really good one - high code coverage, well documented and easy API initially was a must.

Yesterday, I've released it - now available on PyPi and on github

Toolchain

Here is list of basic tools which I've used to create the driver

  1. nose - for executing tests, integration and unit. ArangoDB driver contain 103 tests
  2. coverage - to have clear metric how many code lines was covered by tests. Now it's 89%
  3. sphinx - to have good documentation it's a must. Also .. testcode:: and .. doctest :: directives is very useful to keep code examples usable.

Performance

Initially, I've used requests library as main HTTP transport tool. But at some point I've get a letter from ArangoDB team about python driver performance. Here is most important quote:

The guys told me that PHP took 0,5 seconds, JS about a second and our Python script 5 seconds

I've start digging and found that requests is slow. Fast tests shows that urllib2 2x faster than requests, and pycurl 3x faster. Here is performance tests and JS code for arangosh. Below small chart about 1000 documents creation performance



Some details

Installation:
pip install arango

Documentation of the Arango driver for python

ArangoDB official website

Update:: I've improved performance of Arango requests client by using Session Objects. Also updated performance chart.

Wednesday, January 30, 2013

Redis PubSub wrapper for Python

Recently I've found that there's no reasonable simple and useful Redis pub sub examples around. So, here is my dead simple wrapper how to implement it without any unnecessary overhead.

Sunday, August 19, 2012

Sphinx Doc, JSON highlight and Sphinx extensions: kung-fu

At the moment Pygments which used by Sphinx Doc haven't support for JSON code highlight which is really sad.

I've not found any useful information how to do it quickly. So here is my way:
  1. I've found custom pygments lexer which support JSON: pygments-json . I will be part of pygments soon
  2. It wasn't clear to me how to add custom pygments lexer to sphinx, my google-fu isn't good today
  3. A bit more googling gave me Sphinx Extensions API , especially add_lexer method of Sphinx instance
Ok, now it's clear to me how to add new lexer. I've created ext/hijson folder within source, to __init__.py I've added setup function:

Also here is how to add support of ext folder as folder with custom extensions So, pip install pygments-json and use

.. code-block:: json

Nice and smooth:

Thursday, July 19, 2012

PyFlakes-clean import of local_settings.py

Any project have own bunch of settings for different environments: for dev, production, staging. Everyone use it on daily basis. But annoying thing is that PyFlakes, great code analysis tool, warn about that. And it's reasonable.

So, to have this functionality, but without warning I use this pattern:

try:
    import local_settings
    for setting in dir(local_settings):
        if setting.startswith('_'):
            continue
        locals()[setting] = getattr(local_settings, setting)
except ImportError:
    pass

Simple and useful!


Update: I've added ignore of attributes which starting with underscore. Thanks to Igor Davydenko

Sunday, July 8, 2012

Unstable HTTP services: what we can do to easily handle that?

The story: I have several HTTP service providers which works quite unstable. Yes, I had to have in mind this during development. But we're all thought that issues are "temporary" and will gone when we going to production. We accurately added logging.error in every place and move on with other stuff.

But our expectation about temporary nature of service behavior will never happen. Service sometimes work slowly, sometimes return HTTP errors and so on. We receive tons of exceptions every day. We had to do something with that.

The solution: Here is safe_exec decorator which help solve this problem. You can specify how many times you want to try execute function, what's timeout between them and what exceptions are expected during execution decorated function. For example, urllib2.urlopen may generate urllib2.URLError or  urllib2.HTTPError.

import logging
import time

from functools import wraps

__all__ = ("safe_exec",)


def safe_exec(exceptions, shakes=3, timeout=1, title="", **kwargs):
    """
    Decorator to safely execute function or method
    within `shakes` trying.

    In case provide argument `default` exception will not
    be raised and will return provided value.
    """
    def wrap(func):
        if not isinstance(exceptions, tuple):
            raise TypeError(
                "First argument of safe_exec should be tuple of exceptions"
            )

        @wraps(func)
        def wrapped(*args, **kwargs):
            result = None
            orig_exception = None
            for shake in range(shakes):
                try:
                    result = func(*args, **kwargs)
                    break
                except exceptions, orig_exception:
                    logging.warn("%s: Sorry, can't execute %s, shake #%d",
                        title,
                        func.__name__,
                        shake,
                        exc_info=True
                    )
                    time.sleep(timeout)
            else:
                logging.error(
                    "%s: Can't execute `%s` after %d shakes",
                    title,
                    func.__name__,
                    shakes
                )

                if "default" in kwargs:
                    return kwargs.get("default")

                raise orig_exception

            return result
        return wrapped
    return wrap


Sample usage:

import urllib2

@safe_exec((urllib2.URLError, urllib2.HTTPError), shakes=2)
def download(url):
    return urllib2.urlopen(url).read()

download("http://slow-resource.com/")

Saturday, June 9, 2012

Testing with Jenkins, Selenium and Continuous Deployment


My presentation at KharkivPy#4 about Selenium, Jenkins and Continuous Deployment. A lot of people asket me about code samples. There's no samples. There's no python code at all in this presentation. The main purpose of this presentation is that you can build full cycle testing for your project easy enough.

For example by using tools like Fabdeploy or even raw Fabric

Saturday, March 24, 2012

Kharkiv.Py #3: JavaScript and Python Back-end presentation

I will talk at Kharkiv.Py #3 about JavaScript and Python as back-end.

In short, I will talk  about experience with next things:
  • Assets management
  • JavaScript Templates
  • Constants within JS files
  • Browser-specific assets
  • SCSS/LESS
  • CDN
Start ar 2:30PM

Update: I've published slides on SlideShare and published Sample  Project on Github


Sunday, February 12, 2012

Headless JavaScript testing, Sinon.js, Fake Timers and Rhino

Recently I've started to use TDD approach with development of client-side code/JavaScript. The code on front-end become more complex and it have to be tested along with back-end code.

One of first things which I've found was Sinon.js - the mock library to make my life easier. Btw, I'm using Jasmine for writing tests. So, everything looks great until I've tried to integrate my tests into our CIA (Jenkins).

The tests which working great within browser failed using rhino/env.js. That was weird and unfortunately tracebacks were useless. I've started to go deeper and found an article which make things clear to me: Sinon.js have issues with timers.

Update: I've fixed issue with Sinon.JS Fake Timers and Rhino.js

I've made small stub to replace functionality of fake timeouts from Sinon.js and wow, tests are passed!


// setTimeout/clearTimeout stub using Underscore.js
var FakeTimeout = function () {
  var self = this,
  timers = [],
  counter = 1,
  timeoutOrig = setTimeout,
  clearOrig = clearTimeout;

  // add new timeout to the queue
  this.setTimeout = function (f, timeout) {
    var id = counter++;
    timers.push({
      'callback': f,
      'timeout': timeout,
      'id': id
    });
    return id;
  };

  // cleanup timeout
  this.clearTimeout = function (id) {
    timers = _.filter(timers, function (item) {
      return item.id !== id;
    });
  };

  // tick the clock by timeout
  this.tick = function (timeout) {
    var timerToRemove = [];
    _.each(timers, function (timer) {
      if (timer.timeout - timeout <= 0) {
        timer.callback();
        timerToRemove.push(timer);
      } else {
        timer.timeout = timer.timeout - timeout;
      }
    });
    timers = _.difference(timers, timerToRemove);
  };

  // install fake timers
  this.install = function () {
    setTimeout = _.bind(this.setTimeout, this);
    clearTimeout = _.bind(this.clearTimeout, this);
  };

  // restore the original timers
  this.restore = function () {
    setTimeout = timeoutOrig;
    clearTimeout = clearOrig;
  };
};

Here is how to use it:
// Usage:
var timeout = new FakeTimeout();

// Somewhere in your code
timeout.install();

// ... some functionality which use setTimeout/clearTimeout
timeout.tick(1000);

// restore the state of setTimeout/clearTimeout
timeout.restore()


The code require Underscore.js 

The conclusion is to keep it simple. I don't get why you guys need so much code for such simple thing.

Friday, December 30, 2011

Flask-Jasmine: Execute Jasmine tests within Flask

Just finished Flask-Jasmine extension to execute beautiful Behavior Driven tests for Jasmine in JavaScript.

Such extensions already exists for Django and for Rails. Now it's available for Flask too.

Install with pip:

pip install Flask-Jasmine

Detailed instruction about configuration and usage