ESClient: Python ElasticSearch Client

Posted by on February 25, 2012 No comments

I’ve been working on a little project: a Python client for ElasticSearch. Are there other clients out there? Yeah sure, and they are even pretty decent. Especially pyes is OK. But I’m missing documentation on that one and I don’t like the approach they took on implementing the API. I wanted to create a simple client that stays close to the ElasticSearch REST API. It allows you to directly submit either JSON or a tree of Python objects that can be converted to JSON. ESClient has documented code and supplied unittests that should get you a long way. I plan to write good documentation to get people started quickly as soon as the API starts to reach a stable state.

In a few days I have learned a lot about:

It’s simply awesome how quick and easy you can get something up and running these days. With just a few lines of code and Pythons distutils you can create a source package and a Windows binary, upload it all to PyPI and make it installable for everyone in the world with a single command:

pip install esclient
or:
easy_install esclient

ESClient is still in its early stages and I can assure you the API will change over the coming weeks. I need to implement more API methods and I want to implement bulk indexing. If all API methods are implemented and bulk indexing works I will work towards a stable 1.0.0 release that will also have a stable API. I also want ESClient to handle errors well. This is something that I missed in some other libraries that I have found, like pyelasticsearch.

Share

Install Canon software without the original CD

Posted by on August 28, 2011 No comments

Canon has an annoying and useless policy which makes it impossible to install the updates from their website without having an original cd. Apparently they want to make the lives of their customers harder than it already is.  (My Mackbook Air doesn’t even have a DVD player!)

Lucky for us this website has some tricks to circumvent this. For me simply removing an update.plist file was enough to get ImageBrowser 6.7.2-updater (Mac OS 10.5-10.6) to installing without original cd.

Share

Twitter’s list of 401 banned passwords

Posted by on August 19, 2011 No comments

I was looking at the source of a Twitter.com page when I noticed a javascript array with banned passwords. They were ROT13 encoded, so I decided to decode them and have a look. This has been done before but the list is now a bit longer, so here you go:

(warning: this list contains quite some dirty words, which could be the reason why Twitter does a rot13 enconding :-0) Read more »

Share

Macbook Air fan stuck and making ticking noise?

Posted by on August 4, 2011 No comments

Does your Macbook Air fan makes a repeating, predictable ticking noise? Does it get hotter than usual when watching YouTube video’s? Does iStat Pro reports a fan speed of 0?

A few taps with your finger help! If you have an older model (2009) tick on the right side. If you have a newer model (mine is late 2010) then tap on the right side. Don’t overdo it, but don’t be too gentle either.

See also this discussion: https://discussions.apple.com/thread/1860264

Share

Want to beat the hackers at their own game?

Posted by on May 7, 2010 No comments

Google shows how web application vulnerabilities can be exploited and how to defend against these attacks. The best way to learn things is by doing, so you’ll get a chance to do some real penetration testing, actually exploiting a real application.

http://jarlsberg.appspot.com/

Share

Anti Counterfeiting Treaty Agreement leaked

Posted by on March 27, 2010 No comments 
The leaked ACTA agreement!
Soon to be copy pasted into a law near you.

Check out this page on The piratebay

Share

Another privacy invading monster (LSO cookies)

Posted by on August 18, 2009 2 comments

Recently I came across something called LSO cookies (read about it, I’m not going to explain them in detail here). Since more and more browsers, virus scanners and security software block cookies, these LSO cookies are a real treat for advertisers and tracking companies for several reason:

  • most people never heard of them
  • they are difficult to block
  • they provide lots of storage (100KB per site)
  • they are not removed by your browser, ever
  • they work and track you, even with your browsers “privacy mode”

Dangers

There are several dangers to these cookies. First and foremost, we block cookies for a reason. We don’t want to be tracked everywhere on the web and we don’t want companies to build profiles of our web usage for whatever reasons they have. These companies shamelessly track us anyway by using all kinds of tricks, like these LSO cookies, instead of respecting our explicit choice to not be tracked and monitored.

Another problem is that this will leave tracks of your Internet usage on your computer, even if you try to cover those tracks by deleting cookies, browser cache and temporary files.

So what can we do about this?

First of all, the best thing would be to not use flash but that ain’t an option. (We want our youtube to work!). So the second best option is to block or at least remove the cookies. There is an excellent Firefox plugin called Better Privacy that will give you all kinds of options to remove or block LSO cookies.

If you don’t have Firefox, your third option is to go to Abobe’s Flash player settings page – you never heard of it, neither did I – and set the storage space to zero KB. Next, go to the last tab there or use this link, and be amazed at the amount of sites that use LSO cookies to store whatever they want to store on your PC. Next, click the remove all button to remove it all. Note that setting the storage to zero prevents sites from storing cookies, but Flash will still create directories for each site that tries. So next time you visit that shameful pr0n site, be aware that Flash will keep track of it.

Share

WikiBench master thesis

Posted by on May 20, 2009 No comments

I officially got my WikiBench project graded with an 8, with which I’m of course very satisfied. You can now read my thesis called WikiBench: A distributed, Wikipedia based web application benchmark.

People interested in this project already found their way to my blog. For those who are wondering: I will publish the code and I will do so very shortly (within days). It will most probably appear on Google Code and you won’t have to search for it since I will devote a post to it right here and include the URL. I will probably release it under a BSD-style license which should give you lots of freedom. Unfortunately I’m not sure yet if I am allowed to release some of the trace files obtained from Wikipedia.

Share

Ghostnet

Posted by on April 14, 2009 No comments

After all the buzz around Ghostnet, it’s fun to look back and read the origal document describing the spy network. It’s an interesting read, and if you don’t have the time to read this you can also check out the Security Now! podcast from April 9th in which Steve Gibson explains how the research group found out about the spy network and how amateurish the (open source) Gh0trat software actually is.

One very important lesson learned from this story is that attackers no longer  control these networks by using IRC as we have seen in the past. Ghostnet used plain old http requests to periodically check for new commands. The startling thing about this is that this is exactly the kind of traffic that gets through firewalls and even proxy servers without any problems. HTTP replies consisting of jpg images contained the actual, encoded commands.

Share

Bloom filters

Posted by on March 3, 2009 No comments

This article by Broder and Mitzenmacher gives a good description of how bloom filters work and what they can do for you. The bloom filter basically replaces a dataset with a filter that can tell you if an item is a member of that set or not. It will not give false negatives, but it might give false positives. In practise, this is a negative property that can be outweighted by the space savings a bloom filter introduces; after all, you do not need to query the dataset to determine membership. The most important and summarizing quote you should remember from the article:

The Bloom filter principle: Wherever a list or set is used, and space is at a premium, consider using a Bloom filter if the effect of false positives can be mitigated.

The article also gives a number of examples in which bloom filters are used. E.g. to aid resource location in P2P and cache systems.

Share