Silas Sewell

Getting Started with Cloud Foundry's Warden

Silas Sewell (Silas Sewell) — Sat, 06 Oct 2012 00:00:00 PDT

warden is a framework that allows you to spawn containers in seconds and programmatically control resource isolation (memory, bandwidth, disk), mounts, proccesses and other things.

At its core warden is a Ruby daemon and collection of clients developed and maintained as part of VMware's open source PaaS Cloud Foundry.

In this post I'll cover setting up warden in a Vagrant box and laundry list of commands via builtin repl.

To get started you'll need Vagrant and Git.

Once those are installed grab vagrant-warden and up the box:

$ git clone git://github.com/silas/vagrant-warden.git
$ cd vagrant-warden
$ git submodule init
$ git submodule update
$ vagrant up

After the box finishes booting you want to open another terminal, one to run the server and the other to run the repl.

In the server terminal ssh to the vagrant box and start the warden server:

$ vagrant ssh
$ sudo warden-server

In the client terminal ssh to the box again and start the warden repl:

$ vagrant ssh
$ sudo warden-repl

Finally step through the commands below to get an idea of what you can do with warden.

Create container

warden> create
handle : 16bcqds1452

List containers

warden> list
handles[0] : 16bcqds1452

Get info on a container

warden> info --handle 16bcqds1452
state : active
host_ip : 10.254.0.9
container_ip : 10.254.0.10
container_path : /opt/warden/warden/data/containers/16bcqds1452
...
memory_stat.rss : 2863104
memory_stat.swap : 0
...
cpu_stat.usage : 201273681
cpu_stat.user : 0
cpu_stat.system : 0
disk_stat.bytes_used : 20480
...

Map a host port to a container port

warden> net_in --handle 16bcqds1452 --container_port 8000
host_port : 61001
container_port : 8000

Spawn a process in a container

warden> spawn --handle 16bcqds1452 --script 'echo hello | nc -l 8000'
job_id : 2

Interact with a mapped port (in a third terminal)

$ vagrant ssh
$ echo world | nc 10.254.0.9 61001
hello

Get results of spawned proccess

warden> link --handle 16bcqds1452 --job_id 2
exit_status : 0
stdout : world

stderr :

Copy data into and out of a container

warden> copy_in --handle 16bcqds1452 --src_path /tmp --dst_path /tmp/bla
warden> copy_out --handle 16bcqds1452 --src_path /tmp/bla --dst_path /tmp/bla-out

And more via the help command

warden> help

Updated liquid-inheritance (liquid-blocks)

Silas Sewell (Silas Sewell) — Fri, 07 Sep 2012 00:00:00 PDT

I've updated Dan Webb's liquid-inheritance to work with the latest version of the Liquid templating language and published the result as liquid-blocks.

It includes an example Sinatra project for anyone looking to get started quickly.

Building RPMs using brpm

Silas Sewell (Silas Sewell) — Sun, 15 Jul 2012 00:00:00 PDT

Building RPMs for Fedora or the Enterprise Linux variants is a pain in the butt. My attempt at remedying this problem is a little command line tool called brpm.

The idea is to have sane defaults and do as much of the manual work as possible.

So if you have a spec file named bla.spec in the current working directory, you just run brpm and the tool will download all your sources, create a source and binary RPMs for your current distro (ex: Fedora 17) and architecture (ex: x86_64). The log files for the build will be in ./build and a yum repository with the build results will be created in ../build/<distro-name>/<distro-version>/<arch>.

The reason for creating a yum repository is to make it easy to chain builds of RPMs that require their precursed to build.

Setup

Install the latest version of Fedora (local or vm)

Add the following yum repository

sudo curl http://dl.sewell.org/rpm/sewell/fedora/sewell.repo \
  -O /etc/yum.repos.d/sewell.repo

Install brpm RPM
```
sudo yum install brpm -y
```
Add yourself to the mock group
```
sudo usermod -G mock $USER
```

Usage

Create an rpm and spec directory
```
mkdir -p rpm/python-ssh
```

Switch to the directory and grab a spec

cd rpm/python-ssh

curl https://raw.github.com/silas/rpms/master/python-ssh/python-ssh.spec \
  -O python-ssh.spec

Build the RPM
```
brpm
```

For more information see brpm --help and the brpm page.

Setup Gitolite on Ubuntu (Maverick)

Silas Sewell (Silas Sewell) — Sat, 08 Jan 2011 00:00:00 PST

Below is a quick guide to installing Gitolite on Ubuntu.

Create an SSH public/private key pair
```
[silas@client]$ ssh-keygen -t rsa
```

Upload the public key to the server

[silas@client]$ scp ~/.ssh/id_rsa.pub user@server.example.org:/tmp/user.pub

SSH to the server and become root

[silas@client]$ ssh user@server.example.org
[silas@server]$ sudo su -

Install gitolite

[root@server]# apt-get install gitolite

Switch to the gitolite user
```
[root@server]# su - gitolite
```

Run the Gitolite setup script

[gitolite@server]$ gl-setup /tmp/user.pub

From your client computer clone the gitolite-admin repository

[silas@client]$ git clone gitolite@server.example.org:gitolite-admin
Cloning into gitolite-admin...
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 6 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (6/6), done.

Switch to the gitolite-admin directory
```
[silas@client]$ cd gitolite-admin
```

Create a test repository

[silas@client]$ vim conf/gitolite.conf
[silas@client]$ git commit -a -m "Add test repository"
[master 507045a] Add test repository
 1 files changed, 3 insertions(+), 0 deletions(-)
[silas@client]$ git push
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 376 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Already on 'master'
remote: creating test...
remote: Initialized empty Git repository in /var/lib/gitolite/repositories/test.git/
To gitolite@server.example.org:gitolite-admin
   87cc470..507045a  master -> master

Clone the test repository

[silas@client]$ git clone gitolite@server.example.org:test
Cloning into test...
warning: You appear to have cloned an empty repository.

Add a README file to the test repository

[silas@client]$ echo "Test Repo" > README
[silas@client]$ git commit -a -m "Initial commit"
[master (root-commit) 4a49ee0] Initial commit
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 README

Push the changes to the server

[silas@client]$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 218 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To gitolite@server.example.org:test
 * [new branch]      master -> master

Check out the Gitolite wiki for more documentation.

txsrv: Message-based Twisted Services

Silas Sewell (Silas Sewell) — Sun, 12 Sep 2010 00:00:00 PDT

I've started work on txsrv, a Python library that aims to make developing message-based services in Twisted easy.

The code is still pretty rough, but might be of interest to someone hacking on AMQP-based Twisted services.

Sample Usage

Create a new service and change to the project directory.
```
$ txsrv create mytest
$ cd mytest
```

Edit the mytest.conf file so that the spec_file option is valid.

[connection:amqp]
type = amqp
host = localhost
port = 5672
vhost = /
user = guest
password = guest
spec_file = /usr/share/amqp/amqp.0-8.xml

[handler:hello]
connection = amqp
exchange = hello
routing_key = default

Edit the mytest.py file so that it prints the message body twice.

import txsrv

class Service(txsrv.Service):
    @txsrv.handler('hello')
    def hello(self, message):
        print message.body * 2

class ServiceMaker(txsrv.ServiceMaker):
    tapname = 'mytest'
    description = 'A mytest txsrv example.'
    service_type = Service

Start the mytest service and send a message to the hello exchange.

$ twistd -n mytest -c mytest.conf
2010-09-12 00:41:10-0400 [-] Log opened.
2010-09-12 00:41:10-0400 [-] twistd 10.1.0 (/usr/bin/python 2.6.4) starting up.
2010-09-12 00:41:10-0400 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2010-09-12 00:41:10-0400 [-] Starting factory <txsrv.protocol.amqp.AmqpFactory instance at 0x2cce950>
2010-09-12 00:41:10-0400 [AmqpProtocol,client] hellohello

Go Programming Language: Hello World (Fedora 13)

Silas Sewell (Silas Sewell) — Wed, 28 Jul 2010 00:00:00 PDT

If you're interested in trying the Go Programming Language I've created an RPM for Fedora 13 to get you started quick (this package is in no way suitable for submission to Fedora).

Download the x86_64 (or i386) RPM.

Install the Go package using yum.

sudo yum localinstall --nogpgcheck go-0.0.0*.rpm

Open a new shell.

Create a file named hello.go and enter the following code:

package main

import "fmt"

func main() {
  fmt.Printf("Hello World!")
}

Compile, link and run (replace 6 with 8 if you're using i386).
```
$ 6g hello.go
$ 6l hello.6
$ ./6.out
```

You can view the latest RPM source on GitHub.

Update: It looks like someone has already created a much better package.

GNU Screen Profiles

Silas Sewell (Silas Sewell) — Wed, 14 Jul 2010 00:00:00 PDT

If you're a regular GNU Screen user might find this hack for creating profiles useful.

Create a .screen directory in your home directory.
```
mkdir "$HOME/.screen"
```

Add the following to your .bashrc or .bash_profile.

sp() {
  if [[ -z "$1" ]]; then
    screen
  elif [[ -f "$HOME/.screen/$1" ]]; then
    screen -c "$HOME/.screen/$1"
  else
    echo "Unknown screen profile '$1'."
  fi
}

Create a file in the ~/.screen directory with whatever profile name you would like to use (below is a sample profile named tyrion).

You should copy the first line into each profile as it loads your default GNU Screen settings.
```
source "$HOME/.screenrc"

chdir "$HOME/src/tyrion/src"
screen -t code  0

chdir "$HOME/src/tyrion"
screen -t build  1

chdir "$HOME/src/tyrion/tests"
screen -t tests  2

chdir "$HOME/src/tyrion"
```
Reload your shell and test your profile (replace tyrion with whatever name you picked).
```
sp tyrion
```

Introduction to Tyrion

Silas Sewell (Silas Sewell) — Wed, 07 Jul 2010 00:00:00 PDT

Today I wanted to introduce Tyrion to the Fedora Planet and anyone who happens to read my blog.

Tyrion is in short a lightweight Func replacement.

Why Tyrion?

I created Tyrion because Func didn't meet some of my needs, specifically the points outlined below.

(1) Asynchronous

Tyrion was built using a protocol which is inherently asynchronous. This has lead to lower latency operations and better scalability.

That isn't to say Func couldn't be updated to be asynchronous (using the Twisted XML-RPC libraries for example), but I think current toolsets highly favor protocols which were designed from the beginning to be asynchronous.

(2) Language Agnostic

From a technical perspective Func is also language agnostic (it uses XML-RPC as a transport protocol) and although you could write a client or server in any language, from a pragmatic standpoint you're stuck writing services in Python.

On the other hand Tyrion is written in C++ and each service is a simple executable. This lends itself nicely to writing services in whatever language is most appropriate.

(3) Low memory footprint

Func is written in Python and as such suffers from a dynamic language's higher memory usage.

For example, the following Python application uses ~4.2MB memory on my Fedora 13 box.

#!/usr/bin/env python

import time

time.sleep(60)

While the complete tyrion-node application uses ~4.5MB.

For many systems administrators the difference between Func and tyrion-node's memory usage will be negligible, but if you running an infrastructure with several hundred or several thousand hosts the resource difference can be enormous.

A lower memory footprint is also advantageous to companies who heavily utilize virtualization. If you have 5 servers and you slice those up into 4 virtual machines each, you've just jumped from 5 to 20 daemons. Lets say Func uses 15MB memory, that's a 60MB to 300MB jump compared to tyrion-node's 22.5MB to 90MB.

(4) Permissions

The protocol that Tyrion uses has built-in support for users. This allows Tyrion to support user specific permissions on a service level and opens up some really cool possibility, such as exposing services to partially trusted users.

In addition because Tyrion uses a process model to run each service, a configuration setting can be defined to specify under which user and group a service is run.

(5) Cloud

This is a somewhat stupid bullet, but it does have some credibility. Func was created to rely on a hostname/certificate combindation and the assumption that we could communicate to a managed host on a defined port. This setup works great when you have full control over your network.

Unfortunately many of us use external resources (AWS, Linode, Slicehost, etc...) where some or much of the network and infrastructure configuration is out of our control.

Tyrion was built on the XMPP protocol, this means that Tyrion connects to a XMPP server and just listens for events.

With XMPP we also get authentication, an encrypted transport, some decentralization, federation, a solid selection of open source and scalable servers and clients for pretty much every language.

(6) Hot Reloads

The ability to update configuration settings and install services without a daemon restart.

Overview

This will briefly overview how Tyrion works.

The Tyrion package has two primary parts, the tyrion client and the tyrion-node.

tyrion-node is daemon which runs on each managed server. It boots with the host, connects to an XMPP server and listens for service requests.

tyrion-node Configuration

Below is the general configuration settings for tyrion-node which are located in the node.conf file:

[general]
acl_path = /etc/tyrion/acl.conf                 ; ACL configuration file
log_path = /var/log/tyrion/node.log             ; Logging path
log_level = info                                ; Logging level
service_path = /usr/share/tyrion/service        ; Service directory

[xmpp]
jid = test.node@example.org/tyrion              ; Tyrion node JID
password = test.node.pass                       ; Tyrion node password
;server = example.org                            ; Server address if different from JID domain

The node.conf file also lets you define default values for installed services.

[service:org.tyrion.service.bash]
timeout = 300                                   ; Default service timeout (seconds)
timeout_lock = false                            ; Lock default timeout setting
user = mike                                     ; Default user to run service as
user_lock = false                               ; Lock default user setting

This tells Tyrion that the org.tyrion.service.bash service should have a default timeout of 300 seconds and run as the user mike. In this case because we haven't locked the timeout or user option they can be overriden by the requester.

tyrion-node Service

If you were to look in the service_path directory you would see an executable file named org.tyrion.service.bash.

[silas@example.org]$ cat /usr/share/tyrion/service/org.tyrion.service.bash 
#!/usr/bin/env sh

/usr/bin/env bash

Obviously this service is a bit redundant and not all that interesting, but serves as a good example of how a service works.

tyrion Client

First we'll cover the basics of the tyrion client. Then we'll use that information to make a basic service request and follow that request from the client to the node and back again.

tyrion Client Configuration

First the client.conf file:

[xmpp]
jid = test.client@example.org                   ; Tyrion node JID
password = test.client.pass                     ; Tyrion node password
;server = example.org                            ; Server address if different from JID domain

[profile:test]
jid = test.node@example.org/tyrion              ; Default destination JID
service = org.tyrion.service.bash               ; Default service type
timeout = 300                                   ; Default timeout
user = usertest                                 ; Default user
group = grouptest                               ; Default group

We can see that like the node.conf file we have our XMPP options under the xmpp section. Below that we also have a default profile called test. Similiar to the service section in the node.conf file this lets us setup some basic defaults. Here we're defining a destination jid, service, timeout, user and group.

tyrion Client Options

Additionally if we ran the help option on tyrion we would see the following:

[silas@example.org]$ tyrion --help
Usage: tyrion [OPTION]...
Example: tyrion -c client.conf

Configuration options:
  -c, --config-file         the node configuration file
  -p, --profile             default service options

Service options:
  -j, --jid                 destination JID(s)
  -s, --service             service type
  -t, --timeout             max service run time
  -u, --user                run service as user
  -g, --group               run service as group

Misc options:
  --debug                   show debug information

tyrion Client Usage

So an example usage of tyrion might be:

[silas@example.org]$ echo "echo test" | tyrion -c client.conf -p test
test.node@example.org/tyrion (0): test

Here we've run tyrion with the client.conf configuration file and the profile test.

We sent echo test to the org.tyrion.service.bash service and got the exit code 0 and the result test.

tyrion-node Handler

From the other side tyrion-node sees a request from test.client@example.org to use the service org.tyrion.service.bash.

[org.tyrion.service.bash]
test.client@example.org = true

[org.tyrion.service.python]
test.client@example.org = true

[org.tyrion.service.ruby]
test.client@example.org = true

After validating that request against the above acl_path file it runs the /usr/share/tyrion/service/org.tyrion.service.bash executable and pipes echo test to stdin.

When the service request completes or reaches the defined timeout it sends the exit code, stdout and stderr back to the client.

txtyrion

The tyrion client is really just for testing and one-off hacks. It not suppose to be used in scripts or anything in production.

Because of this I created txtyrion, a Twisted-based Tyrion client library.

A simple client application might look like this:

from twisted.internet import defer, reactor

import txtyrion

class ExampleClient(txtyrion.Client):

    def success(self, response):
        print '=' * 80
        print '     ID:', response.id
        print '    JID:', response.jid
        print '   Code:', response.code
        print 'Service:', response.service
        print ' Output:', response.output
        print '  Error:', response.error

    def error(self, error):
        print '=' * 80
        print 'Error:', error

    def done(self, x):
        print '=' * 80
        reactor.stop()

    def run(self):
        jid = 'test.node@example.org/tyrion'

        command = """
        sleep 1
        echo bash says hello stdout
        echo bash says hello stderr >&2
        false
        """
        d1 = self.request(jid, 'org.tyrion.service.bash', command)
        d1.addCallbacks(self.success, self.error)

        d2 = self.request(
            jid,
            'org.tyrion.service.python',
            'print "python says hello"',
        )
        d2.addCallbacks(self.success, self.error)

        dl = defer.DeferredList([d1, d2])
        dl.addCallback(self.done)

if __name__ == '__main__':
    client = ExampleClient(
        'test.client@example.org',
        'test.client.pass',
        debug=True,
    )
    reactor.run()

txtyrion also comes with a node library for creating custom Tyrion nodes (see example).

TyrionHub

It's my goal to keep Tyrion simple; to let people utilize it for as much or as little as is appropriate for their situation.

That said I've created a separate project called TyrionHub which will provide a Twisted-based system for managing nodes and a library of Python-based services for systems automation.

The immediate goals for TyrionHub are:

User authentication using twisted.cred
Node authentication using twisted.cred and an integration script for ejabbard
Node enrollment

I'm currently building TyrionHub with the following technologies:

Server: Twisted
Database: MongoDB (although this may change)
Web GUI: Google Closure Tools (JavaScript/Ajax/JSON)
Services: Python with JSON serialization (simple as possible, avoid frameworks and all unnecessary libraries)

Status

Tyrion - working and packaged (not in any repo), although it could use some code reviews by people comfortable with C++
TyrionHub - just getting started, lots of work to do
txtyrion - working with various tweaks here and there

Participation

If anyone is interesting in participating in the development/testing process or would like to be contacted when TyrionHub is more feature complete, please contact me at silas@sewell.org.

License

Everything will be BSD.

Follow-up

I'll be posting a follow-up with a step-by-step explanation on how to setup a basic working version of Tyrion in the next week or two.

txmpp: A BSD Licensed C++ XMPP Library

Silas Sewell (Silas Sewell) — Wed, 16 Jun 2010 00:00:00 PDT

txmpp is a permissively licensed (BSD) C++ XMPP library based on libjingle.

The goals for txmpp and why it's a separate project from libjingle are as follows:

All code is under a single, unified namespace
All code is permissively licensed
Good build and install support for POSIX operating systems
Focus on core XMPP components (the XEP-0166 related code has been removed)

I've currently got the library building on Fedora 13, CentOS 5 and OS X. In the near future I'm going to focus on removing the Windows code and reducing the amount of code not directly tied to core XMPP functionality.

With all that said I want to keep the core as close to libjingle as possible. I think it's important that txmpp can track and integrate changes from libjingle in a relatively simple way.

If you're interested in messing around with the library I'd recommend checking out the sample code in the examples directory.

node.js HTTPS (SSL) Server Example

Silas Sewell (Silas Sewell) — Thu, 03 Jun 2010 00:00:00 PDT

A simple HTTPS server using node.js:

const crypto = require('crypto'),
      fs = require("fs"),
      http = require("http");

var privateKey = fs.readFileSync('privatekey.pem').toString();
var certificate = fs.readFileSync('certificate.pem').toString();

var credentials = crypto.createCredentials({key: privateKey, cert: certificate});

var handler = function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
};

var server = http.createServer();
server.setSecure(credentials);
server.addListener("request", handler);
server.listen(8000);

You can generate the privatekey.pem and certificate.pem files using the following commands:

openssl genrsa -out privatekey.pem 1024 
openssl req -new -key privatekey.pem -out certrequest.csr 
openssl x509 -req -in certrequest.csr -signkey privatekey.pem -out certificate.pem

Update: Thanks to Kord Campbell for spotting an issue and sending a fix!

Auto-reload node.js (OS X)

Silas Sewell (Silas Sewell) — Sun, 16 May 2010 00:00:00 PDT

restarter is a simple Python application which runs a specified command and restarts the command each time a file event occurs in a specified directory.

I created restarter because there isn't auto-reload functionality built into node.js (at the time of this post).

Example

[silas@pluto keyfu.js]$ restarter node keyfu.js
Express started at http://localhost:3000/ in development mode

Usage

Usage: restarter [options] command

Options:
  -h, --help     show this help message and exit
  --path=PATH    directory to watch for file event changes
  --ignore=PATH  specifies events to ignore

Jinja2 Markdown Extension

Silas Sewell (Silas Sewell) — Mon, 10 May 2010 00:00:00 PDT

Below is an example of a Markdown extension for Jinja2.

import jinja2
import jinja2.ext
import markdown2

class Markdown2Extension(jinja2.ext.Extension):
    tags = set(['markdown2'])

    def __init__(self, environment):
        super(Markdown2Extension, self).__init__(environment)
        environment.extend(
            markdowner=markdown2.Markdown()
        )   

    def parse(self, parser):
        lineno = parser.stream.next().lineno
        body = parser.parse_statements(
            ['name:endmarkdown2'],
            drop_needle=True
        )
        return jinja2.nodes.CallBlock(
            self.call_method('_markdown_support'),
            [],
            [],
            body
        ).set_lineno(lineno)

    def _markdown_support(self, caller):
        return self.environment.markdowner.convert(caller()).strip()

env = jinja2.Environment(extensions=[Markdown2Extension])

text = """ 
{ % markdown2 %}
Hello World
===========

 1. One
 2. 
 3. Three
{ % endmarkdown2 %}
"""

html = env.from_string(text).render(two='Two')

print html

Which would result in the following output:

<h1>Hello World</h1>

<ol>
<li>One</li>
<li>Two</li>
<li>Three</li>
</ol>

Google App Engine Middleware (gaem)

Silas Sewell (Silas Sewell) — Tue, 29 Dec 2009 00:00:00 PST

Google App Engine Middleware (gaem) is a collection of utilities for hosting external sites on Google App Engine.

I created gaem because I wanted to host my blog on an unreliable server and let Google App Engine serve the content while my server was down or in maintenance mode.

The basic design is similar to that of Django's middleware. A request travels through a list of components which can either manipulate the request or provide a response. The response is filtered back through the components and served to the requester.

A basic implementation looks something like

import gaem
import gaem.middleware
import gaem.middleware.cache
import gaem.middleware.dos
import gaem.middleware.fetch

URL = 'http://www.example.org'

MIDDLEWARE_CLASSES = (
    gaem.middleware.cache.Memcache(),
    gaem.middleware.cache.Datastore(),
    gaem.middleware.dos.DoS(),
    gaem.middleware.fetch.Fetch(url=URL),
)

application = gaem.Application(MIDDLEWARE_CLASSES)

if __name__ == '__main__':
    application.run()

When a request first hits this application it is handled by the Memcache component. This basic caching component checks the Google App Engine Memcache service to see if the page is in the cache. If so it simply returns the cached response, otherwise the request proceeds to the Datastore component which does pretty much the same thing except it uses Google Datastore for persistence. If both caching components fail the the request goes through a DoS protection component and is finally retrieved from the backend server by the Fetch component.

Using a setup like this gaem acts as a rudimentary reverse proxy and while most of the functionality I've implemented thus far is for this type setup, I plan on adding features not traditionally found in a reverse proxy.

Current Features

Caching
Backend fetch
Gracefully handle a dead backend
DoS protection
API for expiring content

Possible Future Features

Initial and periodic crawler (pre-cache and other hooks)
RESTful API for managing content (put, delete, etc...)
Web interface
XSS protection
Some ESI support

But what makes gaem really cool for developers is how easy it is to create custom components.

A basic component looks like

from gaem import middleware

class DoesNothing(middleware.Base):

    def process_request(self, request):
        return request

    def process_response(self, response, request):
        return response

A default gaem component should inherit from gaem.middleware.Base and implement process_request and/or process_response.

The process_request function lets you either manipulate the request (ex: strip the "cookie" header) and pass it along or use the request to create and return a response (ex: do a fetch to the backend server). The process_response function lets you filter the response (ex: translate body into pig latin) and do side affects (ex: store response in memcache) before the response is sent to the requester.

The Request/Response objects are borrowed from WebOb and can be manipulated using the interfaces defined in the WebOb documentation.

gaem is currently pretty rough and likely to change over the next couple of months, but if you're a Python hacker I'd love to hear some feedback (contact me or submit an issue).

P.S. You can see the application I created for this site in the examples directory.

Stomping with Python and ActiveMQ (Stomp Framework)

Silas Sewell (Silas Sewell) — Thu, 27 Aug 2009 00:00:00 PDT

python-stomping is a simple Stomp framework I wrote which makes creating stomp services in Python easy.

Install requirements

$ sudo yum install python-twisted python-stomper

Checkout stomping

$ svn export http://silassewell.googlecode.com/svn/trunk/projects/python-stomping python-stomping
$ cd python-stomping

Update the example.py file to use your host, port, username and password ActiveMQ/Stomp broker settings

from stomping import Stomping, route

class MyStomping(Stomping):

def init(self):
    self.send('/queue/test1', 'init test1')
    self.send('/queue/test2', 'init test2')

@route('/queue/test1')
def a_test(self, message):
    print 'a_test: %s' % message['body']

@route('/queue/test1')
@route('/queue/test2')
def b_test(self, message):
    print 'b_test: %s' % message['body']

if __name__ == '__main__':
    stomp = MyStomping(host='127.0.0.1', port=61613, username='guest', password='guest')
    stomp.run()

Run example.py (Ctrl+c to quit)

$ python example.py 
a_test: init test1
b_test: init test1
b_test: init test2

NOTE: python-stomping is just a light wrapper around Twisted and stomper and is based on an example provided in the stomper code.

Qpid on CentOS 5

Silas Sewell (Silas Sewell) — Tue, 16 Jun 2009 00:00:00 PDT

Build using Fedora

Install the RPM development tools
```
sudo yum install rpmdevtools mock
```

Add yourself to the mock group

sudo /usr/sbin/usermod --groups mock --append $USER

Setup build directories
```
rpmdev-setuptree
```

Install spec and source files (or get latest here)

rpm -i ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/RHEMRG/SRPMS/qpidc-0.5.752581-17.el5.src.rpm
rpm -i ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/RHEMRG/SRPMS/amqp-1.0.750054-1.el5.src.rpm
rpm -i ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/RHEMRG/SRPMS/python-qpid-0.5.752581-3.el5.src.rpm

Build qpidc

rpmbuild -bs --nodeps ~/rpmbuild/SRPM/qpidc.spec
mock -vr epel-5-x86_64 ~/rpmbuild/SRPMS/qpidc-*.src.rpm
scp /var/lib/mock/epel-5-x86_64/result/*.rpm CENTOS_HOST:SOME_PATH

Build amqp

rpmbuild -bs --nodeps ~/rpmbuild/SPECS/amqp.spec
mock -vr epel-5-x86_64 ~/rpmbuild/SRPMS/amqp-*.src.rpm
scp /var/lib/mock/epel-5-x86_64/result/*.rpm CENTOS_HOST:SOME_PATH

Build python-qpid

rpmbuild -bs --nodeps ~/rpmbuild/SPECS/python-qpid.spec
mock -vr epel-5-x86_64 ~/rpmbuild/SRPMS/python-qpid-*.src.rpm
scp /var/lib/mock/epel-5-x86_64/result/*.rpm CENTOS_HOST:SOME_PATH

Install on CentOS

Become root
```
sudo su -
```

Install Qpid server and Python client packages

cd SOME_PATH
yum localinstall --nogpgcheck qpidc-*.el5.x86_64.rpm qpidd-*.el5.x86_64.rpm amqp-*.el5.noarch.rpm python-qpid-*.el5.noarch.rpm

Disable auth for testing
```
echo 'auth=no' >> /etc/qpidd.conf
```
Start qpidd
```
service qpidd start
```
Test using these instructions.

NOTE: I realize you can build the SRPMs without installing them.

Python Line-by-line Profiler (line_profiler and kernprof)

Silas Sewell (Silas Sewell) — Thu, 28 May 2009 00:00:00 PDT

The following is a quick and dirty guide to getting started with line_profiler, a Python line-by-line profiler, on Fedora.

Build and install the python-line_profiler package

Create a file called test.py with the code below

import random, time

def sleep():
    seconds = random.randint(0, 5)
    print 'Sleeping %s seconds' % seconds
    time.sleep(seconds)

@profile
def test():
    sleep()
    sleep()
    sleep()

test()

Profile test.py

[silas@pluto ~]$ kernprof.py -l test.py
Sleeping 4 seconds
Sleeping 5 seconds
Sleeping 2 seconds
Wrote profile results to test.py.lprof

View the results

[silas@pluto ~]$ python -m line_profiler test.py.lprof
Timer unit: 1e-06 s

File: test.py
Function: test at line 8
Total time: 10.9994 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     8                                           @profile
     9                                           def test():
    10         1      3999416 3999416.0     36.4      sleep()
    11         1      4999982 4999982.0     45.5      sleep()
    12         1      1999990 1999990.0     18.2      sleep()

NOTE: I have a package review up for line_profiler and it should be available via yum eventually.

Pipe Apache (or any) Logs to Scribe

Silas Sewell (Silas Sewell) — Tue, 12 May 2009 00:00:00 PDT

I created a simple Python script called scribe_log to tail a log file and pipe it to Scribe.

I use supervisor to start and keep the pipe running.

Relevant supervisord.conf configuration:

[program:scribe.apache.access]
command=/usr/local/sbin/scribe_log --category apache.access --file /var/log/httpd/access_log

Options

usage: scribe_log [options]

options:
  -h, --help           show this help message and exit
  --file=FILE          file to tail into Scribe
  --category=CATEGORY  Scribe category
  --host=HOST          destination Scribe host server
  --port=PORT          destination Scribe port
  --prefix=PREFIX      add to the beginning of each log line
  --postfix=POSTFIX    add to the end of each log line

Scribe - Scalable Real Time Log Aggregation for CentOS 5 / RHEL 5

Silas Sewell (Silas Sewell) — Thu, 07 May 2009 00:00:00 PDT

"Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine. Scribe was developed at Facebook and released as open source."

I've packaged Thrift, fb303 and Scribe for CentOS 5 / RHEL 5.

SRPM

Scribe depends on fb303 which in turns depends on Thrift so you'll need to setup a local repository and build them in that order. To build the above packages for Fedora 9+ you'll need to tweak the Python sub-packages to include the egg files.

Both fb303 and Scribe need some tweaks upstream before they're suitable for a package review, but I'd like to get them in Fedora shortly (I've already submitted Thrift).

If you'd like to hack around with my latest Fedora specs you can grab them here.

Managing Large Networks with Puppet

Silas Sewell (Silas Sewell) — Mon, 04 May 2009 00:00:00 PDT

Puppet is an open source configuration management tool written in Ruby. It allows a systems administrator to define how a system should be configured using Puppet's declarative language. Each Puppet client pulls its catalog at a regular interval and figures out how to make the catalog definitions true for the local operating system.

The Puppet Introduction uses the following diagram to show how Puppet works.

Currently Puppet is most useful when you have lots of nodes with similar setups. Unfortunately Puppet's declarative language is a bit weak when it comes to inheritance and users familiar with true object oriented systems will soon become frustrated (at least I did). I'm assuming this issue will be addressed in the future, but for now I'm going to tell you how I setup a 100+ node system while adhering to DRY principles.

A quick note, I'm assuming you already know how to use Puppet.

First lets start with the Puppetmaster configuration layout:

puppet/
 - fileserver.conf
 - manifests/
   - classes/
     - initialize.pp
   - nodes/
     - net/
       - example/
         - web01.pp   # web01.example.net
   - site.pp
   - templates.pp
 - modules/           # application specific modules
   - httpd/
     - files/         # static assets (ex: default HTML file)
     - manifests/     # application configuration logic
     - templates/     # dynamic configuration files (ex: httpd.conf)

The layout was adapted from recommendations in Pulling Strings with Puppet by James Turnbull.

When I was initially designing my Puppet setup the common practice was to define and initialize the various Puppet types and classes throughout the inheritance tree. So if you had a generic resolv.conf configuration you would include it at the top of the inheritance tree. This worked great until I needed to change one attribute of a class further down. I initially tackled this problem by hacking in if/case statements, but eventually it became unmanageable.

After a couple of rewrites I came up with the idea of wrapping the internals of each class in a conditional statement and initializing all classes at the end of each node.

The key components of my setup were:

import all modules in the site.pp manifest
wrap the code of each class in a conditional statement
define and extend attributes throughout the inheritance tree (including the class conditional)
include all classes at the end of each node

This lets you both arbitrarily redefine or extend class attributes (including the on/off state) throughout the inheritance tree.

I've created a simple example to show how this setup works. The main files are listed below (you'll want to go through each in order):

site.pp - import modules
templates.pp - define and extend class attributes
web01.pp - make use of defaults
ns01.pp - disable a class
initialize.pp - initialize all classes
client.pp - class implementation

Also note that I designed this setup about 8 months ago and I haven't kept as up-to-date on recent developments as I should. If there is a better way to do this please let me know.

funcshell - A Shell Interface to Func

Silas Sewell (Silas Sewell) — Wed, 22 Apr 2009 00:00:00 PDT

Features

advanced client selection (you can use + and - to add/remove hosts/groups)
tab completion
persistent history
a command sub-shell
quick help (just hit the "?" key)

funcshell took a similar approach to features as Func; it allows developers to easily extend and customize its functionality using a plugin system. It also assumes the most useful commands will be the ones you write yourself. That being said, funcshell currently comes with both the command and service module.

Here are some things you can do with funcshell.

Get a list of all clients (not very useful if you have hundreds):

funcshell> set clients *
funcshell> get clients
webapp001.example.net
webapp002.example.net
webapp003.example.net
webstatic001.example.net
webstatic002.example.net

Assuming webapp00[1-3].example.net are in the app and bpstatic00[1-2].example.net1 are in the static groups; remove some clients and restart a service:

funcshell> set clients - @static;webapp001*
funcshell> get clients
webapp002.example.net
webapp003.example.net
funcshell> service httpd restart
==> webapp003.example.net <==
True
==> webapp002.example.net <==
True

Run a command and get the results:

funcshell> set clients - webapp003*
funcshell> set clients + webstatic002*
funcshell> command run grep MemTotal /proc/meminfo | awk '{ print $2/1024 }'
==> webstatic002.example.net :: 0 <==
4099.44
==> webapp002.example.net :: 0 <==
8198.88

Get help:

funcshell> command <TAB><TAB>
exists   run      shell
funcshell> command ?
  exists Check if a command exists
  run    Run a command
  shell  Run a command shell

Run a couple of commands in a row:

funcshell> set c<TAB> @static
funcshell> command shell
> du -sh /tmp
==> webstatic002.example.net :: 0 <==
7.2M    /tmp
==> webstatic001.example.net :: 0 <==
3.4M    /tmp
> cp -r /tmp /tmp
==> webstatic002.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
==> webstatic001.example.net :: 1 <==
cp: cannot copy a directory, '/tmp', into itself, '/tmp/tmp'
> asdf
==> webstatic002.example.net :: 127 <==
/bin/sh: asdf: command not found
==> webstatic001.example.net :: 127 <==
/bin/sh: asdf: command not found
> <CTRL-D>
funcshell> exit

funcshell is a shell interface to Func which provides some useful features for managing a large number of machines.

Getting Started with AMQP, Qpid, Python & Fedora

Silas Sewell (Silas Sewell) — Sun, 29 Mar 2009 00:00:00 PDT

The Advanced Message Queuing Protocol (AMQP) is the first open standard for Enterprise Messaging and has been adopted by Cisco, Microsoft, Novel, Red Hat and others.

One implementation of AMQP currently packaged by Red Hat and Fedora is Qpid. Qpid is a collection of brokers (C++, Java) and clients (C++, Java, Python, Ruby) which aims to be 100% AMQP Compliant.

Below is a step-by-step tutorial on getting started with Qpid and the Python client in Fedora.

Install the Qpid broker and Python client
```
yum install qpidd python-qpid
```
Start the broker
```
service qpidd start
```

Create a message producer (producer.py)

from qpid.connection import Connection
from qpid.datatypes import Message, uuid4
from qpid.util import connect

# Create connection and session
socket = connect('localhost', 5672)
connection = Connection(sock=socket, username='guest', password='guest')
connection.start()
session = connection.session(str(uuid4()))

# Setup queue
session.queue_declare(queue='message_queue')
session.exchange_bind(exchange='amq.direct', queue='message_queue', binding_key='routing_key')

# Setup routing properties
properties = session.delivery_properties(routing_key='routing_key')

# Send messages
session.message_transfer(destination='amq.direct', message=Message(properties, 'Message one'))
session.message_transfer(destination='amq.direct', message=Message(properties, 'Message two'))
session.message_transfer(destination='amq.direct', message=Message(properties, 'Done'))

# Close session
session.close(timeout=10)

Create a message consumer (consumer.py)

from qpid.connection import Connection
from qpid.datatypes import RangedSet, uuid4
from qpid.util import connect

# Create connection and session
socket = connect('localhost', 5672)
connection = Connection(sock=socket, username='guest', password='guest')
connection.start()
session = connection.session(str(uuid4()))

# Define local queue
local_queue_name = 'my_local_queue'

# Create local queue
queue = session.incoming(local_queue_name)

# Route messages from message_queue to my_local_queue
session.message_subscribe(queue='message_queue', destination=local_queue_name)
queue.start()

content = ''

while content != 'Done':
    # Get message from the local queue
    message = queue.get(timeout=10)
    # Get body of the message
    content = message.body
    # Accept message (removes it from the queue)
    session.message_accept(RangedSet(message.id))
    # Print message content
    print content

# Close session
session.close(timeout=10)

First run the producer and then the consumer
```
Message one
Message two
Done
```

Source: Qpid direct example

String Slicing in Bash (like Python)

Silas Sewell (Silas Sewell) — Thu, 12 Mar 2009 00:00:00 PDT

A simple function to slice strings in Bash similar to Python's string slicing functionality.

Examples

[silas@pluto ~]$ string_slice "12345" 0 1
1
[silas@pluto ~]$ string_slice "12345" 0 3
123
[silas@pluto ~]$ string_slice "12345" 2 3
3
[silas@pluto ~]$ string_slice "12345" 2 -2
3
[silas@pluto ~]$ string_slice "12345" -3
345

Implementation

function string_slice {
    STRING="$1"
    declare -i LENGTH="${#STRING}"
    declare -i START="$2"
    declare -i END="$3"
    if [ $START -lt 0 ]; then
        START=$[ $LENGTH + $START ]
    fi
    if [ $END -le 0 ]; then
        END=$[ $LENGTH + $END ]
    fi
    START=$[ $START + 1 ]
    (printf "%s\n" "$STRING" | cut -c $START-$END) 2> /dev/null
}

Profile Management with Git and GitHub

Silas Sewell (Silas Sewell) — Sun, 08 Mar 2009 00:00:00 PST

The following describes a simple way to manage you profile configuration files using GitHub.

Features

Centralized configuration management
Files live in their native locations (no symbolic linking)
Home directory is not a Git repository
All the power of git with a simple alias

Setup Repository

Log into GitHub and create a repository named config
Add your public keys to GitHub (if you haven't done so already)
Open a terminal and switch to your home directory
```
cd ~
```
Create a configuration directory
```
mkdir .config.git
```

Add the following alias to your current session and your .bash_profile

alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'
echo "alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'" >> .bash_profile

Add your .bash_profile to the configuration repository
```
config add .bash_profile
```
Commit the changes
```
config commit -m 'Initial commit'
```

Change the origin to GitHub

config remote add origin git@github.com:GITHUB_USERNAME/config.git

Push the changes
```
config push origin master
```

If you get an error when running config pull to the effect of You asked me to pull without... run the follow:

    echo -e '[branch "master"]\n  remote = origin\n  merge = refs/heads/master' >> ~/.config.git/config

Setup Configuration Management on a Different System

Add your public keys to GitHub (if you haven't done so already)
Switch to your home directory
```
cd ~
```
Backup your local configuration files, example:
```
mv .bash_profile .bash_profile.bk
```

Clone your configuration repository

git clone git@github.com:GITHUB_USERNAME/config.git config.git

Move the git metadata to ~/.config.git
```
mv config.git/.git .config.git
```
Enable dotglob
```
shopt -s dotglob
```
Move your configuration files to your home directory
```
mv -i config.git/* .
```
Delete the config.git directory
```
rmdir config.git
```
Logout and log back in

Basic Usage

config pull - get latest configuration changes
config add FILENAME - add a configuration file
config commit -a - save all configuration changes
config push - push configuration changes to GitHub
and any other config GIT_OPTION

You can see my configuration repository at http://github.com/silas/config.

Source: Manage your $HOME with git by Robert Escriva

Upgrade the Android Developer Phone (ADP) to RC33 Manually

Silas Sewell (Silas Sewell) — Tue, 17 Feb 2009 00:00:00 PST

Please note that you will lose all data on your phone when performing these steps.

Mount the ADP MicroSD card
Download hboot-0.95.0000.zip to the mount and rename it to update.zip
Unmount the card and shutdown the phone (unplug the USB cable if connected)
Hold Home+Power to boot
When prompted press Alt+S and wait for the update to finish
Press Home+Back to reboot and wait for second update to finish and boot into the Android recovery system
Press Home+Back to reboot again
Mount the MicroSD card
Delete update.zip
Download signed-holiday_devphone-ota-130444-debug.55489994.zip to the mount and rename it to update.zip
Unmount the card and shutdown the phone
Hold Home+Power to boot
Press Alt+S to update and wait for update to finish
Press Home+Back to reboot and wait for the second update to finish and boot into the Android system recovery utility
Press Alt+W to wipe all data
Press Home+Back to reboot and wait for the reboot to finish (this could take a couple of minutes)
Re-setup the ADP (hint: press menu to setup custom APN settings)

Introducing the FuncShell

Silas Sewell (Silas Sewell) — Fri, 30 Jan 2009 00:00:00 PST

Update: I am currently refactoring FuncShell; please use the following code: shell.py.

I created funcshell for running Func modules in a more intuitive manner.

The current implementation only supports the command module, but more will be released shortly.

Code: http://github.com/silas/funcshell

Example Usage

[root@pluto ~]# python shell.py
fs> use web*.example.org
fs> get hosts
web01.example.org
web02.example.org
fs> !du -sh /tmp
================================================================================
== web02.example.org                                                          ==
================================================================================

236K    /tmp

================================================================================
== web01.example.org                                                          ==
================================================================================

217M    /tmp

fs> !cat /proc/meminfo | grep MemTotal | awk '{ print $2 }'
================================================================================
== web02.example.org                                                          ==
================================================================================

1027116

================================================================================
== web01.example.org                                                          ==
================================================================================

1027116

fs> exit
[root@pluto ~]#

Android Dev Phone 1 - AT&T

Silas Sewell (Silas Sewell) — Wed, 17 Dec 2008 00:00:00 PST

I received my Android Dev Phone 1 today and wanted to report that with the following APN settings it works great (setup instructions come with the phone).

Name: AT&T
APN: wap.cingular 
Proxy: <Not set>
Port: <Not set>
Username: wap.cingulargprs.com 
Password: CINGULAR1 
Server: <Not set>
MMSC: mmsc.cingular.com 
MMS proxy: <Not set>
MMS port: 80 
MCC: 310
MNC: 410
APN type: <Not set>

Source Jackson Miller

Adroid Dev Phone 1

Silas Sewell (Silas Sewell) — Sat, 06 Dec 2008 00:00:00 PST

I just ordered the Adroid Dev Phone 1, an unlocked Android device available through the Developer Console.

Price: $399.00 USD

Overview

Run and debug your Android applications directly on a device. Modify and rebuild the Android operating system, and flash it onto a phone. The Android Dev Phone 1 is carrier independent, and available for purchase by any developer registered with Android Market.

Development Platform Features

SIM unlocked - Put any SIM in the device
Hardware unlocked - Rebuild the Android operating system, and load it onto the device
Test out your apps on real hardware before publishing them to the world

Hardware Features

Touch screen
Trackball
3.2 Megapixel camera with auto focus
Wi-Fi
Bluetooth v2.0
- Handsfree profile v1.5
- Headset profile v1.0
3G WCDMA (1700/2100 MHz)
Quad-band GSM (850/900/1800/1900 MHz)
QWERTY slider keyboard
Includes 1GB MicroSD card (Can be replaced with up to 16GB card)
Included in the box
- HTC Android Dev Phone 1
- USB Cable
- AC Adapter (with US plug)
- Stereo Hands-Free Headset
- Battery
- Getting Starting Guide
- 1G Micro SD Card (inserted into Device)

Software Features

Real web browsing
Customizable home screen
One-touch Google Search
Android Market
Full-featured Google applications:
- Google Maps
- Gmail
- YouTube
- Google Calendar
- Google Talk
SMS and MMS
Music Player

IPython + Python: Single Bash Command

Silas Sewell (Silas Sewell) — Mon, 01 Dec 2008 00:00:00 PST

The following is a simple Bash function you can paste into your bashrc file to start Python/IPython depending on the context.

~/.bashrc

function python {
    IPYTHON="/usr/bin/ipython"
    PYTHON="/usr/bin/python"

    if [[ -n $1 ]]; then
        $PYTHON $@
    elif [[ -e $IPYTHON ]]; then
        $IPYTHON
    else
        $PYTHON
    fi
}

Push: Func Module to Run Arbitrary Python Code

Silas Sewell (Silas Sewell) — Tue, 18 Nov 2008 00:00:00 PST

Func is a nifty, although not yet polished, Python-based service for running tasks on many hosts at once. It lets you do things like restart Apache instances, run yum updates or provision virtual machines.

You can accomplish theses tasks via the command line interface

func web-*.example.net call service restart httpd

or through the Python API

import func.overlord.client as fc

client = fc.Client('web-*.example.net')

print client.service.restart('httpd')

One of the annoying issues I'm dealing with right now is the distribution of Func modules, which for various reasons isn't as simple as packaging the modules and pushing them to all the hosts (it is, just not in my network).

My solution is Push, a Func module which lets you instantly run Python code on any or all hosts in your network.

Example usage:

import func.overlord.client as fc

client = fc.Client('web-*.example.net')

source = """
def main():
    return 'Hello World'
"""

print client.push.code(source)

I realize this isn't the most elegant solution, but its very useful and extremely simple.

I also realize that I could use the copyfile module to push modules via Func, but I haven't found a way to restart funcd via Func (required to initialize modules), so until that happens I need a solution which doesn't require restarting Func.

Get Push Module

Adobe Flash 10 64-bit SPEC File for Fedora 9

Silas Sewell (Silas Sewell) — Tue, 18 Nov 2008 00:00:00 PST

I hacked up a quick SPEC file for the alpha release of Adobe Flash 10 64-bit for Linux.

Name:           adobe-flash
Version:        10.0.d20.7
Release:        1%{?dist}
Summary:        Adobe Flash player.

Group:          Application/Internet
License:        Proprietary
URL:            http://labs.adobe.com/technologies/flashplayer10/
Source0:        http://download.macromedia.com/pub/labs/flashplayer10/libflashplayer-10.0.d20.7.linux-x86_64.so.tar.gz
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

BuildArch:      x86_64
Requires:       firefox >= 3
Requires:       mozilla-filesystem

%description
The Adobe Flash player.

%prep
%setup -T -c -a 0

%install
install -D -m 644 libflashplayer.so $RPM_BUILD_ROOT%{_usr}/lib64/mozilla/plugins/libflashplayer.so

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
%{_usr}/lib64/mozilla/plugins/libflashplayer.so

%changelog
* Tue Nov 18 2008 Silas Sewell <silas@sewell.ch> - 10.0.d20.7
- Created package.

C GMP Hello World on Leopard (OS X 10.5)

Silas Sewell (Silas Sewell) — Sun, 19 Oct 2008 00:00:00 PDT

A simple GMP Hello World example in C.

Makefile

all:
    gcc -o gmp_hello_world gmp_hello_world.c -lgmp -m64

clean:
    rm gmp_hello_world

gmphelloworld.c

/*
 * GMP Hello World on OS X 10.5
*/
#include <gmp.h>
#include <stdio.h>

int main() {
  mpz_t add_total1, add_total2, sub_total1, sub_total2, mul_total1, mul_total2;
  mpz_t num1, num2;

  // Initialize variables
  mpz_init_set_str(num1, "345192567923875922375736284875732", 10);
  mpz_init_set_str(num2, "937929298382994742939293857584837", 10);
  mpz_init(add_total1);
  mpz_init(add_total2);
  mpz_init(sub_total1);
  mpz_init(sub_total2);
  mpz_init(mul_total1);
  mpz_init(mul_total2);

  // Do arithmetic
  mpz_add(add_total1, num1, num2);
  mpz_add_ui(add_total2, num1, 10);
  mpz_sub(sub_total1, num1, num2);
  mpz_sub_ui(sub_total2, num1, 10);
  mpz_mul(mul_total1, num1, num2);
  mpz_mul_ui(mul_total2, num1, 10);

  // Display results
  gmp_printf("Add Total 1: %Zd\n", add_total1);
  gmp_printf("Add Total 2: %Zd\n", add_total2);
  gmp_printf("Subtract Total 1: %Zd\n", sub_total1);
  gmp_printf("Subtract Total 2: %Zd\n", sub_total2);
  gmp_printf("Multiply Total 1: %Zd\n", mul_total1);
  gmp_printf("Multiply Total 2: %Zd\n", mul_total2);

  // Free space
  mpz_clear(num1);
  mpz_clear(num2);
  mpz_clear(add_total1);
  mpz_clear(add_total2);
  mpz_clear(sub_total1);
  mpz_clear(sub_total2);
  mpz_clear(mul_total1);
  mpz_clear(mul_total2);

  return 0;
}

Persistent Queuing in Python

Silas Sewell (Silas Sewell) — Fri, 26 Sep 2008 00:00:00 PDT

Peafowl is a Python port of Ruby's Starling.

Common Usage Database (cudb)

Silas Sewell (Silas Sewell) — Wed, 24 Sep 2008 00:00:00 PDT

A couple of months ago I was searching for an excuse to write an application on Google App Engine. After mucking around with the Datastore and authentication API I decided to rewrite Most Common Usage on Google App Engine.

The end result was the Common Usage Database and like Most Common Usage the website is focused on the usage of CLI commands in Linux and Unix-like operating systems.

The website uses Google for authentication so you don't need to signup if you already have a Google account. I've also setup a Google Group and the #cudb channel on Freenode.

I look forward to creating a repository of useful commands and providing an easy and open way for developers to use that repository.

Link: http://www.cudb.org/

Python SDK for PhoneFactor

Silas Sewell (Silas Sewell) — Sat, 07 Jun 2008 00:00:00 PDT

A first (and very alpha) attempt at creating a Python SDK for PhoneFactor.

Link: http://github.com/silas/graveyard/tree/master/python-phonefactor/

Universal Feed Parser to JSON

Silas Sewell (Silas Sewell) — Mon, 05 May 2008 00:00:00 PDT

My attempt at wrangling a Universal Feed Parser object into JSON.

import feedparser

def json_make_normal(obj):
    if type(obj) in [str, unicode, int, float, bool, dict, set, list, tuple]:
        return obj
    try: return dict(obj)
    except: pass
    try: return list(obj)
    except: pass
    return None

def json_handle(obj):
    obj = json_make_normal(obj)
    if type(obj) in [str, unicode]:
        obj = obj.replace('\\', '\\\\')
        obj = obj.replace('"', '\\"')
        obj = obj.replace('\b', '\\\b')
        obj = obj.replace('\f', '\\\f')
        obj = obj.replace('\n', '\\\n')
        obj = obj.replace('\r', '\\\r')
        obj = obj.replace('\t', '\\\t')
        return '"%s"' % obj
    elif type(obj) in [int, float]:
        return obj
    elif type(obj) is bool:
        if obj: return 'true'
        else: return 'false'
    elif type(obj) is type(None):
        return 'null'
    elif type(obj) is dict:
        temp = ''
        for key in obj.keys():
            temp += '%s:%s, ' % (json_handle(key), json_handle(obj[key]))
        return '{\%s}' % temp[:-2]
    elif type(obj) in [set, list, tuple]:
        temp = ''
        for value in obj:
            temp += '%s, ' % json_handle(value)
        return '[%s]' % temp[:-2]
    return 'null'

data = feedparser.parse('http://digg.com/rss/index.xml')

print json_handle(data)

Link: feedparserto_json.py

Update: According to John Paulett jsonpickle now supports Universal Feed Parser objects. Thanks for letting me know!

OS X’s GNU Screen - Open in the Current Working Directory

Silas Sewell (Silas Sewell) — Sat, 05 Apr 2008 00:00:00 PDT

How to get GNU Screen working properly in Mac OS X.

~/.bashrc

alias screen='export SCREENPWD=$(pwd); /usr/bin/screen'
export SHELL='/bin/bash -rcfile ~/.bash_profile';

case "$TERM" in 
    'screen')
         cd $SCREENPWD
         ;; 
esac

~/.screenrc

startup_message off
shell -$SHELL

Four Ways to Make Safari Better

Silas Sewell (Silas Sewell) — Fri, 04 Apr 2008 00:00:00 PDT

Force Safari to open a new window in a tab
Change Safari's default search engine (use with Keyfu to get Firefox-like keywords)
Set a search shortcut in Safari
Install AdBlock for Safari

Post a Get

Silas Sewell (Silas Sewell) — Sun, 30 Mar 2008 00:00:00 PDT

I've created a PHP script called Post a Get which can be copied to any PHP capable server and used to created a POST request using GET syntax.

Let's say you use GoDaddy as your domain registrar and you would like to see if a domain is available using Firefox keywords, but they don't offer a GET search.

For a traditional keyword, you could construct a GET request like:

https://www.godaddy.com/gdshop/registrar/search.asp&checkAvail=1&fblur=1&tld=.com&domainToCheck=%s

But as of right now Firefox doesn't offer a way to create POST keywords. What Post a Fix does is allow you to construct a POST request using GET syntax and pass it to the Post a Get code.

So if you wanted to submit the above GET as a POST, you would construct the URL and append the special __action parameter with target URL. So the final URL would look something like:

OS X GNU Screen Proper Bash Prompt

Silas Sewell (Silas Sewell) — Sat, 23 Feb 2008 00:00:00 PST

How to get GNU Screen working properly in Mac OS X.

~/.bashrc

alias screen='export SCREENPWD=$(pwd); /usr/bin/screen'
export SHELL='/bin/bash -rcfile ~/.bash_profile';

case "$TERM" in
    'screen')
         cd $SCREENPWD
         ;;
esac

~/.screenrc

startup_message off
shell -$SHELL

Gracefully restart Apache in Debian

Silas Sewell (Silas Sewell) — Sun, 27 Jan 2008 00:00:00 PST

alias graceful="sudo /usr/sbin/apache2 -t && sudo /etc/init.d/apache2 stop && sleep 2 && sudo /etc/init.d/apache2 start"

Simple Shell Script to Manage SSH Tunnels

Silas Sewell (Silas Sewell) — Sat, 15 Dec 2007 00:00:00 PST

Manage SSH Tunnel is a simple shell script to start and stop ssh tunnels from the terminal.

Setup

Note: the setup assumes bash as the default shell

Download tunnel.sh and save it to ~/bin/

Add to ~/.bash_profile or ~/.bashrc

alias tunnel_home='/bin/sh ~/bin/tunnel.sh example.net sam 22 8888'

Replace example.net with the hostname of the server, sam with the username on the server, and 8888 with the local tunnel port
Type source ~/.bash_profile or source ~/.bashrc

Usage

To open the tunnel
```
tunnel_home start
```
To close the tunnel
```
tunnel_home close
```

Better Terminal Font in OS X (GNOME)

Silas Sewell (Silas Sewell) — Sun, 18 Nov 2007 00:00:00 PST

With the release of Leopard (OS X 10.5) came a sufficiently usable Terminal (aka tabs). The only real complaint I had was the font.

I ended up tracking down the default font used in the GNOME terminal and installed it using the steps below:

Download Bitstream Vera Mono 1.10
Extract the archive
Double click on VeraMono.ttf
Click Install Font
From Terminal click Terminal->Preferences
To the right of Font click Change...
Select Bitstream Vera Sans for the Family
Select 12 for the Size
Close the Fonts window
Enable Antialias text under Text
Restart the Terminal

iGoogle - Hide Google Search

Silas Sewell (Silas Sewell) — Thu, 15 Nov 2007 00:00:00 PST

NOTE: This gadget will no longer work, Google has deprecated the inline property which this gadget required.

Hide Google Search (source) is a gadget which hides the search area of iGoogle.

I've tested the gadget with the latest versions of Firefox, IE and Safari.

The above image is how the iGoogle page will load by default. You can toggle the hidden area by clicking the line right below the navigation bar.

opml2html Updates

Silas Sewell (Silas Sewell) — Sun, 05 Aug 2007 00:00:00 PDT

I've made some tweaks to opml2html to make it a little cleaner and more useful.

The main two additions are a class so opml2html can be called programmatically and a command line argument so the input file isn't static.

Open Source Games

Silas Sewell (Silas Sewell) — Sat, 04 Aug 2007 00:00:00 PDT

X-Moto - 2D motocross platform game

GWT Webmail Experiment

Silas Sewell (Silas Sewell) — Fri, 03 Aug 2007 00:00:00 PDT

A Java/GWT webmail experiment.

Link: ssmail

Setting up Ricochet in Linux

Silas Sewell (Silas Sewell) — Fri, 20 Jul 2007 00:00:00 PDT

NOTE: The follow instructions are meant for the Linux CLI, and should be run with sudo or as root.

Install wvdial

Debian/Ubuntu: apt-get install wvdial
Fedora/RHEL: yum install wvdial

Edit /etc/wvdial.conf and replace its content with:

[Dialer Defaults]
Modem = /dev/ttyACM0
Baud = 460800
Init1 = ATZ
Init2 = ATQ0

[Dialer ricochet]
Username = *
Password = *
Phone = 3333
Inherits = Dialer Defaults

Replace /dev/ttyACM0 with the device your modem connected to:
```
$ dmesg | grep cdc_acm
cdc_acm 2-1:1.0: ttyACM1: USB ACM device
```
Take the device name and prepend dev, so you get:
```
Modem = /dev/ttyACM1
```
Start wvdial
```
wvdial ricochet &
```

Convert Google Reader subscriptions (OPML) to HTML

Silas Sewell (Silas Sewell) — Mon, 16 Jul 2007 00:00:00 PDT

I've created a simple Python script to convert Google Reader’s OPML to HTML.

Link: http://github.com/silas/graveyard/tree/master/python-opml2html/