Printer Access RC1

Posted by Samuel Williams Mon, 21 Jul 2008 09:18:00 GMT

Hi, I’ve recently made some software for administrators to help manage many printers on the network. This software currently only has a Mac OS X client, but I’ll be looking to make a Window and Linux clients too. For more information, please see the main Printer Access website.

Easy backups with lsync. 1

Posted by Samuel Williams Sat, 06 Oct 2007 23:57:00 GMT

Backups are not always the easiest thing to setup and maintain, especially when you have complex requirements. rsync, a tool probably known by most unix geeks, is a convenient way to setup and administrate backups. This tool I have created, called ‘sync’ (but installed as lsync so as not to cause problems), is a simple set of python and ruby scripts to provide an additional layer to rsync, to address some additional requirements/features:

  • Mounting and unmounting local and remote filesystems, and also, mounting local filesystems on a remote machine.
  • Running the backup either on the source system, or destination system, without changing the configuration.
  • Run arbitrary script on source or destination before/after backups.
  • Sending backups to multiple destinations.
  • Using DNS CNAMEs to specify source and destination servers, and have backup clusters reconfigure automatically.

rsync provides a good foundation - it has good error reporting, performs well, widely deployed and tested, integrates with ssh, so this is used as the basis for lsync. On top of this, is a custom configuration parser written in python, a set of methods for using rsync, and Actions, which are sets of scripts which can be executed to perform various tasks.

The configuration file establishes a simple hierarchy. For example, take a look at Examples/site.conf:

[Server]
# We will resolve printserver1a and printserver1b to find which one is equivalent to printserver.
    Hostname: "printserver"
    User: "root"

[PrintA:Server]
    Hostname: "printserver1a"

[PrintB:Server]
    Hostname: "printserver1b"

[SyncConfig:Directory]
    Source: "/etc/sync/"

[PykotaConfig:Directory]
    Source: "/etc/pykota/"

[ApacheConfig:Directory]
    Source: "/etc/apache2/"

[ApacheWWW:Directory]
    Source: "/usr/apache/"

This configuration theoretically synchronises two servers, printserver1a and printeserver1b. One server will be designated a master, while all other servers are designated as destinations. The CNAME “printserver” is used to discover which server is the master (or you can specify it explicitly by using the exact hostname in the [Server] section.

There are also a list of directories. These are directories that will be synchronised on the servers - other directories are ignored.

When this configuration is executed on printserver1a, and printserver resolves to printserver1a, we will use the default method ‘push’, and establish an ssh connection with all other servers, in this case, printserver1b. We will then push out the data for the four directories listed.

The idea behind this is that you can have a cluster of machines, and use this to synchronise them. If the master server falls over, you can designate a new master using DNS CNAME record, and everything will keep on going.

To install this script:

# svn co http://svn.oriontransfer.org/Scripting/Sync
# cd Sync
# setup

This installs files to /usr/libexec/sync/ and a symlink lsync for the main executable. Because I am strange, I wrote part of this program in python, and the other part in ruby, so you’ll need to have both installed.

This script has more features, which I will cover in a future article. Please be aware that this set of scripts is currently in beta - it has been deployed and used successfully in a number of situations, but be aware it is far from complete.

Check the Examples directory for some simple configurations.

Managing Subversion 2

Posted by Samuel Williams Tue, 21 Aug 2007 11:24:00 GMT

I wanted to have a nice front-end to my subversion repositories, not only for my own use, but also to allow others to easily browse the code I make public. I made a simple ruby CGI script which you can see in action at svn.oriontransfer.org.

The script actually consists of a number of pieces. Firstly, there is the installation of mod_ruby. This can be accomplished by using a2enmod ruby. This needs some minor tweaking to work well though. My /etc/apache2/mods-enabled/ruby.conf looks like this:

<IfModule modruby.c>
  RubyRequire apache/ruby-run
  RubyRequire apache/eruby-run

  # Execute .rbx files as Ruby scripts
  <Files *.rbx>
    SetHandler ruby-object
    RubyHandler Apache::RubyRun.instance
  </Files>

  <Files *.rhtml>
    SetHandler ruby-object
    RubyHandler Apache::ERubyRun.instance
  </Files>

  AddType text/html .rhtml
</IfModule>

Then, I have my subversion repositories in /srv/svn/repositories/[Category]/[Repository]. This structure is important, because the CGI script builds the page based on this information.

My Apache2 configuration for SVN looks like this:

<VirtualHost *>
        ServerName svn.oriontransfer.org

        CustomLog /var/log/apache2/svn.log "%t %u %{SVN-ACTION}e" env=SVN-ACTION

        DocumentRoot /srv/svn/www/
        <Directory /srv/svn/www/>
                Options Indexes FollowSymLinks MultiViews ExecCGI
                AllowOverride None
                Order allow,deny
                Allow from all
        </Directory>

        <Location /Applications/>
                DAV svn
                SVNParentPath /srv/svn/repositories/Applications/
                SVNListParentPath on
                SVNIndexXSLT "/svn-index.xsl"

                AuthName "Orion Transfer Subversion"
                AuthType Digest
                AuthUserFile /etc/apache2/users/svn.htdigest

                Satisfy Any
                Require valid-user
                AuthzSVNAccessFile /srv/svn/repositories/Applications/svn.auth
        </Location>
</VirtualHost>

This configuration is carefully crafted to allow non-password access to public repositories, but require digest authentication when accessing a non-public repository. You can duplicate the Location section as many times as needed for different categories of projects.

The first important file to consider is svn-index.xsl (/srv/svn/www/svn-index.xsl). This file contains the XSL for rendering the output from SVN directory lists. You can grab a template of this from your default apache2 site (should be called svnindex.xsl and svnindex.css). I’ve modified these for my own design and style. The next important file is the actual CGI itself, /srv/svn/www/index.rhtml.

% require 'rubygems'
% gem 'BlueCloth'
% require 'bluecloth';
% Repositories = "/srv/svn/repositories"

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
        "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" version="-//W3C//DTD XHTML 1.1//EN" xml:lang="en">
<head>
        <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8" />

        <title>Orion Transfer Subversion Server</title>

        <link rel="stylesheet" href="/stylesheets/oriontransfer.css" type="text/css" media="screen" charset="utf-8">

</head>

<body>
        <div id="header"></div>
        <div id="tableOfContents"></div>

        <div id="page">
% Top = Dir[Repositories + "/*"].delete_if { |p| !File.directory?(p.untaint) }.sort

        <h1>Orion Transfer Subversion Repositories</h1>

        <p>Listed below are public SVN repositories. Please submit any changes you make
                if appropriate. Unless otherwise indicated, source code is released under the
                <a href="http://www.gnu.org/copyleft/gpl.html">GNU GPLv2</a>.</p>

% Top.each do |p|
%       name = File.basename(p)
%       next if File.exists?(File.join(p, ".hidden"))
        <h2><a href="/<%= name %>/"><%= name %></a></h2>
        <ul>
%       Dir["#{p}/*".untaint].each do |r|
%               next unless File.directory?(r.untaint)
%               next if File.exists?(File.join(r, ".hidden"))

%               repo = File.basename(r)
                <li><h3><a href="/<%= name %>/<%= repo %>/"><%= repo %></h3></a>
%               readme_path = File.join(r, "README.txt").untaint
                <%= (BlueCloth.new(File.read(readme_path)).to_html if File.exists? readme_path) || "" %>
                </li>
%       end
        </ul>

% end
        </div>

        <div id="footer">
                [ &copy; Orion Transfer Ltd ]
        </div>

</body>
</html>

This file is incredibly simple and basically just summarises the repositories based on the path information. It skips any repository or category with the file .hidden in it, so you can hide some private repositories that are mostly for your own use.

Keyword Extraction and Ruby Benchmarks

Posted by Samuel Williams Mon, 16 Jul 2007 09:40:00 GMT

I was interested to test the performance of pure ruby implementation vs using a set of piped commands for a simple task: extract all simple keywords from a file. Initially, I thought that using a set of piped commands would be faster than ruby, due to the nature of ruby being interpreted, and the shell commands written in compiled C. However, the results are very interesting.

Basically, I have created a simple script to test the performance of the two different methods, and used the benchmark module for displaying the results:

#!/usr/bin/env ruby

require 'set'
require 'benchmark'

FILE = "/etc/passwd"

def shell_keywords
    pipe = IO.popen("strings #{FILE} | tr -c '[a-zA-Z]' '
' | sort -u")
    buf = pipe.read
    pipe.close
    return buf
end

def ruby_keywords
    kwds = Set.new
    File.open(FILE).read.gsub(/[^a-zA-Z]/,"
").split("
").each { |i| kwds.add i.strip }
    kwds.to_a.join("
")
end

Benchmark.bm do |x|
    x.report("shell") { 100.times do shell_keywords end }
    x.report("ruby") { 100.times do ruby_keywords end }
end

The end results are that it is faster to use the ruby implementation by quite a margin. My guess is that fork is probably where the most time is spent for the shell implementation.

 $ ./benchmark.rb 
      user     system      total        real
shell  0.020000   0.110000   2.060000 (  3.589461)
ruby   0.460000   0.030000   0.490000 (  0.873792)

I found this page a great resource for implementing the benchmark.

USA / UK Date formats for ActiveRecord

Posted by Samuel Williams Tue, 10 Oct 2006 04:39:00 GMT

I have a client who are based in the UK and want to put UK formatted %d/%m/%y formatted dates into Ruby on Rails applications.

The solution is not very clear, but quite simple.

There is a module called ‘parsedate’ used by ActiveRecord to parse dates:

# parsedate.rb: Written by Tadayoshi Funaba 2001, 2002
# $Id: parsedate.rb,v 2.6 2002-05-14 07:43:18+09 tadf Exp $

require 'date/format'

module ParseDate

  def parsedate(str, comp=false)
    Date._parse(str, comp).
      values_at(:year, :mon, :mday, :hour, :min, :sec, :zone, :wday)
  end

  module_function :parsedate

end

We simply need to make some modifications. I have added this to the top of my application.rb

module ParseDate
  def parsedate(str, comp=false)
    match = /(d+)/(d+)/.match(str)
    str = match.pre_match + match[2] + '/' + match[1] + match.post_match if match

    Date._parse(str, comp).
      values_at(:year, :mon, :mday, :hour, :min, :sec, :zone, :wday)
  end

  module_function :parsedate
end

This code basically maps dd/mm to mm/dd in any date passed in, which means it will handle dates with or without years, times etc.

I hope this is useful to someone else. I think that the core Date class could be expanded to deal with both USA and UK dates.

observe_field not working correctly?

Posted by Samuel Williams Tue, 18 Jul 2006 00:17:00 GMT

Using observe_field for receiving server-side notifications of select boxes doesn’t seem to be working correctly in Ruby on Rails + prototype.js. On most browsers which conform, it will work as expected - ie, changing the value of a select will send an AJAX request upstream.. but on Internet Explorer (all versions that I have tested on) the behavior is broken.

There is a simple fix which can be made to prototype.js. Firstly, search for the class Abstract.EventObserver and look for the method registerCallback. This is where events are bound from the field to the observe_field callback. There is a case statement. Under the case for select boxes, we need to add a new event binding:

Abstract.EventObserver.registerCallback
 registerCallback: function(element) {
   if (element.type) {
     switch (element.type.toLowerCase()) {
       case 'checkbox':
       case 'radio':
         Event.observe(element, 'click', this.onElementEvent.bind(this));
         break;
       case 'password':
       case 'text':
       case 'textarea':
+        Event.observe(element, 'change', this.onElementEvent.bind(this));
+        break;
       case 'select-one':
       case 'select-multiple':
         Event.observe(element, 'change', this.onElementEvent.bind(this));
+        Event.observe(element, 'click', this.onElementEvent.bind(this));
         break;
     }
   }
 }

If you look carefully, you will notice there are a number of different field types that will have this event bound (due to the drop-through nature of a select/case block) - ie ‘password’, ‘textarea’, etc. So, we have firstly made sure these are still only bound to the onChange event.

Secondly, we have added a new event for select fields - onClick. This works because the onClick event for Internet Explorer is similar to the onChange event in other browsers, and also Abstract.EventObserver does change tracking (ie, it keeps track of the last value so if the value is the same, it does not send a notification), so that multiple clicks will not cause spurious events to be sent upstream.

There may be other useful events to bind to (ie, keyboard events) but this is not so useful in my case - feel free to comment on this if you have implemented different events.