Ruby 1.9 and Unicode: The BOM Will Fuck Your Shit Up

So I’ve been playing around with the things mentioned in the title, and I found out something unfortunate when I moved a UTF-8 encoded file from a Ruby 1.9 machine to a Ruby 1.8 machine.

There’s this thing called a Byte Order Marker (BOM) that text editors use, apparently to remind themselves of the file’s UTF-8 encoding. I’m pretty sure it’s useless, because UTF-8 doesn’t actually have a variable byte order to keep track of, but there you go.

Basically, it’s 3 bytes that the text editor inserts at the beginning of a text file, and then hides from you. It might look like a plain text file, but it’s actually got 3 hidden bytes for no good reason. When you try to run it through the Ruby 1.8 interpreter, it’ll see 3 invalid characters on Line 1 and throw an error right away.

This sort of error message is pretty unhelpful, especially when you appear to have nothing at all on Line 1. You might enable visible whitespace: still nothing. You might try opening it in another text editor or IDE: you will likely still not see the problem, as the only program I’ve tried so far that doesn’t hide the BOM is NetBeans.

SciTE has two different UTF-8 encoding settings: UTF-8 and UTF-8 Cookie. In theory, the plain UTF-8 setting uses a Byte Order Marker, while UTF-8 Cookie setting doesn’t. In practice, the choice doesn’t seem to affect whether or not the Ruby interpreter chokes on the file, at least not with Ruby 1.8.

With 1.9 I’ve still had problems one or two times, but of the kind that could be fixed by closing the text editor, opening the file in NetBeans, removing the BOM, and restarting the text editor.

It’s not perfect, but at least it works now, even if it’s very slightly buggy.

A basic genetic algorithm (part 3)

I’ve posted the updated code here, on Refactor :my => ‘code’, a cool little site I just found.

I’ll go refactor someone else’s code on there later, could help me with my coding too.

The program works now, although it’s not too efficient. I need to add sexual reproduction, ’cause right now they’re asexual and that’s slower.

EDIT: Okay, I’ll just leave it up on the other site then, because WordPress was throwing HTML in my Ruby, like this :

def initialize(copy_genome=‘off’ <img src=“http://s.wordpress.com/wp-includes/images/smilies/icon_wink.gif” alt=“;)” class=“wp-smiley”>

A basic genetic algorithm (part 2)

Okay, so I’ve broken this up into two files :

equation.rb

class Array
  def random
    self[rand(self.length)]
  end
end

class Genome
  attr_accessor :code

  @@decode = {   '0000' => '0.0', '0001' => '1.0',
  '0010' => '2.0', '0011' => '3.0', '0100' => '4.0',
  '0101' => '5.0', '0110' => '6.0', '0111' => '7.0',
  '1000' => '8.0', '1001' => '9.0', '1010' => '+',
  '1011' => '-', '1100' => '*', '1101' => '/' }
  @@operators = %w[+ - * /]
  @@numbers = %w[0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0]

  def initialize(min_length=4, max_length=32)
    length = min_length + rand(1 + max_length - min_length)
    @code = generate_code(length)
  end

  def generate_code(length)
    code = ''
    1.upto(length) do
      code += ['0', '1'].random
    end
    code
  end

  def decode
    decoded = ''
    expected = :number

    1.upto(@code.length) do |i|
      if i % 4 == 0
        coded = @code.slice((i-4)..(i-1))
        symbol = @@decode[coded].to_s
        puts symbol
        if expected == :number && @@numbers.include?(symbol)
          decoded += symbol
          expected =  perator
        elsif expected ==  perator &&
        @@operators.include?(symbol)
          decoded += symbol
          expected = :number
        else
          puts "#{expected} expected, #{symbol} found."
        end
      end
    end

    if expected == :number
      decoded.chop!
    end

    return decoded
  end
end

class Equation
  attr_accessor :genome, :phenotype

  def initialize
    @genome = Genome.new(4, 64)
    @phenotype = @genome.decode
  end
end

formula_ga.rb

require 'equation'

class Population
  def initialize(size=50, target_number=11)
    @equations = []
    @size = size
    @target_number = target_number
    1.upto(size) { @equations << Equation.new }
  end

  def members
    @equations
  end

  def evaluate_fitness(equation)
    answer = eval(equation.phenotype)
    deviation = @target_number - answer.to_f
    fitness = 1 / deviation
  end

  def sort_by_fitness(equation_array)

  end

  def next_generation
    reproduction_pool = []
    1.upto(@size / 2) do
      reproduction_pool << @equations.random
    end
  end
end

pop = Population.new
pop.members.each do |member|
  puts member.genome.code
  puts member.phenotype
  x = eval(member.phenotype)
  puts x.to_s
  puts "Fitness : #{pop.evaluate_fitness(member)}"
  puts ""
end

A basic genetic algorithm (part 1)

This here is an intro to genetic algorithms with a nice little biological analogy and everything. It starts by explaining the evolution of blind, clumsy, algae-eating, cave-dwelling creatures called Hooters into light-seeing, eagle-dodging, moss-eating machines.

It then goes on to explain how this is relevant to computing, and gives a simple example of a problem that can be solved with a genetic algorithm. In this case it involves evolving equations that give a number you’re looking for.

I’m already a little familiar with GAs, I’ve written one that generated words out of random characters, but it was slightly ugly. Especially the fitness function. I’ll to do this one properly.

I’m going to solve the equation problem and post my code later (it’ll be in Ruby).

UPDATE : Here’s the code so far. I’m off to play DOTA.

class Array
  def random
    self[rand(self.length)]
  end
end

class Genome
  attr_accessor :code

  def initialize(min_length=4, max_length=32)
    length = min_length + rand(1 + max_length - min_length)
    @code = generate_code(length)
  end

  def generate_code(length)
    code = ''
    1.upto(length) do
      code += ['0', '1'].random
    end
    code
  end
end

class Equation
  def initialize
    @genome = Genome.new
  end

  def genome
    @genome.code
  end
end

class Population
  def initialize(size=50)
    @equations = []
    1.upto(size) { @equations << Equation.new }
  end

  def members
    @equations
  end
end

Population.new.members.each { |member| puts member.genome}

Anyone know of a better way to easily manipulate a series of bits than by storing it in a string? It’s fine if I just have a few but I like to make these things scale, if possible, and I know from experience that this kind of thing tends toward total memory consumption.