MATH AND PROGRAMMING!

Rescaling

2011-12-20T08:38:00.000-08:00

I keep forgetting/losing this function. I need to put it in a module somewhere.

def rescale( data, newmin, newmax ):
   oldmin, oldmax = np.min( data ), np.max( data )
   rows, cols = data.shape
   nd = np.reshape( data, (rows*cols,) )
   newdata = list()
   newscale = newmax-newmin
   oldscale = oldmax-oldmin
   ratio = newscale/float(oldscale+.005)
   for i in nd:
   newdata.append((i-oldmin)*ratio+newmin)
   newdata = np.reshape( newdata, (rows,cols) )
   return newdata

Merging Segments in an Image

2011-12-08T10:12:00.001-08:00

I'll walk though some code for merging segments in an image using a two classes, Segment and Segments. The first class is for

class Segment:
   def __init__( self, elements, neighbors, class ):
   self.elements = [ elements ] # tuple addresses
   self.neighbors = neighbors # class labels
   self.class = class # personal class
   self.quality = None # either textured or smooth
   self.mean = None # mean reflectance
   self.texture = None # texture vector

<< OTHER CODE >>

    def merge( self, a, b ):

        # enhance readabilityness

        newclass = a.class

        oldclass = b.class

        # a subsumes b's elements 

        a.elements += b.elements 

        # set of class labels

        # the class labels neighboring a and the class label of a

        mine = set( a.neighbors + [ newclass ] )

        # set of class labels

        # the class labels neighboring b and the class lebel of b

        theirs = set( b.neighbors + [ oldclass ] )

        # set of class labels for elements of a (after adding b to a)

        ours = set( [ self.element_segment[i] for i in a.elements ] )

        # update a's new neighbors

        a.neighbors = list( mine.union( theirs ) - ours ) 

        # update class elementship of b's elements

        for m in b.elements:

            self.element_segment[ m ] = newclass 

            self.segment_element[ newclass ].append( m ) 

        # remove the oldclass from the dict of classes

        del self.segment_element[ oldclass ] 

        # remove the oldclass from the dict of segments

        del self.segments[ oldclass ]

        # update old neighbors of the oldclass

        for s in self.segments:

            if oldclass in self.segments[s].neighbors:

                while oldclass in self.segments[s].neighbors:

                    self.segments[s].neighbors.remove( oldclass )

                if newclass not in self.segments[s].neighbors:

                    self.segments[s].neighbors.append( newclass )

What the merge() function does, is figure out the new elements of the new segment (easy), figure out the new neighbors of the new segment (harder), and then do some cleaning operations.

In the merge() function, newclass = a.class and oldclass = b.class take the class labels of the segments to be merged, a and b. In this implementation, a is dominant, and subsumes b's class label.

Then, with a.elements += b.elements, we add the elements of b to the elements of a. These elements are tuple addresses for image chips, or individual pixels.

Imagine the following setup:

|---|---|---|---|

| 3 | 8 | 8 | 8 |

|---|---|---|---|

| 3 | 1 | 2 | 7 |

|---|---|---|---|

| 4 | 1 | 2 | 7 |

|---|---|---|---|

| 4 | 4 | 5 | 6 |

|---|---|---|---|

Here, segment a is denoted by the ones, and segment b is denoted by the twos. The line mine = set( a.neighbors + [ newclass ] ) is just the set {3,4,2,8,1}, the class labels for the n,s,e,w neighbors of segment a, which are {3,4,2,8}, and the class label for a, which is {1}. Similarly for theirs we have the class labels for the neighbors of b, and the class label for b, and this ends up being the set {1,8,7,5,2}. Then ours is the class labels in the new segment, which will be the set {1,2}, since 1 and 2 are the class labels of segment a and segment b.

Then the new neighbors of the new segment will be found by taking the union of mine and theirs, which is {3,8,4,7,5,1,2}, and then tossing ours, giving us the set {3,4,8,7,5}, and these are the tiles ordinally neighboring the tiles labeled 1 and 2. Note that tile 6 is not considered a neighbor because it is to the south-east of tile 2; only tiles that share a whole side are counted as neighbors.

After cleaning up the requisite dictionaries that keep track of these things, we should have the following diagram,

|---|---|---|---|

| 3 | 8 | 8 | 8 |

|---|---|---|---|

| 3 | 1 | 1 | 7 |

|---|---|---|---|

| 4 | 1 | 1 | 7 |

|---|---|---|---|

| 4 | 4 | 5 | 6 |

|---|---|---|---|

pywt versus PyWT

2011-11-22T18:49:00.001-08:00

To be specific, I am talking about the difference between pywt (PyWavelets) and PyWT (Python Web Toolkit).

I thought I had installed the wavelets package, as I had used pywt freely for several years; however, when I executed,

$ pip install pywt

I got the web toolkit package instead, even though it goes by PyWF now. Executing the inverese:

$ pip uninstall pywt

I got the reassuring,

$ pip uninstall pywtUninstalling PyWT:Proceed (y/n)? y Successfully uninstalled PyWT

But when I went to use the wavelet package, it was still accessing the ununinstalled web toolkit, which does absolutely nothing for wavelets. So... I imported pywt at the terminal again and looked at the path:

>>> pywt.__path__
['/usr/local/lib/python2.6/dist-packages/pywt']

And then I went in like gangbangers and shot up that joint with,

$ sudo rm -r /usr/local/lib/python2.6/dist-packages/pywt/

Then everything worked out okay.

   >>> import pywt
   >>> from pywt import dwt
   >>> dwt([1,2,3,4,5,6,7,8],'db4')
   (array([ 7.06453146, 4.23073611, 1.41360717, 2.83605428,
   5.6633906 , 8.49718595, 11.3143149 ]),
   array([ 2.37131306e-02, 4.09620864e-02, -6.46752170e-02,
   -9.90107302e-13, -2.37131306e-02, -4.09620864e-02,
   6.46752170e-02]))
   >>>

Matrix Inversion with HDF5 files

2011-11-07T09:12:00.000-08:00

#!/usr/bin/env python

import sys
import h5py
import numpy as np

def doolittle_h5( a_filename, side_size ):

import h5py
import numpy as np

fa = h5py.File( a_filename, 'r' )
a = fa['chunked']

fl = h5py.File( 'l.h5', 'r+' )
l = fl[u'data']

fu = h5py.File( 'u.h5', 'r+' )
u = fu[u'data']

n = side_size

for i in range( n ):
    for j in range( i, n ):
        u[i,j] = a[i,j]
        for k in range( i ):
          u[i,j] = u[i,j] - l[i,k] * u[k,j]
    for j in range( i, n ):
        l[j,i] = a[j,i]
        for k in range( i ):
        l[j,i] = l[j,i] - l[j,k] * u[k,i]
        l[j,i] = l[j,i] / float( u[i,i] )
    print 'd: '+i

fa.close()
fl.close()
fu.close()

return None

def solve_L_system_h5( a_filename, side_size, b ):

import h5py
import numpy as np

f = h5py.File( a_filename, 'r' )
a = f[u'data'][ :side_size, :side_size ]

x = np.zeros_like( b ).astype( np.float32 )

for i in range( b.size ):
    x[i] = b[i]
    for j in range( i ):
        x[i] -= a[i,j] * x[j]

f.close()

return x

def solve_U_system_h5( a_filename, side_size, b ):

import h5py
import numpy as np

f = h5py.File( a_filename, 'r' )
a = f[u'data'][ :side_size, :side_size ]

x = np.zeros_like( b ).astype( np.float32 )

for i in range( b.size-1,-1,-1 ):
    x[i] = b[i]
    for j in range( i+1, b.size ):
        x[i] -= a[i,j] * x[j]
    x[i] /= float( a[i,i] )

f.close()

return x

def inverse_h5( a_filename, side_size ):

import h5py
import numpy as np

doolittle_h5( a_filename, side_size )

print "Done doing little."

finv = h5py.File( 'ainv.h5', 'w' )
dinv = finv.create_dataset( u'data', ( side_size, side_size ), np.float32, chunks=(side_size,1) )
inv = finv[u'data']

for i in range( side_size ):
    b = np.zeros((side_size,)) ; b[i] = 1.0
    b.astype( np.float32 )
    y = solve_L_system_h5( 'l.h5', side_size, b )
    x = solve_U_system_h5( 'u.h5', side_size, y )
    inv[:,i] = x
    print "inverting: "+str(i/float(side_size))+"% done"

finv.close()

return None

#fa = h5py.File( 'a.h5', 'w' )
#d = fa.create_dataset( 'chunked', ( 3,3 ), np.float32 )
#a = fa['chunked']

#da = [ [ 2,-1,-2 ], [ -4,6,3 ], [ -4,-2,8 ] ]
#for i in range( 3 ):
#    for j in range( 3 ):
#        a[i,j] = da[i][j]

#fa.close()

#inverse_h5('a.h5',3)

# at command line: d0=date; nice -n -20 python doolittle.py xx.h5 250000; d1=date;
inverse_h5( sys.argv[1], int( sys.argv[2] ) )

Deleting tons of files at once

2011-10-19T11:36:00.001-07:00

Okay, suppose you have tons of files you need to delete.. so many that you can't open up the folder, or use ls to delete them in one fell swoop. Suppose further that your files are labeled like, "x_0_0_1_2.dat". Then you can write a loop in you bash terminal as follows:

$ for i in 0 1 2 ; do rm x_${i}_* ; done

The key thing to note here is the curly braces around the iterating variable. If you leave off the braces, it won't work.

$ for i in 3 4 5 ; do rm x_$i_* ; done
bash: /bin/rm: Argument list too long
bash: /bin/rm: Argument list too long
bash: /bin/rm: Argument list too long

An Online Physics Textbook!

2011-04-22T12:59:00.000-07:00

It claims to be a work in progress, but it seems pretty complete to me. You can find it here at http://physics.info/

PyCUDA Documentation

2011-04-18T08:31:00.000-07:00

Here is the PyCUDA documentation.

Maxima

2011-04-12T15:58:00.000-07:00

Maxima is a free computer algebra system developed at MIT. Here is a great tutorial to get you up and running.

L-System Fractals in Python using PIL

2011-04-07T12:39:00.000-07:00

Here is some code from ActiveState detailing some L-System fractal stuff

Pretty neat site on fractals

2011-04-07T11:43:00.000-07:00

Go here.

A good source for the Python Imaging Library

2011-04-07T08:00:00.000-07:00

I've run across this site multiple times, but I don't think I've ever posted it here. Anyway, it has lots of stuff about the PIL module.

Python Sorting

2011-04-05T11:01:00.000-07:00

So... sort() sorts an interable in place, whereas sorted() returns a sorted copy. This also works on an iterable of objects! So, here's the run-down.

PyOpenCV - Access OpenCV Through Python!

2011-03-30T12:27:00.000-07:00

OpenCV is an open source computer vision library written in C/C++. Here is the Python module for it.

Create Image with NumPy

2011-03-29T12:46:00.000-07:00

This is a good resource. Apparently better than using PIL putpixel(), etc.

A Nice Introduction to R

2011-03-27T18:09:00.000-07:00

Here is a really good R time series resource.

Some RPy Examples

2011-03-27T03:00:00.000-07:00

This is a pretty good resource for RPy. It has a lot of economics data. It also uses json, which is something I've never heard of.

This is another great introduction to RPy!

Installing Packages in R, using Ubuntu

2011-03-25T19:47:00.000-07:00

R... is the unintuitive statistics "programming language". I'm being overly disdainful; just because you don't like something doesn't mean it isn't useful. It has a fractal dimension package, fdim, which you can read about on CRAN.

I found this useful in getting fdim up and running, even though it has deprecated dependencies and everything.

PCA in Python

2011-03-23T17:30:00.000-07:00

Here is the stack-overflow page on PCA in Python.

Hypertext Help With LaTeX

2011-03-22T09:30:00.001-07:00

This is a great general site on LaTeX.

LaTeX Standard Environments

2011-03-22T09:22:00.000-07:00

Here is a good page on the LaTeX standard environments.

Example of using threading using Python

2011-03-17T08:48:00.000-07:00

#!/usr/bin/env python

import Queue
import threading
import mahotas
import time
import sys

# this is the list where
# the final data goes
f = list()

# create a Queue
queue = Queue.Queue()

# the code that needs to be executed
# we override the init function in order to pass parameters to the threads
class MyThread( threading.Thread ):
def __init__( self, f, queue ):
self.f = f
self.queue = queue
threading.Thread.__init__( self )
def run( self ):
while True:
task = self.queue.get()
self.f.append( mahotas.features.haralick( task ) )
self.queue.task_done()

# reads the image and converts it to a numpy int ndarray
img = mahotas.imread( sys.argv[1] )

# chip size dimensions
chipx = int( sys.argv[2] )
chipy = int( sys.argv[3] )

# figure out image size
imgx, imgy, imgz = img.shape

# the number of chips in each direction
nbxchips = imgx/chipx
nbychips = imgy/chipy

start = time.time()

# create some threads
for i in range( 3 ):
t = MyThread( f, queue )
t.setDaemon( True )
t.start()

# put chips into the queue
for i in range( nbxchips ):
for j in range( nbychips ):
queue.put( img[i*chipx:(i+1)*chipx,j*chipy:(j+1)*chipy,:] )

# wait for all the work to finish
queue.join()

print 'Elapsed Time: %s' % ( time.time() - start )

print len( f )

Haralick Features, and other features, too!

2011-03-15T13:36:00.000-07:00

I found this site that describes a Python module that calculates Haralick textures, SURF features, and others.

Notable algorithms:

watershed.
convex points calculations.
hit & miss. thinning.
Zernike & Haralick, LBP, and TAS features.
freeimage based numpy image loading (requires freeimage libraries to be installed).
Speeded-Up Robust Features (SURF), a form of local features.
thresholding.
convolution.
Sobel edge detection

PythonVision is another good site.

Python PCA (ii)

2011-03-15T09:25:00.000-07:00

# Code from Chapter 10 of Machine Learning: An Algorithmic Perspective
# by Stephen Marsland (http://seat.massey.ac.nz/personal/s.r.marsland/MLBook.html)

# You are free to use, change, or redistribute the code in any way you wish for
# non-commercial purposes, but please maintain the name of the original author.
# This code comes with no warranty of any kind.

# Stephen Marsland, 2008

# An algorithm to compute PCA. Not as fast as the NumPy implementation
from pylab import *
from numpy import *

def pca(data,nRedDim=0,normalise=1):

# Centre data
m = mean(data,axis=0)
data -= m

# Covariance matrix
C = cov(transpose(data))

# Compute eigenvalues and sort into descending order
evals,evecs = linalg.eig(C)
indices = argsort(evals)
indices = indices[::-1]
evecs = evecs[:,indices]
evals = evals[indices]

if nRedDim>0:
evecs = evecs[:,:nRedDim]

if normalise:
for i in range(shape(evecs)[1]):
evecs[:,i] / linalg.norm(evecs[:,i]) * sqrt(evals[i])

# Produce the new data matrix
x = dot(transpose(evecs),transpose(data))
# Compute the original data again
y=transpose(dot(evecs,x))+m
return x,y,evals,evecs

Python PCA (i)

2011-03-15T09:23:00.000-07:00

#!/usr/bin/env python

""" a small class for Principal Component Analysis
Usage:
p = PCA( A, fraction=0.90 )
In:
A: an array of e.g. 1000 observations x 20 variables, 1000 rows x 20 columns
fraction: use principal components that account for e.g.
90 % of the total variance

Out:
p.U, p.d, p.Vt: from numpy.linalg.svd, A = U . d . Vt
p.dinv: 1/d or 0, see NR
p.eigen: the eigenvalues of A*A, in decreasing order (p.d**2).
eigen[j] / eigen.sum() is variable j's fraction of the total variance;
look at the first few eigen[] to see how many PCs get to 90 %, 95 % ...
p.npc: number of principal components,
e.g. 2 if the top 2 eigenvalues are >= `fraction` of the total.
It's ok to change this; methods use the current value.

Methods:
The methods of class PCA transform vectors or arrays of e.g.
20 variables, 2 principal components and 1000 observations,
using partial matrices U' d' Vt', parts of the full U d Vt:
A ~ U' . d' . Vt' where e.g.
U' is 1000 x 2
d' is diag([ d0, d1 ]), the 2 largest singular values
Vt' is 2 x 20. Dropping the primes,

d . Vt 2 principal vars = p.vars_pc( 20 vars )
U 1000 obs = p.pc_obs( 2 principal vars )
U . d . Vt 1000 obs, p.obs( 20 vars ) = pc_obs( vars_pc( vars ))
fast approximate A . vars, using the `npc` principal components

Ut 2 pcs = p.obs_pc( 1000 obs )
V . dinv 20 vars = p.pc_vars( 2 principal vars )
V . dinv . Ut 20 vars, p.vars( 1000 obs ) = pc_vars( obs_pc( obs )),
fast approximate Ainverse . obs: vars that give ~ those obs.

Notes:
PCA does not center or scale A; you usually want to first
A -= A.mean(A, axis=0)
A /= A.std(A, axis=0)
with the little class Center or the like, below.

See also:
http://en.wikipedia.org/wiki/Principal_component_analysis
http://en.wikipedia.org/wiki/Singular_value_decomposition
Press et al., Numerical Recipes (2 or 3 ed), SVD
PCA micro-tutorial
iris-pca .py .png

"""

from __future__ import division
import numpy as np
dot = np.dot
# import bz.numpyutil as nu
# dot = nu.pdot

__version__ = "2010-04-14 apr"
__author_email__ = "denis-bz-py at t-online dot de"

#...............................................................................
class PCA:
def __init__( self, A, fraction=0.90 ):
assert 0 <= fraction <= 1
# A = U . diag(d) . Vt, O( m n^2 ), lapack_lite --
self.U, self.d, self.Vt = np.linalg.svd( A, full_matrices=False )
assert np.all( self.d[:-1] >= self.d[1:] ) # sorted
self.eigen = self.d**2
self.sumvariance = np.cumsum(self.eigen)
self.sumvariance /= self.sumvariance[-1]
self.npc = np.searchsorted( self.sumvariance, fraction ) + 1
self.dinv = np.array([ 1/d if d > self.d[0] * 1e-6 else 0
for d in self.d ])

def pc( self ):
""" e.g. 1000 x 2 U[:, :npc] * d[:npc], to plot etc. """
n = self.npc
return self.U[:, :n] * self.d[:n]

# These 1-line methods may not be worth the bother;
# then use U d Vt directly --

def vars_pc( self, x ):
n = self.npc
return self.d[:n] * dot( self.Vt[:n], x.T ).T # 20 vars -> 2 principal

def pc_vars( self, p ):
n = self.npc
return dot( self.Vt[:n].T, (self.dinv[:n] * p).T ) .T # 2 PC -> 20 vars

def pc_obs( self, p ):
n = self.npc
return dot( self.U[:, :n], p.T ) # 2 principal -> 1000 obs

def obs_pc( self, obs ):
n = self.npc
return dot( self.U[:, :n].T, obs ) .T # 1000 obs -> 2 principal

def obs( self, x ):
return self.pc_obs( self.vars_pc(x) ) # 20 vars -> 2 principal -> 1000 obs

def vars( self, obs ):
return self.pc_vars( self.obs_pc(obs) ) # 1000 obs -> 2 principal -> 20 vars

class Center:
""" A -= A.mean() /= A.std(), inplace -- use A.copy() if need be
uncenter(x) == original A . x
"""
# mttiw
def __init__( self, A, axis=0, scale=True, verbose=1 ):
self.mean = A.mean(axis=axis)
if verbose:
print "Center -= A.mean:", self.mean
A -= self.mean
if scale:
std = A.std(axis=axis)
self.std = np.where( std, std, 1. )
if verbose:
print "Center /= A.std:", self.std
A /= self.std
else:
self.std = np.ones( A.shape[-1] )
self.A = A

def uncenter( self, x ):
return np.dot( self.A, x * self.std ) + np.dot( x, self.mean )

#...............................................................................
if __name__ == "__main__":
import sys

csv = "iris4.csv" # wikipedia Iris_flower_data_set
# 5.1,3.5,1.4,0.2 # ,Iris-setosa ...
N = 1000
K = 20
fraction = .90
seed = 1
exec "\n".join( sys.argv[1:] ) # N= ...
np.random.seed(seed)
np.set_printoptions( 1, threshold=100, suppress=True ) # .1f
try:
A = np.genfromtxt( csv, delimiter="," )
N, K = A.shape
except IOError:
A = np.random.normal( size=(N, K) ) # gen correlated ?

print "csv: %s N: %d K: %d fraction: %.2g" % (csv, N, K, fraction)
Center(A)
print "A:", A

print "PCA ..." ,
p = PCA( A, fraction=fraction )
print "npc:", p.npc
print "% variance:", p.sumvariance * 100

print "Vt[0], weights that give PC 0:", p.Vt[0]
print "A . Vt[0]:", dot( A, p.Vt[0] )
print "pc:", p.pc()

print "\nobs <-> pc <-> x: with fraction=1, diffs should be ~ 0"
x = np.ones(K)
# x = np.ones(( 3, K ))
print "x:", x
pc = p.vars_pc(x) # d' Vt' x
print "vars_pc(x):", pc
print "back to ~ x:", p.pc_vars(pc)

Ax = dot( A, x.T )
pcx = p.obs(x) # U' d' Vt' x
print "Ax:", Ax
print "A'x:", pcx
print "max |Ax - A'x|: %.2g" % np.linalg.norm( Ax - pcx, np.inf )

b = Ax # ~ back to original x, Ainv A x
back = p.vars(b)
print "~ back again:", back
print "max |back - x|: %.2g" % np.linalg.norm( back - x, np.inf )

# end pca.py

Threading and Queues in Python!

2011-02-27T16:06:00.000-08:00

Here is a good site on the Queue module.

Here is a good site on the Threading module.

Here is the IBM site on Threading and Queues.

Instantaneous Amplitude, Phase and Frequency

2011-02-13T18:54:00.001-08:00

Here is a link to a good discussion on this with MATLAB code.

Here is another place with a good discussion on doing a phase shift.
The Hilbert transform is fft ---> phase shift ---> ifft.
Doing fft and ifft are easy in numpy, and a phase shift is just multiplying each term in the signal by exp(j*p), where j is the imaginary unit, and p is the desired phase shift.

ROC Curves

2011-02-09T14:47:00.000-08:00

Here is a good site on ROC curves.

Project Results!

2011-02-06T10:44:00.001-08:00

This is the raw data
This is the data after subtracting the reference signal
This is the data after each row has been differenced
This density function represents the sum of each column of the previous image
This is the result of using a five point smoothing function on the previous data

NASDAQ, NYSE, AMEX Data

2011-01-26T18:47:00.000-08:00

Here is a site that provides historical data over 10 years for three exchanges, the NYSE, NASDAQ and AMEX. Go to 'Stock Chart,' and then 'Historical Quotes' or something can be found in the sidebar. Link is here.

LaTeX Crashing Course

2011-01-26T06:17:00.000-08:00

This is a good tutorial that walks through things pretty slowly. I have so far only used it to fill in fundamental gaps; I haven't perused it for anything fancy. Here.

This is a good tutorial on tables.

Best Bash Tutorials Ever. Period.

2011-01-25T13:04:00.000-08:00

I can stop looking for Bash tutorials forever now. This is better than the expensive book I bought on this topic.

http://www.faqs.org/docs/abs/HTML/index.html

Colors

2011-01-17T12:26:00.001-08:00

Here is a good color scheme designer.

Paul McCartney II

2011-01-06T10:46:00.000-08:00

Awesome opossum. Some selected tracks can be found, here.

Coming Up, the album's single, can be found, here.

Digital Signal Processing

2011-01-06T06:42:00.000-08:00

A free book on digital signal processing (DSP) can be found, here.

Computing Talk

2011-01-03T06:30:00.000-08:00

A page on Python software is found here.

Best Curry Yet.

2010-12-20T08:38:00.000-08:00

Start some brown rice with mashed ginger. Chop some yams or sweet potatoes and carrots and roast them halfway. Lightly toast some cumin seeds in a skillet and then grind them in a mortar. Put the half-roasted root vegetables into the skillet with some minced onion and garlic. Add dried basil, cayenne, cinnamon, cardamom, corriander, and cumin. Cook until the onions are translucent and then add to the rice. Mix in siracha, green onions, peanuts, banana chips and raisins. (The peanuts should add enough salt.)

I don't think Indians use cayenne, but instead rely on the combination of other spices and cinnamon to create a heat. Whatever. It was my birthday and I liked it.

White/Pink/Red Noise Generation

2010-12-20T08:37:00.000-08:00

This guy uses some scripts and sox to produce different kinds of noise, here.

This guy has some Matlab/Octave code to generate pink noise, here.

The wikipedia article on spectral density is helpful. It is found here.

The scholarpedia (?) article on 1/f noise is here.

A python pink noise generator is here.

A Matlab script for pink noise is here.

Command Line Audio Manipulation

2010-12-20T08:26:00.000-08:00

Sound eXchange can be found, here. I think you need to convert ogg -> wav using sox, and then wav -> mp3 using lame.

Soundconverter, a GNU GUI for translating between formats can be found, here. It appears to also have a command-line capability.

Create Windows Executables from Python Scripts

2010-12-13T13:22:00.000-08:00

Py2exe is found here.

Tutorial Ideas

2010-12-09T17:01:00.000-08:00

+ Predator-Prey Modeling
+ Other Dynamic Models
+ Differential Equations
+ Kalmann Filter
+ Using Cyphers
+ Image Processing
+ Color Schemes
+ Random Numbers and Chaos

Handling Images in Python via Matplotlib

2010-12-09T08:45:00.000-08:00

This page describes how to use the matplotlib.image module to load and work with images. Here.

Something about Python GUI

2010-12-06T13:54:00.001-08:00

Har.

Micro-controllers

2010-12-01T06:47:00.001-08:00

http://focus.ti.com/mcu/docs/mcuprodmsptoolsw.tsp?sectionId=95&tabId=1203&familyId=342&toolTypeId=1

cat is for concatenate (tfw?)

2010-11-30T08:41:00.001-08:00

I always forget how to use the cat utility. Here's how.

Lotka-Volterra Predator-Prey Model in Three Part Harmony

2010-11-29T11:23:00.000-08:00

#!/usr/bin/env python

import os

def system( u, v, w, uinit, vinit, winit, time ):
ut = list(); ut.append( uinit )
vt = list(); vt.append( vinit )
wt = list(); wt.append( winit )
for i in range( time ):
ut.append( u.update( ut[-1], vt[-1], wt[-1] ) )
vt.append( v.update( vt[-1], ut[-1], wt[-1] ) )
wt.append( w.update( wt[-1], ut[-1], vt[-1] ) )
return ( ut, vt, wt )

class Component:
def __init__( self, a, b, g, h ):
self.alpha = a
self.beta = b
self.gamma = g
self.step = h
def update( self, x, y, z ):
self.body = self.alpha*x + self.beta*x*y + self.gamma*x*z
return x + self.step*self.body

step = 0.08
u = Component( -1.0, 0.025, 0.035, step )
v = Component( 1.5, -0.45, -0.024, step )
w = Component( 1.45, -0.75, 0.026, step )
( ut, vt, wt ) = system( u, v, w, 5.0, 50.0, 20.0, 4000 )

ubuff = open( "ut.txt", "w" )
vbuff = open( "vt.txt", "w" )
wbuff = open( "wt.txt", "w" )

for i in range( len( ut ) ):
ubuff.write( str( ut[i] ) )
ubuff.write('\n')
vbuff.write( str( vt[i] ) )
vbuff.write('\n')
wbuff.write( str( wt[i] ) )
wbuff.write('\n')

ubuff.close(); vbuff.close(); wbuff.close()

foo = list()
foo.append('#!/usr/bin/env octave\n')
foo.append('\n')
foo.append('load ut.txt ; load vt.txt ; load wt.txt ;\n')
foo.append('t = 0:4000 ; plot( t, ut, t, vt, t, wt ) ;\n')
foo.append('print -djpeg foo.jpeg;\n')

f = open( "foo.o", "w" )
f.writelines( foo )
f.close()

os.system( 'octave -qf foo.o' )
os.system( 'evince foo.jpeg &' )

os.system( 'rm ut.txt' )
os.system( 'rm vt.txt' )
os.system( 'rm wt.txt' )
os.system( 'rm foo.o' )

Robot Scientist

2010-11-24T10:46:00.000-08:00

A robot scientist, here.

Genetic algorithm equation maker, here.

"Distilling Free-Form Natural Laws from Experimental Data." By Michael Schmidt and Hod Lipson. Science, Vol. 324, April 3, 2009.

"Automating Science." By David Waltz and Bruce Buchanan. Science, Vol. 324, April 3, 2009.

Pthreads!

2010-11-24T08:22:00.000-08:00

A good introductory tutorial to pthreads can be found here.

Sanderson and Croft, Deriving Concept Hierarchies from Text

2010-11-18T17:35:00.000-08:00

Monothetic clusters: membership is based only on one feature.

Polythetic clusters: a document's membership to a given cluster is defined by its possession of a sufficient fraction of terms from that cluster. See Scatter/Gather.

The topic of a monothetic cluster is usually more intuitive than the topic of a polythetic cluster.

Five basic principles:

1) Terms in hierarchy extracted from documents, and reflect topics covered in documents

Color Schemes

2010-11-10T11:06:00.000-08:00

Kuler?

Misc Linux Commands

2010-11-10T07:27:00.000-08:00

Commands found, here, banner and cal are pretty neat.

GPU Gems 2

2010-11-10T06:30:00.001-08:00

Book available in HTML format, here.

GPGPU Shtuff

2010-11-05T09:41:00.000-07:00

Supercomputing for the Masses can be found, here.

stdint.h with g++ on Ubuntu

2010-11-01T12:19:00.000-07:00

The header file stdint.h, needed for larger integer types, has at line 24:

#include

but on Ubuntu you need to change this to:

#include

Cheat Sheets

2010-10-20T08:45:00.000-07:00

Here is a good place to find some math related cheat sheets.

Here is a fine place to get any cheat sheet you could want. Programming, math... just about everything.

C++ Sources

2010-10-15T07:57:00.000-07:00

Cpluplus dot comma is found here.

GSL - GNU Scientific Library

2010-10-13T15:47:00.000-07:00

A set of scientific libraries for C/C++ can be found here.

Installation and compilation instructions found here.

Cython

2010-10-13T12:07:00.000-07:00

The Cython documentation can be found here.

Gantt Chart in OpenOffice

2010-10-12T12:39:00.000-07:00

Go here.

SSH and SCP

2010-10-11T15:43:00.000-07:00

This is my ssh and scp goto page. (goto. lol.)

Microsoft fonts in Ubuntu

2010-10-08T10:47:00.000-07:00

Here is a good tutorial for loading the traditional Microsoft fonts into Ubuntu.

Open Course Work from MIT

2010-10-08T10:40:00.001-07:00

MIT's open course work website can be found here.

Call for Abstracts

2010-10-08T10:37:00.000-07:00

A good resource for writing abstract can be found here.

Variance

2010-10-07T15:27:00.000-07:00

Different programs implement variance in different ways. This can lead to confusing results when you move from one set of functions to another.

scipy.stats.cov() normalizes by (N-1), by default, bias=False. By setting bias=True, you will normalize by N.

However, scipy.var() normalizes by N.

In Octave, var( x, 0 ) biases by N-1, and var( x, 1 ) biases by N.

Matlab and R, I think, do things slightly differently.

NumPy and Input/Output

2010-10-01T14:58:00.000-07:00

Super dooper resource for NUmPy I/O can be found here. Includes information on how to interpret binary data, and use the NumPy alias of the the MATLAB fread() function.

GNU Octave

2010-10-01T12:47:00.000-07:00

GNU Octave Manual, version 3, is found here.

Tilings

2010-09-29T12:28:00.000-07:00

Girih tiles can be found here.

Penrose tiles can be found here.

Aperiodic tiles in general are found here.

CUDA, PyCUDA, and PyCUBLAS

2010-09-27T07:05:00.000-07:00

Andreas Klöckner's PyCUDA page.

Some documentation for PyCUDA and PyOpenCL.

Moire Patterns using PyCUDA and PyGame.

Use PyCUBLAS to multiply matrices smaller than 65536-by-65536.

Statistics Examples

2010-08-26T10:38:00.000-07:00

Some useful examples can be found at Dr. Hockings STAT 636 webpage.

Examples of Statistical Computing, including R and SAS and SPLUS.

Gnuplot

2010-08-26T08:32:00.001-07:00

Not So Frequently Asked Questions is a large and accessible set of examples for gnuplot.

Data Sets

2010-08-26T08:29:00.000-07:00

UCI Machine Learning Repository has a wide variety of popular data sets and lists of publications that have used those data sets.

Statistics Websites

2010-08-25T14:44:00.000-07:00

StatSoft Electronic Statistics Textbook

Social Research Methods Knowledge Base

Statistics Solutions

Scipy and NumPy Documentation

2010-08-24T08:09:00.000-07:00

Python has modules several modules, Scipy and NumPy, that offer very similar functionality to the functions offered by Matlab. (This is not a conclusive list of math/science related Python modules, but this is a good place to start.)

N.B. I found that to use scipy.linalg.eig(A), I needed to say: import scipy; from scipy import linalg. That did the old tricking.

The Tentative NumPy Tutorial may be found here.

NumPy for Matlab users may be found here.

Self Organising Neural Networks for Remote Sensing

2010-03-28T18:12:00.000-07:00

http://www.alphagalileo.org/ViewItem.aspx?ItemId=71339&CultureCode=en

LaTeX to Image

2010-03-20T14:05:00.000-07:00

Roger's Online Equation Editor lets you create images from LaTeX code.

Wavelet Coherence

2009-12-03T07:14:00.000-08:00

Wavelet coherence website: http://www.pol.ac.uk/home/research/waveletcoherence/
Guide to wavelet analysis: http://paos.colorado.edu/research/wavelets/

Research Fiasco

2009-11-04T13:13:00.000-08:00

Today, in the midst of organizing my data and images to send to Dr. Dai, I accidentally erased over half of my research. I still have the raw data, and the Fortran code to produce it, but I lost my images, sampled data, sampling scripts and gnuplot scripts for making the images possible. That rm utility really removes the shit out of things. I think I can redo all the scripts in three hours.

%%%%%

Scripts in three hours? Perhaps not so much. I wrote a better Ruby script this time for organizing the Fortran data. Its still not beautiful, or even homely, but its better.

%%%%%

Ok, its been about six hours since I thought I could do this in three hours and I've (hopefully) finished writing my Ruby scripts for organizing the Fortran data... now to tackle the gnuplot scripts. Not so frequently asked questions is a great resource for gnuplotery. This is a good site for adding LaTeX magic to your gnuplots.

%%%%%

I just finished tweaking the gnuplot script. It is now 1:32 AM. Fantastic. Just in case something happens again, I'm going to paste my codes here. (I'm kind of embarassed. I know this code is ugly, I know that it can be prettified... but I can't spend time right now making it pretty/readable.)

Ruby:

#!/usr/bin/env ruby
#
# This script takes the .dat files output by the Fortran codes and organizes them into
# several figure .dat files to be turned into figures by the figures.plt gnuplot script.
#
# To run: ruby combine-data.rb
#
def process i1E, i2E, i3E, i1L, i2L, i3L
r = /(\d*\.\d+)(\s+)(\d+\.\d+)/
o1E = i1E.readlines; o1E.map!{|l| l.scan(r)}.flatten!(1); o1E.each{|l| l.delete_at(1)}
o2E = i2E.readlines; o2E.map!{|l| l.scan(r)}.flatten!(1); o2E.each{|l| l.delete_at(1)}
o3E = i3E.readlines; o3E.map!{|l| l.scan(r)}.flatten!(1); o3E.each{|l| l.delete_at(1)}
o1L = i1L.readlines; o1L.map!{|l| l.scan(r)}.flatten!(1); o1L.each{|l| l.delete_at(1)}
o2L = i2L.readlines; o2L.map!{|l| l.scan(r)}.flatten!(1); o2L.each{|l| l.delete_at(1)}
o3L = i3L.readlines; o3L.map!{|l| l.scan(r)}.flatten!(1); o3L.each{|l| l.delete_at(1)}
a = [o1E, o2E, o3E, o1L, o2L, o3L]
end

def print_data a, out1, out2, in1E, in2E, in3E, in1L, in2L, in3L
(0..380).each do |i|
if i%12 == 0 then
out1.print a[0][i][0]+" "+a[0][i][1]+" "+a[1][i][1]+" "+a[2][i][1]+"\n"
out2.print a[3][i][0]+" "+a[3][i][1]+" "+a[4][i][1]+" "+a[5][i][1]+"\n"
else
out1.print a[0][i][0]+" "+a[0][i][1]+"\n"
out2.print a[3][i][0]+" "+a[3][i][1]+"\n"
end
end
in1E.close; in2E.close; in3E.close
in1L.close; in2L.close; in3L.close
out1.close; out2.close
end

idata1E = File.open "ord1_m201_te_results.dat", "r"
idata2E = File.open "ord2_m51_te_results.dat", "r"
idata3E = File.open "ord4_m21_te_results.dat", "r"
idata1L = File.open "ord1_m201_tl_results.dat", "r"
idata2L = File.open "ord2_m51_tl_results.dat", "r"
idata3L = File.open "ord4_m21_tl_results.dat", "r"
out1 = File.open "fig-1.dat", "w"
out2 = File.open "fig-2.dat", "w"
a = process idata1E, idata2E, idata3E, idata1L, idata2L, idata3L
print_data a, out1, out2, idata1E, idata2E, idata3E, idata1L, idata2L, idata3L

idata1E = File.open "ord1_m201_te_dim_results.dat", "r"
idata2E = File.open "ord2_m51_te_dim_results.dat", "r"
idata3E = File.open "ord4_m21_te_dim_results.dat", "r"
idata1L = File.open "ord1_m201_tl_dim_results.dat", "r"
idata2L = File.open "ord2_m51_tl_dim_results.dat", "r"
idata3L = File.open "ord4_m21_tl_dim_results.dat", "r"
out1 = File.open "fig-3.dat", "w"
out2 = File.open "fig-4.dat", "w"
a = process idata1E, idata2E, idata3E, idata1L, idata2L, idata3L
print_data a, out1, out2, idata1E, idata2E, idata3E, idata1L, idata2L, idata3L

idata2E = File.open "ord2_te_peak_results.dat", "r"
idata2L = File.open "ord2_5ps_tl_results.dat", "r"
out1 = File.open "fig-5b.dat", "w"
out2 = File.open "fig-6b.dat", "w"
a = idata2E.readlines
a.each_with_index{|x,i| if i%6 == 0 then out1.print x+"\n" end }
a = idata2L.readlines
a.each_with_index{|x,i| if i%6 == 0 then out2.print x+"\n" end }

idata1E = File.open "ord2_m26_te_results.dat", "r"
idata2E = File.open "ord2_m51_te_results.dat", "r"
idata3E = File.open "ord2_m101_te_results.dat", "r"
idata1L = File.open "ord2_m26_tl_results.dat", "r"
idata2L = File.open "ord2_m51_tl_results.dat", "r"
idata3L = File.open "ord2_m101_tl_results.dat", "r"
out1 = File.open "fig-7.dat", "w"
out2 = File.open "fig-8.dat", "w"
a = process idata1E, idata2E, idata3E, idata1L, idata2L, idata3L
print_data a, out1, out2, idata1E, idata2E, idata3E, idata1L, idata2L, idata3L

idata1E = File.open "ord4_m21_te_results.dat", "r"
idata2E = File.open "ord4_m41_te_results.dat", "r"
idata3E = File.open "ord4_m51_te_results.dat", "r"
idata1L = File.open "ord4_m21_tl_results.dat", "r"
idata2L = File.open "ord4_m41_tl_results.dat", "r"
idata3L = File.open "ord4_m51_tl_results.dat", "r"
out1 = File.open "fig-9.dat", "w"
out2 = File.open "fig-10.dat", "w"
a = process idata1E, idata2E, idata3E, idata1L, idata2L, idata3L
print_data a, out1, out2, idata1E, idata2E, idata3E, idata1L, idata2L, idata3L

Gnuplot:

#
# This is a gnuplot script to produce the .ps and .tex files
# for use in LaTeX. It must be run in the same folder as the
# figure .dat files.
#
# To run: gnuplot figures.plt
#

set term pslatex auxfile

set style line 1 lt -1 lw 1
set style line 2 lt 3 pt 6 ps 2
set style line 3 lt 3 pt 10 ps 2

set output "fig-1.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{e}\;(K)$'
set xrange [0.1:2]; set yrange [300:1400]
plot "fig-1.dat" u 1:2 t "1st Order, M = 201" w l ls 1, \
"fig-1.dat" u 1:3 t "2nd Order, M = 51" w p ls 2, \
"fig-1.dat" u 1:4 t "4th Order, M = 21" w p ls 3

set output "fig-2.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{l}\;(K)$'
set xrange [0.1:2]; set yrange [300:310]
plot "fig-2.dat" u 1:2 t "1st Order, M = 201" w l ls 1, \
"fig-2.dat" u 1:3 t "2nd Order, M = 51" w p ls 2, \
"fig-2.dat" u 1:4 t "4th Order, M = 21" w p ls 3

set output "fig-3.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{e}/(\Delta T_{e})_{max}\;(K)$'
set xrange [0.1:2]; set yrange [0:1.1]
plot "fig-3.dat" u 1:2 t "1st Order, M = 201" w l ls 1, \
"fig-3.dat" u 1:3 t "2nd Order, M = 51" w p ls 2, \
"fig-3.dat" u 1:4 t "4th Order, M = 21" w p ls 3

set output "fig-4.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{l}/(\Delta T_{l})_{max}\;(K)$'
set xrange [0.1:2]; set yrange [0:1.4]
plot "fig-4.dat" u 1:2 t "1st Order, M = 201" w l ls 1, \
"fig-4.dat" u 1:3 t "2nd Order, M = 51" w p ls 2, \
"fig-4.dat" u 1:4 t "4th Order, M = 21" w p ls 3

set output "fig-5.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel '$r$, (mm)'; set ylabel '$T_{e}\;(K)$'
set xrange [0:0.0001]; set yrange [750:1400]
plot "fig-5a.dat" u 1:2 t "1st Order, M = 201" w l ls 1, \
"fig-5b.dat" u 1:2 t "2nd Order, M = 51" w p ls 2, \
"fig-5c.dat" u 1:2 t "4th Order, M = 21" w p ls 3

set output "fig-6.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel '$r$, (mm)'; set ylabel '$T_{l}\;(K)$'
set xrange [0:0.0001]; set yrange [306:310]
plot "fig-6a.dat" u 1:2 t "1st Order, M = 201" w l ls 1, \
"fig-6b.dat" u 1:2 t "2nd Order, M = 51" w p ls 2, \
"fig-6c.dat" u 1:2 t "4th Order, M = 21" w p ls 3

set output "fig-7.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{e}\;(K)$'
set xrange [0.1:2]; set yrange [300:1400]
plot "fig-7.dat" u 1:2 t "2nd Order, M = 101" w l ls 1, \
"fig-7.dat" u 1:3 t "2nd Order, M = 51" w p ls 2, \
"fig-7.dat" u 1:4 t "2nd Order, M = 26" w p ls 3

set output "fig-8.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{l}\;(K)$'
set xrange [0.1:2]; set yrange [300:310]
plot "fig-8.dat" u 1:2 t "2nd Order, M = 101" w l ls 1, \
"fig-8.dat" u 1:3 t "2nd Order, M = 51" w p ls 2, \
"fig-8.dat" u 1:4 t "2nd Order, M = 26" w p ls 3

set output "fig-9.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{e}\;(K)$'
set xrange [0.1:2]; set yrange [300:1400]
plot "fig-9.dat" u 1:2 t "4th Order, M = 51" w l ls 1, \
"fig-9.dat" u 1:3 t "4th Order, M = 41" w p ls 2, \
"fig-9.dat" u 1:4 t "4th Order, M = 21" w p ls 3

set output "fig-10.tex"
set size 1,1
set xtics nomirror; set ytics nomirror
set format x "$%g$"; set format y "$%g$"
set xlabel 'Time, (ps)'; set ylabel '$T_{l}\;(K)$'
set xrange [0.1:2]; set yrange [300:310]
plot "fig-10.dat" u 1:2 t "4th Order, M = 51" w l ls 1, \
"fig-10.dat" u 1:3 t "4th Order, M = 41" w p ls 2, \
"fig-10.dat" u 1:4 t "4th Order, M = 21" w p ls 3

%%%%%

Like I said, its not pretty.

CSC-579 Image Classification Project

2009-11-03T10:34:00.000-08:00

I'm going to cluster the features of each image and take the medioid of the clusters to be representative features. If need be, I'll take a few random outliers to complete the set. I've made a csv list of the cleaned keys, the keys of the images that had between 300 and 1200 keypoints. I'm going to take that list and break it up into 300 individual parts so that it can be clustered by Matlab. I might figure out some batch method in Weka and use that to cluster the features for each image.

%%%%%

Dude, omg. I found Ruby for Scientific Computing and I think it might save my balls on this project. There's a post about using jRuby and WEKA and I think I might go that route. I just unistalled WEKA. I was having trouble accessing it by `java -jar weka.jar`. I had originally downloaded via my synaptic package manager, and I think it installed it so that I could pull it up with just `weka`, but all the code snippets online use the java route. I don't know, I think this will work. I'm not really sure what's going on. Anyway, I'm going to have to figure out what jRuby is all about, but that doesn't look too hard.

%%%%%

Bingo. Downloading from the site made everything better. One day I'm going to be a Linux baller and just figure this stuff out in like a minute or something.

%%%%%

I have 300 images. Each image has 300 features. Each feature is a 132-tuple.

I can't handle all of this information at once. I can't reduce the number of images. Thus, I must reduce the number of features. I can do the following: (1) feature subset selection (3) discrete wavelet transform (2) principal component analysis. I might do some sort of tree grouping thing like DIANA or something, but I can't remember where that is in the book.

%%%%%

Attribute selection in Weka is in the "Filter" button. Here is a good site to start with.

%%%%%

I did the attribute selection on one of my arff files that contained the 300 keypoints for an image, and it returned the 7 best attributes - which corresponded to the columns. Basically, the 7 best dimensions for each keypoint, not the 7 best keypoints for the whole image. The solution: transpose the arff file so that the keypoints are the attributes listed (@attr... k1, k2, k3,...,k300), and the instances are the features of each keypoint.

%%%%%

I think I'm going to use Java to process the images for key features. This site has some Java/Weka codes and applets, and miscellaneous machine learning data sets. Here is a page from the Weka site concerned with using Jython with Weka. This page talks about calling the Weka API from Matlab. Apparently Matlab is built up from the JVM, I had no idea.

%%%%%

I've tried all day to run use Weka in an automated fashion. I'm going to try using it from Matlab, and then I'm going to just do it by... by hand. I had a the jruby.jar and weka.jar file in the directory, and some APIs just wouldn't work. I got some kind of exception like this:

Exception in thread "main" java.lang.NoClassDefFoundError: weka/filters/supervised/attribute/AttributeSelection
Caused by: java.lang.ClassNotFoundException: weka.filters.supervised.attribute.AttributeSelection
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: weka.filters.supervised.attribute.AttributeSelection. Program will exit.

I don't know what any of that means... it looks like it can't find the classes I need? I don't know. I ran the kmeans script from Scientifice Computing with Ruby, but I couldn't figure out how to adapt that to attribute selection, or even filtering.

I'm going to spend the rest of the night (a) writing a filtering script (b) mining all of my 300 classes by hand. Fortunately I have plenty of Diet Coke.

%%%%%

The other group got awesome results by averaging all of their descriptor vectors into one vector per image? Did I hear that right? What the fuck?

%%%%%

Okay, I found this site that has a bunch of Weka links including manuals and powerpoints. I think I finished scripting everything. I think that all we have to do now is figure out how to use weka effectively.

%%%%%

Here's a page that shows you how to use Weka in your Java code.

%%%%%

I was having a lot of trouble with exceptions in the Java code I used in the above website. Also, apparently in JRuby arrays come in two flavors: Ruby, and Java. I needed to pass a Java String[] array to a Weka API thingy, but I couldn't figure out how to declare a strictly Java array as opposed to a Ruby array. Eventually I found this wonderful site, the JRuby wiki. I don't know how I didn't find it earlier. Anyway, to Javafy something in JRuby, just do thing.to_java.