Posts on Blog Updates

Working With Opaque Data in Go

Tue, 12 Apr 2022 21:08:42 +1000

Opaque data, is data where you don’t know what information it contains. For this article we are going to assume it has some structure: the data is in key, value form. The keys are strings. The values can be maps, arrays or single values.

In Go this can be modeled as map[string]interface{}. Most encoding packages will encode and decode into values using this type. To be more precise the two types used are map[string]interface{} for maps and []interface{} for arrays. Using these limited types to model opaque data is a good idea, even if you are not encoding/decoding, rather than using struct.

The meaning of the keys and values are not defined. But we still want to be able to work with data like this.

Merge Example:

This function will merge any data in src into dst preserving the structure, but will not overwrite plain values. All that is required is to use a type switch to deal with maps and array structure. Having this limited set of map and array types to deal with makes this easy.

 1// doesn't overwrite
 2func mergeMapStringInterface(dst, src map[string]interface{}) {
 3  for k, v := range src {
 4    if _, ok := dst[k]; ok {
 5      switch v2 := v.(type) {
 6      case map[string]interface{}:
 7        if v3, ok := dst[k].(map[string]interface{}); ok {
 8          mergeMapStringInterface(v3, v2)
 9        }
10      case []interface{}:
11        if v3, ok := dst[k].([]interface{}); ok {
12          dst[k] = append(v3, v2...)
13        }
14      default:
15        // don't overwrite plain values
16      }
17    } else {
18      dst[k] = src[k]
19    }
20  }
21}

Zero Values

It’s worth remembering that in Go indexing a map with a key which does not exist evalutes to the zero value for the type defined in the map. In other languages doing this can raise an exception.

Also when using a type assertions that assign to two variables (the special form, with boolean check), if the type assertions fails it also results in the assignment to be the zero value of the asserted type.

Combining these two features can make working with data easier

Checking a Value

In the following code if the value for “expires” doesn’t exist or if it’s not of type time.Time then we don’t run the expiry check. When using this method, what the zero value represents needs to be ruled out as being signficant, in many cases a zero value time.Time or empty string is invalid. (Note the brackets around the if clause, needed because of literal here, the parser needs this, a bit of 👀❓with Go, oldschool)

1if expires, _ := fields["expires"].(time.Time); (expires != time.Time{}) {
2  if time.Now().Before(expires) {
3    loggedIn = true
4  }
5}

The Meaning of Things

Here we creating meaning out of “public_metrics.like_count” and “likeCount”. So we don’t know the data, it could be from some API, it’s opaque. But we’ve created meaning here, they are likes 👍📈. The way of thinking about this is treating data as opaque even though you may know roughly what the data is. Probably the best use of this, is giving the user of your app the power to give meaning to data they are interested in.

 1var likeSemantics = map[string][]string{"likes" : {"public_metrics.like_count", "likeCount"}}
 2
 3func getLikes(data map[string]interface{}) (out map[string]interface{}) {
 4  for k, v := range data {
 5    switch v2 := v.(type) {
 6    case map[string]interface{}:
 7        getLikes(v2)
 8    case []interface{}:
 9        for _, v3 := range v2 {
10          if v4, ok := v3.(map[string]interface{}); ok {
11            getLikes(v4)
12          }
13        }
14    case int:
15      for k2, v5 := range likeSemantics {
16        for _, name := range v5 {
17          if k == name {
18            return map[string]interface{}{k2: v2}
19          }
20        }
21      }
22    }
23  }
24  return
25}

Top Reddit Demo

Tue, 01 Dec 2020 01:48:28 +0000

Top Reddit

This runs locally on this server as a Go service running in Docker behind an HAProxy that is routing the URL.
It’s page scraping the top links by domain over last 24 hours page on Reddit for selected domains. It uses GoQuery to do this which gives you JQuery like syntax in Go to select elements.
It updates every 30 minutes, you can see when it last ran at the bottom of the page.

Rsync File Matching

Tue, 19 Jun 2018 06:26:35 +1200

Rsync is a super effective file synchronization tool. It is versatile in how it synchronizes a source file tree to a destination file tree, updating, adding and deleting files. It uses file meta-data and block diffing of contents to provide efficient file transfers. Selecting the list of files to transfer can be difficult, for exmaple doing backups of file trees where you want to exclude some paths.

Basics

When transferring files usually what you want to do is select a local directory and specify a remote directory to copy the contents of the local directory into. In the example below copying the contents of /home/tim/ on local filesystem into /backup/tim_home on the server.

1rsync -a /home/tim/ server_hostname:/backup/tim_home

Note

tim_home will be created if it does not already exist on the server in the /backup directory.
Note that the trailing slash in the source path /home/tim/ is important without it tim_home/tim will be created on the server.
rsync -a server_hostname:/backup/tim_home/ /home/tim is reverse remote to local.

Filter Rules

Filter rule matching is a big topic in rsync, reading the man page can be useful, but it is very long and can be hard to read. This is a discussion of the most common uses. Rules are made up of patterns that match file paths (file or directories names) in a tree.

Include/Exclude

The -a flag to rsync is archive mode, that almost always should be set when syncing trees, it means that the whole tree of files and subdirectories under the source path will be included.

To exclude a path use a --exclude= argument with a pattern, this will override the -a inclusion. But this too can be overridden by using --include=. To do this it must precede the --exclude= on the command line and all parent directories specified in the rule must be included. This is shown in the example below.

`*` and `**` pattern operators

* and ** can be used in patterns to match any path. * will match any path excluding slashes. ** Will also include slashes.

Transfer root

If a pattern starts with a / it will match against transfer root, otherwise it will match from the end of the path. The example matches against transfer root.

Example

For example let’s say you have a directory /home/tim/project with a sub directory editor that you want to select but you want exclude all other files and directories in /home/tim/project from the home directory backup.

1rsync -a --include=/project/editor** --exclude=/project/** /home/tim/ server:/backup/tim_home

The important parts:

The include comes first.

The exclude does not match /project directory. If the exclude was --exclude=/project** instead, /project would be excluded even with the include rule. Because the include rule only matches the subdirectory, all parents of the directories it matches must be included for it to take effect.

The root of the patterns is from the perspective of the transfer, since we have a trailing slash in the source /home/tim/ the root is the contents of this directory.

Note

Note that in this example another directory with “editor” as the prefix would be matched too, this could be fixed by having --include=/code/editor --include=/code/editor/**.

Trying it out

Easiest way to try this out is to use the -v and -n options to rsync these add verbosity (v) which lists the files that matched and only do a dry run (n) with no actual transfer, respectively. Create a small directory tree in a temporary directory and just try to sync it including and excluding different sets of files to see how this works.

AWS EBS Magnetic HDD storage I/O performance

Wed, 31 Aug 2016 02:19:03 +1200

Using http://www.coker.com.au/bonnie++/ as a benchmark. Sequential write performance seems to be what the AWS magnetic EBS is best at.

AWS Setup

20Gb Magnetic HDD EBS volume
t1.micro instance in availability zone: eu-west-1b
command bonnie++ -s 2G -d /mnt/data/tmp

Local Setup

750Gb Toshiba Magnetic HDD 5200 RPM, 8mb buffer
Core 2 Duo P7550 2.26Ghz , 8Gb RAM
command bonnie++ -s 16G -d /tmp

The size options here are double the systems RAM, this is to try to prevent reads from disk cache.

Results

AWS EBS VS Local HDD

Sequential IO Test	AWS EBS	Local Magnetic HDD
Write Block	20,595	62,549
Read Block	10,571	63,776
Rewrite Block	8,109	30,285
Per Char
Write Per Char	17,273	585
Read Per Char	10,213	2,322
Random Seeks	399	109.2

Values are in Kb/s except for Random Seeks which is seeks/s.

Observations

Sequential write performance seems to be what the AWS magnetic EBS is best at, it reads at half the rate. Local HDD read/write is more symmetrical.

Inefficient I/O code (reading/writing one byte each I/O call) really hurts performance on local HDD, but on AWS EBS it seems this is mostly negated by the underlying virtualization technology doing the i/o.

In general Magnetic EBS is three times slower at writing and six times slower at reading data than a local HDD.

Not sure about seek performance see the notes below.

Notes

There are some differences in bonnie++ versions, also the filesystem on the local HDD was nearly full a the time of the tests.

Bonnie++ Output

AWS:

Version 1.03e       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ip-172-31-12-48  2G 17273  29 20595   3  8109   1 10213  17 10571   0 399.5   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
ip-172-31-12-48,2G,17273,29,20595,3,8109,1,10213,17,10571,0,399.5,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Local:

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
minerva         16G   585  90 62549  10 30285   5  2322  95 63776   7 109.2   3
Latency             22012us    2442ms    3533ms   19712us   32693us    1609ms
Version  1.97       ------Sequential Create------ --------Random Create--------
minerva             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18445  34 +++++ +++ +++++ +++ 27279  49 +++++ +++ 30763  45
Latency             14623us    1209us    1105us    1175us     438us     516us
1.97,1.97,minerva,1,1485665620,16G,,585,90,62549,10,30285,5,2322,95,63776,7,109.2,3,16,,,,,18445,34,+++++,+++,+++++,+++,27279,49,+++++,+++,30763,45,22012us,2442ms,3533ms,19712us,32693us,1609ms,14623us,1209us,1105us,1175us,438us,516us
t

SWT

Wed, 03 Aug 2016 02:28:28 +1200

Eclipse SWT cross platform (OSX/Linux/Windows) native controls for Go: https://github.com/timob/swt. Check the Readme, on how to get started.

This package is used by Cyan for its UI. Go's concurrency, synchronization and function closure features are useful in UI event/callback code. It is very suited to this kind of programming. UIs can be made very responsive, by easily running things in the background from an event handler.

Having a native Go UI option would be better, but this will take time.

Example

Demonstrates adding tab and tree controls to a window. (from examples dir)

package main

import (
	"github.com/timob/javabind"
	"github.com/timob/swt"
	"log"
	"os"
)

// Example as shown at https://www.eclipse.org/swt/

func main() {
	err := javabind.SetupJVM(os.Getenv("CLASSPATH"))
	if err != nil {
		log.Fatal(err)
	}

	var SWT *swt.SWT

	display := swt.NewWidgetsDisplay()
	shell := swt.NewWidgetsShell3(display)
	shell.SetLayout(swt.NewLayoutGridLayout2(2, true))

	tabFolder := swt.NewWidgetsTabFolder(shell, SWT.NONE())
	tabFolder.SetLayoutData(swt.NewLayoutGridData5(SWT.FILL(), SWT.FILL(), true, true, 2, 1))
	item := swt.NewWidgetsTabItem(tabFolder, SWT.NONE())
	item.SetText("Widget")

	composite := swt.NewWidgetsComposite(tabFolder, SWT.NONE())
	composite.SetLayout(swt.NewLayoutGridLayout())

	tree := swt.NewWidgetsTree(composite, SWT.BORDER())
	item.SetControl(composite)
	tree.SetHeaderVisible(true)
	tree.SetLayoutData(swt.NewLayoutGridData4(SWT.FILL(), SWT.FILL(), true, true))
	column1 := swt.NewWidgetsTreeColumn(tree, SWT.NONE())
	column1.SetText("Standard")
	column2 := swt.NewWidgetsTreeColumn(tree, SWT.NONE())
	column2.SetText("Widget")
	branch := swt.NewWidgetsTreeItem(tree, SWT.NONE())
	branch.SetText2([]string{"Efficient", "Portable"})
	leaf := swt.NewWidgetsTreeItem2(branch, SWT.NONE())
	leaf.SetText2([]string{"Cross", "Platform"})
	branch.SetExpanded(true)
	branch = swt.NewWidgetsTreeItem(tree, SWT.NONE())
	branch.SetText2([]string{"Native", "Controls"})
	leaf = swt.NewWidgetsTreeItem2(branch, SWT.NONE())
	leaf.SetText2([]string{"Cross", "Platform"})
	branch = swt.NewWidgetsTreeItem(tree, SWT.NONE())
	branch.SetText2([]string{"Cross", "Platform"})
	column1.Pack()
	column2.Pack()

	item = swt.NewWidgetsTabItem(tabFolder, SWT.NONE())
	item.SetText("Toolkit")

	button := swt.NewWidgetsButton(shell, SWT.CHECK())
	button.SetText("Totally")
	button.SetSelection(true)
	button.SetLayoutData(swt.NewLayoutGridData4(SWT.CENTER(), SWT.CENTER(), false, false))
	listener := NewSelectionListener(button)

	button = swt.NewWidgetsButton(shell, SWT.PUSH())
	button.SetText("Awesome")
	button.SetLayoutData(swt.NewLayoutGridData4(SWT.CENTER(), SWT.CENTER(), false, false))

	shell.Pack()
	shell.Open()

	for !shell.IsDisposed() {
		if !display.ReadAndDispatch() {
			display.Sleep()
		}
	}
}

Example apps:

AWS Plus: small utility to work with AWS

Windows 10

Cyan IDE

Fri, 29 Jul 2016 11:05:05 +1200

Cyan is an upcoming Go IDE/Editor written in Go. Designed with a minimalist / responsive UI. It's designed not interfere with your CLI tools.

I use this daily for coding in Go while developing it along the way, so the features at the moment are what I find I most need. All the UI components are keyboard navigatable (why use a mouse?). The file navigation is inspired from IntelliJ, access to the directory where you are working, with quick navigation to other directories from the buttons on top. All the undo history is written in Go, ie. if you save edit then undo back to save point it won't ask you to save. The scrolling is done with Go callbacks to give you nice clamped at 60 fps scrolling. Dialog boxes get in the way so are not there, all messages are shown in responsive way. The design is to use Go routines to do all non-interactive tasks in the background and prioritize user input. What's shown here is just the start.

It uses JNIGI to use Java which is used for SWT UI components.

Features

Cross platform Linux/Windows/OSX
Syntax highlighting using core Go parser
Persistent saving of application state over restarts
File browser, quick access to directories in the current file path
Search/Replace
Smart indentation, home
UTF 8 support

Download

coming soon

Future

Improved syntax highlighting (right now it limits knowledge to current file)
Always on Gofmt
Auto sync of editor buffers with files on the filesystem
Code completion
Indicate compiler errors in editor
Debugger with breakpoint support
Custom code analysis / scripting
Source code navigation / sidebar code outlines
Multi file refactoring, renaming etc..
Display Git revisions/branches

Android USB Network Connection

Thu, 23 Jun 2016 08:43:03 +1200

It's possible to create a network connection between an Android phone and your computer over USB. Also some later Android versions have this option in the settings menu. This method uses the RNDIS protocol on a a jail broken phone. This information is available elsewhere but it's somewhat hard to find. So I am reposting here.

echo 0 > /sys/class/android_usb/android0/enable
echo 18d1 > /sys/class/android_usb/android0/idVendor
echo 4e23 > /sys/class/android_usb/android0/idProduct
echo rndis > /sys/class/android_usb/android0/functions
echo 224 > /sys/class/android_usb/android0/bDeviceClass
echo 1 > /sys/class/android_usb/android0/enable
setprop sys.usb.state rndis
netcfg rndis0 up
ifconfig rndis0 10.100.100.1 netmask 255.255.255.0

This is the code that works on my Samsung Galaxy Nexus (yes a very old phone) running Android 4.3. I imagine the process is similar on other phones, but maybe the Vendor/Product codes need to change.

After the above script is run on your phone just connect it's USB cable to your computer. On Linux the interface will show up as for example usb0. I think the interface is provided by the rndis_host, cdc_ether and usbnet kernel modules which hopefully load automatically. Just configure it with a corresponding address on the subnet and you have a IP over USB to your phone.

Useful for sharing your phones 3g/Wifi or app development/hacking.

Let's Encrypt

Thu, 05 May 2016 08:50:59 +1200

Tekao.net now uses Letsencrypt! Let's Encrypt issues short term certificates, three months i think, for websites that can prove they own a domain. You can check the letsencrypt.org certificate for the page you are reading now by clicking on the padlock to left of the address bar in your browser.

Using rsc.io/letsencrypt it's extremely simple to have a Golang webserver that uses the standard HTTP package be supported. It all happens on demand with no configuration: Given a HTTPS request to your server, it will ask for a certificate and prove to Let's Encrypt that you own the host whose name is specified in the request. It proves this by serving specific content reachable from Let's Encrypt servers using the host name.

This certificate will be used on subsequent requests. I guess until it expires.

Check out Let's Encrypt website for details of the certificate issuing protocol.

Very cool.

Modern ls

Thu, 25 Feb 2016 08:18:08 +1300

List directory Unix utility written in pure Go. Has some extra options: Humanized relative timestamps, path mode and cross platform.

"I have it in my PATH!"👍

https://github.com/timob/ls

Some Features:

Compatiblity with GNU ls
-R has changed, meaning recursively lists all files with relative path names as one group. This means sort options such as by time (-t) can show most recent file modified under a path. (Useful for example to see latest logs in /var/log)
-O only list entries starting with .
Color output
Works on Linux/Windows/OSX

Screenshot showing -1PRtr options in effect, showing 1 column, only file paths, recursive subdir, sort by time reversed:

Showing now relative Humanized timestamps for last modified:

Improved layout:

Compared to standard MacOS ls:

Sindex

Thu, 25 Feb 2016 05:42:40 +1300

Slice Indexing library for Golang. SIndex is a slice indexing library. It maintains an ordered list, by mapping list positions to slice indexes. sindex.Interface implementation types: List (Basic slice list), LinkedList (Todo)

https://github.com/timob/sindex

Documentation

https://godoc.org/github.com/timob/sindex

Example

func ExampleList() {
    bytes := []byte("helloworld")
    bl := sindex.NewList(&bytes)
    bytes[bl.Insert(5)] = ' '
    bytes[bl.Append()] = '!'

    fmt.Println(string(bytes))

    for iter := bl.Iterator(0); iter.Next(); {
        fmt.Print(string(bytes[iter.Pos()]))
    }

    // Output:
    // hello world!
    // hello world!
}

Nvidia proprietary drivers and Ubuntu 12.04 redraw

Fri, 20 Jun 2014 16:38:48 -0700

One thing that took me a long time to fix was, small of regions windows not getting redrawn when modified. After trying countless things, the option was in compiz config settings manager (ccsm).

To fix, in CCSM->Workarounds it i set:
"Force complete redraw on initial damage"
"Force fullscreen redraws (buffer swap) on repaint"

Now this works very nicely indeed....

Thanks to all the people working on Ubuntu/Linux software!!!

Update:
Using nvidia-settings, changing power saving mode to maximum performance fixes this problem. It gives better performance than below but i assume uses more power.

Tweaking Ubuntu 12.04 to run on a Macbook

Mon, 05 Aug 2013 01:28:11 -0700

Configuration tweaks for Ubuntu.

Update 2

Ubuntu looks better with it's default color profile. Probably not worth changing.
With the latest NVidia drivers I've found the options I set below are not needed (apart from the brightness one).
Also see new post on how to get Nvidia/Compiz to work together nicely.

Updated

After using the above setup for a while, I've updated some things.

I found you can't really replicate the way the fonts look in OSX, so it's better to set your font configuration to using hinting (I am using medium hinting). This makes the fonts look crisper more like Windows. With this setting using a default of DejaVu Sans as your default Sans font in Chrome looks good. In fact most of the sans fonts that are installed by default look better with some hinting.

The trackpad mtrack driver works much better if you install it from source:

sudo apt-get build-dep xserver-xorg-input-mtrack

Download the source from https://github.com/BlueDragonX/xf86-input-mtrack/ .

cd xf86-input-mtrack/
libtoolize
aclocal
autoconf
automake -c --add-missing
./configure
make

I then just replaced the driver

sudo mv .libs/mtrack_drv.so /usr/lib/xorg/modules/input/

Original post:

I have a 2009 Macbook its model number 6,2. Running Ubuntu 12.04 Precise Pangolin.

The OS install is pretty straightforward. The instructions are here. Notes on those install instructions:

You have to burn a CD, booting off of USB never worked from me.
I used the specific install ISO for 64bit Mac computers. Available here.
The Ubuntu installer asks you to create an an extra parition for booting, in addition to the normal filesystem/swap parititons.

At the end you should end up with a working Ubuntu Install. But here are some tweaks to make the hardware work well.

Graphics card

The model I have has a NVidia 9400M graphics card which shares memory with system RAM. The Nouveau drivers as well as the NVidia driver version that is installed by default cause instability after a while resulting in lockups. I ended up installing the latest Linux driver from the NVidia website. I am using version 304.37. This worked well but I still had some X server crashes, while not as bad as complete lock ups are still annoying. I've disabled some options in my xorg.conf in the nvidia device section:

Option "IndirectMemoryAccess" "Off"
Option "NoFlip" "True"
Option "DynamicTwinView" "Off"

This seems to make my system more stable. I also added:

Option "RegistryDwords" "EnableBrightnessControl=1"

To fix the brightness controls.
If the Nouveau driver works for you I would stick to this.

Display

By default the color profile of the screen is bluish, different from the warmer profile in OSX. To get the same color profile, boot OSX and save the default color profile to an ICC file. This can be done by using system preferences -> display -> color -> open profile -> save as. Then you can load this file in Ubuntu, I used the xcalib utility to do this, available from the apt repositories.

I found the fonts by default didn't look as good as on OSX, especially in the web browser using the Macbook laptop display. The best results I achieved was to use grayscale anti-antialiasing, except for font sizes 13 point and smaller which I set to use subpixel anti-aliasing. And to not use any hinting. The second thing was to set the default sans-serif font in the Chrome web browser to "Nimbus Sans L" this seems most like Helvetica/Arial used on the web alot. (See Update below.)

I configured the fonts using /etc/fonts/conf.d. Here is my listing of that directory:

10-no-sub-pixel.conf 10-unhinted.conf 11-lcd-filter-lcddefault.conf 20-fix-globaladvance.conf 20-unhint-small-vera.conf 30-subpixel-for-small.conf 30-urw-aliases.conf 40-nonlatin.conf 45-latin.conf 49-sansserif.conf 50-user.conf 51-local.conf 53-monospace-lcd-filter.conf 60-latin.conf 64-01-tlwg-kinnari.conf 64-02-tlwg-norasi.conf 64-11-tlwg-waree.conf 64-12-tlwg-loma.conf 64-13-tlwg-garuda.conf 64-14-tlwg-umpush.conf 64-21-tlwg-typo.conf 64-22-tlwg-typist.conf 64-23-tlwg-mono.conf 65-fonts-persian.conf 65-khmer.conf 65-nonlatin.conf 69-unifont.conf 70-force-bitmaps.conf 80-delicious.conf 89-tlwg-garuda-synthetic.conf 89-tlwg-kinnari-synthetic.conf 89-tlwg-loma-synthetic.conf 89-tlwg-umpush-synthetic.conf 89-tlwg-waree-synthetic.conf 90-fonts-nanum.conf 90-synthetic.conf 90-ttf-bengali-fonts.conf 90-ttf-devanagari-fonts.conf 90-ttf-gujarati-fonts.conf 90-ttf-kannada-fonts.conf 90-ttf-malayalam-fonts.conf 90-ttf-oriya-fonts.conf 90-ttf-punjabi-fonts.conf 90-ttf-tamil-fonts.conf 90-ttf-telugu-fonts.conf 99pdftoopvp.conf README

The only custom file in that directory is 30-subpixel-for-small.conf which contains:

I also used the myunity utility ovailable from the repositories to configure the fonts using the GUI to match the fontconfig files. I just set the Hinting to none and Antialiasing to grayscale.

Trackpad

The default Xorg synaptics driver was not working that well, it was too sensitive and didn't handle multitouch well. I ended up removing the xserver-xorg-input-synaptics package and installing the xserver-xorg-input-mtrack using apt.

This driver has a configuration file at /usr/share/X11/xorg.conf.d/50-mtrack.conf.

I am still tweaking the options here, but the trackpad is ok to use now.

Day to day use

So due to some stupidity on my part my OSX partiton doesn't boot any more. So I am using my Ubuntu/Mac every day, and I think it's interface is nearly as good as OSX. That combined with IMHO a better underlying infrastructure in Linux ie system/network software, open file systems and better compatibility and user software that works for you, like being able to copy songs from your Ipod, makes the system on balance better than OSX/Mac.

Macbook RAID

Wed, 08 May 2013 02:18:40 -0700

With some Macbooks it is possible to remove the DVD drive and install a second HDD. You need to buy a slimline SATA adapter to attach the HDD to the DVD power cable.

http://www.amazon.com/Micro-SATA-Cables-Slimline-Adapter/dp/B003YVP8VS/ref=aag_m_pw_dp?ie=UTF8&m=A1UCLUF7KW7AYG

I've installed a 128GB Vertex 4 SSD and a 750GB HDD Drive in my Macbook using the above method. I've partitioned the HDD into a 128GB partition and another partition with the remaining space (actually there is a small 100mb boot partition on each drive too). I've then setup a Hybrid SSD/HDD RAID 1 array as described here http://www.tansi.org/hybrid/. This method gives you the read speeds of an SSD with the reliability of RAID 1.

On the RAID device i have my system/home filesystems, and then on the big non-RAID partition on the HDD i've got my big media files/data files etc. This setup is really nice.

Once thing I've notice though is sometimes when a write happens to the RAID device i get some IOwaits if i am reading from the non-RAID part of the HDD. This tends to be noticeable when playing a movie. So to get around this you can set the read-ahead for the device using hdparm. Eg hdparm -a 1024 /dev/sdb. This seems to fix the problem as enough data is present for the movie player to keep reading data while the IOwaits happen.

Indexing time periods using MySQL and GNU date

Tue, 28 Feb 2012 17:38:29 -0800

Periodic data in a database is something i use quite a bit. The problem is using a date time column or a date and period column for each table is quite cumbersome. To make things easier create a time dimension table:

CREATE TABLE time (
  time_id mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
  date date NOT NULL,
  tp tinyint(3) unsigned NOT NULL,
  year smallint(5) unsigned NOT NULL,
  month tinyint(3) unsigned NOT NULL,
  day_of_year smallint(5) unsigned NOT NULL,
  day_of_week tinyint(3) unsigned NOT NULL,
  PRIMARY KEY (time_id),
  KEY date (date) USING BTREE
);

This indexes every date, and tp column to a time_id. In this case tp is a number from 1 to 48 for half hour periods through out the day. The rest of the columns are use to make queries using time clauses on the data easier.

To populate this table, use the excellent gnu date command to specify a starting date and a count of the number of days and a format string that can be used to insert into the database. Here are the commands typed into bash to create the sql file to load into the table:

$ daycount=$((366*17))
$ tpcount=48
$ { for ((i=0;$i<$daycount;i=$i+1)); do 
          for ((j=1;j<=$tpcount;j=$j+1)); do 
              gdate -d "1996-01-01 + $i days" +"insert into time (date, tp, year, month, day_of_year, day_of_week) values ('%Y-%m-%d', $j, %Y, %m, %j, %u);"; 
          done; 
      done } > time_data.sql

This creates a file with insert statements for 17 years starting from 1996-01-01 for every half hour. After loading into the database, a check is to see how many periods / year there are:

mysql> select year, count(1) from time group by year order by year;
+------+----------+
| year | count(1) |
+------+----------+
| 1996 |    17568 |
| 1997 |    17520 |
| 1998 |    17520 |
| 1999 |    17520 |
| 2000 |    17568 |
| 2001 |    17520 |
| 2002 |    17520 |
| 2003 |    17520 |
| 2004 |    17568 |
| 2005 |    17520 |
| 2006 |    17520 |
| 2007 |    17520 |
| 2008 |    17568 |
| 2009 |    17520 |
| 2010 |    17520 |
| 2011 |    17520 |
| 2012 |    17568 |
| 2013 |      576 |
+------+----------+
18 rows in set (0.11 sec)

Here the leap years have 48 more periods.
Given this price, quantity test data for the first 3 months of the year 2000:

create table price (time_id mediumint not null, price decimal(8,2));
create table quantity (time_id mediumint not null, quantity int);

insert into price select time_id, cast(rand()*1000 as decimal(8,2)) from dim.time where year = 2000 and month in (1,2,3);
insert into quantity select time_id, round(rand()*1000) from dim.time where year = 2000 and month in (1,2,3);

Queries like this make it easy to get information:

mysql> select date, tp, price*quantity cost from price natural join quantity natural join dim.time order by date,tp limit 3;
+------------+----+-----------+
| date       | tp | cost      |
+------------+----+-----------+
| 2000-01-01 |  1 | 111661.86 |
| 2000-01-01 |  2 | 452187.36 |
| 2000-01-01 |  3 |  57724.85 |
+------------+----+-----------+
3 rows in set (1.28 sec)

or:

select date, tp, a.price, b.price, round((b.price / a.price),2) * 100 price_change from price a, price b, dim.time where b.time_id = a.time_id + 48 and b.time_id = time.time_id limit 10;
+------------+----+--------+--------+--------------+
| date       | tp | price  | price  | price_change |
+------------+----+--------+--------+--------------+
| 2000-01-02 |  1 | 979.49 | 923.78 |        94.00 |
| 2000-01-02 |  2 | 509.22 |  10.22 |         2.00 |
| 2000-01-02 |  3 | 607.63 | 279.77 |        46.00 |
| 2000-01-02 |  4 | 510.47 | 368.20 |        72.00 |
| 2000-01-02 |  5 | 729.49 |   1.70 |         0.00 |
| 2000-01-02 |  6 | 116.03 | 903.87 |       779.00 |
| 2000-01-02 |  7 | 391.66 | 514.25 |       131.00 |
| 2000-01-02 |  8 | 610.23 | 859.65 |       141.00 |
| 2000-01-02 |  9 | 876.16 | 755.51 |        86.00 |
| 2000-01-02 | 10 | 550.10 | 198.58 |        36.00 |
+------------+----+--------+--------+--------------+

As well as easy to query, it can make your data tables smaller by only having a integer of the right length to specify the time.

OWA Mail Checker, keep alive

Wed, 25 Jan 2012 17:45:28 -0800

This Greasemonkey script keeps your Outlook 2010 OWA session alive by refreshing every 60s. It's basically an update of http://blog.hacker.dk/2007/10/greasemonkey-owa-mailcheck-keepalive/ to work with Outlook 2010. Make sure you use the light version of the app when logging in.

It will also show the number of unread emails in the title:

Download here:
http://userscripts.org/scripts/source/124122.user.js

Posts on Blog Updates

Working With Opaque Data in Go

Merge Example:

Zero Values

Checking a Value

The Meaning of Things

Top Reddit Demo

Top Reddit

Rsync File Matching

Basics

Note

Filter Rules

Include/Exclude

* and ** pattern operators

Transfer root

Example

The important parts:

Note

Trying it out

AWS EBS Magnetic HDD storage I/O performance

AWS Setup

Local Setup

Results

Observations

Notes

Bonnie++ Output

SWT

Example

Example apps:

Windows 10

Cyan IDE

Features

Download

Future

Android USB Network Connection

Let's Encrypt

Modern ls

Sindex

Documentation

Example

Nvidia proprietary drivers and Ubuntu 12.04 redraw

Tweaking Ubuntu 12.04 to run on a Macbook

Update 2

Updated

Graphics card

Display

Trackpad

Day to day use

Macbook RAID

Indexing time periods using MySQL and GNU date

OWA Mail Checker, keep alive

`*` and `**` pattern operators