Pulse APM

Pulse – Application Heartbeat Monitor

Pulse is a new library I wrote in a weekend because I couldn’t find anything out there which performs this simple service. And I was tired of getting calls from GoGet on my day off (from my job as CTO of Fleetcutter).

I needed to monitor a somewhat fragile, legacy Windows application server which already had a heartbeat database record it updates every minute. Great, but nothing was monitoring the heartbeat record, so no cigar there. I also have a bunch of other servers written in different languages hosted on different platforms. I also have a few separate support companies (customer service, managed hosting and 24-hr IT support and application triage) so I’d like to notify each of specific levels/types of problems, and each has their own management infrastructure and contact preferences.

Therefore I wanted something that was simple and reliable but extendable and flexible.

So I wrote Pulse: https://bitbucket.org/scipilot/pulse

I believe the architecture is a good example of Dependency Injection, the Strategy Pattern, the Repository Pattern, IoC Container (flexible), and automated testing (reliable). In fact, during initial development I wrote all the interfaces first, then the test platform, then began to fill in the functionality while writing the tests, in a loose form of TDD. I like the idea that the code was initially run solely via the test platform (i.e. no dummy manually run top-level-scripts). This meant that when I eventually deployed the system it pretty much worked first time! The main issues discovered on deployment were (apart from one major foobar) deployment-related things like the autoload location and the need to ‘park’ work-in-progress configurations.

In fact, in its first day of operation it caught a major outage seconds after the cause and several minutes before any humans noticed. But… I ignored the notification emails thinking they were false positives!

But now I’m out of the loop, I’m fully off-duty on an island in the Indian Ocean. Now my 24hr support has a new pair of ears stethoscoped directly into the butterflies that beat in the heart of our veteran application.

Overview

Pulse is a standalone component which provides simple application/service heartbeat monitoring.

Simply put: it checks your application is still alive.

  1. A timestamp is updated by your service or the daemon, and a monitor daemon regularly checks the hearbeats are current.
  2. If a service fails to register a heartbeat within the configured thresholds, notifications are triggered.

Client service/applications can have one or multiple heartbeats monitored – called Pulses.

All components are customisable: notifiers, config, log, storage (via dependency-injection in an IoC container). The notifications are the most likely to need customisation, e.g. sending alerts to your own dashboard API. If you customised the storage engine, you could perhaps integrate saving heatbeats from another source.

e.g. 1 – Integrated Application Heartbeat Update and Monitor Service Sending Email Notifications

This example shows a PHP application using the Pulse class to store regular heatbeats. The monitor service is using the email notifier and regularly checks the heartbeat database.

example 1

Note if the heartbeat database is a simple file (serialised, sqlite) then the monitor service would need to be installed locally.

e.g. 2 – Multiple Application Heartbeats and Multiple Notification Types

This example demonstrates scaling the application to monitor multiple applications and use different notification strategies to send different levels of alerts, or notificiations from the different applications, to different support staff.

example 2

e.g. 3 – Remote Probe Ability for Legacy or non-PHP Apps

This example shows the Probe feature querying a remote application’s status, without integrating into it.

example 3

Alarms

There are three states your service can be in:

  1. OK – All is well. Nothing to report.
  2. ALERT – Service degraded. Things are getting bumpy, but you might not want to call out the troops yet…
  3. ALARM – System down. Urgent panic, wake up the boss.
  4. OK again – on returning to normal you will get one notification.

You only get a notification when the service changes state.

You don’t have to use the ALERT state, but it gives you a bit of extra granularity to triage any flapping. Often a system will have a ‘grey area’ where you don’t want to really panic everyone, but perhaps you should keep an eye on things. This ALERT state can therefore be targetted at a different response team (see below).

Installation

Standard composer installation:

> composer require scipilot/pulse

or more likely you will fork it and require your own.

Usage

There are two halves to the system:

1. Heatbeat Update

This can be done in one of two ways:

a) “Locally”: Each service can require the component and call Pulse->beat() regularly. Of course this only works for PHP services, and ones you can modify by integrating with this component.

b) “Remotely”: The Pulse daemon can poll your system using “Scopes” to update the heartbeats. This is the most unintrusive method, and can interface with existing database timetamps for example.

Scopes allow you to implement custom probes into other systems to extract heartbeats. Currently there is only one: a DatabaseRecordScope which queries an SQL DB for a single timestamp.

The IoC model allows you to easily implement another interface to query an API, top, PID file etc.

2. Pulse Monitoring

The monitoring daemon must be run in a supervised process (or it can be polled by your own scheduler).

Both uses obviously must share the same config, by default these are files in the storage folder, so the same installation must be used. If you need to separate the daemon from your services, you would need to implement another Storage strategy (i.e. implement a Storage class and inject into a modified container).

Usage Examples

See /bin/example_service.php and /bin/daemon.php respectively.

  1. Copy the sample config to config.json
  2. Edit your email address in the default ‘to’.
  3. Run php /bin/daemon.php &; php /bin/example_service.php &
  4. Check your inbox!

Also see the Unit Tests for usages (although I’ve always hated reading people say that).

Config

Each Pulse is configured with alert and alarm thresholds, friendly name, unique ID.

Notifications also require some configuration too (see below).

See storage/config-SAMPLE.json for a bootstrap config.

You can either hand-craft a json config file, or programmatically create Pulse or Scope objects, which can persist their own configuration for re-load later. See the PulseRegistry and ScopeRegistry.

Notifications

Email Notifications

Initially there is only an EmailNotify implementation. it requires the following configuration:

notify.EmailNotify.default.email.to
notify.EmailNotify.default.email.subject            - default subject
notify.EmailNotify.default.email.body               - with placeholders: _TYPE_, _LEVEL_, _NAME_, _MESSAGE_

Body template e.g.:

"Heads up! Alert type: _TYPE_, at level: _LEVEL_, was received for service _NAME_. \n Details are as follows: _MESSAGE_"

Optionally you can customise any specific messages for both monitoring alerts, and internal errors:

notify.EmailNotify.<TYPE>.<LEVEL>.email.to          - specific alert overrides
notify.EmailNotify.<TYPE>.<LEVEL>.email.subject
notify.EmailNotify.<TYPE>.<LEVEL>.email.body        - with placeholders: _TYPE_, _LEVEL_, _MESSAGE_

e.g. you might want outage alarms sent to L1 Support pagers, but internal warnings sent to developers or devops.

See ‘INotify’ for these constants.

Logs

The system logs INFO by default into the storage folder.

This will just be service status changes, and any system errors (e.g. notification send failures). They will be pretty small (if your services are stable), but you might want to rotate them anyway.

You can change the level of detail up to debug or down to just warnings, by calling app->log->setVerbosity(). (There is no config option for this, at present)

Architecture

- IoC App Container
    - Config DI
    - Log DI
    - Storage DI
    - Notification DI
- PulseRegistry
    - Pulse->beat
- ScopeRegistry
    - Scope->Listen
- Monitor Daemon->run:
    - Listen to Scopes
    - Scan Pulses
- Feed API (TODO!)

The Feed API has not yet been implemented. The idea is to allow remote applications both register Beats, and fetch Pulse status.

Roadmap

  • NewRelicNotify
  • SMSNotify
  • DatabaseStorage
  • RestNotify
  • StatusPageIONotify
  • Alternate language service bindings (Python, JavaScript)

Tests

There are extensive (complete?) PHPUnit tests which are mostly standalone, except for some email and database configurations required for full-system testing.

To run the DB-oriented tests you will need the example service database installed:

CREATE DATABASE pulse_test;
CREATE TABLE `system` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `last_seen` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;
INSERT INTO `system` (`id`, `last_seen`) VALUES ('1', NOW());

CREATE USER 'pulse_test'@'localhost' IDENTIFIED BY 'pulse_test';
GRANT SELECT, UPDATE ON `pulse\_test`.* TO 'pulse_test'@'localhost';
FLUSH PRIVILEGES;

See storage/config-UNITTEST.json for more details or to customise it.

Licence

This library is licenced under the LGPL v3.

Simply put: you can use this software pretty much as you wish as long as you retain this licence, make the source available and don’t blame me for anything.

I’d also really like to see any changes / fixes / suggestions – thanks!

https://bitbucket.org/scipilot/pulse

Posted in Projects | Tagged , , , | Leave a comment

Cutlass – Cut Less!

While re-building my house I became obsessed with getting efficiency out of the timber 10materials I was using when planning cutting lists. Most people would want to achieve an efficient cutting plan because of cost-efficiency, effort-reduction or ecological conservation, and of course that was true for me but being a huge nerd I was even more intrigued by the devilishly difficult planning problem itself. Surely an algorithm could be produced to make cutting lists easy to produce, and thereby save enormous amounts of trees?

So what’s the problem?

The Problem

Well I have a) a long list of measured lengths to make tongue and grooved lining boards fit horizontally around our many doors and windows and b) a bunch of raw timber in standard lengths. The challenge is to cut the timber so the waste offcut each time can also be used.

Cutlass Stock Illustration

The Solutions

The woodyard-recommended solution is to order 10% more timber than you need and hope. Then you just throw away the waste. Of course they would recommend you spend 10% more, but I just don’t find this an acceptable attitude.

After trying to figure out a process myself for three complex rooms and 150m of timber I almost had a brain meltdown, I did some research into online tools and then common algorithms. Of course this is a very old problem!

At first I researched the “cutting stock problem” but this is mostly oriented towards reducing waste from cutting 2D sheets of fabric, timber or metal and incorporates complex requirements like reducing knife-changing interruptions.

I found it difficult to find a 1D cutting algorithm, until I came across the “bin packing problem”, which is functionally equivalent to this problem. All you have to do is tilt your head on one side, and view the ‘bins’ as the original stock and the vertical heights of the objects being placed in the bins as the planned lengths required.

Cutlass Stock Illustration (1)

The first-fit algorithm is pretty simple: you find the next longest length in your requirement plan and take that away from the first stock with enough left (i.e. put it in the first bin). Repeat.

It’s not guaranteed to find the overall best usage for all items. In theory, if you tried all possible combinations you could probably get a lot more zero-waste planks, but this sounds like one of those more-atoms-than-the-universe scenarios to me. I might add some level of re-try to it later or do more research into simultaneous equations, which I suspect could offer a solution.

My Implementation

Of course programming this will have a few more practical details like initially sorting the required lengths and handling problems like running out of stock, but that’s the basic algorithm. So I cracked open a new Bootstrap theme and wrote it in Javascript to avoid any unnecessary back-end requirements:

/*
 * Best fit tries to fill the available slots with the minimum remaining.
 * Slots are only brought online when no previous slots can be fitted into.
 * You start with one slot online, and put the first item into it.
 */
bestFitAlgorithm: function(aTimberList, aCuttingList){
  // first we assume there's only one entry in the timber list - phase one!
  var iPlankSize = aTimberList[0][0];
  var iNoPlanks = aTimberList[0][1];
  var aPlanks = [];
  var count, len, min, p, remaining;

  // iterate over the required lengths
  for(var i in aCuttingList){
    len = aCuttingList[i][0];
    count = aCuttingList[i][1];
    while(count--){
      // Check all online planks in order
      min = iPlankSize;
      p = null;
      for(var j in aPlanks){
        remaining = aPlanks[j].remaining - len;
        if(remaining >= 0 && remaining < min){
           min = remaining;
          p = j;
        }
      }
      if(p != null){
        // p is the best fit.
         aPlanks[p].remaining = min;
        aPlanks[p].pieces.push(len);
      }
      else {
        // No fit found, bring another plank online
        aPlanks.push({remaining:iPlankSize-len, pieces:[len]});
      }
    }
  }
  return aPlanks;
}

I surrounded this with some self-unit-testing, a JQuery plugin wrapper and a basic form UI to talk to the humans, and went to work.

The Real World Beta Test – My Lining Boards

The plan that I actually needed was as follows, with length of item in mm (len) and how many (off).

len  off
3100 2
2900 1
360 1
305 16
1300 2
245 13
3740 4
440 8
1070 1
190 7
900 1
4000 2
3820 5
140 9
110 15
1200 2
230 15
120 15
1450 23
500 15
2820 1
120 15
380 15
200 9
1150 3
1150 6
140 9
1270 18

My stock was 5400mm lengths of pre-primed T&G grooved panelling. The algorithm came up with the following cutting list, where the first number is the stock plank index, then the length to cut and finally the waste.

I added a default 3mm kerf (the width of the cut made by the saw) which is easy to forget when planning cuts and soon adds up to a significant amount – especially when you are trying to achieve near zero-wastage with many small pieces (see bottom planks).

0: 4000,1300 waste:100
1: 4000,1300 waste:100
2: 3820,1450,120 waste:10
3: 3820,1450,120 waste:10
4: 3820,1450,120 waste:10
5: 3820,1450,120 waste:10
6: 3820,1450,120 waste:10
7: 3740,1450,200 waste:10
8: 3740,1450,200 waste:10
9: 3740,1450,200 waste:10
10: 3740,1450,200 waste:10
11: 3100,1450,500,305 waste:45
12: 3100,1450,500,305 waste:45
13: 2900,1450,900,140 waste:10
14: 2820,1450,1070 waste:60
15: 1450,1450,1450,500,500 waste:50
16: 1450,1450,1450,500,500 waste:50
17: 1450,1450,1450,500,500 waste:50
18: 1450,1270,1270,1270,140 waste:0
19: 1270,1270,1270,1270,305 waste:15
20: 1270,1270,1270,1270,305 waste:15
21: 1270,1270,1270,1270,305 waste:15
22: 1270,1270,1270,1200,380 waste:10
23: 1200,1150,1150,1150,500,245 waste:5
24: 1150,1150,1150,1150,500,245 waste:55
25: 1150,1150,500,500,500,500,500,440,140 waste:20
26: 440,440,440,440,440,440,440,380,380,380,380,380,380 waste:40
27: 380,380,380,380,380,380,380,380,360,305,305,305,305,305,305,140 waste:30
28: 305,305,305,305,305,245,245,245,245,245,245,245,245,245,245,245,230,230,230,230,230 waste:30
29: 230,230,230,230,230,230,230,230,230,230,200,200,200,200,200,190,190,190,190,190,190,190,140,140,140,140,140 waste:70
30: 140,140,140,140,140,140,140,140,140,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,120,110,110,110,110,110,110,110,110,110,110 waste:40
31: 110,110,110,110,110 waste:4850

Total timber length: 167005
Total waste:5795
Percentace waste:3.47%

So that was pretty impressive 3.5% wastage which is a third of the accepted industry standard.

In theory!

The Messy Real World

When I actually got cutting with my proudly printed-out list, I discovered a few “real world” shortcomings.

  1. Job planning: My initial list was ordered in a real-world sequence from room to room, making it easy to follow while you work from the floor-up (grooves into tongues), and around window features etc. However the algorithm had scattered the pieces all over the stock planks, meaning I had to a) track them down in the list, b) ‘open’ up a stock plank by cutting out that one length, c) keep track of the remaining stock with chalk/tape/pencil which is physically challenging in a small house and 160m of timber!
  2. Mistakes: I had mis-measured a couple of items in my plan! This meant the whole cutting list was wrong and needed updating. So surely a quick re-run through the algorithm? No, because I was now half way through the cutting process and had already cut several planks, so the stock was now of various sizes. Well, I had planned this as a feature for phase-2 but I had taken the kitchen apart and was all covered in sawdust and sweat and wasn’t about to stop and do a couple of hours programming!
  3. Timber quality: no timber stock is perfect (even DAR or this pre-primed luxury). Sometimes it gets knocked, dented or a knot pops. So again we need to be able to discard a piece, and re-run the algorithm as per 2).
  4. Other similar problems: leading to the need for flexibility, better planning and keeping tally on progress and stock/inventory. Often your plan changes while you’re doing carpentry as you work around issues or get into the detail (we have sloping ceilings and other oddities) which just needs a hands-on decision.

Being the Computer

Of course all of these things could be resolved in phase 2, but to finish the project I actually ran the algorithm myself:

  • Select the next longest item in the (current room/job’s) plan
  • Select the shortest stock which this can be cut from
  • Cut it, re-measure and write the measurement on the stock. (This is essentially an index, as you will be reading this measurement many times as you iterate through the stock and it means you can keep the timber stacked which makes it difficult to measure).

This is effectively what experienced carpenters do anyway – I found out later from my father-in-law! But from a craftsman perspective it’s reassuring to know that a rule of thumb is far more practically effective than an algorithm based on advanced mathematical theory and these new-fangled computers.

DIY!

Try out (the alpha version) for yourself here: http://scipilot.org/cutlass

Lessons

While I would like to work on phase 2, I don’t have any major carpentry projects planned at the moment, so I predict lack of necessity will mean this project may sadly stagnate.

However I really think this real-world problem would be a great tutorial or project for a software-engineering course.

  • It encapsulates a very difficult problem with potentially fun mathematical depth,
  • it has no perfect solution which is a reality young programmers have to face,
  • it highlights some real-world complexities requiring good UI/UX to prevent the wonderful maths being totally useless practically
  • and finally it would benefit from expert-interviews (stakeholder input)

i.e. by talk to your father in-law before starting a hair-brained programming project!

Posted in articles, Projects | Tagged , , , | Leave a comment

From Desktop to Dashboard – Your Car is the New Digital Frontier

After several years of incremental developments, the world of in-car technology is about to explode into the consumer market. Just as the general public have adapted to managing and exploiting the myriad functions of desktops, smart phones and tablets, they are about to be faced with a similar plethora of competing technologies, standards and options in the dashboard of their car.

Read my full article at Deepend’s Chrysalis

Posted in Miscellaneous | Tagged , , , | Leave a comment

Techniques in Fostering Innovation

In this article on Deepend’s Chrysalis, I’d like to present some of the interesting topics which have surfaced at Deepend while implementing a drive to foster a culture of innovation in our organisation. These include dealing with failure, redefining the deliverable of innovation, using a scientific approach to research, exploring corporate psychology and respecting cultural differences.

Read my full article at Chrysalis

Posted in articles | Tagged , , , , , | Leave a comment

Dusted off: God Options

God Options

godoptionsdialogue

 

– (c) Pip 2007

Posted in Dusted Off, Miscellaneous | Tagged , | Leave a comment

End of the road?

After a few tense days of radio silence, Mignon and “The Internets” team finally arrived in Darwin safe and in one piece! The guys have had a blast, smashed their $20,000 fundraising target and are happy the shitbox and its equipment survived the harsh journey, much of it off-road.

Shitbox - Full journey

The first thing I did of course was to download the GPS data and see if there’s anything interesting to visualise from the trip.

First off some quick graphs of the raw data from 30/05/14 to 06/06/14.

Speed of course is the most interesting, next up I will plot this against who was driving, and do some more interesting mashups.

Shitbox Data - Speed

Here’s the altitude graph. The slopes seem steep until you remember the graph spans 3,000kms!ShitBox Altitude

I was interested to see the GPS accuracy was fairly consistent throughout the trip, except one patch early on. I couldn’t see any correlation between altitude and accuracy, but we’re only talking 600m.

HDOP is an error multiplier, so low is better. Unfortunately I’ve not been able to find the base accuracy of our GPS device (the hardware specs are mysteriously silent) so I’m assuming it’s 5m which is pretty common (or a pessimistic 10m). For most of the journey the HDOP was around 2-3 meaning an estimated accuracy of around 10-30m. Certainly zooming into the map does usually show the car on the road at the highest zoom, so it could be as low as 5m A 3D-fix from 8 satellites was typical.Shitbox GPS Accuracy

 

I’m hoping to release the data so anyone can use it to do interesting things with. After all we’ve just logged satellite and phone coverage across half a continent!

Read more at http://rallythefuture.com.au.

Posted in Miscellaneous | Tagged , , , , | Leave a comment

And they’re off! The rally to the future begins…

The Shitbox Rally is finally underway – Dave and Matt picked up “Mignon” in Perth yesterday and drove down south to literally get some new wheels, while us back-room boys hurried to reassemble the ridiculous plethora of technology covering the car.

While we know mobile coverage is going to be a problem on the long road north, I was pleased to see the Fleetcutter system (provided by GoGet) kick in automatically when the car was unboxed from it’s transport container. While I had wound down the ping-times to preserve battery life during transportation from Sydney to Perth, now was the time to crank up the logging and see what kind of precision we’d get from the various telematics.

I was pleased to see a good set of GPS data after the first impressive run to Meekatharra.

Mignon's First day

 

 

The Rasperry Pi’s and the web cam feeds are working! Although quite what’s going on is not clear…

But all is not going smoothly! We’re having trouble with the prototype satellite messaging system (our backup Internet path for  when the car is out of range) which we blagged from the company trialling it (worth many thousands), and some of the other data feeds and the “donate to turn of the air-con” system isn’t currently working properly.

Hopefully we can get enough online time at the next pitstop to dial into the car and debug these teething troubles, but it’s very difficult in such patch coverage and time delays.

Last we heard was a tweet that they’re heading off road…

Mignon-heading-offroad

 

Good luck guys!

http://rallythefuture.com.au

 

 

 

Posted in Miscellaneous | Tagged , , , , | Leave a comment

Shitbox Rally – this shit just got real

Our unmovable hardware deadline has passed!

“The Internets” team car Mignon is now winging its way to Perth for the start of the Shitbox Rally. Well, when I say winging it’s more like trundling its way locked inside a steel container which is operating as a very effective Faraday Cage. We can’t contact the beast!

IMG_6135

After weeks of frantic development and hardware installation, fraught with battery problems, Canbus “1.5” incompatibilities, odometer failures, intermittent connectivity, door-lock relay problems and general shitboxiness, we just had to stop tinkering with the car and pack it up for its long trip across our expansive continent.

But at least we could continue developing the backend APIs right? Well no, not if the car is refusing to respond to us and provide the all essential data streams to test with…

The last we saw of it via the remote telematics (provided by the Fleetcutter system via the GoGet sponsorship) it was being shunted around in a car transport yard in Blacktown:

Mignon in Blacktown

But since then it’s been suspiciously silent:

Mignon-failed ringing during transport

We phoned the transport company and they confirmed they transport the cars across the country in shipping containers! So until it arrives in Perth we won’t know if the battery has survived the week-long convoy across the Nullarbor. Especially if we forgot to turn off the heated foot-massager and the disco lights – again.

IMG_6138 IMG_6137

With only 6 days to go to polish and integrate the systems we’re still not looking at sleeping any time soon.

But we’ve raised almost $20,000 and are currently sitting just shy of the #1 fundraiser slot so we feel like we’ve already achieved the most important goal. The rest is just technology right?

http://rallythefuture.com.au/

Posted in Miscellaneous | Tagged , , , | Leave a comment

Dusted off – Behemoth Cottontail Dynamo

Behemoth Cottontail Dynamo

In memory of silent storms

The sky is full of fire tonight
The passive clouds stage the show
Openmouth eyes flash with light
Behemoth cottontail dynamo

Drifting north along the shore
A silent minstrel’s midnight chore
As orchestrated siblings play
With breakdown voltage sound delay

No applause nor recompense
No guide nor captive audience
No directions, lighting calls
Clapper loader, greasepaint, stalls

Asleep before the show is done
Evaporated actors run
Into a sea of ‘resting’ drops
To find the story never stops.


(c) 2003 Lazykate (aka. Pip Jones)

Posted in Dusted Off, Poetry | Tagged , | Leave a comment

Dusted off – A 90’s IDEa

Today I found this note on an old hard-drive in a file simply named “THOUGHTS”.

In 1995 I was tired of the manual debug process in writing (Borland) C programs and imagined a graphical debugger which allowed you to visualise tree-structures of data, but never implemented it. Of course every IDE, browser and toaster has one now.

======================== COMPUTER IDEAS======================================

** See ideas.shtml **

1995 GRAPHICAL DEBUGGING TOOL------------------------------------------------------

dISPLAYING values of variables and structures in the way one sees them in your
head. Thats what people want to see. May as well make the representation
configurable.

?? What to look like.

foo-+--footoo 24 (int)
    |
    +--bar 5 (long)
          +-----------+----------+----+----------+
"Earth"         |  Type |  Value   |Size|Address   |
planet          +-----------+--------------------------+
      +-radius        | ulong |   1435   |  4 |4C68,3544 |
      +-orbit_radius  | ulong |  4245645 |  4 |4C68,3548 |
      +-eccentricity  | float |  0.104   |  4 |4C68,354D |
      +-periods       |     |    |    |    |
      |        |-axial      | ulong |  24      |  4 |4C68,3552 |
      |        |-solar      | ulong |  80000   |  4 |4C68,3556 |
      |        |-p_lunar    | pointer | 4C68,3643|  4 |4C68,355A |
      +-p_atmosphere_list   |   | 4C68,3775|  8 |4C68,355F |
          +-----------+-----------+---+----------+

2004 Update
Well of course this was before it's time and did get done and integrated into all
good compilers, and interpreters. Even Flash has one now.

Sadly ideas.shtml has been lost. I vaguely remember it and am now sad its gone – it had dozens of really “good” ideas, games, inventions, stupid stuff written up in some detail. When will I learn to back stuff up!

Posted in Dusted Off, Ideas | Tagged , , | Leave a comment