I believe the architecture is a good example of Dependency Injection, the Strategy Pattern, the Repository Pattern, IoC Container (flexible), and automated testing (reliable). In fact, during initial development I wrote all the interfaces first, then the test platform, then began to fill in the functionality while writing the tests, in a loose form of TDD. I like the idea that the code was initially run solely via the test platform (i.e. no dummy manually run top-level-scripts). This meant that when I eventually deployed the system it pretty much worked first time! The main issues discovered on deployment were (apart from one major foobar) deployment-related things like the autoload location and the need\u00a0to ‘park’ work-in-progress configurations.<\/p>\n
In fact, in its first day of operation it caught a major outage seconds after the cause and several minutes before any humans noticed. But… I ignored the notification emails thinking they were false positives!<\/p>\n
But now I’m out of the loop, I’m fully off-duty on an island in the Indian Ocean.\u00a0Now my 24hr support has a\u00a0new\u00a0pair of ears stethoscoped directly into the butterflies that beat in the heart of our\u00a0veteran application.<\/p>\n
Overview<\/h2>\n
Pulse is a standalone component which provides simple application\/service heartbeat monitoring.<\/p>\n
Simply put: it checks your application is still alive<\/em>.<\/p>\n Client service\/applications can have one or multiple heartbeats monitored – called Pulses.<\/p>\n All components are customisable: notifiers, config, log, storage (via dependency-injection in an IoC container). The notifications are the most likely to need customisation, e.g. sending alerts to your own dashboard API. If you customised the storage engine, you could perhaps integrate saving heatbeats from another source.<\/p>\n This example shows a PHP application using the Pulse class to store regular heatbeats. The monitor service is using the email notifier and regularly checks the heartbeat database.<\/p>\n Note if the heartbeat database is a simple file (serialised, sqlite) then the monitor service would need to be installed locally.<\/p>\n This example demonstrates scaling the application to monitor multiple applications and use different notification strategies to send different levels of alerts, or notificiations from the different applications, to different support staff.<\/p>\n This example shows the There are three states your service can be in:<\/p>\n You only get a notification when the service changes state<\/em>.<\/p>\n You don’t have to use the ALERT state, but it gives you a bit of extra granularity to triage any flapping. Often a system will have a ‘grey area’ where you don’t want to really panic everyone, but perhaps you should keep an eye on things. This ALERT state can therefore be targetted at a different response team (see below).<\/p>\n Standard composer installation:<\/p>\n or more likely you will fork it and require your own.<\/p>\n There are two halves to the system:<\/p>\n This can be done in one of two ways:<\/p>\n a) “Locally”: Each service can require the component and call Pulse->beat() regularly. Of course this only works for PHP services, and ones you can modify by integrating with this component.<\/p>\n b) “Remotely”: The Pulse daemon can poll your system using “Scopes” to update the heartbeats. This is the most unintrusive method, and can interface with existing database timetamps for example.<\/p>\n The IoC model allows you to easily implement another interface to query an API, top, PID file etc.<\/p>\n The monitoring daemon must be run in a supervised process (or it can be polled by your own scheduler).<\/p>\n Both uses obviously must share the same config, by default these are files in the See Also see the Unit Tests for usages (although I’ve always hated reading people say that).<\/p>\n Each Notifications also require some configuration too (see below).<\/p>\n See You can either hand-craft a json config file, or programmatically create Initially there is only an Body template e.g.:<\/p>\n Optionally you can customise any specific messages for both monitoring alerts, and internal errors:<\/p>\n e.g. you might want outage alarms sent to L1 Support pagers, but internal warnings sent to developers or devops.<\/p>\n See ‘INotify’ for these constants.<\/p>\n The system logs INFO by default into the This will just be service status changes<\/em>, and any system errors (e.g. notification send failures). They will be pretty small (if your services are stable), but you might want to rotate them anyway.<\/p>\n You can change the level of detail up to debug or down to just warnings, by calling The Feed API has not yet been implemented. The idea is to allow remote applications both register Beats, and fetch Pulse status.<\/p><\/blockquote>\n There are extensive (complete?) PHPUnit tests which are mostly standalone, except for some email and database configurations required for full-system testing.<\/p>\n To run the DB-oriented tests you will need the example service database installed:<\/p>\n\n
e.g. 1 – Integrated Application Heartbeat Update and Monitor Service Sending Email Notifications<\/h3>\n
![\"example](\"https:\/\/bitbucket.org\/scipilot\/pulse\/raw\/master\/doc\/img\/pulse-doc1.png\")
<\/p>\ne.g. 2 – Multiple Application Heartbeats and Multiple Notification Types<\/h3>\n
![\"example](\"https:\/\/bitbucket.org\/scipilot\/pulse\/raw\/master\/doc\/img\/pulse-doc2.png\")
<\/p>\ne.g. 3 – Remote Probe Ability for Legacy or non-PHP Apps<\/h3>\n
Probe<\/code> feature querying a remote application’s status, without integrating into it.<\/p>\n![\"example](\"https:\/\/bitbucket.org\/scipilot\/pulse\/raw\/master\/doc\/img\/pulse-doc3.png\")
<\/p>\nAlarms<\/h2>\n
\n
Installation<\/h2>\n
><\/span> composer<\/span> require<\/span> scipilot<\/span>\/<\/span>pulse<\/span>\r\n<\/pre>\n<\/div>\nUsage<\/h2>\n
1. Heatbeat Update<\/h3>\n
Scope<\/code>s allow you to implement custom probes into other systems to extract heartbeats. Currently there is only one: a DatabaseRecordScope which queries an SQL DB for a single timestamp.<\/p>\n2. Pulse Monitoring<\/h3>\n
storage<\/code> folder, so the same installation must be used. If you need to separate the daemon from your services, you would need to implement another Storage<\/code> strategy (i.e. implement a Storage<\/code> class and inject into a modified container).<\/p>\nUsage Examples<\/h3>\n
\/bin\/example_service.php<\/code> and \/bin\/daemon.php<\/code> respectively.<\/p>\n\n
php \/bin\/daemon.php &; php \/bin\/example_service.php &<\/code><\/li>\nConfig<\/h2>\n
Pulse<\/code> is configured with alert and alarm thresholds, friendly name, unique ID.<\/p>\nstorage\/config-SAMPLE.json<\/code> for a bootstrap config.<\/p>\nPulse<\/code> or Scope<\/code> objects, which can persist their own configuration for re-load later. See the PulseRegistry<\/code> and ScopeRegistry<\/code>.<\/p>\nNotifications<\/h2>\n
Email Notifications<\/h3>\n
EmailNotify<\/code> implementation. it requires the following configuration:<\/p>\nnotify<\/span>.<\/span>EmailNotify<\/span>.<\/span>default<\/span>.<\/span>email<\/span>.<\/span>to<\/span>\r\nnotify<\/span>.<\/span>EmailNotify<\/span>.<\/span>default<\/span>.<\/span>email<\/span>.<\/span>subject<\/span> -<\/span> default<\/span> subject<\/span>\r\nnotify<\/span>.<\/span>EmailNotify<\/span>.<\/span>default<\/span>.<\/span>email<\/span>.<\/span>body<\/span> -<\/span> with<\/span> placeholders:<\/span> _TYPE_<\/span>,<\/span> _LEVEL_<\/span>,<\/span> _NAME_<\/span>,<\/span> _MESSAGE_<\/span>\r\n<\/pre>\n<\/div>\n\"Heads up! Alert type: _TYPE_, at level: _LEVEL_, was received for service _NAME_. \\n Details are as follows: _MESSAGE_\"<\/span>\r\n<\/pre>\n<\/div>\nnotify<\/span>.<\/span>EmailNotify<\/span>.<<\/span>TYPE<\/span>>.<<\/span>LEVEL<\/span>>.<\/span>email<\/span>.<\/span>to<\/span> -<\/span> specific<\/span> alert<\/span> overrides<\/span>\r\nnotify<\/span>.<\/span>EmailNotify<\/span>.<<\/span>TYPE<\/span>>.<<\/span>LEVEL<\/span>>.<\/span>email<\/span>.<\/span>subject<\/span>\r\nnotify<\/span>.<\/span>EmailNotify<\/span>.<<\/span>TYPE<\/span>>.<<\/span>LEVEL<\/span>>.<\/span>email<\/span>.<\/span>body<\/span> -<\/span> with<\/span> placeholders:<\/span> _TYPE_<\/span>,<\/span> _LEVEL_<\/span>,<\/span> _MESSAGE_<\/span>\r\n<\/pre>\n<\/div>\nLogs<\/h2>\n
storage<\/code> folder.<\/p>\napp->log->setVerbosity()<\/code>. (There is no config option for this, at present)<\/p>\nArchitecture<\/h2>\n
-<\/span> IoC<\/span> App<\/span> Container<\/span>\r\n -<\/span> Config<\/span> DI<\/span>\r\n -<\/span> Log<\/span> DI<\/span>\r\n -<\/span> Storage<\/span> DI<\/span>\r\n -<\/span> Notification<\/span> DI<\/span>\r\n-<\/span> PulseRegistry<\/span>\r\n -<\/span> Pulse<\/span>-><\/span>beat<\/span>\r\n-<\/span> ScopeRegistry<\/span>\r\n -<\/span> Scope<\/span>-><\/span>Listen<\/span>\r\n-<\/span> Monitor<\/span> Daemon<\/span>-><\/span>run:<\/span>\r\n -<\/span> Listen<\/span> to<\/span> Scopes<\/span>\r\n -<\/span> Scan<\/span> Pulses<\/span>\r\n-<\/span> Feed<\/span> API<\/span> (<\/span>TODO!<\/span>)<\/span>\r\n<\/pre>\n<\/div>\nRoadmap<\/h3>\n
\n
NewRelicNotify<\/code><\/li>\nSMSNotify<\/code><\/li>\nDatabaseStorage<\/code><\/li>\nRestNotify<\/code><\/li>\nStatusPageIONotify<\/code><\/li>\nTests<\/h2>\n
CREATE<\/span> DATABASE<\/span> pulse_test<\/span>;<\/span>\r\nCREATE<\/span> TABLE<\/span> `<\/span>system<\/span>`<\/span> (<\/span>\r\n `<\/span>id<\/span>`<\/span> int<\/span>(<\/span>11<\/span>)<\/span> unsigned<\/span> NOT<\/span> NULL<\/span> AUTO_INCREMENT<\/span>,<\/span>\r\n `<\/span>last_seen<\/span>`<\/span> datetime<\/span> DEFAULT<\/span> NULL<\/span>,<\/span>\r\n PRIMARY<\/span> KEY<\/span> (<\/span>`<\/span>id<\/span>`<\/span>)<\/span>\r\n)<\/span> ENGINE<\/span>=<\/span>InnoDB<\/span>;<\/span>\r\nINSERT<\/span> INTO<\/span> `<\/span>system<\/span>`<\/span> (<\/span>