Playing Around with Visuals: CanvasJS & FusionCharts

I was off from work this past week (and sick and recovering from a sprained ankle), so I've had lots of time to sit in front of the computer and try new things. I usually use CanvasJS for charting, but it has some limitations that bug me. I've looked into other charting packages, but always end up coming back to CanvasJS.  In the embedded below, I show the input bandwidth chart for my HA server using CanvasJS.  One thing you'll notice is the spike that dominates the chart. CanvasJS doesn't support log scale so you end up with large spikes drowning out the rest of the bars.  People have been asking for this feature for years but it's not in their roadmap.  Also, CanvasJS does automatic time/date labeling on the X-axis. There are 3-4 different days in the chart, only the times are shown. It would be nice if it would show the dates too, but apparently only does so when there are even more days of data.  It does have a nice zoom feature, which works great because I usually store a year's worth of data in MySQL.

FusionCharts promised log scales, for line and bar charts, and you can see that at the end of the clip.  Unfortunately, the X-axis doesn't auto-format for dates. My data is stored in MySQL by UNIX timestamp which means for every bit of data, I convert the timestamp in PHP before sending it over to chart. Log scale is nice, but there isn't any zoom/pan ability for these type of charts.  Apparently, only some line and bar charts with linear scales have zooming.

I didn't have any chance to pretty up the FusionChart, but I have to say, in spite of the log scale feature, I much prefer CanvasJS.  I also did some work with D3 but I'll save that for another post.

Updating HA Server to Windows 10 At This Moment

Given the update of our SageTV/media server to Windows 10 went so smoothly, I decided it's time to upgrade the HA server before the free upgrade expires on July 29th. I'm a little concerned because it's not just the HA server, but I also do a lot of development on it. Hopefully all the SDKs, libraries, dlls, drivers and other dev tools will just work after the upgrade. I cloned the OS drive to a new SSD and am installing to the new drive, like I did with the media server. Right now it's 25% of the way through installation. Fingers crossed...

[2 hours later...] For the most part, things are working. A couple .ocx files had to be re-registered for the xAPIntranet stuff. IIS was enabled which caused Apache not to start. A file from the 1Wire SDK, vjsnativ.dll, was removed so my 1Wire app wouldn't work, but I restored it from my Win 7 drive. Some admin permission things had to be resolved and some firewall re-training had to be done. It looks good. Now, time to disable all the useless & privacy invading new features.

So Much To Learn

As they say, so much to learn and not much time. This is just a short list of things I want to learn: - Better JavaScript. I'm reading Learning JavaScript Design Patterns. Our HA GUI is a homebrew AJAX app and I'm trying to clean up the JavaScript portion that started almost 9 years ago!
- Node.js - Our HA GUI uses PHP and I've thought about migrating that portion to Node. It would be nice to code in the same language for server side as client side.
- Natural Language Parsing (NLP) - I have done some work with Python & NLTK but didn't make much progress. Google has released their NLP, SyntaxNet that looks interesting.
- Android development - Hard to believe I haven't written a single Android app. Java isn't my thing. Some day.

I've also been optimizing my usage of MySQL. I noticed my queries from various apps and web GUIs caused a lot of disk I/O. I've actually been able to cut it down about 10x by restructuring queries and also restructuring tables. My server is considerably quieter with less disk activity.

Jawbone Up Move vs Misfit Flash Activity Trackers

A few months ago, I picked up a Jawbone Up Move activity tracker, mainly to see how much I run during my weekly basketball games. I'm not a fan of the wrist trackers and I figure they'd get in the way when I'm playing basketball.

What good is a tracker if all the data is trapped in the app? Jawbone has an API and I found this Python wrapper to facilitate access to the API. I was able to extract the hourly activity data, load it into my MySQL database and generate my own graphs:



This chart compares my current day's cumulative step count to the average for Sundays. Since I usually do the same activities on certain days, I'm able to see how I'm doing step-wise for today (and potentially adjust my activity level if I'm watching too much basketball or writing too much Python).

After about 2.5 months of use, my Jawbone was dislodged from my waistband during a game of basketball, fell off somewhere and is unfortunately gone forever. I wasn't completely thrilled with it, so I opted for a different tracker as a replacement - the Misfit Flash. I found a Python wrapper for the Misfit got it up & running. It's been about 5 days, but that's enough time for me to decide it's time to switch back to Jawbone.

What are the key differences between the Up Move and the Flash?
- The Up Move clasp is rubber coated and feels durable. The Flash clasp is hard plastic with hard edges and feels like it could snap in half.
- Jawbone data is hourly when downloaded through the API. The Misfit API only gives daily data.
- Up Move activity tracking is much more accurate than the Flash. When we play basketball, we may play a game for 15 minutes then take a few minutes break before the next game and this goes on for a couple hours. Up Move accurately calls the 2 hour duration one activity. The Flash thinks it's 6 or 7 short activities, unless I manually tag the activity start and end with a double press.
- Up Move doesn't automatically detect sleep and waking up (requires a press & hold to change mode). Flash supposedly does, but so far it's not so hot on sleep detection. The lazy hours of watching TV on the sofa before bed are included as part of my sleep even though I do get up and about for about 30 minutes before I actually climb into bed. It does detect waking up OK.
- Up Move is bulkier and ugly. I tucked the nicer looking Flash into my sock when I played basketball to avoid losing it. The Up Move probably won't be as comfortable in my sock.
- Flash is waterproof but the Up Move is only splash resistant.
- Jawbone syncs in the background whenever my phone wakes up (usually. had occasional issues with it not syncing). Misfit annoyingly requires you to push the sync button in their app. Misfit has background syncing for iPhone but not for their 2nd class Android users.
- Misfit offers a sleep SDK to enable automation when sleeping or waking but it lamely only works on iOS. There is also a notification API that allows you to get real time notifications based on your activity...I think...but I haven't seen a single notification and it's useless to me since there's no background sync on Android. So these potential pluses are of no value to me.
- The Flash can be used as a trusted device to unlock your Android phone. The Up Move can't.
- The Flash, through the Misfit Link app, can perform operations on double or triple presses, like being a selfie shutter or doing some IFTTT action. Not so for the Up Move.

Overall, the Up Move's more accurate activity tracking and better syncing are the most important features to me. The automation feature of the Misfit Link app is nice as is the Flash's bluetooth trusted device feature, but less important. Looks like the Misfit Flash is getting returned and I'm going to have to buy another Jawbone Up Move.

Free SSL Certificate

I've been running my own public webserver since about 2000, but I've never gave much thought to setting up SSL and getting a proper certificate to encrypt the content from my site. Earlier this week, I did do just that thanks to Let's Encrypt. They offer free certificates and provides some command line tools that help generate them for you & register them with their certificate authority. It was really easy and the steps are clearly documented here. Now I'll just have to remember to renew in 90 days, although it looks like they have scripts that can automate this process.

Updating to Windows 10

I've always been slow to update to a new OS, mainly because stuff inevitably becomes broken especially since I write my own programs. Moving my HA server from XP to Windows 7 was painful for my .NET apps, so migrating from 7 to 10 wasn't high on my priority list. I haven't written many things in .NET lately due to problems I encountered migrating 7 or various Windows updates. Instead, I've been writing a bunch of Python scripts and running them on Raspberry Pis.

I started upgrading my kids' laptops to Windows 10. This was easy. I make them use Linux and they barely use Windows so I did clean installs for them. However, for my servers, I wanted to just do an upgrade and pray everything works. I started with the media server. I cloned the boot drive to a new SSD, stashed the original & then installed Windows 10. Surprisingly, everything worked. The big project will be doing the same for the HA server, but I'm not ready for that yet. Maybe as we get closer to the end date for the free Windows 10 upgrade, I'll feel more urgency. Until then, I'm happily surprised by how smoothly the media server upgrade went.

I'm back

Yes, it's been a while since I've updated here. We sold our house last year and it took a good month to strip out all vestiges of home automation. Nothing was left except dual Cat-5 to every room homerun back to a patch panel in the office closet. In the meantime, we're in a rental in one of the most overheated markets in Silicon Valley, waiting for things to cool off a bit (so far it's not!) This limits what automation I can do, but when you're used to living in an automated house, anything less seems like the stone age. We're relying a lot more on wifi than before when everything at our old house was hardwired. I did drag Cat-5 along the baseboards to link the living room, office and master with GigE, and I also got a few things up and running pretty quickly: - HA and media server were brought back up. - Installed a couple outdoor IP cameras and recording is handled by Blue Iris (Avermedia NV3000 has been retired). - Squeezeboxes are back online - A couple Infocasts were brought back up helping with zoned TTS announcements - Only ZWave is being used - only modules, no switches - and no UPB or X10 as I don't want to deal with powerline signal issues - 1Wire serial adapter for with a couple sensors attached - Humidity, pressure and light level sensing is online via Arduinos - Bluetooth presence and RFID are online - X10 wireless motion and DS10A's are online via the W800 - Installed a temporary outdoor antenna and SageTV is running - Some current sensing is running on most AV equipment - the usual xPL apps for GMail, GCal, noise sensing, gamepad contact closure - Voice and IM interfaces I've added a few new things: - the obvious one is Blue Iris (BI) because the Avermedia software I had couldn't handle more than 1 IP cam - xPL interface to BI JSON interface - added MQTT to my automation engine - converted BI interface to MQTT instead of xPL - converted Arduino script from xPL to MQTT - HDMI-CEC control via Raspberry Pi & Python A few things I miss: - IR control - VOIP intercom - Power monitoring I've been playing around with some cloud API devices and doing other random things I'll hopefully get around to mentioning in later posts.

The End is Near

We'll be moving soon and I've been slowly packing up pieces of our home automation system along with our regular belongings. We've been here over 15 1/2 years and automation is so embedded in the house it's going to take some time to extract it all. We'll be moving into a rental for a little while so HA will be on hold. This was in the cards for a while and part of the reason I've slowed down on automation projects recently. We're planning to sell the house, although we have discussed renting it out. Even though this is Silicon Valley, it's hard to say how much automation I will leave intact. On to new adventures...

Just Tweaking

I haven't felt like blogging for a while, as I've been distracted with life in general. My son's first school basketball season dominated winter, then both kids were playing Spring league. At work, there was turmoil and uncertainty for the first 6 months of the year, leading my group to be acquihired. So now, I just started work at the new place while my kids are gearing up for Summer league.

I have been doing some minor HA stuff during that time. The TC+ Python code was stabilized and rock solid a few days after my last post. The new HA server was brought online shortly after and I quickly ported my xEngine app to x64. The Haswell based server is so much more efficient, our power usage dropped considerably and I'm now considering another one to upgrade the media server. The BeagleBone Black has been reliably running ZWave, RFID, UPB and the W800. I deployed my first Arduino, just to try out using a DHT11 based humidity sensor. It's not as accurate as I hoped, so I'm still looking for other methods.

Software-wise, I haven't felt the inspiration to undertake any neat, new projects. I'm mostly in maintenance mode, occasionally finding some obscure bug introduced when I ported over to xEngine. Otherwise, everything just works. I did modify the JavaScript and PHP for our AJAX based HA GUI to use JSON, instead of the comma delimted data format I had before. I still have a couple Kinects I haven't done anything with and I'm learning AngularJS right now. With the new job, I'm not sure how much time I'll have to play. Hopefully, I'll have some time and find some inspiration.

HA Revamp Progress

It took about 3 days to get the W800 code migrated over from starCOMUltra and debugged. That's now sitting on an Edgeport 4 port serial to USB adapter connected to the Beaglebone Black, along with 1Wire, UPB and the RFID reader. The last major interface to write was for the JDS TimeCommander+, which I started on January 1. I completed most of it by Friday the 3rd so I was able to shut off starCOMUltra and let my engine run overnight all by itself for the first time. Since I had to go back to work the following week, I probably spent 15 hours a day working on it that last weekend. It's been over week now and I'm still cleaning up code, making the Python interface to the TC+ more reliable and fixing some events that didn't get correctly ported over from starCOMUltra. Last night, I put in the final piece of TC+ support by adding control of the IR Xpander. The next steps will be porting this engine over to x64. After that, hopefully I can start migrating everything to the new server.

Saying Goodbye to 2013 and 32-bit Windows

2013 is almost over and it really wasn't a very prolific year for me as far as HA was concerned. Work has been overwhelming this year and last, but I got some sporadic cycles to do some HA. I am, however, finishing the year with a flourish as I have some time off.

Just before the holidays, I picked up a new Haswell based i3 server to eventually replace our aging HA server. I installed Windows 7 x64 on it, but moving forward from 32-bit XP means leaving things behind due to compatibility. It's something I've dreaded having to do, but it's also a chance to rid myself of some of these Windows dependencies, as much as possible. The WinEVM software used to program our JDS TimeCommander+ (TC+) will get left behind, and starCOMUltra (SCU), which is THE heart of our HA system is based, won't be coming along either. I don't want the hassle of running XP in a VM just for a couple programs. While I've got an XP based netbook to run WinEVM and program the TC+, there's nearly 15 years of HA scripting tied into SCU that has to be migrated.

I have my own app, xPL SCU, which is a scripting engine bridging SCU and xAP/xPL where about 1/4 of my HA code resides and where I'll be moving most of the nearly 10,000 lines of code I've written in SCU. I already have variations of xPL SCU running on x64 in the form of my other scripting apps, xScript and xPLScript (basically the scripting engine minus the SCU piece). In order to leave SCU, I'll need to hopefully code my own interface to the TimeCommander+, and when that happens, I'll remove those references from xPL SCU and everything should just go. We'll see what roadblocks I bump into.

In the meantime, since I built all the interfaces to other hardware into the SCU script, I have to rebuild that somewhere else. As I mentioned in my previous post, I moved ZWave and 1Wire to a BeagleBone Black, and I'm continuing in that direction. I'm putting all the serial devices on the BBB and will use xPL to communicate with the HA server - to keep their interfacing code independent of whatever HA software I use now or in the future. I already prototyped this path with the ZWave interface, and since I can run the same Python code on the Windows server and the BBB (just changing the serial port reference), I can debug it without moving the hardware to the BBB.

As a result, I've made some good progress in a short time. Three days ago, I wrote a Python script to interface xPL and the RFID reader, put that on the BBB, and moved the related logic from SCU to xPL SCU. The next day, I did the same with UPB. Yesterday, I started doing this for the WGL 800. I have so much stuff triggering off of X10 wireless (DS10As, MS10As, etc.), debugging the migrated code is taking a while. In addition, there's a lot of cleanup to do. Lots to do to keep me busy well into 2014.

Happy New Year.

Time Flies

Has it really been almost four months since my last post? Things have been incredibly busy at work, and I haven't had much time to blog since summer. We had a major liquidity event recently, and the build up to that and the aftermath, has been an increased focus on the product I'm working on. Needless to say, the stupid crazy work schedule for a grossly understaffed project has been ramped up again. While it's great to have a cash reward for 2 1/2 years of manic, startup work, what I really want is time off.

I have been able to do some things here and there. One much needed housekeeping chore I did was cleaning up a bunch of wiring around the house and garage and putting all the exterior cameras and motions on a single 12V power supply (so they can easily be put on a UPS). Another project I've been working on is gradually cleaning up the jobs that run on my HA server, mainly in preparation to build a new one. It's a 6.5 year old Core2 Duo system running XP. Eventually I want to put in a new i3 or i5 server.

As part of that cleanup, I started testing putting some hardware on one of the BeagleBone Blacks I bought. I put the Aeon Labs ZStick on the BBB, installed OpenZWave and added xPL functionality to their Python sample code. After some testing, I moved ZWave permanently over from the HA server to the BBB and turned off my .NET app. Another thing I moved was 1Wire. I installed DigiTemp on the BBB to read the DS9097U 1Wire network and wrote more Python to dump the temperature data to mySQL. With that, I turned off another of my .NET apps.

Is there a pattern here? Possibly. I have gotten tired of .NET apps running on XP but not on Windows 7 or breaking after a Windows update. The ever growing, bloated Visual Studio tools are another negative. I don't think I can get away completely from .NET, but I can reduce some of the dependencies I have by coding in Python. Of course, coding in Python means a certain Premise user on CocoonTech (who doesn't understand blogs, thinks they're hard to read and should be organized by topic instead of date) will have an easy time stripping the headers and credits from my code and passing it off as his own.

I also funded this Arduino project on indiegogo getting 5 boards, 2 for the kids and 3 for future projects. Haven't had much time to work on them, other than powering up and downloading sample code to make sure they work. Recently, I've been searching EBay for various sensors for these boards to play around with. Borderless Electronics, has now come back with a follow up campaign, featuring various combinations of a kit, board and shield. I'm also contributing, to get some shields for my Arduinos, so they can be network enabled.

While on the topic of crowdfunded projects, I also backed the Pressy project. This is going to be a great way to kick things off on my Note II, like triggering my voice recognition home automation script.