Monday, July 18, 2016

A Little HA A.D.D.

I've been busy working on a whole bunch of different projects. It's been a little HA A.D.D. as I keep jumping from project to project and so far, not completing any.

Referring to my "So Much to Learn" post, I went through the Learning JavaScript Design Patterns book and cleaned up the JavaScript in our HA GUI, making it not only cleaner to read but more efficient and easier to add on to.  There's still work to do, but I distract easily.  Next, I did the freeCodeCamp course on frontend design.  I zooooooomed through it, completing all the exercises in less than a week, but I never did any of the projects.  I'd rather take what I learn and feed it back into our HA GUI as my project.

After I completed the frontend coursework, I saw freeCodeCamp had some not yet ready exercises on D3, which I hadn't heard of.  So after some digging, I got around to learning a little D3.  It has some neat visuals that CanvasJS doesn't offer, and I used it with my Jawbone activity tracking data.  I went a little overboard charting that data, but that will be the topic of a future post.

I also have an Ouya game console which I plan on making a standalone Android box for who knows what.  I went through the process of rooting it, installing a new bootloader, wiping the Ouya OS and installing CyanogenMod 11.  Once I finished that, I parked this project and went on to the next.

Last week, I received my C.H.I.P.s, and I played around with my first one.  I did a complete reflash of the OS (which has an interesting browser plugin approach to do this).  Then I started installing my usual packages like Perl, Python, VNC, Emacs, etc.  The first thing I plan to do with it is interface to a few current sensors using its GPIO pins.  I've already found an appropriate library to do the interfacing, I just need to decide if I'll use xPL or MQTT to notify the HA system of changes.  I really need to get the other 3 up and running to make sure they're good, but for now, it's on the backburner.

The Amazon Echo I got on Prime Day also arrived last week, and last night, I got an Alexa app server framework up and running with my own local endpoint.  It was my first time really playing with NodeJS and it was pretty straightforward.  Configuring things on the Amazon side was simple too.  Right now, I need to figure out how to securely get messages from my endpoint (in the DMZ) to my LAN in order to control devices.  Hopefully, I'll do that before starting on a new project!

Monday, July 11, 2016

Got My C.H.I.P.s Today

Just got home and found my 4 C.H.I.P.s had arrived.  I've only unboxed them:

Each comes with an adapter that connects to a jack on the C.H.I.P. and fans out to 3 RCA jacks for composite video and stereo audio.

A nice touch is the header pins are labeled unlike Pis or BeagleBones.

It even comes premounted to a nice plastic base!

It also includes builtin WiFi B/G/N, Bluetooth 4.0 and 4GB of onboard flash. DRAM is a skimpy 512MB however. The CPU is a 1GHz R8.  Hardware-wise I think it's a really good deal for ~$41 shipped for 4.  You can read more about C.H.I.P. at

Not sure what I will do with them yet. I have 2 BeagleBone Blacks and a Pi Model B lying around unused still, but I'll update after I've had a chance to play with them.

Sunday, July 10, 2016

Playing Around with Visuals: CanvasJS & FusionCharts

I was off from work this past week (and sick and recovering from a sprained ankle), so I've had lots of time to sit in front of the computer and try new things. I usually use CanvasJS for charting, but it has some limitations that bug me. I've looked into other charting packages, but always end up coming back to CanvasJS.  In the embedded below, I show the input bandwidth chart for my HA server using CanvasJS.  One thing you'll notice is the spike that dominates the chart. CanvasJS doesn't support log scale so you end up with large spikes drowning out the rest of the bars.  People have been asking for this feature for years but it's not in their roadmap.  Also, CanvasJS does automatic time/date labeling on the X-axis. There are 3-4 different days in the chart, only the times are shown. It would be nice if it would show the dates too, but apparently only does so when there are even more days of data.  It does have a nice zoom feature, which works great because I usually store a year's worth of data in MySQL.

FusionCharts promised log scales, for line and bar charts, and you can see that at the end of the clip.  Unfortunately, the X-axis doesn't auto-format for dates. My data is stored in MySQL by UNIX timestamp which means for every bit of data, I convert the timestamp in PHP before sending it over to chart. Log scale is nice, but there isn't any zoom/pan ability for these type of charts.  Apparently, only some line and bar charts with linear scales have zooming.

I didn't have any chance to pretty up the FusionChart, but I have to say, in spite of the log scale feature, I much prefer CanvasJS.  I also did some work with D3 but I'll save that for another post.

Saturday, June 18, 2016

Updating HA Server to Windows 10 At This Moment

Given the update of our SageTV/media server to Windows 10 went so smoothly, I decided it's time to upgrade the HA server before the free upgrade expires on July 29th. I'm a little concerned because it's not just the HA server, but I also do a lot of development on it. Hopefully all the SDKs, libraries, dlls, drivers and other dev tools will just work after the upgrade. I cloned the OS drive to a new SSD and am installing to the new drive, like I did with the media server. Right now it's 25% of the way through installation. Fingers crossed...

[2 hours later...] For the most part, things are working. A couple .ocx files had to be re-registered for the xAPIntranet stuff. IIS was enabled which caused Apache not to start. A file from the 1Wire SDK, vjsnativ.dll, was removed so my 1Wire app wouldn't work, but I restored it from my Win 7 drive. Some admin permission things had to be resolved and some firewall re-training had to be done. It looks good. Now, time to disable all the useless & privacy invading new features.

Saturday, May 28, 2016

So Much To Learn

As they say, so much to learn and not much time. This is just a short list of things I want to learn: - Better JavaScript. I'm reading Learning JavaScript Design Patterns. Our HA GUI is a homebrew AJAX app and I'm trying to clean up the JavaScript portion that started almost 9 years ago!
- Node.js - Our HA GUI uses PHP and I've thought about migrating that portion to Node. It would be nice to code in the same language for server side as client side.
- Natural Language Parsing (NLP) - I have done some work with Python & NLTK but didn't make much progress. Google has released their NLP, SyntaxNet that looks interesting.
- Android development - Hard to believe I haven't written a single Android app. Java isn't my thing. Some day.

I've also been optimizing my usage of MySQL. I noticed my queries from various apps and web GUIs caused a lot of disk I/O. I've actually been able to cut it down about 10x by restructuring queries and also restructuring tables. My server is considerably quieter with less disk activity.

Sunday, May 8, 2016

Jawbone Up Move vs Misfit Flash Activity Trackers

A few months ago, I picked up a Jawbone Up Move activity tracker, mainly to see how much I run during my weekly basketball games. I'm not a fan of the wrist trackers and I figure they'd get in the way when I'm playing basketball.

What good is a tracker if all the data is trapped in the app? Jawbone has an API and I found this Python wrapper to facilitate access to the API. I was able to extract the hourly activity data, load it into my MySQL database and generate my own graphs:

This chart compares my current day's cumulative step count to the average for Sundays. Since I usually do the same activities on certain days, I'm able to see how I'm doing step-wise for today (and potentially adjust my activity level if I'm watching too much basketball or writing too much Python).

After about 2.5 months of use, my Jawbone was dislodged from my waistband during a game of basketball, fell off somewhere and is unfortunately gone forever. I wasn't completely thrilled with it, so I opted for a different tracker as a replacement - the Misfit Flash. I found a Python wrapper for the Misfit got it up & running. It's been about 5 days, but that's enough time for me to decide it's time to switch back to Jawbone.

What are the key differences between the Up Move and the Flash?
- The Up Move clasp is rubber coated and feels durable. The Flash clasp is hard plastic with hard edges and feels like it could snap in half.
- Jawbone data is hourly when downloaded through the API. The Misfit API only gives daily data.
- Up Move activity tracking is much more accurate than the Flash. When we play basketball, we may play a game for 15 minutes then take a few minutes break before the next game and this goes on for a couple hours. Up Move accurately calls the 2 hour duration one activity. The Flash thinks it's 6 or 7 short activities, unless I manually tag the activity start and end with a double press.
- Up Move doesn't automatically detect sleep and waking up (requires a press & hold to change mode). Flash supposedly does, but so far it's not so hot on sleep detection. The lazy hours of watching TV on the sofa before bed are included as part of my sleep even though I do get up and about for about 30 minutes before I actually climb into bed. It does detect waking up OK.
- Up Move is bulkier and ugly. I tucked the nicer looking Flash into my sock when I played basketball to avoid losing it. The Up Move probably won't be as comfortable in my sock.
- Flash is waterproof but the Up Move is only splash resistant.
- Jawbone syncs in the background whenever my phone wakes up (usually. had occasional issues with it not syncing). Misfit annoyingly requires you to push the sync button in their app. Misfit has background syncing for iPhone but not for their 2nd class Android users.
- Misfit offers a sleep SDK to enable automation when sleeping or waking but it lamely only works on iOS. There is also a notification API that allows you to get real time notifications based on your activity...I think...but I haven't seen a single notification and it's useless to me since there's no background sync on Android. So these potential pluses are of no value to me.
- The Flash can be used as a trusted device to unlock your Android phone. The Up Move can't.
- The Flash, through the Misfit Link app, can perform operations on double or triple presses, like being a selfie shutter or doing some IFTTT action. Not so for the Up Move.

Overall, the Up Move's more accurate activity tracking and better syncing are the most important features to me. The automation feature of the Misfit Link app is nice as is the Flash's bluetooth trusted device feature, but less important. Looks like the Misfit Flash is getting returned and I'm going to have to buy another Jawbone Up Move.

Thursday, May 5, 2016

Free SSL Certificate

I've been running my own public webserver since about 2000, but I've never gave much thought to setting up SSL and getting a proper certificate to encrypt the content from my site. Earlier this week, I did do just that thanks to Let's Encrypt. They offer free certificates and provides some command line tools that help generate them for you & register them with their certificate authority. It was really easy and the steps are clearly documented here. Now I'll just have to remember to renew in 90 days, although it looks like they have scripts that can automate this process.

Wednesday, May 4, 2016

Updating to Windows 10

I've always been slow to update to a new OS, mainly because stuff inevitably becomes broken especially since I write my own programs. Moving my HA server from XP to Windows 7 was painful for my .NET apps, so migrating from 7 to 10 wasn't high on my priority list. I haven't written many things in .NET lately due to problems I encountered migrating 7 or various Windows updates. Instead, I've been writing a bunch of Python scripts and running them on Raspberry Pis.

I started upgrading my kids' laptops to Windows 10. This was easy. I make them use Linux and they barely use Windows so I did clean installs for them. However, for my servers, I wanted to just do an upgrade and pray everything works. I started with the media server. I cloned the boot drive to a new SSD, stashed the original & then installed Windows 10. Surprisingly, everything worked. The big project will be doing the same for the HA server, but I'm not ready for that yet. Maybe as we get closer to the end date for the free Windows 10 upgrade, I'll feel more urgency. Until then, I'm happily surprised by how smoothly the media server upgrade went.

Tuesday, May 3, 2016

I'm back

Yes, it's been a while since I've updated here. We sold our house last year and it took a good month to strip out all vestiges of home automation. Nothing was left except dual Cat-5 to every room homerun back to a patch panel in the office closet. In the meantime, we're in a rental in one of the most overheated markets in Silicon Valley, waiting for things to cool off a bit (so far it's not!) This limits what automation I can do, but when you're used to living in an automated house, anything less seems like the stone age. We're relying a lot more on wifi than before when everything at our old house was hardwired. I did drag Cat-5 along the baseboards to link the living room, office and master with GigE, and I also got a few things up and running pretty quickly: - HA and media server were brought back up. - Installed a couple outdoor IP cameras and recording is handled by Blue Iris (Avermedia NV3000 has been retired). - Squeezeboxes are back online - A couple Infocasts were brought back up helping with zoned TTS announcements - Only ZWave is being used - only modules, no switches - and no UPB or X10 as I don't want to deal with powerline signal issues - 1Wire serial adapter for with a couple sensors attached - Humidity, pressure and light level sensing is online via Arduinos - Bluetooth presence and RFID are online - X10 wireless motion and DS10A's are online via the W800 - Installed a temporary outdoor antenna and SageTV is running - Some current sensing is running on most AV equipment - the usual xPL apps for GMail, GCal, noise sensing, gamepad contact closure - Voice and IM interfaces I've added a few new things: - the obvious one is Blue Iris (BI) because the Avermedia software I had couldn't handle more than 1 IP cam - xPL interface to BI JSON interface - added MQTT to my automation engine - converted BI interface to MQTT instead of xPL - converted Arduino script from xPL to MQTT - HDMI-CEC control via Raspberry Pi & Python A few things I miss: - IR control - VOIP intercom - Power monitoring I've been playing around with some cloud API devices and doing other random things I'll hopefully get around to mentioning in later posts.

Wednesday, December 31, 2014

The End is Near

We'll be moving soon and I've been slowly packing up pieces of our home automation system along with our regular belongings. We've been here over 15 1/2 years and automation is so embedded in the house it's going to take some time to extract it all. We'll be moving into a rental for a little while so HA will be on hold. This was in the cards for a while and part of the reason I've slowed down on automation projects recently. We're planning to sell the house, although we have discussed renting it out. Even though this is Silicon Valley, it's hard to say how much automation I will leave intact. On to new adventures...

Saturday, July 5, 2014

Just Tweaking

I haven't felt like blogging for a while, as I've been distracted with life in general. My son's first school basketball season dominated winter, then both kids were playing Spring league. At work, there was turmoil and uncertainty for the first 6 months of the year, leading my group to be acquihired. So now, I just started work at the new place while my kids are gearing up for Summer league.

I have been doing some minor HA stuff during that time. The TC+ Python code was stabilized and rock solid a few days after my last post. The new HA server was brought online shortly after and I quickly ported my xEngine app to x64. The Haswell based server is so much more efficient, our power usage dropped considerably and I'm now considering another one to upgrade the media server. The BeagleBone Black has been reliably running ZWave, RFID, UPB and the W800. I deployed my first Arduino, just to try out using a DHT11 based humidity sensor. It's not as accurate as I hoped, so I'm still looking for other methods.

Software-wise, I haven't felt the inspiration to undertake any neat, new projects. I'm mostly in maintenance mode, occasionally finding some obscure bug introduced when I ported over to xEngine. Otherwise, everything just works. I did modify the JavaScript and PHP for our AJAX based HA GUI to use JSON, instead of the comma delimted data format I had before. I still have a couple Kinects I haven't done anything with and I'm learning AngularJS right now. With the new job, I'm not sure how much time I'll have to play. Hopefully, I'll have some time and find some inspiration.

Monday, January 13, 2014

HA Revamp Progress

It took about 3 days to get the W800 code migrated over from starCOMUltra and debugged. That's now sitting on an Edgeport 4 port serial to USB adapter connected to the Beaglebone Black, along with 1Wire, UPB and the RFID reader. The last major interface to write was for the JDS TimeCommander+, which I started on January 1. I completed most of it by Friday the 3rd so I was able to shut off starCOMUltra and let my engine run overnight all by itself for the first time. Since I had to go back to work the following week, I probably spent 15 hours a day working on it that last weekend. It's been over week now and I'm still cleaning up code, making the Python interface to the TC+ more reliable and fixing some events that didn't get correctly ported over from starCOMUltra. Last night, I put in the final piece of TC+ support by adding control of the IR Xpander. The next steps will be porting this engine over to x64. After that, hopefully I can start migrating everything to the new server.