Saturday, June 18, 2016

Updating HA Server to Windows 10 At This Moment

Given the update of our SageTV/media server to Windows 10 went so smoothly, I decided it's time to upgrade the HA server before the free upgrade expires on July 29th. I'm a little concerned because it's not just the HA server, but I also do a lot of development on it. Hopefully all the SDKs, libraries, dlls, drivers and other dev tools will just work after the upgrade. I cloned the OS drive to a new SSD and am installing to the new drive, like I did with the media server. Right now it's 25% of the way through installation. Fingers crossed...

[2 hours later...] For the most part, things are working. A couple .ocx files had to be re-registered for the xAPIntranet stuff. IIS was enabled which caused Apache not to start. A file from the 1Wire SDK, vjsnativ.dll, was removed so my 1Wire app wouldn't work, but I restored it from my Win 7 drive. Some admin permission things had to be resolved and some firewall re-training had to be done. It looks good. Now, time to disable all the useless & privacy invading new features.

Saturday, May 28, 2016

So Much To Learn

As they say, so much to learn and not much time. This is just a short list of things I want to learn: - Better JavaScript. I'm reading Learning JavaScript DesignPatterns. Our HA GUI is a homebrew AJAX app and I'm trying to clean up the JavaScript portion that started almost 9 years ago!
- Node.js - Our HA GUI uses PHP and I've thought about migrating that portion to Node. It would be nice to code in the same language for server side as client side.
- Natural Language Parsing (NLP) - I have done some work with Python & NLTK but didn't make much progress. Google has released their NLP, SyntaxNet that looks interesting.
- Android development - Hard to believe I haven't written a single Android app. Java isn't my thing. Some day.

I've also been optimizing my usage of MySQL. I noticed my queries from various apps and web GUIs caused a lot of disk I/O. I've actually been able to cut it down about 10x by restructuring queries and also restructuring tables. My server is considerably quieter with less disk activity.

Sunday, May 8, 2016

Jawbone Up Move vs Misfit Flash Activity Trackers

A few months ago, I picked up a Jawbone Up Move activity tracker, mainly to see how much I run during my weekly basketball games. I'm not a fan of the wrist trackers and I figure they'd get in the way when I'm playing basketball.

What good is a tracker if all the data is trapped in the app? Jawbone has an API and I found this Python wrapper to facilitate access to the API. I was able to extract the hourly activity data, load it into my MySQL database and generate my own graphs:

This chart compares my current day's cumulative step count to the average for Sundays. Since I usually do the same activities on certain days, I'm able to see how I'm doing step-wise for today (and potentially adjust my activity level if I'm watching too much basketball or writing too much Python).

After about 2.5 months of use, my Jawbone was dislodged from my waistband during a game of basketball, fell off somewhere and is unfortunately gone forever. I wasn't completely thrilled with it, so I opted for a different tracker as a replacement - the Misfit Flash. I found a Python wrapper for the Misfit got it up & running. It's been about 5 days, but that's enough time for me to decide it's time to switch back to Jawbone.

What are the key differences between the Up Move and the Flash?
- The Up Move clasp is rubber coated and feels durable. The Flash clasp is hard plastic with hard edges and feels like it could snap in half.
- Jawbone data is hourly when downloaded through the API. The Misfit API only gives daily data.
- Up Move activity tracking is much more accurate than the Flash. When we play basketball, we may play a game for 15 minutes then take a few minutes break before the next game and this goes on for a couple hours. Up Move accurately calls the 2 hour duration one activity. The Flash thinks it's 6 or 7 short activities, unless I manually tag the activity start and end with a double press.
- Up Move doesn't automatically detect sleep and waking up (requires a press & hold to change mode). Flash supposedly does, but so far it's not so hot on sleep detection. The lazy hours of watching TV on the sofa before bed are included as part of my sleep even though I do get up and about for about 30 minutes before I actually climb into bed. It does detect waking up OK.
- Up Move is bulkier and ugly. I tucked the nicer looking Flash into my sock when I played basketball to avoid losing it. The Up Move probably won't be as comfortable in my sock.
- Flash is waterproof but the Up Move is only splash resistant.
- Jawbone syncs in the background whenever my phone wakes up (usually. had occasional issues with it not syncing). Misfit annoyingly requires you to push the sync button in their app. Misfit has background syncing for iPhone but not for their 2nd class Android users.
- Misfit offers a sleep SDK to enable automation when sleeping or waking but it lamely only works on iOS. There is also a notification API that allows you to get real time notifications based on your activity...I think...but I haven't seen a single notification and it's useless to me since there's no background sync on Android. So these potential pluses are of no value to me.
- The Flash can be used as a trusted device to unlock your Android phone. The Up Move can't.
- The Flash, through the Misfit Link app, can perform operations on double or triple presses, like being a selfie shutter or doing some IFTTT action. Not so for the Up Move.

Overall, the Up Move's more accurate activity tracking and better syncing are the most important features to me. The automation feature of the Misfit Link app is nice as is the Flash's bluetooth trusted device feature, but less important. Looks like the Misfit Flash is getting returned and I'm going to have to buy another Jawbone Up Move.

Thursday, May 5, 2016

Free SSL Certificate

I've been running my own public webserver since about 2000, but I've never gave much thought to setting up SSL and getting a proper certificate to encrypt the content from my site. Earlier this week, I did do just that thanks to Let's Encrypt. They offer free certificates and provides some command line tools that help generate them for you & register them with their certificate authority. It was really easy and the steps are clearly documented here. Now I'll just have to remember to renew in 90 days, although it looks like they have scripts that can automate this process.

Wednesday, May 4, 2016

Updating to Windows 10

I've always been slow to update to a new OS, mainly because stuff inevitably becomes broken especially since I write my own programs. Moving my HA server from XP to Windows 7 was painful for my .NET apps, so migrating from 7 to 10 wasn't high on my priority list. I haven't written many things in .NET lately due to problems I encountered migrating 7 or various Windows updates. Instead, I've been writing a bunch of Python scripts and running them on Raspberry Pis.

I started upgrading my kids' laptops to Windows 10. This was easy. I make them use Linux and they barely use Windows so I did clean installs for them. However, for my servers, I wanted to just do an upgrade and pray everything works. I started with the media server. I cloned the boot drive to a new SSD, stashed the original & then installed Windows 10. Surprisingly, everything worked. The big project will be doing the same for the HA server, but I'm not ready for that yet. Maybe as we get closer to the end date for the free Windows 10 upgrade, I'll feel more urgency. Until then, I'm happily surprised by how smoothly the media server upgrade went.

Tuesday, May 3, 2016

I'm back

Yes, it's been a while since I've updated here. We sold our house last year and it took a good month to strip out all vestiges of home automation. Nothing was left except dual Cat-5 to every room homerun back to a patch panel in the office closet. In the meantime, we're in a rental in one of the most overheated markets in Silicon Valley, waiting for things to cool off a bit (so far it's not!) This limits what automation I can do, but when you're used to living in an automated house, anything less seems like the stone age. We're relying a lot more on wifi than before when everything at our old house was hardwired. I did drag Cat-5 along the baseboards to link the living room, office and master with GigE, and I also got a few things up and running pretty quickly: - HA and media server were brought back up. - Installed a couple outdoor IP cameras and recording is handled by Blue Iris (Avermedia NV3000 has been retired). - Squeezeboxes are back online - A couple Infocasts were brought back up helping with zoned TTS announcements - Only ZWave is being used - only modules, no switches - and no UPB or X10 as I don't want to deal with powerline signal issues - 1Wire serial adapter for with a couple sensors attached - Humidity, pressure and light level sensing is online via Arduinos - Bluetooth presence and RFID are online - X10 wireless motion and DS10A's are online via the W800 - Installed a temporary outdoor antenna and SageTV is running - Some current sensing is running on most AV equipment - the usual xPL apps for GMail, GCal, noise sensing, gamepad contact closure - Voice and IM interfaces I've added a few new things: - the obvious one is Blue Iris (BI) because the Avermedia software I had couldn't handle more than 1 IP cam - xPL interface to BI JSON interface - added MQTT to my automation engine - converted BI interface to MQTT instead of xPL - converted Arduino script from xPL to MQTT - HDMI-CEC control via Raspberry Pi & Python A few things I miss: - IR control - VOIP intercom - Power monitoring I've been playing around with some cloud API devices and doing other random things I'll hopefully get around to mentioning in later posts.

Wednesday, December 31, 2014

The End is Near

We'll be moving soon and I've been slowly packing up pieces of our home automation system along with our regular belongings. We've been here over 15 1/2 years and automation is so embedded in the house it's going to take some time to extract it all. We'll be moving into a rental for a little while so HA will be on hold. This was in the cards for a while and part of the reason I've slowed down on automation projects recently. We're planning to sell the house, although we have discussed renting it out. Even though this is Silicon Valley, it's hard to say how much automation I will leave intact. On to new adventures...

Saturday, July 5, 2014

Just Tweaking

I haven't felt like blogging for a while, as I've been distracted with life in general. My son's first school basketball season dominated winter, then both kids were playing Spring league. At work, there was turmoil and uncertainty for the first 6 months of the year, leading my group to be acquihired. So now, I just started work at the new place while my kids are gearing up for Summer league.

I have been doing some minor HA stuff during that time. The TC+ Python code was stabilized and rock solid a few days after my last post. The new HA server was brought online shortly after and I quickly ported my xEngine app to x64. The Haswell based server is so much more efficient, our power usage dropped considerably and I'm now considering another one to upgrade the media server. The BeagleBone Black has been reliably running ZWave, RFID, UPB and the W800. I deployed my first Arduino, just to try out using a DHT11 based humidity sensor. It's not as accurate as I hoped, so I'm still looking for other methods.

Software-wise, I haven't felt the inspiration to undertake any neat, new projects. I'm mostly in maintenance mode, occasionally finding some obscure bug introduced when I ported over to xEngine. Otherwise, everything just works. I did modify the JavaScript and PHP for our AJAX based HA GUI to use JSON, instead of the comma delimted data format I had before. I still have a couple Kinects I haven't done anything with and I'm learning AngularJS right now. With the new job, I'm not sure how much time I'll have to play. Hopefully, I'll have some time and find some inspiration.

Monday, January 13, 2014

HA Revamp Progress

It took about 3 days to get the W800 code migrated over from starCOMUltra and debugged. That's now sitting on an Edgeport 4 port serial to USB adapter connected to the Beaglebone Black, along with 1Wire, UPB and the RFID reader. The last major interface to write was for the JDS TimeCommander+, which I started on January 1. I completed most of it by Friday the 3rd so I was able to shut off starCOMUltra and let my engine run overnight all by itself for the first time. Since I had to go back to work the following week, I probably spent 15 hours a day working on it that last weekend. It's been over week now and I'm still cleaning up code, making the Python interface to the TC+ more reliable and fixing some events that didn't get correctly ported over from starCOMUltra. Last night, I put in the final piece of TC+ support by adding control of the IR Xpander. The next steps will be porting this engine over to x64. After that, hopefully I can start migrating everything to the new server.

Tuesday, December 31, 2013

Saying Goodbye to 2013 and 32-bit Windows

2013 is almost over and it really wasn't a very prolific year for me as far as HA was concerned. Work has been overwhelming this year and last, but I got some sporadic cycles to do some HA. I am, however, finishing the year with a flourish as I have some time off.

Just before the holidays, I picked up a new Haswell based i3 server to eventually replace our aging HA server. I installed Windows 7 x64 on it, but moving forward from 32-bit XP means leaving things behind due to compatibility. It's something I've dreaded having to do, but it's also a chance to rid myself of some of these Windows dependencies, as much as possible. The WinEVM software used to program our JDS TimeCommander+ (TC+) will get left behind, and starCOMUltra (SCU), which is THE heart of our HA system is based, won't be coming along either. I don't want the hassle of running XP in a VM just for a couple programs. While I've got an XP based netbook to run WinEVM and program the TC+, there's nearly 15 years of HA scripting tied into SCU that has to be migrated.

I have my own app, xPL SCU, which is a scripting engine bridging SCU and xAP/xPL where about 1/4 of my HA code resides and where I'll be moving most of the nearly 10,000 lines of code I've written in SCU. I already have variations of xPL SCU running on x64 in the form of my other scripting apps, xScript and xPLScript (basically the scripting engine minus the SCU piece). In order to leave SCU, I'll need to hopefully code my own interface to the TimeCommander+, and when that happens, I'll remove those references from xPL SCU and everything should just go. We'll see what roadblocks I bump into.

In the meantime, since I built all the interfaces to other hardware into the SCU script, I have to rebuild that somewhere else. As I mentioned in my previous post, I moved ZWave and 1Wire to a BeagleBone Black, and I'm continuing in that direction. I'm putting all the serial devices on the BBB and will use xPL to communicate with the HA server - to keep their interfacing code independent of whatever HA software I use now or in the future. I already prototyped this path with the ZWave interface, and since I can run the same Python code on the Windows server and the BBB (just changing the serial port reference), I can debug it without moving the hardware to the BBB.

As a result, I've made some good progress in a short time. Three days ago, I wrote a Python script to interface xPL and the RFID reader, put that on the BBB, and moved the related logic from SCU to xPL SCU. The next day, I did the same with UPB. Yesterday, I started doing this for the WGL 800. I have so much stuff triggering off of X10 wireless (DS10As, MS10As, etc.), debugging the migrated code is taking a while. In addition, there's a lot of cleanup to do. Lots to do to keep me busy well into 2014.

Happy New Year.

Tuesday, November 5, 2013

Time Flies

Has it really been almost four months since my last post? Things have been incredibly busy at work, and I haven't had much time to blog since summer. We had a major liquidity event recently, and the build up to that and the aftermath, has been an increased focus on the product I'm working on. Needless to say, the stupid crazy work schedule for a grossly understaffed project has been ramped up again. While it's great to have a cash reward for 2 1/2 years of manic, startup work, what I really want is time off.

I have been able to do some things here and there. One much needed housekeeping chore I did was cleaning up a bunch of wiring around the house and garage and putting all the exterior cameras and motions on a single 12V power supply (so they can easily be put on a UPS). Another project I've been working on is gradually cleaning up the jobs that run on my HA server, mainly in preparation to build a new one. It's a 6.5 year old Core2 Duo system running XP. Eventually I want to put in a new i3 or i5 server.

As part of that cleanup, I started testing putting some hardware on one of the BeagleBone Blacks I bought. I put the Aeon Labs ZStick on the BBB, installed OpenZWave and added xPL functionality to their Python sample code. After some testing, I moved ZWave permanently over from the HA server to the BBB and turned off my .NET app. Another thing I moved was 1Wire. I installed DigiTemp on the BBB to read the DS9097U 1Wire network and wrote more Python to dump the temperature data to mySQL. With that, I turned off another of my .NET apps.

Is there a pattern here? Possibly. I have gotten tired of .NET apps running on XP but not on Windows 7 or breaking after a Windows update. The ever growing, bloated Visual Studio tools are another negative. I don't think I can get away completely from .NET, but I can reduce some of the dependencies I have by coding in Python. Of course, coding in Python means a certain Premise user on CocoonTech (who doesn't understand blogs, thinks they're hard to read and should be organized by topic instead of date) will have an easy time stripping the headers and credits from my code and passing it off as his own.

I also funded this Arduino project on indiegogo getting 5 boards, 2 for the kids and 3 for future projects. Haven't had much time to work on them, other than powering up and downloading sample code to make sure they work. Recently, I've been searching EBay for various sensors for these boards to play around with. Borderless Electronics, has now come back with a follow up campaign, featuring various combinations of a kit, board and shield. I'm also contributing, to get some shields for my Arduinos, so they can be network enabled.

While on the topic of crowdfunded projects, I also backed the Pressy project. This is going to be a great way to kick things off on my Note II, like triggering my voice recognition home automation script.

Friday, July 12, 2013

What I Automate

This is something I've been meaning to write down, but never got around to. Let's see what I can remember:

  • Usual lighting control - control from switch, keypads, PCs, mobile devices
  • Night lighting - all rooms except the bedrooms have motion activated night lighting, which is overridden by light sensors
  • Back lighting - turning on backyard lighting when the back door deadbolt is unlocked and turning off when it's locked
  • Protocols - X10, UPB and ZWave

    Notification methods:
  • Whole house text-to-speech (TTS) announcements - for announcing weather, reminders, warnings, etc during normal waking hours
  • Whole house sound effects - various different notifications sounds for mailbox being opened, doorbell ring (changes with the season), motion on the porch, etc.
  • Zoned TTS announcements - Insignia Infocasts and SqueezeBoxes in bedrooms and living areas for targeted notifications
  • On screen displays (OSD) - Infocasts, PCs and TVs also display announcements in case they can't be heard (watching a movie, blasting the stereo)
  • IM - notifications are sent via IM when away from home and Internet connected
  • Email - when attachments are part of the notification
  • SMS - Google Voice sent SMS for when we're out and not Internet connected
  • multiple - depending on the severity of the notification, multiple methods may be combined. OSD & TTS typically happen for most notifications when someone's home

  • HVAC - disable HVAC when alarm armed in away mode or when a window is opened, warn if someone's trying to turn it on when a window is left open via TTS, OSD, IM
  • Fans - 1Wire temperature sensors in each bedroom control when to turn off any fans running on summer nights
  • Exhaust fans - turn on/off laundry room & bathroom fan based on humidity, guestbath fan based on duration of occupancy ;)
  • Leaks - wetness sensors under sinks & in bathrooms and alerts sent via all notification methods

    Info/Status notifications:
  • Temperature & weather for the day - TTS announcement in the morning as we come downstairs
  • Google calendar appointments for the day - same as above. as calendar events hit their reminder time, notifications are sent via IM and OSD.
  • Phone calls while we're out are announced as we enter the family room
  • Mailbox - OSD & sound on event. TTS reminder later to check it if we haven't gotten the mail.
  • Doorbell - OSD & sound effect on event. TTS to tell us the doorbell rang while we were out
  • Temperature - sound effect when it is cool enough to open the windows on days over 80°F
  • Washer finished - TTS, OSD, IM
  • Garage door left open with no activity for a period of time - TTS, OSD, IM, SMS
  • Front door left unlocked for a period of time - TTS, OSD, IM, SMS

  • Google Voice forwarding - when arriving home, Google Voice forwarding for the home phone is enabled, when leaving it's disabled. arriving & leaving work also changes GV forwarding to my work phone.
  • Caller ID - incoming calls are checked against all our Google contacts, and if it's a match, the person's name (number if no match) is displayed & announced via TTS, OSD and IM.
  • Intercom - wired phones and softphones for calling specific rooms from other house/softphones
  • Mute/Pause AV - when the main phone is picked up.

  • TV - automatically turn on when DVD player is turned on (but not off since we may watch something else after). game consoles are connected to one tv that is turned on & off when the consoles are.
  • Whole house audio - via Squeezeboxes
  • Uptime - all AV equipment in the house are monitored by current sensors so their actual power on status is known. uptime, last on time, and last off time are tracked.

  • Feeding - automated feeding 1 hour after our morning run, when I've already left for work. when the feeder is used, I get an SMS confirmation and a picture of the happy eater
  • Outdoor bark deterrent - when barking is detected by the side gate, verbal corrections are played by the garage speaker and a stream of water is shot at the gate area.

  • Kids' laptops - when on, the kids are reminded every 20 minutes to get up walk around and look away from the screen
  • Game consoles - when on, the same 20 minute reminder is announced (and the TV is muted to reinforce the message)

  • RFID - tracking home & away status of cars - TTS, OSD, IM announcement when a vehicle arrives home

  • Run time - adjusted based on rain, weather forecast & past temperatures

  • CCTV + DVR - typical motion based recording of cameras
  • Away notifcation - Email snapshots of particular events (doorbell, mailbox, porch activity, etc) when we're out
  • Triggered - all sorts of notifications

    Power monitoring:
  • Real-time - whole house power monitoring split into 7 zones
  • Tracking - local via rrdtool & canvas.js, cloud via automatic updating to Google Drive spreadsheet
  • Oven - warning via TTS, SMS if arming house in away mode and the oven is on
  • Solar production comparables - in progress, comparing current production of array to past to determine if cleaning is necessary

  • Locks - deadbolt state monitoring
  • Reminder - warning if we unlock a door and the alarm is still armed in home mode (so we don't accidentally set off the alarm and wake up the neighborhood)
  • Away lighting - replay lighting events from some randomly chosen log file
  • AV - turn off AV equipment when armed in away mode

  • Backups - typical automated local and remote backups
  • Monitoring - cpu utilization & temperature, hard drive utilization & temperature, uptime, low disk & memory notifications, automatic killing of runaway processes on servers
  • Internet - bandwidth & connection monitoring, auto power cycle modem & router if connectivity is lost
  • Power - automatic hibernation of certain powered on machines when alarm is armed in away
  • Wake from hibernation - automatic waking of my laptop from hibernation upon first motion in the master bathroom in the morning or arrival home of my cellphone
  • Email - TTS, OSD notification of new emails
  • Craigslist bot - periodically scans CL for things I'm looking for (found a $10 and $20 Squeezebox this way!)

  • Motion sensors - some hardwired, many wireless all over the house & around the house
  • Audio - sound detectors upstairs and in the living room
  • Video - one camera connected to a hacked Seagate Dockstar is being used for a specific motion detection purpose
  • Cellphone - via bluetooth, mentioned above
  • Car - via RFID, mentioned above

    Control methods:
  • Android speech recognition - using our phones & tablets (and SL4A+Python) to voice control lighting, AV equipment and various appliances, query status of devices and control audio players and music selection
  • IM - using the same language parsing as the speech recognition, everything that's voice controllable can be controlled via IM
  • AJAX Floorplan GUI - control & view status of light/appliances/AV/HVAC from cellphones, tablets, PCs - basically anything with a modern browser. see status of doors/windows/locks/temperature
  • Insignia Infocast (Chumby) - used as touchscreen control panels with our Panel Builder app and "offline firmware" (since Chumby service is essentially dead)
  • Wireless keypads - X10RF & ZWave based keypads
  • IR - all Squeezeboxes are IR receivers and can broadcast the IR codes they receive via the xPL plugin

    2nd Home:
  • Minimal automation - lighting, occupancy, cellphone presence, leak sensors
  • Remote link - all status is updated to the main home via IM and lighting can be controlled over IM

  • Cellphone tracking - running on my office laptop, IMs my presence to the HA server when my cellphone is around, to adjust Google Voice forwarding

    Old stuff (no longer in use):
  • Baby monitor - broadcast through whole house speaker system and local Shoutcast channel (for listening on a PocketPC PDA when I was out in the yard - yes it's been a long time since the kids were babies), automatically turns on when drop side crib (now banned) door raised and off when someone enters the room (and back on after a period of inactivity if the crib door is still raised)

    This is just a quick list. I'm sure I'll add to it when I have more time... I need to annotate this with some pics and links too (in progress)
    This is a screen cap of the main screen for our Insignia Infocasts (aka Chumby) that we have around the house