Using SL4A and Android Speech Recognition for Home Automation

My latest project has been experimenting with SL4A (Scripting Language For Android) and Python on my Galaxy Note II. I started with the included saychat.py sample to build a simple script that kicks off Android speech recognition. It takes the text result returned from Google and sends it over IM to our HA server. The HA server does some basic natural language processing on the text, extracting commands and performing the operations if any valid ones are found. It then returns a response over IM to the phone with the result of the command(s). Back on the phone, the Python script has been waiting for this confirmation and uses TTS to read it back. The cycle repeats until the user says "goodbye" or it gets two consecutive recognition results with no speaker. Here's a short YouTube video of it in action:



From the video, you can see I've tried to parse the speech so that it can find the commands and devices even if the command is spoken differently. I used three different phrases:

"Can you turn on the kitchen light and dining room light?"

"Can you turn off the lights in the kitchen?"

"Turn off the dining room light."

I was trying to avoid having only simplistic commands like the last one. The first one demonstrates that ability to speak a command for multiple devices, and the ability to preface the command with "Can you" or pretty much anything like "The dog wants you to" ;) The 2nd command shows that it's not restricted to parsing "kitchen light" together. The last command is a typical HA VR command. My parser also has the ability to decode multiple commands in one, such as "Turn off the kitchen light, the guest bath fan and living room light and turn on the back floods." The only challenge is saying everything you want to say without much of a pause, otherwise recognition stops and the partial command will be sent.

A few advantages of using this setup:

Google's speech recognition in the cloud is probably the best, most up-to-date system. They started building up their system with the now closed GOOG-411 service. Further fine tuning gets done on the millions of voicemails their Google Voice service transcribes. Their Chrome browser also uses their speech recognition and of course, so do the millions of Android users. All this input goes into tuning their accuracy, and what you end up with is one of the best performing, up to date speech recognition systems. If you're using Microsoft's Windows VR, you're probably getting something that gets updated every few years with each OS release - if you're upgrading. With HAL, you've getting a 1990s VR engine. I'm not even sure if that gets updated anymore.

Google's free form speech recognition allows the most flexibility in speaking commands. Granted, that makes the parsing more difficult, but it allows a system that can more accurately respond to the different ways different people phrase commands. Most speech recognition engines I've worked with require you to pre-program canned phrases in order to recognize commands. If you deviate just a little from what's programmed, good luck getting your command recognized.

By using Jabber IM as a transport mechanism for the recognized commands, the same system that works at home, works when you're away. You just turn on your mobile data - there's no VPN or SSH tunnels to set up every time you want to speak a command. There's one level of security for free since your home's IM client must have pre-approved other users to allow communication (adding them to the roster). Another level can be done at the scripting layer of your HA software, by limiting what IM users can issue certain commands. For extra security, you can even encode or encrypt the text being sent over IM if you want, but if you're using Google Talk servers, your communication is already wrapped in SSL.

A few more details. Using SL4A, I cannot control the default speech recognition sounds - it can get annoying after a while. I'm using Nova Launcher as my launcher instead of TouchWiz. Nova Launcher let's you remap the home key behavior on the home screen. When pressed, instead of showing the zoomed out view of all my screens, it kicks off the script. Also, my HA device database is stored in mySQL, which allows for powerful searches and easy matching of what's spoken to actual devices - even when the device name isn't exactly the same as what was spoken. I've been using the mySQL setup, IM interface and command parsing for many years now, (although the parsing was more primitive) so integration was extremely simple. At some point, I would like to implement NLTK, the Natural Language ToolKit, for more complex language processing.

2012 Most Downloaded (3 months late)

Here is our list of popular downloads for 2012:

1. EventGhost xPL Plugin - 99 times
2. xScript - 77
3. BlueTracker - 61
4. xPLGVoice - 39
5. xPLSerial - 17
5. BlueTrackerScript - 17
7. xPLGCal - 16
8. t2mp3 - 15
8. Blabber - 15
8. Noise - 15
8. xPLChumby - 15

The EG plugin continues to be popular despite our stopping development on it years ago. Four of the top five and their downloads are almost exactly the same as last year, with xPLChumby dropping and xPLGVoice getting more interest. There are no new apps on the list since I did virtually no HA last year, but it's nice to see there's still a similar amount of interest in our existing apps. I haven't had the urge to code anything new, let alone time to brainstorm new things. We'll see what 2013 brings...at least I'm blogging again.

Solar Year #3

We just finished our 3rd year of having solar panels, where we generated nearly 75% of electricity usage and saved $1107. After the first year, we've done a good job of reducing our electricity usage by unplugging unnecessary devices and consolidating servers. However, I'm always looking to buy more gadgets :).

year	total usage kWH	solar kWH	grid kWH	savings $	avg monthly kWH	% solar usage
1	8636.35	5811.2	2825.15	$1,234	719.7	67.3%
2	7640.2	5739.2	1901	$1,144	636.7	75.1%
3	7410.95	5548.95	1862	$1,107	617.6	74.9%

Total output from the panels has decreased each year, but that's been mainly due to a larger number of rainy or overcast days. I'm sure some of it is due to dust collecting on the panels. I have yet to get up on the roof to wash them off, and it's something I plan on doing this spring. The wet winter we've had this year has done a decent job of washing the panels for me. Overall, we're very pleased with our solar panels, which have already paid for 28% of the cost (slightly less than I projected due to us using less electricity).

Tasker $1.99 at Google Play

Tasker, an app for automating almost anything on your Android device, is on sale at the moment for $1.99 (regularly $6.49). I was interested in buying it a while ago, but $6.49 for an app seemed a bit high. Glad I waited. Get your copy at Google Play.

Automated Barking Dog Correction

Our Australian Shepherd is a wonderful dog, but he's a bit protective of our property. When he's out in the backyard, he will bark when he senses anything unfamiliar to him in front of our house. He tends to bark near the gate, so I've wired a microphone placed in a garage vent near the gate back to a server. On that server, we've been using our Noise app to detect when he's barking. It generates an xPL message when the sound levels detected exceed a specified threshold. There's a few conditions that are checked before the system decides if he's barking, like if the gate and garage door are closed and there's motion detected near the gate. The server will then play recorded MP3s of our voices telling him to be quiet through a speaker placed near the same garage vent. It worked for a while, but our dog got used to it and started ignoring it. The next step was to purchase a cheap windshield washer pump from Amazon (see below). The pump was connected to a gallon juice jug filled with water, and some plastic tubing was connected to the other end of the pump, routed through the garage vent and aimed at the gate. Finally, the pump was soldered to a 12V wall wart connected to an appliance module. Now in addition to the verbal correction, the pump gets turned on for 3 seconds, shooting a stream of water at the area behind the gate. He hates getting wet, so it's no surprise that the frequency and duration of barking has drastically declined :)

Follow up: Barking has declined from 3-5 times per day to maybe once a week!

It's Alive...Alive!!! (Our Infocast/Chumby that is)

Since the Chumby servers shutdown (and was switched to a stub service), the only available app is a clock. Some time ago, a Chumby user, Zurk, created an offline "firmware" for Chumbies to run without the Chumby servers in case they went down. He also released an app called octopus that downloaded all the apps off the Chumby servers, which was the thing to do when the servers were still up. With the downloaded apps and the "firmware" (more like some scripts and a local collection of Chumby files than actual firmware) Chumby nirvana could mostly be restored. I really only care about 1 app, Panel Builder 2, which runs on all our Infocasts. That rarely changes unless one of the kids wants to use some other app. I didn't even notice the Chumby servers went down because our Infocasts were still running the cached PB2. Slowly, one by one our Infocasts got reset to the default stub server clock. When my nightstand Infocast switched over, and I couldn't control the house from it, that was the last straw! I had to get this Zurk thing working. It was pretty much plug & play - just unzip the contents to a big enough USB disk. Panel Builder 2 was a private app for my testing only, so it couldn't have been grabbed by octopus. Since I wrote it, I just needed to stick the original .swf file on the USB drive, edit a few files to add the app and the Infocasts are useful again :) I've slowly been tweaking things and testing some other apps, but everything's back to normal. One caveat, it looks like apps that need configuration will not work. With PB2, I just hardcoded my server address and compiled a new .swf.

xpllib the CPU sucker

Recently, I had time to reinstall the OS on our media server, which was acting a little sluggish. I upgraded the RAM from 2GB to the max 4GB and installed Windows 7 x64 instead of the 32 bit that was on there. With the extra headroom, I migrated some xPL apps from my HA server to this one, but what I discovered was surprising. The media server has a relatively modern Pentium dual-core E5200, yet periodic bursts of xPL traffic would spike the CPU utilization up to near 100%. Three apps of mine (xPLGMail and 2 instances of xPLGVoice), would each suck up about 25% of the CPU. This didn't happen when those apps were on the E6420 Core2 HA server, but I did see something like this years ago when I was putting some xPL apps on HP T5700 thin clients. The culprit was the xpllib dll. The CPU intensive one is version 4.4.3663.31835, but some other apps I wrote using version 4.3.2737.14049 rarely use 1% of the E5200 CPU. I ended up taking a step back and recompiling xPLGMail and xPLGVoice with 4.3. Just like that, those apps never used more than 1% of the CPU. This reminded me of some years ago, when one of the more recent xPL devs started building a whole new xpllib (V5). At that time, I was working on the T5700s (still in use) so there's no way I would use an even fatter, more CPU intensive xpllib. (V5 at 320KB is almost 8 times bigger than 4.3 at 44KB). Not only that, the new V5 is not backwards compatible. I wasn't going to rewrite 20+ apps to "move forward." In fact, it looks like I will be migrating all my xPL apps backwards from xpllib 4.4 to 4.3. There's no need to waste CPU cycles for equivalent functionality. I'm slowly getting back to HA and it feels good.

Where Have I Been and Where is Chumby?

I've been here...working at another startup. It's been about 18 months and things really got hectic about December 2011 and hasn't let up since. All my HA work has been put on hold, with minor tweaks here and there. It's been my longest time away from HA since I started all this back around 1994, but I've done so much and everything just works. It's quite satisfying. Things are slowly easing up at work and I'm finding a little time and interest to do some more things, but it'll be a while before I'm able to commit as much time as before. One thing that really bums me out is the Chumby service has finally wound down. There's hope that it may come back, but for now there's just a clock widget. At some point, I will probably set up my own local server to serve widgets to our Infocasts, but there's no time for that right now. Chumby RIP for now.

Home Automation 2011

Since I wrote about 2011 downloads, I thought I'd write about the automation & related projects I worked on this year.

1) The biggest effort for the year was the Chumby/Infocast project that resulted in Panel Builder, xPLChumbyTTS, and the xPLChumby Python script. I learned how to program ActionScript/Flash and ended up with these awesome, low power and cheap touch panels that make great picture frames and TTS clients. I also totally revamped Panel Builder with a V2 but never finished it up enough to release. Built screens to control every aspect of the HA system, a lot of the screens automatically generated by PHP scripts populating from from existing MySQL databases. The Infocasts also got updated to run Chumby 8 firmware.

2)Integration of the OBi110 VOIP adapter with our existing Asterisk and MagicJack setup. Wrote a syslog Perl script (the main parts are available here) to monitor all sorts of events from the Obi, including touch tones, line status & caller ID. It also does caller ID lookups from our MySQL db of contacts sending results via xPL. Also migrated Asterisk from a Dockstar to the SageTV server.

3) Upgraded SageTV server from Server 2003 to Win 7 Pro & wrote a new xPL Sage app to replace my xPL Plugin. Also paid for and upgraded SageTV to V7 just weeks before they were acquired by Google and gave away the V7 upgrade for free >8(

4) Built an xPL Facebook app with a scripting engine and created a Doghouse Labs page where certain random events are posted on the fb feed via the app.

5) Did a bit of upgrading my Jab2Twit Jabber-Twitter connector. It's the main way I interact with Twitter since GMail/GTalk is always on my PC or phone. It's great getting an IM whenever Amazon's Appstore announces their free app of the day - no need to start another app or check their webpage to find out.

6) Upgraded all my IM client apps, including Jab2Twit, from using the Jabber-NET library to agsXMPP, which is much more robust.

7) Wet my feet with Android development building in tweaks to gtalksms.

8) Did a lot of tuning & tweaking w/ BlueBlabber, my bluetooth-Jabber connector, which I have running on my work PC so my automation system knows when I'm in my office. I just need to publish the latest version. As you can tell, I love using IM as a communication protocol. It's great for bridging remote networks - no need for firewall holes - plus it has the added security of SSL encryption. You can also encode your messages your own way as well to make things more obscure.

9) Added Z-Wave to the HA system, hacking xPL into the open-zwave demo app. I'm really liking Z-Wave, even more than UPB.

10) Lucked out & got a couple of $99 HP Touchpads in the first wave. Installed CM7 on them & have been extremely pleased. The kids & wife love using them.

11) Got the kids (8 & 10) AKA spoiled brats ;) their own Sandy Bridge based laptops, loaded them with Ubuntu & Google Chrome running AdBlock and WOT extensions, and configured with OpenDNS Family Shield so they can surf safely and smartly. Copied the xPLChumby Python script to their laptops so I can send text-to-speech and on-screen messages to their laptops. Added them to my host monitoring so that every 20 minutes of uptime, they are sent a reminder to get up and walk around to rest their eyes and stretch their legs.

12) Hacked pyrocket to put USB rocket launcher on a Seagate Dockstar and use as an xPL controllable pan-tilt webcam.

13) Wrote an extension of xPLGameport to work with cheap 12 button gamepads, obviously named xPLGamepad.

14) Replaced our public webserver (NSLU2) with Seagate Dockstar and picked up a cheap Pogoplug as a backup for the 3 Dockstars that are deployed. The Pogo's been loaded up with Arch Linux.

15) Finally deployed the WebControl I bought nearly 2 years ago. For now, just using the analog inputs and some CdS sensors to detect light levels in rooms. Of course, it's using xPLWebControl - which got a new feature - a fake SMTP server that the WebControl can send an email to, which will be converted to an xPL message.

16) Wrote a Perl script to screen scrape Yahoo Sports and send TTS over our whole house speaker system so I can listen to play-by-play of football games that aren't on TV while I'm debugging the house.

17) Added IP control of our Samsung TV and Blu-ray player via a Python script.

18) Installed 2 new outdoor bullet cams to upgrade the coverage in front of our house.

19) Rooted and installed a custom ROM on my Galaxy S phone in order to fix GPS that Samsung broke with the Gingerbread update and get rid of the Carrier IQ crapware. Been automating my phone lately with Llama and android scripting.

20) Finally unplugged the last Rio Receivers, which were in the kids' rooms. The kids got migrated to Squeezeboxes and I shut down MediaNet, which, with its predecessor xPLRioNet, have been in use with the Rio's since the early 2000s. Those servers first introduced me to xPL and really opened my eyes to a lot of what can be done with home automation.

21) Lots of minor things:
- hardwiring sensors that were once connected to DS10A's now connected to a distributed network of gamepads/gameports residing on Dockstars, thin clients & servers.
- script to import Android contacts into MySQL DB for caller ID lookups
- constantly refining the 4800 line script that runs in starCOMUltra (my main automation engine) and the 2500+ lines of code running in my xPL scripting engine
- did some natural langugage parsing for the system's IM interface - allowing better voice control via IM using the speech input feature of Android & Google Talk - before people got all hot & bothered about Siri & home automation
- integration of reading Google Voice SMS's based on this

I think that's a pretty good summary of my HA work in 2011. What about you?

Most Downloaded of 2011

Here is our annual list of popular downloads:

1. EventGhost xPL Plugin - 101 times
2. xScript - 67
3. BlueTracker - 54
4. xPLChumby - 51
5. xPLSerial - 21
5. xPLWebControl - 21
7. BlueTrackerScript - 16
8. xPLChumbyTTS - 16
9. xPLGVoice - 16
10. xPLWav - 10

Surprisingly, the EG plugin led the way yet again, but overall downloads were lower than last year. I admit, I haven't been as prolific turning out apps this year. The whole Chumby/Infocast effort was really draining and I don't I think ever recovered from it. It took a lot of the fun out of developing. We do use the Infocasts all the time though. The real interest in Chumby wasn't seen in the download numbers since Panel Builder gets served off of Chumby.com. I did see over 1200 unique IP addresses previewing Panel Builder - although I don't how many actually installed it.

Power Savings and Prepping for an EV

Last week, I had some free time to reassess what devices I have running and whether or not I still use them at all. This was born out of some preliminary research into purchasing an electric vehicle (EV). Nissan Leafs are actually available now to those without reservations. That's the more realistic choice, but I really like the Telsa Model S. I've been missing a sporty car since my Honda roadster was totaled last year.

Taking the Nissan Leaf as an example, with its 24 kWH battery and 100 mile range and my current commute of about 30 miles roundtrip, I could drive 3 days per full charge. I would drive one of our other cars one day a week, so that would leave about 16 days/month on the EV. Rounding it to 15 days, I'd need 5 full charges or an additional 5*24kWH = 120 kWH of electricity per month.

Fortunately, we had solar panels installed 18 months ago. Since then, we've been able to stay in the lower 2 tiers of electricity costs for 11 out of the first 12 months we've had them. Those lower tiers are now 12.2 cents/kWH and 13.9 cents/kWH. The 3rd tier jumps to over 30 cents/kWH and the 4th and 5th tiers more costly. In December 2010, we used 73 kWH of tier 3 electricity. The 2nd worst month was January 2011, but we were 60 kWH below tier 3. The other 10 months we were 150-500 kWH below tier 3.

So if nothing changes, it would be "expensive" to drive an EV in December. The 120 kWH of electricity to charge the EV would all come in tier 3 at a cost of about $36 for 450 miles. That ends up being more than the $33 it would cost to drive our Prius the same distance (assuming $3.70/gallon for gas), but less than the $70 it costs our Subaru. It would cost about $26 in January. April to October would be all tier 1 electricity and cost about $14.50. February, March and November would be about half tier 1 and half tier 2, or about a buck more per month than April-October.

These are hardly savings compared to a hybrid - maybe $150/year. Compared to our non-hybrid, it's more significant - $550/year. When the economy improves, those savings will go up as gas prices will rise. For me, the real savings will be not sending as much of our hard earned money to big oil companies or countries that want to destroy the American lifestyle.

Political statements out of the way, there are some special options our electric utility has for EV owners. One is a time-of-use option where they charge a high rate during peak periods and less during the late night-early morning. This wouldn't work out well for us because of all the HA related servers, electronics, etc running 24x7. Another option is the installation of a 2nd electric meter specifically for charging the EV, but that's at least $2000. A third option would be to install an additional 6 solar panels on the west facing portion of our roof. It's not an optimal position and I haven't priced it out - but I figure it would be around $4000. I could do some of the work myself as I did an install with SunWork before. Of course, the obvious option is to cut our electricity usage - specifically the things that run 24x7.

I have to admit, having solar, I've been lazy about conserving. Rather than run a long Cat-5 cable, I plug in another Ethernet switch. Instead of using an extension cord to get to a UPS already installed, I add another UPS. When security cameras get updated, I keep the old ones plugged in pointing somewhere else - not really someplace necessary. Last weekend, I fixed all that. I pulled 4 UPS's, 4 Ethernet switches, 2 cameras, 3 modulators, one Squeezebox and 1 Slingbox. I disabled Connect24 on our Wii so it can idle at 1W instead of 9W. I unplugged a clock radio in the garage that used 10W. The end result is I cut about 3 kWH/day or 90 kWH/month. That's almost as much as the 120 kWH/month an EV would require!! This was by far the cheapest option. You can really see the results in a chart. In the 5 full days since my unplug-fest, our usage has never exceeded 20 kWH/day.

Now, we have headroom for an EV - I just need to decide whether I'm ready to plunk down all the cash for one.

Random Bits

I just finished my first week at my new startup and I'm already buried. New designs & new methodologies to learn and I need to know it by yesterday. I've spent a lot of time this 3 day weekend going over some training material I got from a co-worker. I did, however, get around to some minor automation stuff.

I finally hardwired some contact closures which were connected to DS10As. I just needed a little more accuracy for these sensors and sometimes I would get collisions with other X10 wireless devices which would cause status updates not to be received. I've wired these sensors to a USB gamepad connected to the thin client that interfaces to our Brultech power monitor. My xPLGamepad app relays the closure state via xPL.

This weekend I also upgraded to the kids from Rio Receivers to Squeezeboxes and turned off Medianet for good. It was a good run, but most of the house was running Squeezeboxes so it was time to consolidate to one music server. Medianet really helped push me deeper into home automation. Years ago, while searching for alternate servers for the Rio Receivers, I stumbled onto Medianet's predecessor, xPLRioNet. Installing it not only gave new life to the Receivers, but introduced me to xPL which has become such an intricate component of our HA system.

Finally, we've got these chef's mats on our kitchen floor, which are pretty thick. The Roomba will go up one of the mats, but will never go down due to its cliff sensors. This makes for one very clean mat, but that's about it. The cliff sensors work by bouncing IR off the ground and if it's not reflected back strong enough, the Roomba thinks it's on an edge and won't continue in that direction. I never use the Roomba upstairs so I chose to disable the cliff sensors. Fortunately, I had some aluminum tape from some HVAC work I did and used it to tape over the cliff sensors. The aluminum works perfectly and reflects the IR back to the cliff sensor and our kitchen floor is now clean again.