Homeseer in a Distributed Environment

[Michael McSharry]

I been working with XAP for the past couple of months with the objective of making a fully distributed HA environment. It started as I
was looking for a solution where my 1-wire data collection could be run independent of Homeseer. I'm continually restarting Homeseer and disrupting the data collection as I evaluate new development activities.

XAP provides a very nice solution to the problem and it has the advantage of being a documented standard with a supportive and growing community. It allows interfaces and applications to be run on any computer of a network without the problems associated with DCOM, authorizations, or object bindings.

The structure I have setup is geared to many individual small applicaitons doing one specific function and interfacing with the XAP framework. It is like an army of agents running on the network. Agents can come and go without disruption of the network overall.

At this time I have a small set of "tools" upon which the environment is being bootstrapped. This is in addition to the core tools available from the XAP organization. I also have a plugin host that allows each homeseer plugin to be run by itself. Each plugin is now its own application and not bound to Homeseer, but interfaces to homeseer via the XAP framework. The binding element is an XAP Homeseer plugin that makes Homeseer act as just another agent on the network.

A this time I have the following working
1. mcsXAP: Homeseer plugin which allows homeseer to participate in the XAP environment. It includes triggering capability and message source capability.
2. xapScope: Application that stores XAP traffic in a database much like one would use a storage scope to observe waveforms
3. xapWritelog: Application that collects all hs.Writelog commands from the network and stores them in a database. Functionally similiar to UltraLog, but applies to any application on the XAP network rather than just Homeseer. Actually could be replaced by Ultralog, but used initally during the bootstrap.
4. xapProcess: Application that can start, stop, restart, and collect RAM and CPU utilization statistics for processes running anywhere on the network.
5. xapConfig: Application to consolidate access to all applications on the XAP network. It provides access to the setup pages and a control GUI to interact with xapProcess. This allows a control console to be run on any computer and prevents the task bar from filling up with icons to get access to the various agents.
6. xapTemp0x: Application that interfaces with Temp05/08. Readings are placed on the XAP network and control of outputs also routed via the same mechanism.
7. xapDataCollection: Application that grabs classes of data and store to a database. It is the companion to xapTemp0x and mcsTemperature plugin. The other 1-wire and CSV input components of mcsTemperature will be split out as well in the future.
8. HomeseerOverThere(HOT): Host for Homeseer Plugins. Multiple instances of HOT are run with each instance holding a Homeseer plugin. I have tried a variety of plugins for basic functionality. The MR26a, CM19A, ADIOcelot are fully tested as I/O type units. The W800 was
shown to not work in this environment. mcsMOvement is tested as an example of one that does not have an IO interface, but provides
value-added processing. Most of the pluigns available from the updater were run just to demonstrate interface compatibiltiy, but no
testing of these beyond that point.

The attached document has been evolving as the development proceeds. I'm posting it now to see the level of interest, obtain any feedback, and see if any others would like to be involved in the development.

[Rupp]

Michael,
In your example<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> . For example, the MR26A plugin can run by itself on Computer A and make all other nodes aware of X10 RF receptions via XAP messages. Homeseer can be on Computer B, and a CMA11 can be on a third computer, if desired, and it will field the MR26A’s retransmit requests if it is configured for retransmission. If you have two MR26A’s then install both on the same or different computers and let the system redundancy management assure the correct information is used at each of the nodes. <HR></BLOCKQUOTE>

How fast is this exchange. Most have complained that the current handling of RF is quite slow. Wouldn't this extra exchange layer be slower?
-Rupp

[This message was edited by Rupp on Sun, 14 March 2004 at 09:08 PM.]

[electron]

As someone who has deployed many firewalls and servers in a distributed environment, I honestly can't see an efficient distributed environment without Homeseer including this functionality directly into the program itself (which means supporting features such as a heartbeat etc so they can failover). That doesn't mean your setup isn't working great, I just believe it's something HST should address.

---
Do you cocoon?
http://www.CocoonTech.com

[Michael McSharry]

I cannot perceive any difference in the timing. It is in the milliseconds during my testing. Everything is interrupt driven. The timing difference is on the order of the difference in timing between an ocx vs. an exe plugin where there is an interprocess schedule to be done within Windows.

Take a scenario where the MR26A is running on computer X and Homeseer is on computer Y. Computer X and Y can be the same or different. In this case the homeseer process is not bogged down by running the MR26A processing, including any critcal lock sections. When the MR26A processing has completed its analysis of the RF signal then it transmits a packet on the network rather than making a callback to Homeseer. This small packet is communicated at ethernet speeds (10/100) and an interrupt generated that makes the data available to the xap plugin and this plugin then makes the callback to homeseer. Note that this packet is received by all nodes at the same time so if another application is interested in X10 traffic then there is no additional delays.

It is true that additional network traffic is generated and additional processing required to send and transmit the messages, however this is so small that it is not noticed and it tends to be random rather than bursty.

Contrast this with using an ocelot for X10, as an example. You are running at serial speeds to get the X10 data packet between homeseer and the ocelot and the serial communication is scheduled with other serial activity in the ocelot. The ocelot cpuxa runs in a polling loop so messages will be acted upon from 0 to 1000 milliseconds after they are actually received. If the ocelot is used in both directions then the delay is doubled.

[Rupp]

Does this use UDP as the transport protocol?

-Rupp

[rocco]

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>The MR26a, CM19A, ADIOcelot are fully tested as I/O type units.<HR></BLOCKQUOTE>Michael:
Are you saying I could control an Ocelot on one of my garage computers from HomeSeer in the house?

And am I correct to assume I can't distribute a CM11A, because it is not controlled by a Plug-in?

I will be reading your Xap-doc-dot-zip

[bdickhaus]

I think this is a fantastic concept. But I agree with Dan that for this to really work well, HST needs to implement this natively in HomeSeer. In the meantime, I don't see any reason not to continue discussion (and development).

I have been contemplating running multiple instances of HomeSeer to deal with some of these issues. This is a more elegant and more flexible solution.

Bill

[zoomkat]

It would look like this would only work in a limited controlled enviroment, other wise the systen would be flooded with packets that may only be of use to a fraction of the machines having to sift through the data to see if it is applicable. Would the broadcast method have error checking on the receiving end? This looks like a massive spam type of setup at the packet level.

[Michael McSharry]

Dan,

Can you illustrate why you believe changes are needed in homeseer to support a distrubuted environment?

With Homeseer open to plugins there really is no difference that I can see between logic that is provided by homeseer.exe vs hspi_xxx.ocx. Both are run in the same process and both have callback mechanisms that make them seemless. The xap plugin supports all the network traffic including the heartbeat.

The problem area that I am aware of is management of system state. While this is a longer range consideration, my primary focus is to separate interface processing from the Homeseer core.

In a Windows environment, or any timesharing environment, it is not possible to achieve voted output among redundant nodes in real-time. This implies an architectural solution where at any point in time there will be one node that contains system state information.

Just asking again... I do not yet understand what could be added to homeseer to better support a distributed environment.

[Rupp]

Can you translate this to us common'ers language. What does this mean?

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> In a Windows environment, or any timesharing environment, it is not possible to achieve voted output among redundant nodes in real-time. This implies an architectural solution where at any point in time there will be one node that contains system state information. <HR></BLOCKQUOTE>

-Rupp

[jhisr]

If I'm reading this correctly, the XAP architecture sounds similar to LonWorks which has been around for at least 10 years.

The premise with the LonWorks network is that all devices on the network broadcast messages and all devices listen for messages and depending on the message, react to them. The amount of packets pale in comparison to traffic you would see on an office network. I like the concept of distributed intelligence.

For example, if the phone rings, the phone will send out a "phone is ringing message." The stereo hears that message and knows it's supposed to mute itself when the "phone is ringing" flag is set. The refrigerator also hears the "phone is ringing" but it's doesn't care and doesn't do anything about it because the refrigerator wasn't programmed to react to the phone flag.

LonWorks is big in Europe (at least it was at one time) but I don't think it ever caught on in the states for home automation. Back in the early 90's there was a big fight to see who would become the so called "standard" in the industry and the two major contenders at that time were CEBus and Echelon's LonWorks. BACnet has made some strides, most notably in HVAV and building management.

I think Echelon made a mistake by making the language proprietary. You had to buy their compiler and tools and in the beginning it cost about 25K if you wanted to become a LID (Lonworks Integrated Developer).

XAP looks intriguing because it's open to everyone. Is XAP similar to LonWorks or am I way off base?

Joe
HomeSeer Rocks!

[Michael McSharry]

rocco,
Yes you can put your ocelot on any computer you want and use it from the Homeseer computer in the house.

The CMA11 has several implementations on XAP. This is one of the first things that was done. I know it is used as a candidate for those new to XAP to get their feet wet. I have not tried it myself, but I'm certain it will work without any problems.

Rupp,
XAP is a protocol specification that does not include the hardware layer. A consideration was given, however, to UDP and the specification states that the maximum message size is around 1024 bytes which is the UDP max. Today's sweet spot for efficient communication is ethernet and my implementation is UDP. I currently only have very small packets defined and do not expect to go beyond that. Entire web pages would not be appropriate using XAP, but an application that queries the web and returns current weather data, news, etc. have already been implemented.

zoomkat,
This is intended for a Home Automation LAN. It would not be appropriate to try to run it in a large company domain. The responses that I have received from the XAP community is that the XAP traffic is in the noise with respect to available network bandwidth. Since I'm relatively new to the technology I have no reason to doubt their inputs.

Other than the periodic heartbeat the only messages that occur are those that are a result of events. If one wanted to flood the network then the capability exists, but that is also true of running scripts that consume 100% of the CPU. One must use the available resources wisely.

As a note, the XAP protocol does define the mechanism to bridge networks. A very common implementation is to bridge RS232 to ethernet. PICs and other dedicated processors will typically do something like this. It is possible to bridge two ethernet networks where only a subset of information is transfered. There is an offshoot of XAP, called XPL, that does not support some of the routing and simplified the packet structure. I'm certain either will do the job for this application. I elected to stay with XAP since it is a little more mature.

The XAP protocol requires a message to be well-formed, however it does not impose a checksum or CRC mechanism on a packet. A well-formed message has a header with certain content, and body with key-value pairs. Messages that are not well-formed are generally discarded.

A message header contains the source/target/class information necessary for a receiver to know if they should pay attention to it. It does not take very much logic to discard messages that are not of interest.

I did not look at the lower-level protocol requirements, but it seems reasonable that hardware CRC would be part of a message that is managed by Winsock. I have not seen any messaging problems, but one could always add a checksum to any "critical" packets. The XAP message classes are quite extensible.

[zoomkat]

From the little I've read in the document, it will be generally less reliable than what is now in place with the tcp/ip and other protocols. I think when this thing gets developed and fine tuned, it will be what is already available now. What is the stated advantage of this type of system? What can this system do that can't already be done now? I'm like that little lady that looked up over the counter and said "where's the beef?" ;-)

[Renegade]

I know I would love to see HS be able to support a distributed environment. Not having to worry about resources on my HS PC would be a definite plus. Whether that be CPU, RAM, serial ports, or whatever else.

Having native support within HS for this type of environment would probably be preferable. But it's definitely not necessary. People are doing some of this now with R232 over IP devices like the UDS100 from lantronix, to name just one.

Eventually some sort of distribution of resources will have to take place. If this able to get me there sooner rather than later, I'm all for it. This is just MHO though.

[rocco]

On a non-routed network, like what most of us use, I don't think UDP would be any less reliable than TCP. If routed over a multi-segment network, like the internet, UDP cannot recover lost packets, as TCP can.

The Ethernet protocol already contains CRC (at the transport layer, I think). On a typical HomeSeer Ethernet, packets might be mangled, but not lost. The CRC will detect the mangling.

At 100mbs, a fifty byte packet would take only 4 microseconds to transmit. That should allow a quarter of a million of them each second. Well, not really, but a lot! I see this as hardly a blip in the network traffic.

Speaking personally, I could really use this.