Share XAP network?

[Steve Q]

My XAP network now includes 33 devices being polled between 1 and 5 minutes. I have 5 DS2438 units that I am using to monitor the power being used by various circuits/appliances in my house. They are being polled every 5 minutes. I would like to increase the poll on some or all of these units to once per 15-30 seconds. This would improve the accuracy of the power usage data I am collecting. I have scripts that write the data to a file. I'm pretty sure if I do this I will use so much CPU/disk drive that it will completely mess up Homeseer.

So my question is: can I connect another computer to my existing XAP network to collect this data? Can I simply tap into the 1-wire network via a DS9097U? Any other method to collect data from 1-wire devices?

Steve Q

[Michael McSharry]

I poll my wattnode at 15 second intervals and that does not cause any problems and I'm certain I have much more xAP traffic than you, albeit I do have HS1 rather than HS2. For specific measurement work I run the wattnode at 2 seconds and that also is no burden for the LAN/HS. The only time you should see any LAN issues are when you stream video or similiar high bandwidth content. The stuff we do with HA DAQ/Control is a drop of water in the ocean. The processing of changes of devices states and properties through HS can place a burdenn on the CPU if data is flooded into HS.

I did do some posts in the mcsXap forum that did a stress test and collected some data when I flooded the xAP network. It has been too long for me to remember the specifics, but some searching should find this analysis. I do know that HS2/mcsXap did not scale as well as HS1/mcsXap.

You can look at the change in message count in xapmcsHub over some interval of time to get an idea of the xAP burden you do have. You can also look at instantaneous rates in the xAP Viewer.

xAP is a UDP broadcast protocol. That means that every computer that has a NIC connected to the same LAN will receive the messages simultaneously. Adding another computer does not change the utilization and marginally increases it with additional heartbeats.

If you do not want your 1-wire info on the LAN then you can use a direct HS 1-wire plugin such as the free .NET version that uses DS9097U or DS9490D, the Embedded Data Systems interfaces supported by the UltraJones plugins, or mcsTemperature. You could even use a hybrid where you split your 1-wire network between flexible vs. high rate and then use a more dedicated interface into HS for the high rate one.

[Steve Q]

Michael,

Thanks for the information. My concern is the handling of the data; I'm sure XAP can handle the transmission of huge amounts of data, but I am not so confident that Homeseer can execute the scripts I am using as fast as the data is coming in! I already see an occasional "permission denied error" when trying to save the data to disk. I should probably rethink my data storage approach; maybe use an accumulator rather than store the individual values.

Thanks for the tip about using XAPHub message count to get an idea of how much of a load XAP is carrying. I also took a look at XAP message viewer on another computer on my LAN and it showed a message rate of 112 messages per minute. Is all this activity on the LAN all the time? Or is it showing up because XAP viewer made the connection?

I'm a bit concerned if all this stuff is always present on my LAN? It may explain some of the odd behavior on my other computers!

I need to research this more.

Steve Q

[Steve Q]

I was mistaken about the poll rate of my 1-wire devices. They are being polled every 5-10 seconds. They are being captured by Homeseer every 1-5 minutes.

Steve Q

[Michael McSharry]

For some reason your xAP message rates are more that 5x the rates I'm experiencing. 1-wire polling will only generate xAP traffic when the value of the sensor changes. This will be often for the energy measurements, but I would think others are not changing that much. When you look at xapmcsHub statistics the Received column is what is going over the LAN. The Forwarded column is what is being routed internally via the fake localhost port.

Even at your data rates your LAN is barely being used for the xAP traffic. For discussion assume each xAP message is 5000 bits of data on the lan. Your rate is 2 messages per second so you are using 10,000 bits/sec on a LAN that probably has an effective throughput of 10,000,000 if you are still on a 100 Mbit NIC. This is 1/1000th of your LAN bandwidth.

I do agree with you that the CPU use at the HS end is the choke point.

[Steve Q]

Originally Posted by Michael McSharry

For some reason your xAP message rates are more that 5x the rates I'm experiencing. 1-wire polling will only generate xAP traffic when the value of the sensor changes. This will be often for the energy measurements, but I would think others are not changing that much. When you look at xapmcsHub statistics the Received column is what is going over the LAN. The Forwarded column is what is being routed internally via the fake localhost port.

Even at your data rates your LAN is barely being used for the xAP traffic. For discussion assume each xAP message is 5000 bits of data on the lan. Your rate is 2 messages per second so you are using 10,000 bits/sec on a LAN that probably has an effective throughput of 10,000,000 if you are still on a 100 Mbit NIC. This is 1/1000th of your LAN bandwidth.

I do agree with you that the CPU use at the HS end is the choke point.

Michael,

Thanks for this info, I agree that xAP traffic on the my LAN is minimal. When I set up the DS2438 units in mcsxAp1wire ver2.4.13, I used 5 second poll rates as an experiment and never changed them. Since everything is working extremely well, I never changed them to a lower rate. So, I think everything regarding xAP is currently setup so that I could get all the data I need. My question is: How could I collect the data, and store it, without using Homeseer? Is there a utility that can act as a data logger?

Steve Q

[mfisher]

Originally Posted by Steve Q

How could I collect the data, and store it, without using Homeseer? Is there a utility that can act as a data logger?

Steve,

Have a look at "xapmcsDatabase" on this page: http://www.mcssprinklers.com/xapXap.htm

[Steve Q]

Originally Posted by mfisher

Steve,

Have a look at "xapmcsDatabase" on this page: http://www.mcssprinklers.com/xapXap.htm

Mark,

Excellent! This is exactly what I was looking for. I have it up and running on my Homeseer computer. I will also try it on a different computer on my LAN. I will have to get reacquainted with Access. I have not used it for years. I prefer Excel.

I notice the voltages are being reported as integers and are scaled by 100. Same thing we saw when bringing the data into Homeseer. I'll see if I can convert them directly within Access; maybe I can also convert the voltages directly to power values (KWhrs [ignoring power factor]).

Thanks,
Steve Q

[Michael McSharry]

xapmcsChart is the companion to produce web pages and jpg images from the database

[Steve Q]

Originally Posted by Michael McSharry

xapmcsChart is the companion to produce web pages and jpg images from the database

Thanks, I will check it out. All of your programs are really great!

Steve Q

[Steve Q]

Michael,

I installed and am running xapmcsdatabase on my XPS400 computer (not homeseer). it seems to be working OK but I now have 1000s of the following messages in my Homeseer log:

9/11/2009 7:37:07 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:37:04 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:37:02 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:56 PM	xapmcsdatabase			mcs.database.xps400 | log_analogdata insert into tblhighrate(sampledate,ae5000000a93d5b26v1,a4c000000a9021a26 v1,a09000000e7833a26v1,af0000000a95c6426v1,a8c000000e935fe26 v1) select #2009-09-11 7:36:57 pm# as qt,335 as q0,60 as q1,39 as q2,4 as q3,13 as q4 line 520 operation must use an updateable query.
9/11/2009 7:36:54 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:52 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:49 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:44 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:40 PM	xapmcsdatabase			mcs.database.xps400 | log_analogdata insert into tblhighrate(sampledate,ae5000000a93d5b26v1,a4c000000a9021a26 v1,a09000000e7833a26v1,af0000000a95c6426v1,a8c000000e935fe26 v1) select #2009-09-11 7:36:42 pm# as qt,331 as q0,60 as q1,39 as q2,3 as q3,13 as q4 line 520 operation must use an updateable query.
9/11/2009 7:36:40 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:35 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:33 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:29 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:25 PM	xapmcsdatabase			mcs.database.xps400 | readmessage line 840 operation must use an updateable query.
9/11/2009 7:36:25 PM	xapmcsdatabase			mcs.database.xps400 | log_analogdata insert into tblhighrate(sampledate,ae5000000a93d5b26v1,a4c000000a9021a26 v1,a09000000e7833a26v1,af0000000a95c6426v1,a8c000000e935fe26 v1) select #2009-09-11 7:36:26 pm# as qt,333 as q0,61 as q1,41 as q2,4 as q3,13 as q4 line 520 operation must use an updateable query.

What's happening?

Steve Q

[Michael McSharry]

Never had this reported before with either Access or SQL Server. I did a google and found this http://databases.aspfaq.com/database...ry-errors.html. There are many hits on the error text. The database connection mode was not explicit set. It is not likely the case, but if you cannot find other solution then I will explicity set it.

[Steve Q]

Michael,

I think I solved the problem, thanks to the information you found with Google. Although I had file sharing turned ON, the folder "xapmcsdatabase" was marked read-only. This apparently prevented MSAccess from creating a database locking file (.ldb), thus generating all the error messages. I changed the file properties to read/write and everything is working now; no more error messages.

I thought this might be a permissions problem that I always seem to run into when using Access on a network. I can only open the database (mcstemperature.mdb) on the local computer. I cannot open it from a networked computer. I have had this problem in the past with other .mdb files and I have never been able to resolve it!

Thanks for your help.

Steve Q

[Steve Q]

Michael,

I am using xapmcsdatabase and it is working very nicely. I have not yet done much analysis of the data. I'm not exactly sure how I want to use this data. In my first attemps to analyze the data, I caused the error messages (post #11) to reoccur. Since I use Excel extensively and know it much better than Access, I opened the database in Excel. This caused the error messages to reappear in the Homeseer log. When I close the excel spreadsheet, the error messages disappear. No error messages appear when I open the database in Access. I think that Excel creates a link to the Access database (perhaps it locks the file) when the data is being used. It is interesting however that xAP data continues to be added to the Access database even when Excel has it open.

This is FYI in case other users run into this issue.

Steve Q

PS: I did find the solution to not being able to open an Access data base on a networked computer. It is an issue caused by the security settings in Internet Explorer. I fixed the problem by turning OFF "Automatically detect intranet network" setting in the security setting of IE 7 - local intranet tab.

[Steve Q]

Originally Posted by Steve Q

Michael,

I installed and am running xapmcsdatabase on my XPS400 computer (not homeseer). it seems to be working OK but I now have 1000s of the following messages in my Homeseer log:


What's happening?

Steve Q

There is a setting in ACCESS called "Open database using record-level locking". If this checkbox is checked, Homeseer will generate an error ("mcs.database.Homeseer3 | createtable: line 160 the database has been placed in a state by user 'admin' on machine 'Homeseer3' that prevents it from being opened or locked.") when Homeseer attempts to write to the database while it is open. Un-checking the box will eliminate this error.

The setting is in Options/Advanced tab.

Steve Q

[Michael McSharry]

Are you trying to use the HS database or some other shared database? It seems like the computer you are running xapmcsdatabase does not have rights to store. xapmcsdatabase should have its own database

[Steve Q]

Michael,

These error messages only occur when I open the database with MS Access to look at the data. As long as the database is not open in Access everything is fine. But since I am collecting data every 15 seconds, there would always be an error in the Homeseer log whenever I look at the data. By unchecking the box, the Homeseer log errors seemed to have stopped.

Thanks,
Steve Q

[Steve Q]

Sorry, I'm wrong, I just checked again and I am still getting the following error:

mcs.database.audrey3 | log_analogdata insert into tblhighrate(sampledate,a09000000e7833a26v3,a4c000000a9021a26 v3,a8c000000e935fe26v3,ace000000e94aac26v3,ae5000000a93d5b26 v3,af0000000a95c6426v3) select #2009-11-19 3:36:55 pm# as qt,0 as q0,22 as q1,1023 as q2,4 as q3,305 as q4,29 as q5 line 520 could not update; currently locked by user 'admin' on machine 'audrey3'.

Steve Q

[Michael McSharry]

You will likely find a file in the same folder with a .ldb file type with same database name. This is Locked Data Base file that is created when you open Access to do management of the tables, queries, or forms. If you have Access closed and the .ldb file is still present then delete it as its purpose is a file lock.

Normally the database is queried with external applications (e.g. xapmcsChart) rather than directly with Access. I normally use SQL Server 2000 and Enterprise Manager allows poking around within it. I use .mdb with mcsSprinklers, but I do not recall every having issues with looking at the tables at the same time mcsSprinklers is running.

[Steve Q]

Yes, the ***.ldb file appears when I open the database file in Access. And it disappears when I close the database file.

The error messages are not really an issue but I have to do maintenence every few days to delete the oldest records when the record count gets above 65,000 records. Excel (I use it to create the graphs) can't handle this much data. I have created an AutoExec Macro for Access that will purge the oldest 5000 records. I am still trying to figure out how to get the macro to close Access. If I can figure it out, I will have Homeseer run it on a daily basis.

Steve Q