Up and coming changes

I've been working on some changes over the xmas break for the next HAH firmware release.
One of the items is the refactoring of the /etc/xap-livebox.ini file into separate files which will be contained in the new directory /etc/xap.d/

Member for

16 years 10 months

brett

12 years 11 months ago

Beta 311.1 posted - this contains quite a number of changes.

I've updated the wiki to already reflect the new API:  plugboard and jeenode wiki pages.

After you update to this beta you can delete /etc/xap-livebox.ini and /etc/ini.conf as these are not required - do make sure that /etc/xap.d was populated with files before you do this.  You might want to just rename those out of the way into say /root for safe keeping.

If the automatic upgrade fails, that is there are no files in /etc/xap.d you can run from the command line.
# ini-migrate
and it will tell you what the issue is.  All is not lost if you get to this point as there is provision to use an external mapping configuration file if there are sections that it can't figure out how to convert.  I'll then include those into the code base before this becomes the next production release.

A quick summary of some of the LUA framework changes that might affect you.

Some changes by example
   xap.init("dbzoo.livebox.hithere","FFDEAD00")
This now becomes
  xap.init{instance="hithere", uid="FFDEAD00"}

You don't need to, and its best you don't, specify the VENDOR or DEVICE components as part of the initialization.  These will be figure out automatically based on some defaults.

Creating BSC endpoints
   bsc.Endpoint{base="dbzoo.livebox.my:relay.3", ...}
This now becomes
   bsc.Endpoint{instance="my:relay.3", ...}

Again to maintain vendor, device portability you don't specify these parts.  The wiki write-up for the BSC API has more examples.

All the examples in /etc_ro_fs/plugboad have been amended to fit into the new way of doing business.

NOTE:  Full backward compatibility for previous API calls has been maintained so nothing should break on upgrading.   *should* but you never know.

This time when I say BETA I really mean it due to the amount of code that has been reworked.

Brett

Member for

15 years 7 months

garrydwilms

12 years 11 months ago

Brett,

Thanks for your work on this, especially over Xmas.

It looks like a substantial update that will enable the project "move on" in the future. Whilst I personally am in no rush to ditch the livebox and onboard rf/input/1wire, the jeenodes (which have been a new addition to my system this year) allowing remote relay control of my heating without wires and ir tx/rx along with the more recent bluetooth stuff has highlighted to me the benefit of "non central" hardware. As always you seem to be one step ahead allowing the rest of us to simply use the things you code for us!

The time both you and Derek put into this project is much appreciated. I hope you both had a good christmas and wishing you all the best for the new year. Let's see what 2014 brings for the HAH :)

Garry

Member for

15 years 1 month

allanayr

12 years 11 months ago

Hear Hear!

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

Your work is very much appreciated. I look forward to trying this beta and I know your plans for 2014 will be good for us all.

Member for

16 years 10 months

brett

12 years 10 months ago

Thanks Mark.  I've also been working on porting the HAH specific parts to the ARM architecture targetting the BeagleBone (armv7), the RaspPI is (armv6).  I figure if I can do one I can do the other without too many drama's.
The plan is to bundle this up as a opkg/ipkg that you can install on a standard distro.

I've comitted some changes that allow all the xAP and LUA(+ addons) to be built and now I'm just waiting on my BeagleBone from China so I can test out my build.  I pick one up one cheap $30 US from seedstudio as they where having some sort of special/runout sale of these.  Now they have run out !  They do have the Blacks for $45 thou.

I also have a RaspPI but I always had issues with the ethernet port, ie it would not work, and I burnt way to much time fecking around with it, and ended up putting it back in its box where its sat for a year.  I hate wasting time.  Anyway the idea of having to attach a monitor and keyboard seems pointless for embedded hardware that sit in a cupboard I might give it another go later...

Brett

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

The BeagleBone looks to be out of stock pretty much everywhere. Looks like most suppliers have them on backorder for February so you have some time to play :)

I have a few alerts set up for when they restock. 

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

The BeagleBone has landed.

Time to play :)

Open for alpha testing too...

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

Brett

I have tried the beta. Check your email. All up and running here again.

Member for

16 years 10 months

brett

12 years 10 months ago

Mark I'm jealous you got your bone before I did.  I'll push up the ARM binaries for the HAH system and post back here with the location when that' done.

The 311.2 beta dropped the all important ini-migrate program due to me missing the dependency to build it in the Makefile.
I've fixed this and posted a new 311.2 beta (yes I didn't bump the version).


NOTE:  the directory /etc/xap.d must be empty before you attempt to upgrade again.
CORRECTION: the /etc/xap.d directory must not be present.

Brett

Member for

16 years 10 months

brett

12 years 10 months ago

For what its worth here are the HAH binaries compiled for ARM.   Once I get my bone I will package these up as a proper opkg file to make installation easier etc. etc..

http://www.dbzoo.com/hah-arm/bb.tar.bz2

They are dynamically linked and I've included the libraries which you will need to check pick to get them running.  You'll at least need libxap2.so as for the others you can use "ldd" on the executables to see if they are required.

As you can see the xap-* binaries for the most part only need the libxap2.so, the other linkables should already in your bone distro. (I'm using a cross compiled ldd if you're doing this natively on your bone just ldd will suffice and -root ../.. isn't required either.)

brett@bb-dev bin]$ /opt/bb/toolchain/bin/arm-linux-ldd --root ../.. xap-hub
        libxap2.so => /lib/libxap2.so (0xdeadbeef)
        libc.so.6 => /lib/libc.so.6 (0x8badf00d)
        ld-linux.so.3 => /lib/ld-linux.so.3 (0x8badf00d)

For LUA I linked in the optional libraries, readline, history, ncurses to make using it from the command line a little easier so these libraries are required.

[brett@bb-dev bin]$ /opt/bb/toolchain/bin/arm-linux-ldd --root ../.. lua
        libm.so.6 => /lib/libm.so.6 (0x8badf00d)
        ld-linux.so.3 => /lib/ld-linux.so.3 (0x8badf00d)
        libc.so.6 => /lib/libc.so.6 (0x8badf00d)
        libdl.so.2 => /lib/libdl.so.2 (0x8badf00d)
        libreadline.so.6 => /lib/libreadline.so.6 (0xdeadbeef)
        libhistory.so.6 => /lib/libhistory.so.6 (0xdeadbeef)
        libncurses.so.5 => /lib/libncurses.so.5 (0xdeadbeef)

Anway lets see how this goes - btw these all use the new /etc/xap.d location for their configuration files.

Brett

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

I'll try the beta again...

As for the BeagleBone...just give me a little while to get to grips with it then I'll see how the distro goes

cheers

Mark

Member for

16 years 6 months

kevin

12 years 10 months ago

From an installed base point of view - and for getting maximum exposure for HAH the Raspberry Pi is obviously the important platform so if anyone has any progress reports on that then do post.    I'm hoping Bretts Pi might leap into life or that he might have access to another board.

K

Member for

15 years 8 months

aivo

12 years 10 months ago

Hi, I have some experience with HAH on Pi for about a year, see if this is type of info you are looking for .. :)

Got into it because was unable to keep Livebox running stable with all wanted applets on it, occasionally found it hanging despite efforts.

I have only complied xaplib, hub, serial + ini and plugboard as this was my main interest.  Lua/pl are via Raspbian.
First binaries were from revision 436 source, later "upgraded" to 466 and lately to 519 just of interest, do not recall stability or other issues.

For now then running xap-hub, xap-serial and applets/plugboard on "hahpi".  To hahpi is also connected Jeelink/basenode, CurrentCost++ still "normally" to Livebox/HAH.

Regards,
Aivo

Member for

16 years 10 months

brett

12 years 10 months ago

Aivo,

That's good to here.  The changes that I made circa r519 try to separate the mega xap-livebox.ini into separate files should tidy this porting excerise up too.  There is another change in r526 which addresses a bug in xaplib in this regard.

Part of the motivation for this work was to remove the word "livebox" as a default in the configuration path, or the files themselves.  This makes the system more host agnostic and friendly for porting.  Separate files in /etc/xap.d/ is a better in this regard than one /etc/xap-livebox.ini file.

Yes I should look more into my PI but it did my head in last time.

Brett

Member for

15 years 7 months

garrydwilms

12 years 10 months ago

Thanks to aivo's news on the Pi, I have had a play with this myself over the last day or so.

I have never done any compiling before so it's real trial by fire but so far, as Aivo has already, I have the main HAH components running nicely on the Pi. (hub, serial, twitter, plugboard)

For interest, I installed subversion and did the source checkout directly to the pi.

I then compiled each component separately using their individual make files, modding them to use the pi complier (gcc) not a cross compiler. Few hiccups along the way but pleased with progress so far.

once I know a little more of what I'm doing I'll write up and maybe try to modify the overall makefile to make this simpler. I see that Brett has already got one for the beagle bone that would be a great starting point. Although it's still beyond me at the moment.

post more info/instructions soon!

Garry

 

(did I say in another post a few weeks back I was in no rush to move from the livebox?  Funny what the temptation to try something new can do! :) 

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

LOL Garry

I know just what you mean. I've got a beaglebone and I have tried to get the code on it but stumbling along so far. 

I'm almost tempted to send my BB to Brett to hurry things along :)

No rush Brett, honestly, I really do need a while to play with this board first :)

Member for

16 years 10 months

brett

12 years 10 months ago

Gary

Let me help you compile everything.
I've made some changes to the Makefile.bone file which I've been using to cross compile for ARM so it will build natively using a couple of modifications.  Revert all the changes you have made and try this:

# vi Makefile.bone

Comment out these 3 lines for native building.
HOST=arm-linux
XHOST=--host=$(HOST)
CROSS_COMPILE=$(TOOLCHAIN)/bin/$(HOST)-
 
# make -f Makefile.bone

I verified with these changes I can build on x86_64 Linux without any problems.
When its done everything will be layed out for you in -> targets/bb.src

Mark I'm still eagerly waiting on my bits and pieces.

Brett

Member for

12 years 10 months

GaryP

12 years 10 months ago

A port of HAH to RPi would be very useful.

I am just starting on the home monitoring journey and have got my feet wet using emonCMS from openenergymonitor.org.

It looks like oem and HAH might be getting closer in your goals as oem are looking at linking their monitoring software into home control which HAH already seems to be doing.  The HUH work on grabbing data from lots of different products would have a lot to offer to this.

One thing to watch out for on RPi and other SD card based boards is failures.  Oem are now advising a hybrid SD card boot and HDD main drive as SD cards seems to be flaking out after 3 months or so of writing.

The emonCMS route might also offer HAH some cloud connectivity as well as they are currently doing free hosting of your emonCMS data as well as you being able to have a local install of this.

Looking forward to seeing where the port gets to

Gary P

Member for

15 years 7 months

garrydwilms

12 years 10 months ago

Thanks Brett, I did try to have a hash at the makefile for the bone myself but got errors on iserver and curl.

I have backed out my alterations and gave your modified makefile a go (after commenting out the 3 lines) but still get the same errors.

 

Curl error is:

 

 /usr/bin/install -c -m 644 './curl-config.1' '/usr/trunk/targets/bb.src/share/m                                             an/man1/curl-config.1'

make[8]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7/docs'

make[7]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7/docs'

make[6]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7/docs'

make[5]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[4]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[3]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[2]: *** [install-recursive] Error 1

make[2]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make[1]: *** [install-strip] Error 2

make[1]: Leaving directory `/usr/trunk/userapps/opensource/curl-7.19.7'

make: *** [libcurl] Error 2

 

 

I have managed to install curl ok via the apt-get install commend though.

 

Iserver error is:

make[1]: Entering directory `/usr/trunk/userapps/hah/iServer'

lex  -t tokenize.l > tokenize.c

/bin/sh: 1: lex: not found

make[1]: *** [tokenize.c] Error 127

make[1]: Leaving directory `/usr/trunk/userapps/hah/iServer'

make: *** [iServer] Error 2

 

I am yet to get around this one.

 

On the plus side, I now have xively, currentcost, twitter, hub, serial (with jeenodes) and plugboard up and running.

I plan to put the HAH hardware onto the GPIO port tonight and attempt the xap-livebox install.

Fingers crossed!!!

 

Garry

Member for

16 years 10 months

brett

12 years 10 months ago

I know that curl error you are getting as I saw that too.   Its because openssl (compiled earlier) is not using -fPIC which means its not relocatable so when this static library is linked with libcurl as a shared library it throws a wobble.

I did fix that when I made my changes and I too ran into the same problem.
Did you "svn revert --depth=infinity" from the top level to get ALL my changes?

Did you also "make clean" as you need to recompile EVERYTHING from scratch any  library compiled without -fPIC will cause you problems.  I think you did not do this step.

lex is also known as flex.   Try installing this package its a std dev tool.  "yum install flex"  or whatever equiv for PI is.

Brett

Member for

15 years 7 months

garrydwilms

12 years 10 months ago

Thanks Brett,

as you thought, I did not make clean!. However, I have now and unfortunately the curl error persists.

I might get a virgin SD card imaged and try again in case it's something I've done with my playing around. 

The flex did fix iserver issue though, cheers. I have now officially powered off my livebox and the Pi is in control!. It better behave itself, the wife is very sceptical?!

 

Garry.

Member for

16 years 10 months

brett

12 years 10 months ago

Gary,

I suspect CURL is not the problem you need to scroll up from the make output and look for another error.
The "install-recursive" error is simply saying that no libcurl.so is found for installation, which means it didn't build.

Check the GCC output for openssl do you see -fPIC ?
See what other errors exist above this one you posted as its more a symptom error of something else failing.

libcurl is a dependency for building  xap-googlecal (broken) and xap-twitter.  So all that is missing for you is twitter.

What you could do is simply remove xap-googlecal and xap-twitter from this list in the Makefile.bone file

HAH_APPS = xaplib2 xap-hub xap-livebox xap-snoop xap-xively xap-sms iServer \
        xap-currentcost xap-googlecal xap-twitter xap-serial xap-plugboard \
        xap-flash xap-urfrx

That should give you a "clean" build avoiding libcurl - at then you are not messing around with individual builds.

Good that flex sorted out the iServer.  I'd still like to get a build without messing around with individual compiles.

Brett

Member for

16 years 5 months

BoxingOrange

12 years 10 months ago

running "apt-get libcurl4-openssl-dev" installs a curl library that allows xap-twitter to be compiled, but googlecal still seems to get an error :-

 

rm -f -rf /home/pi/livebox-hah-read-only/targets/bb.src/share

make -C /home/pi/livebox-hah-read-only/userapps/opensource/libgcal install-strip

make[1]: Entering directory `/home/pi/livebox-hah-read-only/userapps/opensource/libgcal-0.9.6'

make[1]: *** No rule to make target `install-strip'.  Stop.

make[1]: Leaving directory `/home/pi/livebox-hah-read-only/userapps/opensource/libgcal-0.9.6'

make: *** [libgcal] Error 2

 

Once compiled an appropriate directory structure seems to need creating to get the "xap-?" to run.

 

./xap-livebox: error while loading shared libraries: libxap2.so: cannot open shared object file: No such file or directory

pi@raspberrypi ~/livebox-hah-read-only/targets/bb.src/usr/bin $ ./xap-sms

./xap-sms: error while loading shared libraries: libxap2.so: cannot open shared object file: No such file or directory

Once I've got these little issues ironed out I've got a full script of how to setup a RasPi, compile and run the HAH on it.  Awesome work Brett, perhaps Google will take a bit of notice then ;)

 

Member for

15 years 7 months

garrydwilms

12 years 10 months ago

Hi,

try moving the .so file to /usr/lib and the .h files to /usr/include.

this worked for me and it's been running great for a few days now.

still struggling to get a complete compile though as Brett would like, I am gonna try again later and push the errors into a file for the master to look at ;)

 

Garry

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

Brett do you have available the toolchain for the armV7 build? 

/opt/bb/toolchain/bin/arm-linux-gcc

Member for

16 years 10 months

brett

12 years 10 months ago

Mark,

I posted up some of the rough notes I wrote last night whilst I was playing around with my BB.

http://www.dbzoo.com/livebox/beagleboneporting

I didn't write anything up when I built my toolchain so I'll have to go back and rebuild it and make some notes.
Its a little large to upload (130Mb) so it would be best if you build your own.

I believe I just followed this: http://code.google.com/p/my-beagleboard/wiki/Toolchain

I've attached the config.txt (aka .config) which lives in my /opt/cross_arm/bin directory used to build the toolchain these are my exact settings.

Brett

Attachments
config.txt (10.77 KB)

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

I'll take a look

Member for

16 years 10 months

brett

12 years 10 months ago

Mark,

This is the HAH system compiled up for the beaglebone as an .ipkg installation.
These are armv7 binaries and they will not work on the Raspberry pi which is armv6.
Once I get my bone running happy I'll do a armv6 cross compile.

Create you own configuration file:

root@beaglebone:/etc/opkg# cat /etc/opkg/hah.conf
src/gz hah /feeds/ipkg/eglibc/armv7a/hah

(Opps typo on that hah.conf line - I've swapped the ipkg/eglibc fields and adjusted the path on the web server too)

Refresh your package caches and then install the HAH system, its that easy.

root@beaglebone:~# opkg update
root@beaglebone:~# opkg install hah

By default it only starts: xap-hub and kloned.

The kloned is listening on port 8080 instead of 80 which is where the beagle bone http is sitting and I didn't want to disturb it.  On the web gui you'll see it can find its build number as its looking /etc_ro_fs/build which of course is not there - I'll probably fallback to looking /etc/xap.d/build once I see what else is broken.

The filesystem is layed out exactly like the livebox so things will be where you expect them.

I should add I'm running the Angstrom linux distribution on my bone.

# head -1 /etc/angstrom-version
Angstrom v2012.05 (Core edition)

Brett

Member for

15 years 7 months

garrydwilms

12 years 10 months ago

Looks like solid progress on the beaglebone!, good news, can only be good for the future of the HAH project.

As for the pi, I have attempted to compile as one again using the makefile.bone with the three comments for local compiling. I redirected stderr and stdout to a log for investigation.

Brett, you were right, there are earlier errors, I cannot fathom what they are trying to say though :( any ideas?

as for running the HAH system (compiled seperately) on the pi though, its been a week now and all good except for xively. This needs to be restarted a number of times before it becomes stable. once running though it will stay running for days.

when it goes down it complains of "glibc detected" realloc() invalid next size. obviously relating to the memory reallocation in he xively code but I cannot find any indication as to what part of the code its unhappy about.

Is this likely to be due to my compiling? any thoughts?

like I say its intermittent and when xively is working, it works great for days?

 

Garry

Attachments
log.txt (309.33 KB)

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

I'm running Angstrom here too as I have no wish to replace that at the moment.

Angstrom v2012.12 (Core edition)

Running Ubuntu on the Mac in a VM for the cross compiler, so I NEED to get that running. 

The package install ran well. I now have the web gui up on the BB and on my Mac :)

Attachments

Member for

14 years 5 months

mark_baldwin

12 years 10 months ago

I've had an issue where the ip address, subnet mask etc reversed in the gui. They were also reversed in the system.ini file when I viewed that in the terminal. However the ini files downloaded using the backup option were the right way. 

Shortly after I noticed that the BB stopped.  

I also noticed a file named 533745ni in the etc/xap.d folder

Now back up and running after reflashing the eMMC from a backup image.

Just got to redo the wifi drivers and reload HAH

Member for

16 years 10 months

brett

12 years 9 months ago

Mark on the BB pretty much everything in system.ini is ignored.  You cant adjust your IP from the web gui.  Its a straight port from the livebox so there is LOTS of stuff that just won't work.   The port is a proof of concept only.  Having said that you should not get strange files being created like you did.  Sounds like a memory corruption somewhere.   I think when moving to another platform I'm going to drop kloned and rewrite it not in C.

Brett

Member for

16 years 10 months

brett

12 years 9 months ago

Garry,

I had a brief look but nothing obvious comes to mind.   Let me compile native on my BB and see if I can reproduce.  I suspect I will be able to.   You can drop the building of libcurl and libopenssl as the libraries will already be present. I will adjust the makefile for a local build to do this  I might need to rethink how this makefile is put together to allow (livebox vs other platforms) including cross compile and native compile.

Brett

Member for

14 years 5 months

mark_baldwin

12 years 9 months ago

Brett

The HAH on BB seems solid enough now. I took the BB back to a clean install and reloaded all from scratch. It's possible that the erroneous files got there during my meddling.

I've now got the WH node connected and getting xAP messages from that. I'm wondering what route to go down for the RF transmission. Is your plan to release the ethernet RF nodes?

I'm going to hook up the HAH baseboard to the BB and see what comes of that. 

I was thinking for now to use a jeenode board connected to the serial monitor port on the BB and fit a 433Mhz tx to it, as I have no need for the relays (LCD and I2C would be nice though)

Cheers

Mark

Member for

15 years 7 months

garrydwilms

12 years 9 months ago

Cheers for looking Brett. I'm all up and running now so no rush for me. I am happy to test further makefiles, etc to get the porting neater for others to follow.

Mark,

I managed to get my hah board running nicely off the pi gpio ports, it's been great, I just needed a gpio breakout board to wire the hah board into.

It's worth doing if you have the hardware. Like you say, rf, 1wire and lcd are useful. I do also use inputs for alarm monitoring.

Think in the future though rf nodes will be the easier option

Member for

16 years 10 months

brett

12 years 9 months ago

Mark: you wrote "I've now got the WH node connected and getting xAP messages from that. I'm wondering what route to go down for the RF transmission. Is your plan to release the ethernet RF nodes?"

To get the NODE RF transmission into xAP I use this http://www.dbzoo.com/livebox/hah_hahnode/nanode_gateway
It work really well, and this is what I use full time as I have no BaseNode connected to my livebox.
To get RF transmission out:  http://livebox-hah.googlecode.com/svn/trunk/userapps/arduino/EthURF/
Which is a standalone URF transmitter that can be ethernet tethered.  I wrote up the code but there is no wiki link on how this work.  I think from the code is pretty self explanatory.

Garry:  I am curious about xap-xively just dumping.  Perhaps run the process manually and use valgrind to see what it reports.  You might have to compile up xap-xively with -g to give valgrind something to churn on.  I do recall using this previous with various program but I made have introduced a regression recently as I've not valgrind the code for a while now.

Brett

Member for

16 years 10 months

brett

12 years 9 months ago

I hit TWO nasty bugs trying to get this to compile up.

1.   There is no "lex" link, so you need to create one.  I will work around that in the Makefile but for now.
# ln -s /usr/bin/flex /usr/bin/lex

2.   m4 is not installed by default so you need to install it as flex needs this.
# opkg install m4

3. There is a compiled in path in the flex binary that is WRONG !!!

  

I could not get flex to do its thing so I ran strace on the command (after installing strace that is).   

 strace -f flex -o tokenize.c tokenize.l

  

What the f*** is that path doing in there?

[pid 16777] execve("/home/koen/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/m4", ["/home/koen/setup-scripts/build/t"..., "-P"], [/* 14 vars */]) = -1 ENOENT (No such file or directory)
[pid 16777] write(2, "flex: fatal internal error, exec"..., 40flex: fatal internal error, exec failed
) = 40

Something is seriously screwed up with that flex .ipkg. Workaround:

root@beaglebone:# mkdir -p /home/koen/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/
root@beaglebone:# ln -s /usr/bin/m4 /home/koen/setup-scripts/build/tmp-angstrom_v2012_05-eglibc/sysroots/x86_64-linux/usr/bin/m4

That clears the bug.  Now flex does it thing.  Onward and upward for native BB compiling.
Thought I'd share that little pisser with you.

Brett

Member for

15 years 7 months

garrydwilms

12 years 9 months ago

Didn't have these issues on the pi compile. Had to install flex but no issues with links or m4.

Member for

15 years 7 months

garrydwilms

12 years 9 months ago

Valgrind is now running xively but as you might expect, I cannot get xively to break. How annoying!.

will leave it running overnight and if no joy will start doing a few reboots. Guaranteed it will break as soon as I take valgrind off.

 

keep you posted.

Member for

15 years 7 months

garrydwilms

12 years 9 months ago

I've had valgrind running xively as part of the init script for days now and no issues (plus numerous reboots!)  It's almost as though running valgrind has stabilised it?! Is this even possible?

i will remove and see if it breaks again :(

problems I can deal with but pesky intermittent ones with no indication of the issue test my patience! 

G.

Member for

16 years 10 months

brett

12 years 9 months ago

From memory it only reports issues once you stop the process.  So you have to stop it to see what it found.
That fact that nothing is crashing means that valgrind is preventing the problems so it should have something to say about it.
If you are stuggling I'll do the same and see what I can find.    Nigglings bugs are the worse.

I found one very minor leak that happens in xively on startup but its only leaking 382 bytes so its not a showstopper.
I've checked in a fix for this none the less.

==26094==
==26094== HEAP SUMMARY:
==26094==     in use at exit: 10,372 bytes in 213 blocks
==26094==   total heap usage: 511 allocs, 298 frees, 143,320 bytes allocated
==26094==
==26094== 117 bytes in 9 blocks are definitely lost in loss record 44 of 55
==26094==    at 0x4A05E1C: malloc (vg_replace_malloc.c:195)
==26094==    by 0x38C9878C81: strdup (in /lib64/libc-2.5.so)
==26094==    by 0x401C77: getDynINI (main.c:143)
==26094==    by 0x401FA4: parseINI (main.c:190)
==26094==    by 0x402075: main (main.c:204)
==26094==
==26094== 265 bytes in 9 blocks are definitely lost in loss record 48 of 55
==26094==    at 0x4A05E1C: malloc (vg_replace_malloc.c:195)
==26094==    by 0x38C9878C81: strdup (in /lib64/libc-2.5.so)
==26094==    by 0x401C77: getDynINI (main.c:143)
==26094==    by 0x401F79: parseINI (main.c:189)
==26094==    by 0x402075: main (main.c:204)
==26094==
==26094== LEAK SUMMARY:
==26094==    definitely lost: 382 bytes in 18 blocks
==26094==    indirectly lost: 0 bytes in 0 blocks
==26094==      possibly lost: 0 bytes in 0 blocks
==26094==    still reachable: 9,990 bytes in 195 blocks
==26094==         suppressed: 0 bytes in 0 blocks
==26094== Reachable blocks (those to which a pointer was found) are not shown.
==26094== To see them, rerun with: --leak-check=full --show-reachable=yes
==26094==
==26094== For counts of detected and suppressed errors, rerun with: -v
==26094== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 4)

Brett

Member for

14 years 9 months

martynw

12 years 9 months ago

I've been running various bits of your xap code for some time now, mainly on kirkwood devices (sheevaplug, pogoplug) and see the same xap-xively crashing as Gary above.

I compiled the code on an x64 debian box and ran it there to rule out anything platform specific but got the same issue.  So I ran it under valgrind (with xap-xively compiled with the -g flag).

Does the attached shed any light on why it's crashing?

 

Member for

14 years 9 months

martynw

12 years 9 months ago

Actually it crashes pretty consistently for me, so if you want me to try any valgrind options etc let me know!

(attached another valgrind crash log)

 

M

Member for

16 years 10 months

brett

12 years 9 months ago

Martin,

Thanks those are useful... I might also run this on my ARM beaglebone and see if I can reproduce the crash myself.
From those valgrind logs I should be able to figure out what is going on - it points me in the right direction.

Thx.

Member for

14 years 9 months

martynw

12 years 9 months ago

Ok cool, let me know if you need anything else - I removed the log files from my 2 posts as I realised I hadn't sanitised the API keys :-(

 

M

Member for

16 years 10 months

brett

12 years 9 months ago

Thanks.  I'm looking at the code and now I know what to look for I see a glaring defect in my logic.  I don't know what I was thinking at the time this is just so wrong... !
I'll work on fixing this.   I have the logs in a browser tab so feel free to delete them.
Your key is safe with me.

UPDATE: I checked in a change to dscache.c that should fix the issue.  If you want to compile this up and report back that'd be great.   This should nail the issue.

Brett

Member for

14 years 9 months

martynw

12 years 9 months ago

Working well so far, been running for about an hour now with no crashing - previously it only ran for a few minutes or so.

 

I'd say it's fixed :-)

 

Thanks,

M

Member for

16 years 10 months

brett

12 years 9 months ago

Great xively is working!   I've pushed 311.10 for livebox users to get this fix.  

Now I understand why I can see what will excaserbate the bug.   As only 1k of memory was setup for the XML if you have a large number of xively feeds this would run out, the intention was to automatically expand this as requried, however the logic was flawed.   For those with a small number of feeds that fitted withing 1k of XML generation you would never hit this bug but once you start to push this boundary memory overflow and heap corruption occured resulting in random crashes depending on what got clobbered.   I'm suprised that bug has been dormant for so long actually. 

As the fix correctly allocate the right amount of memory  instead of defaulting to 1k of ram it only grabs 200 bytes and then expands as required.  So all in all this fix is more memory friendly if you only have a low feed count.  You get a few hundred bytes of memory back :)  It all counts.

Anyway another one down.  Thx to all for reporting and providing logs, feedback etc.

Brett

Member for

15 years 7 months

garrydwilms

12 years 9 months ago

Ha! Here's me waiting patiently for valgrind to tell me what's up!

When stopped, the logs did indeed appear, but looks like you've already nailed it!


Thanks all for the efforts, I do have a large number of feeds so prob why it affected me, only when I went to the pi though?!


Anyhow, cheers to all involved, I'll update tonight!


Garry