SpreadPi and Powershot SX160

[niram8]

I set up SpreadPi with two PowerShot However, the downloaded images (JPEG) are ot readible neither on windows or linux. I also got the error message "Thumbnail is not in JPEG format".The SX160 works well with CHDKPTP (on Windows 7).Can I set up or modify SpreadPi to work with the SX160.

[markvdb]

Please, in order to provide a somewhat useful bug report, include at least:

• the exact spreadpi version you are using
• the exact chdk version you are using
• the logs resulting from a minimal spreadpi triggering session
• the sample "corrupted" jpeg files

(Also note that this is not a support forum for spreadpi, but a bug tracker for errors in the spreads software. I will gladly help you determine if this is a spreads bug.)

[niram8]

This is my setup:

• Raspberry B+
• USB foot pedal
• Two PowerShot SX 160
• CHCDK 1.3.0 - 4152 (DE)
• SpreadPi immage dated 2015-04-26 (image dated 2015-05-01 does not boot at all)
  

The error logs are on the raspberry pi. I will have to figure out how to transfer them

[niram8]

Unfortunately I do not know how to upload the log files to this thread. Apparently only images files (jpeg, etc) can be uploaded. Can send them by email?
Thanks

[adongy]

Drop them on a pastebin and paste the link here.

On May 7, 2015 8:15:04 PM CEST, niram8 [email protected] wrote:

Unfortunately I do not know how to upload the log files to this thread.
Apparently only images files (jpeg, etc) can be uploaded. Can send
them by email?
Thanks

---

Reply to this email directly or view it on GitHub:
#190 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[niram8]

See the logs of a freshly installed image (2015-04-26) and triggering session:
spreadslog.json:
http://pastebin.com/k6gBiJp4
spreads.log:
http://pastebin.com/w8TaEk9x

[adongy]

As mark said, can you paste a 'corrupted' jpeg file?

[adongy]

Can you also try to take a picture using chdkptp (the lua shell), it's available over at assembla.

[niram8]

Here are the two files:
001.jpg: by spreadpi (not readible)
test.jpg: directly from chdkptp shell with the rs command (readible on Windows)
Hope the upload works

[jbaiter]

Thank you, that clears some things up!
The root cause seems to lie in the Python<->chdkptp library. I just checked the corrupted file with a hex editor, and this is what seems to happen:

The beginning marker of the JPEG file (0xFFD8) only occurs at the very end of the file, but it should be at position 0. Curiously enough, it occurs right after the JPEG end marker (0xFFD9).
So it seems to be the case that when assembling the chunks that are streamed from the device, the first chunk is written last.

After some poking around, it seems that I naively assumed that I received the chunks in order, which does not seem to be the case for your file. So I just pushed a change that sorts the chunks by their ascending offset before assembling them.

Could you try the new version on your SpreadPi and see if it helps? To update, SSH onto your Pi (user: spreads, password: spreads) and run the following:

$ sudo su
# Enter 'spreads' when asked for a password
$ pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta.tar.gz
$ /etc/init.d/spread restart
# or `systemctl restart spreads` if you are running the very latest image

[niram8]

Unfortunately it is not working. After the first update to my current image the cameras did not shoot at all. With a fresh image (2015-04-26) and the update I get the error:
[string""]:4:attempt to compare nil with number

[adongy]

Most chunks do not have an offset... that's why you get this error.

[jbaiter]

Thanks for the hints, @adongy, I think I have found a better and more robust way to assemble the chunks that arrive from the cameras: jbaiter/chdkptp.py@4ff8e53

Here is a new beta release that you should try:
http://jbaiter.de/files/chdkptp.py-0.1.4beta2.tar.gz

[niram8]

With "chdkptp.py-0.1.4beta2.tar.gz" it is working. The images are readible and the thumbnails visible in the web interace.
Shooting is not quite robust. May be still a way.

• Zoom level is not working.
• I frequently get the message "Error tulip index out of range". Then the system (raspberry and cameras) has to be reset completely.
• System/camera crashes in manual focus mode

[adongy]

Can you give us logs for every issue you encounter?

[adongy]

as for zoom, it may be that the sx160 has many zoom step. try to put a high number (eg 60 or more) and try again.

[niram8]

Sorry. The zoom apparently is a parameter issue. With about 30 it is working fine.
Any hints how can I extract relevant parts of the log files. I am afraid the complete log files will become to clumpy.

[markvdb]

paste it to pastie.org or pastebin.com?

2015-05-12 0:03 GMT+02:00 niram8 [email protected]:

Sorry. The zoom apparently is a parameter issue. With about 30 it is
working fine.
Any hints how can I extract relevant parts of the log files. I am afraid
the complete log files will become to clumpy.

—
Reply to this email directly or view it on GitHub
#190 (comment)
.

Mark Van den Borre
Hogestraat 16
3000 Leuven, België
+32 486 961726

[jbaiter]

I just found another set of issues in the chdkptp.py library that were probably the cause for the tuple index out of range errors you were seeing.
Please install the new beta release:

# pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta3.tar.gz

System/camera crashes in manual focus mode

Do you mean that the system completely freezes up? Or is it just the camera that becomes unresponsive?

[niram8]

After install of "chdkptp.py-0.1.4beta3.tar.gz" spreads does not start any more. See the error capture after install:
http://pastebin.com/zvEgZaQV

After reinstalling "chdkptp.py-0.1.4beta2.tar.gz" the system works again.

Here is a fresh log with some capture sessions (foot pedal not working) with "chdkptp.py-0.1.4beta3.tar.gz" :
http://pastebin.com/CREygn87
The last session is with manual-focus. One camera crashes (shuts down) and the system reports the tulip error. Spreads does not freeze. After clicking OK on the tulip error the Web-Interface is still running.

Occasionally the Web-Interace freezes, i.e. does not come come back from shooting. I did not identify it reproducible. I assume that it happens when at least one camera is not reponding any more. I`ll try to capture a log for this.

[niram8]

Sorry, The log is with "chdkptp.py-0.1.4beta2.tar.gz" (NOT chdkptp.py-0.1.4beta3.tar.gz):
http://pastebin.com/CREygn87

[jbaiter]

Ugh, I just noticed that I have a horrible typo in beta3, please try this new one:

# pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz

[nafraf]

Not related to the bug, but to check in the future. Probably, the third method to set focus, where left is pressed during shot_half pressed, does not work for SX cameras.

[jbaiter]

@nafraf: Is there a way to set a manual focus value on the SX cameras over PTP?

[niram8]

The image http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar. works. However foot pedal still is not functioning. Also "Tulip Error" still pops up

I have done the following testing sequence:

• Start cameras
• start SpreadPi
• start web-interface
• 4 shoots autofocus_initial
  -> OK
• 4 shoots autofocus_all
  --> OK
• autofocus_manual
  -> first camera shutdown, Error tulip index out of range
• click OK
  ->return to web-interface
• autofocus_initial
  -> Error tulip index out of range
• click OK
  -> return to web-interface

Resulting spreads.log:
http://pastebin.com/jG0mDW5p

[niram8]

Another Test Sequence resulting in the "tulip index out of range":

• delete all old workflows
• create new workflow with autofocus_initial
• shut down raspberry pi
  *
• Turn on cameras
• turn on SpreadPi
• start web-interface
• start workflow --> cameras start, zoom and focus
• some 60 shoots-->footpedal working!
• stop shooting
• turn off cameras
• wait some time
• turn on cameras again
• initiate shooting --> Error tulip index out of range,

Resulting spreads.log
http://pastebin.com/T2M8tmYH

[nafraf]

@jbaiter: For sx160, set_mf() is an option. See this test log:

connected: Canon PowerShot SX160 IS, max packet size 512
con> rec
con 1> =set_mf(1)
con 2> =set_focus(1200)
con 3> =return get_focus()
4:return:1248
con 4> =set_focus(800)
con 5> =return get_focus()
6:return:797

To detect which subject distance override modes work for each camera: get_sd_over_modes()

[adongy]

That issue should have been fixed in the very last commit by @jbaiter. @niram8, please install latest chdkptp.py from the git repository, using pip install -e git+https://github.com/jbaiter/chdkptp.py.git

[niram8]

Using "pip install -e git+https://github.com/jbaiter/chdkptp.py.git" results in the error:

root@spreadpi /home/spreads # pip install -e git+https://github.com/jbaiter/chdkptp.py.git
--editable=git+https://github.com/jbaiter/chdkptp.py.git is not the right format; it must have #egg=Package
Storing debug log for failure in /root/.pip/pip.log
root@spreadpi /home/spreads # /root/.pip/pip.log

[adongy]

whoops, extra -e is not needed i think

niram8 wrote:

Using "pip install -e git+https://github.com/jbaiter/chdkptp.py.git"
results in the error:

root@spreadpi /home/spreads # pip install -e
git+https://github.com/jbaiter/chdkptp.py.git
--editable=git+https://github.com/jbaiter/chdkptp.py.git is not the
right format; it must have #egg=Package
Storing debug log for failure in /root/.pip/pip.log
root@spreadpi /home/spreads # /root/.pip/pip.log

—
Reply to this email directly or view it on GitHub
#190 (comment).

[niram8]

<pip install git+https://github.com/jbaiter/chdkptp.py.git> runs to install but is not working. There is no web interface and also the IP address of spreadPi is not shown on the cameras.
After re-installation of http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar it worked again.

[jbaiter]

I just uploaded a beta5 for you, @niram8, this should work now. I checked and it seems that the tarball I created had a messed up line.

http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz

[niram8]

I think I am not lucky ...
<http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz > is not working
http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz is working (after re-install)

[jbaiter]

@niram8, for the love of god, please be more specific than just 'is not working' :-(
I really want to help, but without a log or even a description of 'not working' I really can't do anything.

[niram8]

@jbaiter, sorry, you are really right. For whatever reason some parts of my comments are missing. Also the reference to the file which is not running. It should read:

"https://github.com/jbaiter/chdkptp.py.git runs to install but is not working. There is no web interface and also the IP address of spreadPi is not shown on the cameras."

I did not look at the spreads.log file but immediately reinstalled "http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz" which was running again (IP adress on the cameras, Web-Interface running). If the log file is helpfull I can repeat the procedure as from the current log file I cannot identify the sequence of "http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz"

Last part of spreads.log is pasted here: http://pastebin.com/gF1JP1Sc
My internal log of the activities (with copy of teminal output) you will find here (see last two entries on 2015-05-14): http://pastebin.com/Hn6Sj0Eu

[adongy]

Unfortunately these logs do not show anything suspicious, just that you ran spreads and installed various versions of chdkptp.py.

What do you mean by 'is not working'? Does spreads error out when launching ? When capturing ? Does it say anything in the logs ?

[niram8]

I have tested all the versions again. Here is the sequence and the results.
(As described earlier,
working means: after startup cameras show the IP address of the web interface, web interface is starting up;
not working means: no IP address on the cameras, web interface not available but log on to raspberry pi by putty possible)

Sequence start with image "spreadpi-20150514,img"
-> working
pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta.tar.gz
-> working
pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta2.tar.gz
-> Working
pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz
-> working
pip install git+https://github.com/jbaiter/chdkptp.py.git
-> not working (started twice)
pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz
-> not working (started twice)
pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz
-> working again

Resulting spreads.log:
http://pastebin.com/88EFAVxR

Hope that is clearer.

[adongy]

Don't bother with the beta/beta2 files, they are now outdated.

Looks like it doesn't manage to find the camera, thus it doesn't work.

Can you edit spreadsplug/dev/chdkcamera.py, and replace the yield_devices method with this one ?
http://pastie.org/10193030

It should give more info.

[boan-anbo]

Same thing here with A800. I used an earlier version of SpreadPi before. Back then, though one of the two camera dies (automatically shutdown, not freeze) from time to time and requires restarting, it was less frequent. Now with the latest release "tuple out of index" pops up every 20-50 pages, and I need to restart the Raspberry Pi each time...I tried to use separate USB hubs for each of the camera but it didn't work either.
So I am also eagerly waiting for the next update...

[jbaiter]

@niram8: Is this really the full log? How does it e.g. 'crash'/'not work' with beta5? From the logs it looks like everything is running fine up to the point where it tries to find cameras. Does it stop at that point? Is there any console output after that?

@gazagoal: I'm working on it, I'll release a new stable chdkptp.py version once I've got to the bottom of niram8's issue. That should then be available in the next spreadpi build.

[niram8]

@adongy, I have edited "spreadsplug/dev/chdkcamera.py" here is the log (after install of spreadpi-20150514,img / chdkptp.py-0.1.4beta5.tar.gz):
http://pastebin.com/awY2mBwp

@jbaiter, It should be the full log. It just stops. There is no output on the cameras. I cannot see any console output (the pi runs headless). I can still access via putty. Can I look on any other log on the pi?

[jbaiter]

Wow, so it just deadlocks? Can you CTRL+C or /etc/init.d/spread stop or systemctl stop spreads the spreads process?
Are there any suspicious messages in dmesg?

[niram8]

There is no file /etc/init.d/spread on the system (at least I do not find it in that directory).
Here is the output (in putty);
root@spreadpi /home/spreads # /etc/init.d/spread stop
bash: /etc/init.d/spread: No such file or directory
root@spreadpi /home/spreads # systemctl stop spreads
root@spreadpi /home/spreads #

I cannot analyse dmesg and systemctl.
Here is the output of a freshly started system:
dmesg (dmesg > dmesg-output):
http://pastebin.com/qupuBawV
systemctl (systemctl > systemctl-output);
http://pastebin.com/bx2WSv2M

[jbaiter]

@scannopolis, it seems your comments got deleted, but I think you were onto something!

I did some tests last weekend and it turns out that the 'mode' the camera rotary button is set to plays a role. I recommend you all set it to 'P' (or the most 'manual' mode you have on your model), this seems to minimize the errors.

I also did some more work on the error handling and logging, this should help with stability and debugging: http://jbaiter.de/files/chdkptp.py-0.1.4b7.tar.gz

[scannopolis]

I deleted them myself because I wasn't sure if they were useful. I did perform some more test yesterday, using the same version of CHDK (build 1.3.0 ver 4154) and the latest build from Spreads. I did the following:

1. lights off, book far from the glass, autofocus all: got images
2. lights on, book far from the glass, autofocus all: got images
3. lights on, book close to the glass, autofocus all: mistake 'thumb is not in JPEG format'
4. lights off, book close to the glass, autofocus all: got images

It seems to me that nafraf is right about this, it might be a problem with the focus of the camera rather than something else. However, I didn't have time yesterday to fix the bugs with the code that you provided, but I might be able to do it in the weekend

[scannopolis]

I also did:

1. lights on, book close to the glass, manual focus: 'thumb is not in JPEG format'

[jbaiter]

Scann, do you have a log and an example image for the 'thumb is not in JPEG format' error?

[boan-anbo]

Just to follow up on the earlier "tuple out of index" report. The issue has been largely resolved with the latest built and chdk client. Much thanks!

But a new problem is that now when "tuple" problem does occur, which happens now much less frequently, it may result in a bad file for the last photo I scan. This causes the problem that, even after I restart the spreadpi, the Raspberry will crash whenever it loads the last, perhaps badly stored, photo. So either when I want to continue from where I left after the "tuple problem", or when I want to browse the last photo scanned in that workflow, the Raspberry Pi will crash and shows "the server seems to be offline".

This issue can be solved by deleting the whole workflow, or reformat the SD card. But a fix will of course be much appreciated, for it means there is a chance that if the "tuple" problem happens again, you will lose everything you've scanned under the current workflow...

[adongy]

Can you post the last picture saved on such an error along with the logs ? It's really hard to debug without proper input.

[boan-anbo]

Hi Anthony, the error just occurred again and here are the picture files (simply empty...) and the logs. Deleting the last two empty pictures solves the problem; otherwise Raspberry crushes every time I open the webpage.

https://www.dropbox.com/sh/g0w2tofb0x4szv5/AAB8NKqawOAiaUbT267vYNNHa?dl=0

Hope it helps!

[adongy]

Thanks, I think I got it: jpegtran does not handle very well 0 bytes image files, it just segfaults leading to the errors you mentioned.
I'll try to fix it later this week.

Edit: the root cause of it all is that one of the two cameras did not seem to take a shot when asked to, maybe a little bug in CHDK or in the python wrapper. Can you try to put the camera in 'P' mode when taking shots? Usually it gives better results.

[lupa18]

Hello, what of all of this install options solve this issue? Thanks in advance.