Issue 118725 - Pictures replaced by "read error" message and dropped
Summary: Pictures replaced by "read error" message and dropped
Status: CONFIRMED
Alias: None
Product: Writer
Classification: Application
Component: editing (show other issues)
Version: OOo 3.3
Hardware: PC Windows, all
: P2 Major (vote)
Target Milestone: ---
Assignee: AOO issues mailing list
QA Contact:
URL:
Keywords: data_loss
: 126212 (view as issue list)
Depends on:
Blocks:
 
Reported: 2012-01-05 05:23 UTC by Chnutz
Modified: 2019-10-05 22:05 UTC (History)
14 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments
Container with material to work on (276.67 KB, application/zip)
2014-07-21 11:16 UTC, Regina Henschel
no flags Details

Note You need to log in before you can comment on or make changes to this issue.
Description Chnutz 2012-01-05 05:23:56 UTC
When I insert images in a large oO 3.3.0-Writer-odt-Project (>50MB), some pictures are lost after saving or auto-saving. Instead of the images the is an error text "Graphik nicht darstellbar" (image cannot be shown) I have the same problem on my notebook (2GB, WIn XP SP 3) and on my pc (4GB, WIN 8 Professional)
- images are embedded, not only linked;
- project is an odt-Project, not converted from doc;
- the lost images not always be the same, they are often the same, but changed from time to time;
- image formats: jpg and png;
- images in a frame with "Beschriftung" (Sorry, I have a German Version and don`t know the exact translation. I mean, there is a description "Image xy: blabla" below the image);
- images or frames linked to the page or to the paragraph;
- before saving they were shown correctly;
- after the first issue i rise the image cache from 20 to 50 MB without effect;
- after closing an reopening the file the pictures with errors are lost.
The file is too big to attach it, but you can download the document here:
http://www.amtshof.net/Kaub/20120102KaubSNK.odt
I attach an image I try to embedd.
All images are originally filled in, when you'll find a frame with a description and without image: this is the error ;-)

I tried it with filial docs, but the same effect appeared and many others, too.

regards
Chnutz
Comment 1 Chnutz 2012-01-05 05:59:07 UTC
When I try to insert the image at the same or another place without reopening the document a 1x1 picture frame was inserted and when I stretch it the same error was shown
Comment 2 T. J. Frazier 2012-01-05 11:45:06 UTC
(Google Translate, http://translate.google.com/# is your friend. Beschriftung = "caption".)

We at ODFAuthors work with the OO.o user manuals: large files with many pictures. We saw some flaky (= "hard to reproduce exactly") problems like this, until we raised the cache to 256MB. You might try that. Developers might try the opposite, to force the problem.
Comment 3 zhao xia 2012-02-17 08:35:40 UTC
This one is content lost for special case, I tried on several documents and couldn't reproduce it. So not sure if dpend on file size?
Comment 4 Matthias Basler 2012-02-19 13:26:23 UTC
I can confirm the issue on OOo3.3 (on WinXP SP3), just as described.
(Just the message is now "Grafik nicht darstellbar." in German.)
I gather from various forums that other people also have this problem.

Indeed it seems to happens reproducible after every AutoSave or so, and if I then save the file to a new name the file size is smaller than in the previous version, so the images were actually not stored any more, not just "hidden" or anything similar.

It has almost driven me crazy during the last days ...

I consider this a major, if not critical problem, because this essentially makes it impossible to seriously work on larger books (>50MB) with lots of images. Inserting the images as reference is not a workaround either because for such linked images the crop functionality does not work correctly (see f.e. bug 110432 filed 2010 not yet fixed).

P.S: How does "Critical" and "P5" fit together? I thought P5 is lowest priority? Shouldn't this be P3 at least?
Comment 5 Matthias Basler 2012-02-19 18:12:34 UTC
I tested the suggestion of T. J. Frazier and applied following configuration on the memory page:

Graphics cache
- Use for OOo: 256 MB (was: 20MB?)
- Memory per Object: 5MB
- Remove from memory: after 00:10
Cache for inserted objects
- No. of objects: 256 (was: 20)

This did *not* fix the issue for me.
@T. J. Frazier: Exactly what settings do you use?

I'd like to nominate this one for the "3.4_release_blocker" flag, since data loss is involved.
Comment 6 T. J. Frazier 2012-02-19 18:59:58 UTC
(In reply to comment #5)
> I tested the suggestion of T. J. Frazier and applied following configuration on
> the memory page:
> 
> Graphics cache
> - Use for OOo: 256 MB (was: 20MB?) | SAME
> - Memory per Object: 5MB | 1MB
> - Remove from memory: after 00:10 | 01:00 (one hour)
> Cache for inserted objects
> - No. of objects: 256 (was: 20) | 350
> 
> This did *not* fix the issue for me.
> @T. J. Frazier: Exactly what settings do you use?

@mathiasbassler: Our settings shown above. The general idea is to keep everything in memory at once, so the settings may vary according to your usage (for instance, set Memory per Object for your own largest object).
> 
> I'd like to nominate this one for the "3.4_release_blocker" flag, since data
> loss is involved.
Setting "data loss" keyword, and cleaning up other settings.

@ zhao xia: file size (very large, tens of MB) is definitely a triggering factor. Not seen in >1MB individual chapters, but the master ...
Comment 7 Chnutz 2012-02-19 21:14:54 UTC
I think, it's maybe a problem with the "Save"-module/function.
- It needs more than 10 times of MS Word to save the file. Sometimes a hung up appers;
- I tried to fix the problem and saved every time I fill in a picture. After this, nearly every picture was lost except the last one I inserted. As often I did save, as more pictures were lost.
After this I tried only to link the next pictures, but with the same effect: Save = lost pictures.
Comment 8 Matthias Basler 2012-02-19 22:15:08 UTC
Again I tested the settings given by T. J. Frazier, but they made no difference in my case - not even setting the in-memory time to 2 hours.

I have now resorted to following procedure, which at least allows me to continue working on the book with some progress:
1. Open latest document
2. Do some changes, insert now images etc.
3. Save document (under a new name)
4. Close OOo
Repeat from step 1.

Until I have saved the first time everything's fine. It's only then that the problems occur.
I also noted that usually(!) only the images that were last inserted or edited were dropped, although usually not all of them.

@All Thanks for helping on this so far.
Comment 9 Kayo Hamid 2012-05-22 02:30:34 UTC
I believe this is the same of 119360.
Comment 10 stfhell 2012-12-21 10:01:46 UTC
Since the bulk of the image handling code is shared between OpenOffice and LibreOffice, it might be worthwhile to track the LibreOffice metabug issue that tries to collect all bugs that have to do with image caching:

https://bugs.freedesktop.org/show_bug.cgi?id=47148
(#fdo47148 - image caching / management is utterly shambolic.)
Comment 11 Rainer Bielefeld 2014-05-26 17:28:47 UTC
Back to UNCONFIRMED. We have Matthias Basler observations, but without sample documents we can't even find out whether his problem has the same roots as reporter's

@Chnutz: 
Still a problem for you? ORW currently is very busy in that "picture loss" area, so there is a good chance for a solution. But without a sample document we can't find out the roots for your problems. You may send a confidential document to me by email.

@Oliver:
I added you to cc because of your activity in this area.
Comment 12 Matthias Basler 2014-05-27 17:51:06 UTC
I cannot confirm this with an own sample document on AOO 4.1 right now.
Not sure if I could reproduce it there, had I still the original document.


@Rainer Bielefeld
Please note that Chnutz provided a sample document at that time. It is just not accessible any more. Someone should have placed it on a more accessible location back then ...

But since it is not possible to attach serious large documents to this bug tracker, no wonder there is none attached ...
Comment 13 Matthias Basler 2014-05-27 19:01:06 UTC
I can now confirm the bug on AOO 4.1.0, albeit with a larger sample.

Tried to reproduce this behaviour with a scrambled book of 165 pages with ~1 image per page.

Memory consumption is huge ...  ~1.5GB, and after I add another image I am hardly able to "Save as" the document. Opening the dialog takes forever and pressing "Save" does ... absolutely nothing.
... Correction ...
After another 2 minutes and several presses it finally reacts.  :-(

So I save the file into a DIFFERENT directoy. Memory consumption drops to 100MB. Strange - why does it suddenly need less?

I add another picture.

Then again I save the file into a DIFFERENT directoy. This time AOO is more responsive.

After every image I insert into the doc AOO jumps to the end of the document. What the hell ... ?

And then there they are when I scroll through the document: The boxes saying "Lesefehler" ("Read error") where images used to be. A document devoid of almost any images is what is left.

As usual, since the file is far too big (140MB, sigh ...) I'll upload it in my GMX MediaCenter. But please be warned: IT WILL NOT STAY THERE FOREVER EITHER!
Please use the file "TestBook2.odt" there for this issue.

https://mc.gmx.net/guest?path=Meine%20Dokumente%20von%20matthiasbasler%40earthflight.org&token=425CD698469014B4&locale=de_DE&viewType=0

I also tried another, slighly smaller test document, but with that AOO instantly crashes on loading. :-( 
I believe I am wasting my time here trying to avoid other bugs just to test one.


Remarks:
1. If I remember correctly LibO fixed this bug a while ago. (See the meta bug mentioned above.)

2. "image caching / management is utterly shambolic" really hits the point.

3. I am pretty sure - although not 100% - that this is the same effect as described by Chnutz. Same sequence of saving and image loss.

4. Oh, and I believe this makes the issue CONFIRMED again, doesn't it?
Comment 14 Matthias Basler 2014-05-27 19:28:56 UTC
I just realize I forgot the usual info: Reproduced on Win7, 64 Bit, 8GB RAM, SSD drive.
Comment 15 Rainer Bielefeld 2014-05-28 05:35:21 UTC
Original report: with 3.3 every now and then few pictures missing
Your observation: with 4.1.0 reliably all pictures lost (document completely
                  damaged) when save with additional picture.
I am pretty sure that Comment 13 problem is a different one than the one from original report, but more research will be required.

Until we can distinguish the problem in this report from any other problem this one is UNCONFIRMED

I submitted "Issue 124999 - CRASH when scroll through very large and complex .odt" for the memory consumption problem

@Matthias Basler 
Thx. for testing. I switch back to UNCONFIRMED because currently it's completely unclear whether your problem has to do anything with reporter's problem.

Can you install OOo 3.3 in parallel and do additional tests to reproduce a problem with 3.3 and to find exactly the same in AOO 4.1?
Comment 16 Rainer Bielefeld 2014-05-28 06:34:28 UTC
(In reply to Matthias Basler from comment #13)
My results in Issue 124999 are not reproducible with OOo 3.3.0, so that problem (identical with your observations in Comment 13) is different to the one from the original report here.
But may be there are some common early roots
Comment 17 Rainer Bielefeld 2014-05-28 11:57:23 UTC
It seems that the problem also is Reproducible with sever installation of "OOo 3.3.0 English UI / German locale [OOO330m20 (Build 9567)]" on WIN7 Home Premium (64bit) DE during a several hours test:

1. From linked test kit open TestBook2.odt
2. Menu 'Tools -> Options -> Load/Save - General':
   Auto Recovery every minute
3. From now on do some arbitrary smaller edits all over the document, each
   time delete few words or add few words, add a background picture from
   gallery or an arbitrary 0,5 ... 2 MB digicam photo, move a picture few cm.
    Every second edit scroll to check for read error boxes, then save 
    document under new name.
    Every 2 hours save, close and reopen

After few hours first read error boxes should appear.

After save - close - reopen the error boxes (and the pictures) will be lost

@Oliver:
May be you want to check dependencies to your other picture loss fix
activities?
I here nave some different document versions with different sizes caused by picture loss, may be they can give hints concerning the roots?
Comment 18 Rainer Bielefeld 2014-05-28 12:28:50 UTC
Currently because of very special circumstances "major" seems more appropriate, may be we will have to rethink later.
Comment 19 Rainer Bielefeld 2014-05-28 12:30:27 UTC
Currently the root seems to be a loss during the edits (error message), and loss becomes final by save.
Comment 20 Oliver-Rainer Wittmann 2014-06-06 11:59:14 UTC
The initial defect description fits to the issues on which recently had been made corrections - namely issue 114361, issue 124717 and issue 124966.
As I am not able to access the original sample document I am not able to verify, if this one is fixed by the above mentioned corrections.

I am asking for testing recent builds from trunk versions - provided by the build bots - in order to get confidence, if this is the same or another defect. Thanks in advance.
Comment 21 Regina Henschel 2014-07-20 23:52:43 UTC
I have got a debug build of r1610953 and have seen lost pictures right now.
Comment 22 Oliver-Rainer Wittmann 2014-07-21 08:33:33 UTC
(In reply to Regina Henschel from comment #21)
> I have got a debug build of r1610953 and have seen lost pictures right now.

@Regina:
I would be really great, if you could provide a sample document together with corresponding steps to reproduce your observed loss of pictures in recent trunk version. Thanks in advance.
In case you sample document is confidential or too big, you can send it directly to me (orw@apache.org) - be sure that I will keep it confidential and that I will only use it to reproduce the loss of pictures.
Comment 23 Regina Henschel 2014-07-21 11:16:10 UTC
Created attachment 83716 [details]
Container with material to work on

To reproduce the bug do this:
Unpack the attached container. It contains a lot of pictures and an empty presentation show_all.odp, you will need in the next steps. In addition it contains the file damaged.odp, which I get after my first save and damaged_resaved.odp, which I get, when I opened damage.odp and resaved it immediately.

How to produce a damaged file:
1. Open the file show_all.odp
2. Goto Tools > Options > Load/Save > General. Check "Always create backup copy" and "Save AutoRecovery information every" and set the time to 3 Minutes. I guess, the problem is connected with autosave, because I could not reproduce it, when the options were unchecked.

Use the pictures in alphabetic order.
3. Insert the first picture. It is a .png file. Make sure that "Link" is checked.
4. Align the picture left, vertical center.
5. Name the picture "S1".
6. Insert the second picture. It is a .svg file. Make sure that "Link" is checked. You need not name the picture, it gets a name automatically from file name.
7. Align the picture right, vertical centered.
8. Goto next slide.

Repeat steps 3 to 8 with the next pair of .png - .svg files and so on for all eleven slides. It is important, that you really spend some time on it, so that the autosave feature is triggered. You need about 15 min to insert all the given pictures.

Save the file and close it.

Reopen the file. Notice the damage:
Links are broken and references contain a wrong URI.

I have worked with a debug build of AOO420m1(Build:9800)-Rev. 1610953 on Windows 7.
Comment 24 Oliver-Rainer Wittmann 2014-07-21 12:47:28 UTC
(In reply to Regina Henschel from comment #23)
> Created attachment 83716 [details]
> Container with material to work on
> 
> To reproduce the bug do this:
> Unpack the attached container. It contains a lot of pictures and an empty
> presentation show_all.odp, you will need in the next steps. In addition it
> contains the file damaged.odp, which I get after my first save and
> damaged_resaved.odp, which I get, when I opened damage.odp and resaved it
> immediately.
> 
> How to produce a damaged file:
> 1. Open the file show_all.odp
> 2. Goto Tools > Options > Load/Save > General. Check "Always create backup
> copy" and "Save AutoRecovery information every" and set the time to 3
> Minutes. I guess, the problem is connected with autosave, because I could
> not reproduce it, when the options were unchecked.
> 
> Use the pictures in alphabetic order.
> 3. Insert the first picture. It is a .png file. Make sure that "Link" is
> checked.
> 4. Align the picture left, vertical center.
> 5. Name the picture "S1".
> 6. Insert the second picture. It is a .svg file. Make sure that "Link" is
> checked. You need not name the picture, it gets a name automatically from
> file name.
> 7. Align the picture right, vertical centered.
> 8. Goto next slide.
> 
> Repeat steps 3 to 8 with the next pair of .png - .svg files and so on for
> all eleven slides. It is important, that you really spend some time on it,
> so that the autosave feature is triggered. You need about 15 min to insert
> all the given pictures.
> 
> Save the file and close it.
> 
> Reopen the file. Notice the damage:
> Links are broken and references contain a wrong URI.
> 
> I have worked with a debug build of AOO420m1(Build:9800)-Rev. 1610953 on
> Windows 7.

Thanks for the detailed described.

I am able to reproduce the your described picture loss in recent developer snapshot M2 of planned 4.1.1 release and on a recent trunk version, but not in 4.1.0

Thus, I conclude that this picture loss is a new one --> I will submit a new issue.
Comment 25 Regina Henschel 2014-07-21 13:14:29 UTC
I get the error too in AOO411m3(Build:9772)  -  Rev. 1611634, administrative installed, on Windows 7.
Comment 26 Armin Le Grand 2014-07-21 15:28:17 UTC
Did some checks onm Regina's example; we have 22 pics, 11 png's and 11 svg's. The svg's have a corresponding preview png (double pic data) for backward compatibility; in these 2 are double, one is triple, so that 7 different preview png's for svg exist. This can be correct since the ID for these images is equal when the pics are equal. Since these svg's are examples which are targeted to create same content his is probably correct. All seven of these images are patr of the file in the Picures folder.
In the resaved form all but the last svg is missing and only the replacement is still there. All 7 replacements are in the Pictures folder.
When loading damaged.odp the svgs are loaded correctly. The pngs are missing, the link showing the full name used in the content.xml stream. Example for 1st frame:

xlink:href="../attribute_vs_id.png%3FrequestedName=attribute_vs_id"

When correcting this to

xlink:href="../attribute_vs_id.png"

all loads as expected. Checking if this was the case in e.g. AOO410...
Comment 27 Armin Le Grand 2014-07-21 15:36:06 UTC
Continued with trunk version:
When creating a new presentation and adding the 1st png and pic directly (not waiting for save) I get for the png

xlink:href="Pictures/100000000000016A000001E55A5CFEB7.png"

and for the svg

xlink:href="Pictures/100002010000021200000212EE43C7A1.png"
xlink:href="Pictures/10000308000036CC000036CC18EE503F.svg"

2nd slide used links, I get for the png

xlink:href="../PictureLoss/attribute_vs_name.png"

and for the svg

xlink:href="Pictures/100002010000021200000212EE43C7A1.png"
xlink:href="../PictureLoss/attribute_vs_name.svg"

I will now check what happens when the in-betwen save triggers, but my current guess is that the extra text in the form of '%3FrequestedName='filename'' is wrong. It will be interesting to see if the linked and unlinked versions create this...
Comment 28 Armin Le Grand 2014-07-21 15:43:56 UTC
Still on trunk version, for slide1 (non-linked) all is fine, for slide2 (linked) I can indeed reproduce the names for the used graphics with the mentioned extended strings; for the linked png I get


xlink:href="../PictureLoss/attribute_vs_name.png%3FrequestedName=attribute_vs_name"

and for the linked svg I get

xlink:href="Pictures/100002010000021200000212EE43C7A1.png"
xlink:href="../PictureLoss/attribute_vs_name.svg%3FrequestedName=attribute_vs_name"

Something changes the state in a way that this extension is created by error for linked graphics. Need to investigate why this happens...
Comment 29 Oliver-Rainer Wittmann 2014-07-21 15:44:20 UTC
new issue of the wrong URL of linked PNG images - see issue 125289
Comment 30 Oliver-Rainer Wittmann 2014-07-21 15:50:16 UTC
new issue of the embedded SVG of linked SVG image - see issue 125290
Comment 31 Chnutz 2014-07-22 12:56:54 UTC
Sorry, I was very busy and often not at home in the last few months, so I didn't realize the question for the document till today, but at last here's a new link to my first document where the error occured:
http://www.chnutz.de/download/20120102KaubSNK.odt
Comment 32 Pedro 2014-08-03 18:03:44 UTC
(In reply to Chnutz from comment #31)
> Sorry, I was very busy and often not at home in the last few months, so I
> didn't realize the question for the document till today, but at last here's
> a new link to my first document where the error occured:
> http://www.chnutz.de/download/20120102KaubSNK.odt

Loaded the document without any problems with AOO411m4(Build:9774)  -  Rev. 1614049 (aka RC1) on Win7 x64 (Intel i5 4Gb RAM). Inserted new PNG image at page 15, saved, closed, opened, no problem.
Comment 33 Oliver Brinzing 2015-03-30 10:17:57 UTC
.
Comment 34 oooforum (fr) 2015-04-20 09:50:29 UTC
*** Issue 126212 has been marked as a duplicate of this issue. ***
Comment 35 oooforum (fr) 2019-10-05 14:29:49 UTC
(In reply to Chnutz from comment #0)
> The file is too big to attach it, but you can download the document here:
> http://www.amtshof.net/Kaub/20120102KaubSNK.odt
Unable to download this document: username and password required.
Comment 36 Pedro 2019-10-05 16:24:34 UTC
(In reply to oooforum (fr) from comment #35)
> (In reply to Chnutz from comment #0)
> > The file is too big to attach it, but you can download the document here:
> > http://www.amtshof.net/Kaub/20120102KaubSNK.odt
> Unable to download this document: username and password required.

Working link in #Comment31
Comment 37 Keith N. McKenna 2019-10-05 22:05:11 UTC
Using the link from from comment 31 I was able to download the original file. With AOO version 4.1.7 I was able to open the file with no problem and a scan of the file gave no indication of the error message or dropped graphics. There were what appeared to be empty frames for pictures not yet inserted. I was able to insert a picture into one and save and reopen the file with no problem.

System and AOO configuration as follows:

System:
Intel (R) Pentium(R) CPU 4405U @2.10GHz, 2112MHz, 2(Core(s) 4 Logical Processors
Total Physical Memory 8.00 GB
Operating System: Windows 10 Home OS Build 10.0.17763.737 Version 1809

Apache Open Office:
AOO417m1(Build:9800)  -  Rev. 46059c9192 2019-09-03 12:04
Additional Language Packs: None

My belief is that this issue was solved somewhere after the 4.1.1 release. I will do some more testing with Regina's information and report back within the next few days.