Issue 88577 - Cannot open certain DBF (with offset)
Summary: Cannot open certain DBF (with offset)
Status: CLOSED FIXED
Alias: None
Product: Calc
Classification: Application
Component: open-import (show other issues)
Version: DEV300m9
Hardware: All All
: P3 Trivial (vote)
Target Milestone: ---
Assignee: oc
QA Contact: issues@sc
URL:
Keywords: oooqa
Depends on:
Blocks:
 
Reported: 2008-04-22 09:08 UTC by pmike
Modified: 2013-08-07 15:15 UTC (History)
6 users (show)

See Also:
Issue Type: PATCH
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments
zipped sample (173.92 KB, application/octet-stream)
2008-04-22 09:10 UTC, pmike
no flags Details
patch + test dbfs (1.75 KB, application/octet-stream)
2008-05-05 17:57 UTC, bormant
no flags Details
Patch for connectivity which is already include in cws dba30c (7.67 KB, patch)
2008-05-06 09:17 UTC, ocke.janssen
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this issue.
Description pmike 2008-04-22 09:08:09 UTC
There is sample DBF file OOo cannot open (check versions 2.3.1, 2.4.0 and
DEV300_m10)

Some tech info about DBF:
DBF version marker: 0x30
DBF first record offset: 0x0CA0 - this might be the reason of the issue (manual
edit of DBF and removing 'gap' between header and data solves the problem)
Comment 1 pmike 2008-04-22 09:10:33 UTC
Created attachment 53105 [details]
zipped sample
Comment 2 kpalagin 2008-04-25 07:24:26 UTC
Confirming - as described.

From i8857:
"at file offset 8 (zero based) there
is a 16-bit little-endian value telling the overall length of the
header, here 0x0483. The last byte of the header has to be a 0x0d,
which isn't the case here. The 0x0d is at offset 0x0480 instead, and
two more bytes follow to pad the header. Calc does check for the
presence of the 0x0d at the given offset, and if not found refuses to
load the file. The last instance in the chain is the Writer module,
which then loads the file as text. The database module however has a
somewhat more relaxed check and doesn't complain, so the file is
availabel as data source."

http://www.openoffice.org/issues/show_bug.cgi?id=8857
http://www.openoffice.org/issues/show_bug.cgi?id=4991
Comment 3 bormant 2008-05-05 17:53:34 UTC
Solution for this is not complex.
1) modify lcl_MayBeDBase( ) in sc/source/ui/unoobj/scdetect.cxx to accept 
foxpro dbfs as valid dbf-tables without penalty to severity of checking;
2) modify corresponding part of dbase driver (DTable.hxx, DTable.cxx) in 
connectivity/source/drivers/dbase to acts hand-by-hand.
After patch we checks:
1) marker (1 byte at offset 0) corresponding to dbase driver source to 
guarantee that driver can read this type of dbf (NEW);
2) 0x0d after n-th field description (MODIFIED);
3) value of data offset (or header size, that is the same) (2 bytes at offset 
8) checked more strictly (MODIFIED).
It doesn't break current functionality.

Test sequence:
1) start scalc;
2) from main menu select File/Open... or press Ctrl+O;
3) in "files of type" dropdown select "dBASE (*.dbf)";
4) navigate to test_h.dbf from attachment's test_h.dbf or exam.dbf and press 
Open button;
5a) without patch you'll see ASCII filter parameters dialog (wrong behaviour);
5b) with patch you'll see dBASE encoding dialog (right behaviour).

Import of exam.dbf will fail according to another bug 
http://qa.openoffice.org/issues/show_bug.cgi?id=87325, it's another problem and 
has another resolution.

ps. Sorry for my English, I'm not a native speaker.
Comment 4 bormant 2008-05-05 17:57:08 UTC
Created attachment 53391 [details]
patch + test dbfs
Comment 5 ocke.janssen 2008-05-06 09:15:37 UTC
I attached a patch file for the connectivity part. Now Visual FoxPro (0x30) is
accepted as well. The text encoding which can be set inside the dbase is now
also used when of type visual fox pro and the connection character set is set to
system.
Comment 6 ocke.janssen 2008-05-06 09:17:00 UTC
Created attachment 53406 [details]
Patch for connectivity which is already include in cws dba30c
Comment 7 bormant 2008-05-06 11:17:46 UTC
oj, this encodind applays not only for 0x30 marked dbfs, so the condition is 
not only 

case VisualFoxPro:

but

static const char dbf_markers[] = "\x8B\xCB\x03\x30\x43\x63\x83\xF5\xFB";
/* valid "codepaged" markers are with "\x8B\xCB" exception */
static const char *dbf_markers_cp = dbf_markers+2;

if ( strchr(dbf_markers_cp, m_aHeader.db_frei[17]) 
    && !m_aHeader.db_frei[18] && !m_aHeader.db_frei[19])
{
    ...

Condition "&& !m_aHeader.db_frei[18] && !m_aHeader.db_frei[19]" is from 
microsoft cpzero program, shipped with foxpro since 2.6 or earlier and prevent 
corruption future dbf extension that may it's own point of view on reserved 
fields.
Comment 8 ocke.janssen 2008-05-06 11:51:04 UTC
I understand the line 
&& !m_aHeader.db_frei[18] && !m_aHeader.db_frei[19]

but the line

strchr(dbf_markers_cp, m_aHeader.db_frei[17])

confuses me a little bit. Do you mean

strchr(dbf_markers_cp, m_aHeader.db_type)

?


Comment 9 bormant 2008-05-06 12:06:23 UTC
yes, of course, you are right. It must be "m_aHeader.db_type". Copy/Paste is 
wrong method of software developing :-)
Comment 10 oc 2008-05-06 12:51:14 UTC
assigned to OJ
Comment 11 ocke.janssen 2008-05-07 06:45:25 UTC
Hi Oliver,

this patch contains also some pieces of code which have to included in calc. So
I attached my patch as well. As I had a cws at hand I put it also at dba30c. So
please do not assign this issue to me.

Thanks.

Best regards,

Ocke
Comment 12 ocke.janssen 2008-05-07 08:50:54 UTC
Set Owner
Comment 13 oc 2008-05-19 09:53:21 UTC
Hi Eike, please have a look
Comment 14 ooo 2008-06-03 16:50:04 UTC
Waiting with this for integration of cws dba30c.

@bormant: did you verify that with the modified header check the file from issue
9581 is still loaded? And now maybe even the file from issue 4991 as well?
Comment 15 bormant 2008-06-04 17:35:04 UTC
@er: Yes, I did. lcl_MayBeDBase() says true on i9851 and i4991 dbfs.
Comment 16 ooo 2008-06-04 19:34:05 UTC
Nice. Thanks for verification.
Comment 17 ooo 2008-06-17 13:48:30 UTC
In cws odff04:

sc/source/ui/unoobj/scdetect.cxx  1.23.66.1
Comment 18 ooo 2008-06-25 18:56:16 UTC
Reassigning to QA for verification.
Comment 19 ooo 2008-06-25 19:00:57 UTC
...
Comment 20 oc 2008-07-01 10:29:59 UTC
verified in internal build cws_odff4
Comment 21 lohmaier 2008-08-05 18:46:46 UTC
opening of the sample take a while, but works OK in m28 → closing