Issue 98990

.

@brinzing: Could please attach the extension?  

here is what i found - imho the "-f" parameter does not work ...
(and oo 2.4.2 will always work like the frist time installation)

1. first time installation:

unopkg add .\..\share\uno_packages\scriptdispatch.oxt -f --verbose --shared
Listening for transport dt_socket at address: 8000
Kopiere: ScriptDispatch.oxt
pythonloader.Loader ctor
pythonloader.Loader.activate
pythonloader: interpreting url vnd.openoffice.pymodule:pythonscript
pythonloader: after expansion vnd.openoffice.pymodule:pythonscript
Deaktiviere: ScriptDispatch.oxt
 Deaktiviere: ScriptDispatch
Aktiviere: ScriptDispatch.oxt
 Aktiviere: ScriptDispatch
unopkg done.

unopkg-desc.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<unopackages xmlns:unopackages="unopackages.dtd">
    <language value="Java">
     <package value="vnd.sun.star.expand:
       $UNO_SHARED_PACKAGES_CACHE/uno_packages/29D4.tmp_/
       ScriptDispatch.oxt/ScriptDispatch/"/>
 </language>
</unopackages>

2. second time installation:

unopkg add .\..\share\uno_packages\scriptdispatch.oxt -f --verbose --shared
Listening for transport dt_socket at address: 8000
Kopiere: ScriptDispatch.oxt
pythonloader.Loader ctor
pythonloader.Loader.activate
pythonloader: interpreting url vnd.openoffice.pymodule:pythonscript
pythonloader: after expansion vnd.openoffice.pymodule:pythonscript
Deaktiviere: ScriptDispatch.oxt
 Deaktiviere: ScriptDispatch
unopkg done.

unopkg-desc.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<unopackages xmlns:unopackages="unopackages.dtd">
  <language value="Java"/>
</unopackages>

3. third and following installations:

unopkg add .\..\share\uno_packages\scriptdispatch.oxt -f --verbose --shared
Listening for transport dt_socket at address: 8000
Kopiere: ScriptDispatch.oxt
pythonloader.Loader ctor
pythonloader.Loader.activate
pythonloader: interpreting url vnd.openoffice.pymodule:pythonscript
pythonloader: after expansion vnd.openoffice.pymodule:pythonscript
unopkg done.

unopkg-desc.xml still is:
<?xml version="1.0" encoding="UTF-8"?>
<unopackages xmlns:unopackages="unopackages.dtd">
    <language value="Java"/>
</unopackages>

Created attachment 60057 [details]
demo oxt

Created attachment 60058 [details]
use for starting the demo script

just forgot to mention:
i can install all my other extensions with parameter "-f"
(force overwriting existing extensions)

and: unopkg remove org.test.java.scriptdispatch
will give an error too

after some tests with clean installations inside a xp vm i am pretty sure
what's going wrong:

install oo 3.0.1 without "gm_o_Pyuno" and everything works like expected:

C:\Programme\OpenOffice.org 3\program>unopkg add C:\ScriptDispatch.oxt --shared
--verbose -f
Raising process: file:///C:/Programme/OpenOffice.org%203/program/soffice
Arguments: -nologo -nodefault -accept=pipe,name=f311cc459d6d2269f646c9a6cc237dd8
ddab55ffc6456358c7cdd6748e92a6;urp;
Ok.  Connecting...Ok.
Kopiere: ScriptDispatch.oxt
Deaktiviere: ScriptDispatch.oxt
 Deaktiviere: ScriptDispatch
Aktiviere: ScriptDispatch.oxt
 Aktiviere: ScriptDispatch
unopkg done.

install with "gm_o_Pyuno" and you will have trouble again...

and of course it's "remove org.test.java.scriptdispatch --shared"
to remove a shared extension...

.

.

.

Well, I was not even able to install the extension (and a different one) on
windows using OOO310m1. Investigating.

@brinzing: How is the functionality of the extension to be tested? Should there
be an entry among the organize macro dialog?

> I was not even able to install the extension (and a different one) o
> windows using OOO310m1. Investigating.

due to issue http://www.openoffice.org/issues/show_bug.cgi?id=99249
you cannot install an extension using unpgk from command line

use oo 3.0.1 for testing. after installing the extension open the
"StartScriptDispatch.odt" document and click the button

When reinstalling the extension then the extension manager internally removes
the old version and installs the new version. In the latter case it first checks
if the extension is installed. To do this, the "master script provider" is asked
if it knows the new extension. It returns true, which is wrong, and therefore
the new extension will not be installed. The master script provider in turn asks
several special script provider. All of them except 
com.sun.star.script.provider.ScriptProviderForPython, return false. That is, the
python script provider is wrong! So brinzings observation is right.
I will forward this issue to mh, so he can find someone who is responsible for it.

Btw, the function hasByName is called.

to reproduce this issue, issue 99249 needs to be fixed first.

Clarifying: dependency on issue 99249 is only valid for windows. Therefore, one
could use other platforms to investigate this issue.

@MH: Please forward this issue to the maintainer of the python code.

declared as blocker for 3.1 in release status meeting

[the problematic hasByName (that always returns true) mentioned in #desc15 is at
tags/DEV300_m42/scripting/source/pyprov/pythonscript.py l. 775]

tags/OOO310_m3/scripting/source/pyprov/pyhtonscript.py has several severe flaws,
it appears:

1  The PythonScriptProvider has a mapPackageName2Path populated in
getPackageName2PathMap by iterating over the uno_packages/cache file system.
In the case of the first add ScriptDispatch.oxt, the file system iteration is
done at a time when the ScriptDispatch.oxt has already been unpacked in the
uno_packages/cache, so the map has an (erroneous) entry (ScriptDispatch.oxt ->
.../uno_packages/cache/uno_packages/TAG1/ScriptDispatch.oxt/ScriptDispatch/).  A
following
hasByName(.../uno_packages/cache/uno_packages/TAG1/ScriptDispatch.oxt/ScriptDispatch/)
thus erroneously returns true, and the extension is not properly installed (no
basis layer share/Scripts/unopkg-desc.xml is written).
In the case of a subsequent add -f ScriptDipatch.oxt (removing the already
installed ScriptDispatch.oxt and installing it anew), the file system iteration
is done at a time when both the old and new ScriptDispatch.oxt are unpacked in
the uno_packages/cache.  Depending on the file system traversal order, the
resulting map either has a (correct) entry (ScriptDispatch.oxt ->
.../uno_packages/cache/uno_packages/TAGold/ScriptDispatch.oxt/ScriptDispatch/)
or an (erroneous) entry (ScriptDispatch.oxt ->
.../uno_packages/cache/uno_packages/TAGnew/ScriptDispatch.oxt/ScriptDispatch/)---since
both entries have the same key (ScriptDispatch.oxt), they overwrite one another.
In the case where the file system traversal happens to finally record in the map
the TAGold version, a following
hasByName(.../uno_packages/cache/uno_packages/TAGold/ScriptDispatch.oxt/ScriptDispatch/)
correctly returns true, a following
removeByName(.../uno_packages/cache/uno_packages/TAGold/ScriptDispatch.oxt/ScriptDispatch/)
removes the single entry from the map (which is then empty), so that a following
hasByName(.../uno_packages/cache/uno_packages/TAGnew/ScriptDispatch.oxt/ScriptDispatch/)
happens to correctly return false (and everything appears to work fine then).
In the case where the file system traversal happens to finally record in the map
the TAGnew version, a following
hasByName(.../uno_packages/cache/uno_packages/TAGold/ScriptDispatch.oxt/ScriptDispatch/)
erroneously returns false, and the extension re-installation does not work properly.
It appears that on Windows the file system traversal always produces the second
case (on Linux, if the following problem were fixed, the two cases would alternate).

2  Due to the 3-layer basis symlink on Unix, the file URLs produced by
getPackageName2PathMap contain .. segments, while the URLs passed into hasByName
are normalized (in expandUri) and thus no longer contain .. segments.  This
causes hasByName to always return false on Unix systems.  (Each element of
pathes at line 587 would have to be passed through uno.absolutize, as is done in
expandUri.)

3  The code will always fail when there are different extensions that happen to
have the same file name (but different extension identifiers)---see the
discussion of mapPackageName2Path under (1).

So, it seems a substantial redesign of pythonscript.py is necessary to fix this
issue.

so are there any suggestions how to fix sb-comment:1 ?

I am a little confused, is it allowed to have installed two totally  different
components with the same name ? 

If yes, pythonscript.py really needs a complete redesign (actually I dont
understand, how script uri then reference the different incarnations of the same
package name, currently they reference it only by package name AFAIK ).

If no, it is imho a bug in the ooo framework, that the the python script
provider is instantiated, when there is an old and a new component within the
cache directory.

Bye,

Joerg

@jbu:

- "is it allowed to have installed two totally different components with the
same name": yes, see
<http://wiki.services.openoffice.org/wiki/Documentation/DevGuide/Extensions/Extension_Identifiers>

- "I dont understand, how script uri then reference the different incarnations
of the same package name": ab, can you enlighten us?

- "it is imho a bug in the ooo framework, that the the python script provider is
instantiated, when there is an old and a new component within the cache
directory":  the population of the cache directory is imo an implementation
detail of the extension manager, not to be exploited by the various extension
handling backends, but jl is the authority here

but installing the same component again is still illegal, as name + extension
identifier are identical.

@jl: So is there an 'official way' to retrieve installed packages at script
provider instantiation time ( I implemented this by Tomas O'Connors advice some
years ago) ? If the cache directory is to be parsed, the current behaviour would
still be a bug of the ooo script framework ...


BTW: Would be nice to add to
<http://wiki.services.openoffice.org/wiki/Documentation/DevGuide/Extensions/Extension_Identifiers>

whether the tuple filename+identifier is unique or just the identifier is unique
(meaning that two extensions with different names and equal identifiers cannot
be installed side by side). 
Bye,

Joerg

@jbu:

- "installing the same component again is still illegal, as name + extension
identifier are identical":  sure, but what happens here is that two instances of
the same extension are laying unpacked in uno_packages/cache, due to calling
"unopkg add -f" on an already installed extension (not sure from your comment
whether or not that was already clear for you)

- "Would be nice to add [...] whether the tuple filename+identifier is unique or
just the identifier is unique":  "two extensions with the same identifiers
cannot be installed at the same time" implies the latter

>@jl: So is there an 'official way' to retrieve installed packages at script
provider instantiation time ..?


create com.sun.star.deployment.thePackageManagerFactory.
create the XPackageManager for "user" and "shared":
XPackageManagerFactory.getPackageManager

Get all installed Extensions:

Sequence<XPackage> s = XPackageManager.getDeployedPackages();

@jl: I have prepared an about 100 line patch to replace the "package-directory-
traversal" with  XPackageManager.getDeployedPackages(), this seems to work such
a way, that at PythonScriptProvider instantiation time, I only get to know the
"Old package". 

However, this does not solve the problem, that the final hasByName() calls for
the "new package" return false (see above sb's commend at the end) , because the
PythonScriptProvider(PSP) is not informed about the change (no
PSP.insertByName()/PSP.replaceByName() gets called and even the
XModifyListener.modified() call of the packagemanager itself occurs after the
hasByName() - Calls). How shall this work ? 

I do not yet attach the new implementation to this issue, as it needs more
thouroughly testing.

Currently the modified event is fired when the new extension has been completely
installed. I could think of firing this event also after the existing extension
has been removed, that is before the new extension is installed.

I withdraw my previous statement. The extension in question contains Java code.
That is, it is handled by the ScriptProviderForJava. The extension manager calls
insertByName and removeByName at the MasterScriptProvider, which then iterates
over a list of script providers and calls the same functions on every provider.
When one provider successfully processed the call then the iteration stops. It
happens that the ScriptProviderForJava comes before the python provider in the
list, so the python provider is never called. IMO this is the correct behaviour.  

I also believe (or I misunderstood how this all works) that the python provider
should only know  python code. That is the ScriptDispatch extension is of no
interest to the provider. When the provider is called for the first time, it
could build up a list of python extensions using the extension manager API and
the parcel-descriptor.xml. This list is then updated whenever insertByName,
removeByName is called with a valid URL for a python extension.
The problem is probably for the provider to determine the scripting language,
for which it needs to parse the parcel-descriptor.xml

ok, understood. 

BTW, the parcel-descriptor is not mandatory and in fact, the python provider
does not support one. This is solved by throwing exceptions in
insertByName/removeByName. 

So I will now check, whether a .py file is somewhere within the package if no,
the directory gets ignored. This seems to work (but I will traverse the whole
package tree now ).

I need to do more test to verify, I will provide a patch once this is done.

Bye,

Joerg

Created attachment 60839 [details]
Patch of scripting/source/pyprov/pythonscript.py vs DEV300_m42

Created attachment 60840 [details]
Convienience: pythonscript.py with applied patch (for users who can't patch)

@all: I can apply the patch to the python26 cws, however I doubt it will be 
ready in time to get it into 3.1. 
Can anyone else integrate the patch ?

Bye, 

Joerg

---------
the added patch fixes

* installed packages are now queries via the package manager
* PSP.insertByName/removeByName now throw exceptions in case no .py-File is in 
the given directory
* the incoming file url is now always absolutized

thus, this problem is solved. 

However, the pythonscriptprovider still uses the packagename instead of package-
identifier, this will not be solved within this issue.

I added the issue to cws jl119. Provided the fix will be verified it will be
integrated for 3.1

.

is it correct that the patch is not applied into oo 3.1m5 ?

Correct.

@jsk: Please verify.

Ok, the extension now installs ok using extension manager and unopkg.
The sample document with trigger works ok when the extension is installed into
shared layer.
Verified.

@brinzing,

do you want to verify the issue yourself?
The fix should be in 3.1/m8.

it's fixed - no problems with oo 3.1 rc1 :-)

Phew, great! Closing the beast.