Bug 8090 - 4.0.0 prevents GeoIP2 mmdb files to be updated
Summary: 4.0.0 prevents GeoIP2 mmdb files to be updated
Status: NEW
Alias: None
Product: Spamassassin
Classification: Unclassified
Component: Plugins (show other bugs)
Version: 4.0.0
Hardware: PC Windows 10
: P2 normal
Target Milestone: Undefined
Assignee: SpamAssassin Developer Mailing List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-12-08 23:03 UTC by RvdH
Modified: 2023-05-10 18:11 UTC (History)
5 users (show)



Attachment Type Modified Status Actions Submitter/CLA Status

Note You need to log in before you can comment on or make changes to this bug.
Description RvdH 2022-12-08 23:03:10 UTC
installed module:

Dec  8 17:58:19.370 [1896] dbg: diag: [...] optional module installed: MaxMind::DB::Reader, version 1.000014
Dec  8 17:58:19.370 [1896] dbg: diag: [...] optional module not installed: MaxMind::DB::Reader::XS ('require' failed)

I am using 4.0.0-RC4 on Windows, using MaxMind::DB::Reader to utilize ASN en RelayCountry lookups, when running spamd as service it keeps both GeoLite2-City.mmdb and GeoLite2-ASN.mmdb loaded constantly, making it impossible to update those databases with geoipupdate.exe (https://github.com/maxmind/geoipupdate)   

if i ran geoipupdate -v it returns:
 
Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-ASN/update?db_md5=d5194b28eaa48826567fa38a75468c76
error retrieving updates: error while getting database for GeoLite2-ASN: encountered an issue committing database update: error moving database into place: rename C:\GeoIP\GeoLite2-ASN.mmdb.temporary C:\GeoIP\GeoLite2-ASN.mmdb: Access is denied.

This Access is denied error is caused by spamd, if i kill spamd before running geoipupdate it works fine. 

I believe maxmind mmdb files can be loaded into memory, could that possibly prevent the file in use (Access is denied) error?
Comment 1 RvdH 2022-12-08 23:18:14 UTC
FYI, 3.4.6 using GeoIP2::Database::Reader, version 2.006002 doesn't have this issue locking the files in use
Comment 2 Henrik Krohns 2022-12-09 05:53:06 UTC
3.4 only opens database while it's queried, 4.0 keeps the file open for performance since many plugins may use it.

Normal Windows shenanigans locking an opened file:

C:\Strawberry>perl -e "open(FOO, '<C:/temp/test.txt') and sleep(30)"
...
C:\temp>move test2.txt test.txt
Overwrite C:\temp\test.txt? (Yes/No/All): y
Access is denied.

There is no memory load option from what I see and maxmind code requires a physical existing file, only thing we might do is copying the database to some temporary location and use it from there. That would probably get very messy too with processes coming and going.
Comment 3 RvdH 2022-12-09 07:53:23 UTC
Many plugins? You only have a single extra plugin (ASN.pm) besides the existing RelayCountry.pm that could use geoip2 databases, not?

I see the has no option to load in memory, that is a pity, the C# DB readers does have this functionality,  you there can use MemoryMapped or Memory, see: https://github.com/maxmind/MaxMind-DB-Reader-dotnet
Comment 4 Henrik Krohns 2022-12-09 08:09:30 UTC
(In reply to RvdH from comment #3)
> Many plugins? You only have a single extra plugin (ASN.pm) besides the
> existing RelayCountry.pm that could use geoip2 databases, not?

There's URILocalBL also.

It would make no sense to just try loading the databases at the moment of use, as it will still have the same race condition.

There's really no reason to update more than once a week, and one could time a restart then. So I'm inclined to close this, but I'll leave around for a while if someone wants to come up with a simple non-intrusive workaround for Windows.
Comment 5 RvdH 2022-12-09 08:14:48 UTC
OK, so this is a Windows issue only? Doesn't the same error occur on Linux?

I know mmdb files only has to be updated once a week, but recently i checked and noticed those mmdb had not been updated since i start using 4.0.0 (RC2, RC3 and now RC4) as spamd is constantly running (as service)
Comment 6 Henrik Krohns 2022-12-09 08:16:42 UTC
(In reply to RvdH from comment #5)
> OK, so this is a Windows issue only? Doesn't the same error occur on Linux?

Yes it's Windows issue as demonstrated. The locking stuff is completely different there. It's not possible to lock a file on Unix so that it can't be replaced.
Comment 7 Sidney Markowitz 2022-12-09 13:15:19 UTC
With MaxMind no longer supporting the Perl version, there isn't any chance that they will give it the same memory-mapped option that the .NET version has.

I agree with Henrik about leaving this open for a while in case anyone comes up with an idea that would work, but if nobody does for a while I'm inclined to close it as a WONTFIX with the workaround that you have to make the stop and restart of spamd part of your weekly script that calls geopipupdate.
Comment 8 Kevin A. McGrail 2022-12-09 14:56:55 UTC
I think documentation would help if it's windows only but yeah, a file locking issue on a file that updates weekly seems like something to script up locally  Even if we copy to another location, we would have to add code to see if the file has been changed, etc. to use it.
Comment 9 Paul 2023-05-10 14:18:23 UTC
Keeping the DB file open also affects Linux systems using geoip. When a reload using SIGHUP is requested a 2nd path is opened without closing the original path. This eventually leads to spamd stopping when it5 hits open files limit of 1024
Comment 10 Henrik Krohns 2023-05-10 16:45:12 UTC
(In reply to Paul from comment #9)
> Keeping the DB file open also affects Linux systems using geoip. When a
> reload using SIGHUP is requested a 2nd path is opened without closing the
> original path. This eventually leads to spamd stopping when it5 hits open
> files limit of 1024

Please provide proof and exact configuration where this happens? Afaik spamd starts with a completely new process after HUP, it's not even possible to have any old open files.
Comment 11 Paul 2023-05-10 17:00:05 UTC
(In reply to Henrik Krohns from comment #10)
> (In reply to Paul from comment #9)
> > Keeping the DB file open also affects Linux systems using geoip. When a
> > reload using SIGHUP is requested a 2nd path is opened without closing the
> > original path. This eventually leads to spamd stopping when it5 hits open
> > files limit of 1024
> 
> Please provide proof and exact configuration where this happens? Afaik spamd
> starts with a completely new process after HUP, it's not even possible to
> have any old open files.

spamd configured to have 5 children. 

root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root
spamd      766010                            root    9r      REG                
8,5   2579835     260907 /usr/share/GeoIP/GeoIP.dat
root@kanuka:/home/paule# /etc/init.d/spamassassin reload 
Reloading spamassassin configuration (via systemctl): spamassassin.service.
root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root
spamd      767146                            root    9r      REG                
8,5   2579835     260907 /usr/share/GeoIP/GeoIP.dat
spamd      767146                            root   11r      REG                
8,5   2579835     260907 /usr/share/GeoIP/GeoIP.dat
root@kanuka:/home/paule# /etc/init.d/spamassassin reload          
Reloading spamassassin configuration (via systemctl): spamassassin.service.
root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root
spamd      767863                            root    9r      REG                
8,5   2579835     260907 /usr/share/GeoIP/GeoIP.dat
spamd      767863                            root   11r      REG                
8,5   2579835     260907 /usr/share/GeoIP/GeoIP.dat
spamd      767863                            root   13r      REG                
8,5   2579835     260907 /usr/share/GeoIP/GeoIP.dat
Comment 12 Henrik Krohns 2023-05-10 18:11:21 UTC
(In reply to Paul from comment #11)
> spamd configured to have 5 children. 

Thanks, I've moved your report to Bug 8127 as it has nothing to do with this´or GeoIP2.