SA Bugzilla – Bug 8090
4.0.0 prevents GeoIP2 mmdb files to be updated
Last modified: 2023-05-10 18:11:21 UTC
installed module: Dec 8 17:58:19.370 [1896] dbg: diag: [...] optional module installed: MaxMind::DB::Reader, version 1.000014 Dec 8 17:58:19.370 [1896] dbg: diag: [...] optional module not installed: MaxMind::DB::Reader::XS ('require' failed) I am using 4.0.0-RC4 on Windows, using MaxMind::DB::Reader to utilize ASN en RelayCountry lookups, when running spamd as service it keeps both GeoLite2-City.mmdb and GeoLite2-ASN.mmdb loaded constantly, making it impossible to update those databases with geoipupdate.exe (https://github.com/maxmind/geoipupdate) if i ran geoipupdate -v it returns: Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-ASN/update?db_md5=d5194b28eaa48826567fa38a75468c76 error retrieving updates: error while getting database for GeoLite2-ASN: encountered an issue committing database update: error moving database into place: rename C:\GeoIP\GeoLite2-ASN.mmdb.temporary C:\GeoIP\GeoLite2-ASN.mmdb: Access is denied. This Access is denied error is caused by spamd, if i kill spamd before running geoipupdate it works fine. I believe maxmind mmdb files can be loaded into memory, could that possibly prevent the file in use (Access is denied) error?
FYI, 3.4.6 using GeoIP2::Database::Reader, version 2.006002 doesn't have this issue locking the files in use
3.4 only opens database while it's queried, 4.0 keeps the file open for performance since many plugins may use it. Normal Windows shenanigans locking an opened file: C:\Strawberry>perl -e "open(FOO, '<C:/temp/test.txt') and sleep(30)" ... C:\temp>move test2.txt test.txt Overwrite C:\temp\test.txt? (Yes/No/All): y Access is denied. There is no memory load option from what I see and maxmind code requires a physical existing file, only thing we might do is copying the database to some temporary location and use it from there. That would probably get very messy too with processes coming and going.
Many plugins? You only have a single extra plugin (ASN.pm) besides the existing RelayCountry.pm that could use geoip2 databases, not? I see the has no option to load in memory, that is a pity, the C# DB readers does have this functionality, you there can use MemoryMapped or Memory, see: https://github.com/maxmind/MaxMind-DB-Reader-dotnet
(In reply to RvdH from comment #3) > Many plugins? You only have a single extra plugin (ASN.pm) besides the > existing RelayCountry.pm that could use geoip2 databases, not? There's URILocalBL also. It would make no sense to just try loading the databases at the moment of use, as it will still have the same race condition. There's really no reason to update more than once a week, and one could time a restart then. So I'm inclined to close this, but I'll leave around for a while if someone wants to come up with a simple non-intrusive workaround for Windows.
OK, so this is a Windows issue only? Doesn't the same error occur on Linux? I know mmdb files only has to be updated once a week, but recently i checked and noticed those mmdb had not been updated since i start using 4.0.0 (RC2, RC3 and now RC4) as spamd is constantly running (as service)
(In reply to RvdH from comment #5) > OK, so this is a Windows issue only? Doesn't the same error occur on Linux? Yes it's Windows issue as demonstrated. The locking stuff is completely different there. It's not possible to lock a file on Unix so that it can't be replaced.
With MaxMind no longer supporting the Perl version, there isn't any chance that they will give it the same memory-mapped option that the .NET version has. I agree with Henrik about leaving this open for a while in case anyone comes up with an idea that would work, but if nobody does for a while I'm inclined to close it as a WONTFIX with the workaround that you have to make the stop and restart of spamd part of your weekly script that calls geopipupdate.
I think documentation would help if it's windows only but yeah, a file locking issue on a file that updates weekly seems like something to script up locally Even if we copy to another location, we would have to add code to see if the file has been changed, etc. to use it.
Keeping the DB file open also affects Linux systems using geoip. When a reload using SIGHUP is requested a 2nd path is opened without closing the original path. This eventually leads to spamd stopping when it5 hits open files limit of 1024
(In reply to Paul from comment #9) > Keeping the DB file open also affects Linux systems using geoip. When a > reload using SIGHUP is requested a 2nd path is opened without closing the > original path. This eventually leads to spamd stopping when it5 hits open > files limit of 1024 Please provide proof and exact configuration where this happens? Afaik spamd starts with a completely new process after HUP, it's not even possible to have any old open files.
(In reply to Henrik Krohns from comment #10) > (In reply to Paul from comment #9) > > Keeping the DB file open also affects Linux systems using geoip. When a > > reload using SIGHUP is requested a 2nd path is opened without closing the > > original path. This eventually leads to spamd stopping when it5 hits open > > files limit of 1024 > > Please provide proof and exact configuration where this happens? Afaik spamd > starts with a completely new process after HUP, it's not even possible to > have any old open files. spamd configured to have 5 children. root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root spamd 766010 root 9r REG 8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat root@kanuka:/home/paule# /etc/init.d/spamassassin reload Reloading spamassassin configuration (via systemctl): spamassassin.service. root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root spamd 767146 root 9r REG 8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat spamd 767146 root 11r REG 8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat root@kanuka:/home/paule# /etc/init.d/spamassassin reload Reloading spamassassin configuration (via systemctl): spamassassin.service. root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root spamd 767863 root 9r REG 8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat spamd 767863 root 11r REG 8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat spamd 767863 root 13r REG 8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
(In reply to Paul from comment #11) > spamd configured to have 5 children. Thanks, I've moved your report to Bug 8127 as it has nothing to do with this´or GeoIP2.