Created attachment 734 [details] shell script demonstrating the bug The title basically describes the issue. Steps to reproduce the issue: Take an ip list for any country (in most of my tests I've been using the ip list for GB fetched from ipdeny, which currently has 8589 ipv4 ip blocks), measure time taking to load it into a new set, measure time adding the same ip list again in a new set under a new name (without removing the 1st one), repeat n times. Each new iteration takes a little longer. On the OpenWRT installation in VM, 1st iteration takes 0.12s. 10th iteration takes 0.47s. 40th iteration takes 1.59s. Getting a response from the command 'nft list tables' also takes progressively longer with each added set, as well as any other nft CLI operation involving the table under test. Starting with 0.04s after 1st iteration and up to 1.49s after the 40th iteration. Monitoring memory use while running this test also reveals that the transient memory use spikes (similar to those reported in Bug 1584) get progressively larger as you go, reaching about 180MB around the 40th iteration (i've been just monitoring memory use via the 'top' utility which samples once a second at most, so I might have missed the peak of the spike). Using the same ip list is not an essential part of the bug, as well as the specific list for GB. I picked GB because it's a mid-sized list, and re-using it because this way it is possible to get comparable results. The issue also extends to ipv6 ip lists. nft CLI operations involving other tables seem unaffected. This is a pretty big problem for applications like geoip, especially so on embedded devices like routers with limited memory. I'm not sure whether this belongs to the kernel component or nft userspace. Filing this under nft userspace component in the meanwhile. Hardware used for testing: Intel i7-4770 Same behavior on Linux Mint (kernel 6.2.0, nftables 1.0.2) and OpenWRT x86-64 installed in a VM (kernel 5.15.137, nftables 1.0.8). A shell script reproducing the issue is attached.
Also to clarify, the issue happens with or without 'auto-merge', although enabling auto-merge somewhat reduces the magnitude.
Update: the author of BanIP (OpenWRT application doing similar tasks with interval sets) shared their method of populating sets and listing the contents of a table, which avoids the bug. Attaching their version of the test script for comparison. Looking at the differences, I'm starting to suspect that some commands cause nftables to re-process all accumulated sets (which causes spikes in memory use and reduced responsiveness), and some don't. 'nft list tables' does, while 'nft list ruleset' doesn't. BanIP method of populating the sets doesn't, while my method does. I still really don't completely understand why but this looks like a bug to me.
Created attachment 735 [details] shell script avoiding the bug
Pointless set element fetching with 'list tables' command fixed in commit 674eb7fa28958 ("cache: Optimize caching for 'list tables' command"). Sadly, fixing the same for 'add element' command is sadly not as trivial, but I'll leave this ticket open as I think it should be doable.
Thank you for looking into this. Additional affected command: 'nft -t list table inet test' (0.01s after 1st iteration, 0.5s after 40th iteration) To my best understanding, it's not supposed to fetch elements when called with the '-t' option.
'nft -t list set' is also likely affected to some degree, as comparing the time it takes to list a small set (261 ip ranges) to a large set (11655 ip ranges), I'm noticing a significant difference - 0.01s vs 0.07s on my VM (the actual difference is likely higher but I'm limited to 0.01s measurement precision). Note that this is also with the '-t' option so it's not supposed to fetch elements (to my understanding).
(In reply to anton.khazan from comment #6) > 'nft -t list set' is also likely affected to some degree, as comparing the > time it takes to list a small set (261 ip ranges) to a large set (11655 ip > ranges), I'm noticing a significant difference - 0.01s vs 0.07s on my VM > (the actual difference is likely higher but I'm limited to 0.01s measurement > precision). Note that this is also with the '-t' option so it's not supposed > to fetch elements (to my understanding). It is. To see for yourself, use --debug=netlink: | nft --debug=netlink -t list set ... If debug output contains lots of "element" lines while non-debug output doesn't, there likely is a bug (with list-like commands at least).