{"author":"Viktor Szakats","author_email":"commit@vsz.me","author_time":1735342046,"commit_time":1739667599,"committer":"Viktor Szakats","committer_email":"commit@vsz.me","hash":"e7adf3e83747c2915c671f2e560cde6f3d4a4905","message":"cmake: add pre-fill for Unix, enable in GHA/macos, verify pre-fills\n\nTL;DR: Save 10 minutes of CI time for GHA/macos jobs using pre-fills and\nadd pre-fill verification for Apple and Windows. Also restores Xcode job\nand saves 1.5-10 minutes configuring iOS jobs.\n\nPre-filling feature detection results can bring down the CMake configure\nstep to ~5 seconds on most GHA runners, ~10 seconds in slow envs like\nCygwin/MSYS2.\n\nThe potential savings per job are:\n- 5-40 (average 19) seconds on GHA/macos (33 jobs)\n- ~10 seconds on GHA for iOS GNU Makefile (1 job)\n- 1.5-10 minutes on GHA for iOS Xcode generator (1 job)\n- 10 seconds on GHA/linux with native Ubuntu (12 jobs)\n- 40 seconds for Cygwin/MSYS2 (2 jobs)\n- 5-10 seconds for virtualized BSDs, native CPU (3 jobs)\n- ~60 seconds for virtualized BSDs, emulated CPU (1 job)\n\nOn native Windows pre-filling has been in place for a long time and\nsaving 8 minutes (VS2019-VS2015) to 1.5-2 minutes (VS2022), 3 minutes\n(VS2022 UWP), and 30-60 seconds (MinGW), per CI job.\n\nThe downside is that detection results need to be manually collected and\nfiltered to those that universally apply to all platforms that they are\nenabled on. Another downside is that by using a cache, we're not running\nthe actual detections, and thus won't catch regressions in them. It\nmeans we must make sure that the cache is solid and matches with actual\ndetections results. An upside is that it gives a rough overview of which\nfeatures are available on which platforms. Another upside is pre-filled\nvalues do work for feature detections skipped for cross-builds, e.g.\n`HAVE_WRITABLE_ARGV`.\n\nThis PR adds a pre-fill cache that supports all Unixes (except OmniOS)\nused in CI, and makes it usable with an internal option. It also enables\nit for GHA/macos CI jobs, where the maximum savings are. And also for\nthe two iOS [1] and two Cygwin/MSYS2 jobs. The latters don't have\npre-fill checks and we can drop them if they turn into a hassle.\n\nSaving:\n- 10 minutes of CI time per GHA/macos workflow run. [2]\n- ~80 seconds per GHA/windows workflow run with Cygwin/MSYS2.\n  (offsetting the cost of pre-fill verifications)\n- 1.5-10 minutes per GHA/non-native runs with iOS jobs. [3]\n\nYou can enable pre-fill locally with `-D_CURL_PREFILL=ON`. It's\nexperimental, and if you experience a problem, file a PR or an Issue.\n\nThis PR also adds a pre-fill checker for macOS and MinGW/MSVC Windows\nGHA jobs to catch if the cache diverges from real detections. It also\nadds this logic to AppVeyor, but doesn't enable it due to the perf\npenalty of 2 minutes mininum.\n\nThe pre-fill checker works by configuring out-of-tree with and without\npre-fill, then diffing their `lib/curl_config.h` outputs.\n\nExceptions are 3 detection results exposed indirectly [4], and missing\nto expose 2, of which one is the C89 header `stddef.h`. While we assume\nthe C99 `stdint.h` available outside iOS. We can expose them in the\nfuture, if necessary.\n\nThe pre-fill checks cost in total:\n- ~20 seconds for macOS\n- ~40 seconds for MinGW on GHA\n- ~80 seconds for MSVC on GHA (UWP would be 2x this)\n\nAn extra time saving potential is caching type sizes. They are\nwell-known, and seldom change, esp. in CI. GHA/Windows jobs spend 8-17\nseconds per job on these ~12 feature checks. ~5s on Cygwin/MSYS2. Couple\nof seconds on other platforms. (This PR doesn't make this optimization.)\n\nAnother opportunity is doing the same for autotools, which typically\nspends more time in the configuration step than cmake.\n\n[1] Xcode job restored as a\nfollow-up to be5f20202c1618788b3d8f6d255543638f48bd65 #16302\n\n[2] GHA/macos cmake configure times in seconds:\nJob                                              |  Bef. | After |  Gain\n:----------------------------------------------- | ----: | ----: | ----:\nCM clang GnuTLS !ldap krb5                       |  21.2 |   4.5 |  16.7\nCM clang LibreSSL !ldap heimdal c-ares +examples |  13.3 |   3.9 |   9.4\nCM clang OpenSSL +static libssh +examples        |  20.0 |   4.6 |  15.4\nCM clang OpenSSL IDN clang-tidy~ (w/chkprefill)  |  15.7 |  18.6 |  -2.9\nCM clang OpenSSL gsasl rtmp AppleIDN             |  25.0 |   4.7 |  20.3\nCM clang OpenSSL torture !FTP                    |  15.3 |   4.5 |  10.8\nCM clang OpenSSL torture FTP                     |  25.0 |   5.9 |  19.1\nCM clang SecureTransport debug                   |  18.0 |   3.8 |  14.2\nCM clang macos-13 SecureTransport                |  45.8 |  12.4 |  33.4\nCM clang macos-14 SecureTransport                |  15.8 |   4.6 |  11.2\nCM clang macos-15 SecureTransport                |  26.8 |   6.1 |  20.7\nCM clang mbedTLS openldap brotli zstd            |  15.1 |   6.5 |   8.6\nCM clang wolfSSL !ldap brotli zstd               |  27.0 |   4.4 |  22.6\nCM gcc-12 GnuTLS !ldap krb5                      |  39.1 |   8.7 |  30.4\nCM gcc-12 LibreSSL !ldap heimdal c-ares +examples|  23.8 |   7.2 |  16.6\nCM gcc-12 OpenSSL +static libssh +examples       |  20.7 |   8.5 |  12.2\nCM gcc-12 OpenSSL gsasl rtmp AppleIDN            |  23.1 |  10.1 |  13.0\nCM gcc-12 SecureTransport debug                  |  21.1 |   4.8 |  16.3\nCM gcc-12 mbedTLS openldap brotli zstd           |  21.4 |   5.8 |  15.6\nCM gcc-12 wolfSSL !ldap brotli zstd              |  21.1 |   6.9 |  14.2\nCM gcc-14 macos-13 SecureTransport               |  61.9 |  18.7 |  43.2\nCM gcc-14 macos-14 SecureTransport               |  30.5 |   6.4 |  24.1\nCM gcc-14 macos-15 SecureTransport               |  32.7 |   8.4 |  24.3\nCM llvm@15 GnuTLS !ldap krb5                     |  21.1 |   7.5 |  13.6\nCM llvm@15 LibreSSL !ldap heimdal c-ares +exampl~|  24.6 |   6.8 |  17.8\nCM llvm@15 OpenSSL +static libssh +examples      |  19.0 |   6.4 |  12.6\nCM llvm@15 OpenSSL gsasl rtmp AppleIDN           |  19.0 |   8.2 |  10.8\nCM llvm@15 SecureTransport debug                 |  18.0 |   5.4 |  12.6\nCM llvm@15 macos-13 SecureTransport              |  66.2 |  25.7 |  40.5\nCM llvm@15 macos-14 SecureTransport              |  31.9 |   6.1 |  25.8\nCM llvm@15 mbedTLS openldap brotli zstd          |  19.5 |   8.9 |  10.6\nCM llvm@15 wolfSSL !ldap brotli zstd             |  24.3 |   5.9 |  18.4\nCM llvm@18 macos-15 SecureTransport              |  33.8 |   6.4 |  27.4\nTotal                                            | 856.8 | 257.3 | 599.5\nBefore: https://github.com/curl/curl/actions/runs/13311042735/job/37173478424\nAfter: https://github.com/curl/curl/actions/runs/13313927119/job/37183206426?pr=15841\n\n[3] iOS:\nBefore: https://github.com/curl/curl/actions/runs/13326401704?pr=15841\nAfter: https://github.com/curl/curl/actions/runs/13332177764?pr=15841\n\n[4] detection results exposed indirectly in `curl_config.h`:\n- `HAVE_FILE_OFFSET_BITS` via `_FILE_OFFSET_BITS`\n- `HAVE_GETHOSTBYNAME_R_*_REENTRANT` via `NEED_REENTRANT`\n- `HAVE_SOCKADDR_IN6_SIN6_ADDR` via `USE_IPV6`\n\nCloses #15841\n","parents":["1d7c3ab5aae1e8dc31bc089649eabc47ef66b3a6"],"tree_hash":"2a7d2df5c81a875403aa4af495f8dbca6f29bd40"}