autotools: use pkg-config and Homebrew when looking for libcrypto.

[guyharris]

Grab the stuff from libpcap's configure script that looks for libssl (and libcrypto) and adapt it to look for libcrypto.

his includes some macros to check using pkg-config (and other macros, such as macros to save and restore CFLAGS, LIBS, and LDFLAGS; any resemblance between their names and the cmake_push_check_state() and cmake_pop_check_state() commands is entirely coincidental :-)).

Instead of checking for DES_cbc_encrypt(), which we don't use, to determine whether the libcrypto we found is usable, check for EVP_CIPHER_CTX_block_size(), which we do use. (We also check whether the openssl/evp.h header exists; if it doesn't, we might have found the libcrypto that Apple bundles with macOS, for which they do NOT provide the header in newer versions of Xcode.) See also #1174.

This means that we don't need to check whether we have openssl/evp.h at compile time - now, if we don't, we don't even set HAVE_LIBCRYPTO, so there's no need to check HAVE_OPENSSL_EVP_H.

[guyharris]

BTW, this fixes the build on macOS, as a side-effect of

This means that we don't need to check whether we have openssl/evp.h at compile time - now, if we don't, we don't even set HAVE_LIBCRYPTO, so there's no need to check HAVE_OPENSSL_EVP_H.

as HAVE_LIBCRYPTO is now only set if we have a libcrypto that we can and will use, rather than being set if we find the library, even if its headers aren't present (thanks, Apple!).

But it doesn't, with CMake, manage to find the Homebrew libcrypto; that's the next project.

[guyharris]

BTW, this fixes the build on macOS

...but, given that this doesn't change CMakeLists.txt, I'm not sure why.

[guyharris]

...but, given that this doesn't change CMakeLists.txt, I'm not sure why.

It's because:

1. CMake was, for some reason, finding the Homebrew libcrypto library but wasn't finding openssl/evp.h;
2. before this change, print-esp.c and print-isakmp.c were checking whether openssl/evp.h was found and, if it wasn't, undefining HAVE_CRYPTO, and thus not building in the crypto support;
3. that meant that it looked to the "check" script as if tcpdump had crypto support, so it tested the files with ESP traffic as if they would be decrypted, but tcpdump didn't have crypto support, and failed to decrypt the ESP payload, causing a failure;
4. after this change, print-esp.c and print-isakmp.c no longer checked whether openssl/evp.h was found, so they were built with crypto support (the compiler managed to find the header), and the tests no longer failed.

This change might have caused problems when building on macOS without a third-party OpenSSL installed - in those systems, the library is present but openssl/evp.h is not part of the Xcode SDK - but that's fixed by #1198, in which the check for openssl/evp.h will cause FindCRYPTO.cmake to report that and cause CMakeLists.txt to treat libcrypto as not found if the header isn't present.