[checkmk-commits] [tribe29/checkmk] 648313: Build containers: Cache GNU toolchain artifacts

Branch: refs/heads/master Home: https://github.com/tribe29/checkmk Commit: 648313dd4b414cd5fbe0a005f4e97401773e33a5 https://github.com/tribe29/checkmk/commit/648313dd4b414cd5fbe0a005f4e974017… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M buildscripts/infrastructure/build-nodes/debian-8/Dockerfile M buildscripts/infrastructure/build-nodes/gnu-toolchain/bw-build-gnu-toolchain.sh M buildscripts/scripts/build-build-containers.jenkins Log Message: ----------- Build containers: Cache GNU toolchain artifacts Change-Id: Ibacf0a3cdc12fde90ec3dc59a91a43f8d71f003f Commit: 88ad8836c134e499727992fbd7933f65767e04e7 https://github.com/tribe29/checkmk/commit/88ad8836c134e499727992fbd7933f657… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M buildscripts/infrastructure/build-nodes/centos-6/Dockerfile M buildscripts/infrastructure/build-nodes/centos-7/Dockerfile M buildscripts/infrastructure/build-nodes/centos-8/Dockerfile M buildscripts/infrastructure/build-nodes/debian-10/Dockerfile M buildscripts/infrastructure/build-nodes/debian-8/Dockerfile M buildscripts/infrastructure/build-nodes/debian-9/Dockerfile R buildscripts/infrastructure/build-nodes/gnu-toolchain/bw-build-gnu-toolchain.sh A buildscripts/infrastructure/build-nodes/scripts/build_lib.sh A buildscripts/infrastructure/build-nodes/scripts/install-cmake.sh A buildscripts/infrastructure/build-nodes/scripts/install-gnu-toolchain.sh A buildscripts/infrastructure/build-nodes/scripts/install-openssl.sh A buildscripts/infrastructure/build-nodes/scripts/install-python.sh M buildscripts/infrastructure/build-nodes/sles-12sp1/Dockerfile M buildscripts/infrastructure/build-nodes/sles-12sp2/Dockerfile M buildscripts/infrastructure/build-nodes/sles-12sp3/Dockerfile M buildscripts/infrastructure/build-nodes/sles-12sp4/Dockerfile M buildscripts/infrastructure/build-nodes/sles-15/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-14.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-16.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-18.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-19.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-19.10/Dockerfile M buildscripts/scripts/build-build-containers.jenkins M buildscripts/scripts/lib/versioning.groovy Log Message: ----------- Improve build container creation * Standardize local mirroring of external downloads * Standardize build logic of 3rd party packages we need to build to have a single set of tools on all supported platforms. This includes: - GNU toolchain - cmake - OpenSSL - Python 3 The new standard logic introduces a build cache per distro, branch, package which works like the OMD package build cache. * Several cleanups on the way (e.g. in build-build-containers.jenkins) Change-Id: I5740af55abc9552fcb7a8749ffc6ce1a301874bd Commit: 81f7fa4b9b3f2ad18e076ce1350da83b9e83a26d https://github.com/tribe29/checkmk/commit/81f7fa4b9b3f2ad18e076ce1350da83b9… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M buildscripts/infrastructure/build-nodes/debian-8/Dockerfile A buildscripts/scripts/lib/integration.groovy M buildscripts/scripts/lib/versioning.groovy M buildscripts/scripts/test-gui-crawl.jenkins M buildscripts/scripts/test-integration-single.jenkins M buildscripts/scripts/test-integration.jenkins M tests-py3/gui_crawl/test_gui_crawl.py M tests-py3/scripts/run-dockerized.py M tests/testlib/containers.py Log Message: ----------- Consolidate integration test groovy code * All integration tests and integration test like code has now been stripped down to a single implementation (lib/integration.groovy). * Only the execution startup parameters of the common build function differ. * The DISTRO parmeter is now handed over to run-dockerized script. This is work in progress. Next steps: - Make docker image handling work with non debian distros - Revive management of different docker tags Change-Id: I514b9abcfeb792344ab23d7bb0f77532eccc7f6b Commit: c6c1e3662643cd47709c47f930a42bc4a175310c https://github.com/tribe29/checkmk/commit/c6c1e3662643cd47709c47f930a42bc4a… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M buildscripts/scripts/lib/integration.groovy M buildscripts/scripts/lib/versioning.groovy M buildscripts/scripts/test-gui-crawl.jenkins M buildscripts/scripts/test-integration-single.jenkins M buildscripts/scripts/test-integration.jenkins M tests-py3/scripts/run-dockerized.py M tests-py3/scripts/run-integration-test.py M tests/testlib/containers.py Log Message: ----------- Re-enable docker tag handling Change-Id: I7146a8d9c432ffe9fc529608643c3d832769c783 Commit: 75380002ad76ac50bc0b744bc6fd2693725bb193 https://github.com/tribe29/checkmk/commit/75380002ad76ac50bc0b744bc6fd26937… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M tests/testlib/containers.py Log Message: ----------- Push locally built test image to registry The image is based on our standard distro container and is completely prepared for executing Checkmk like integration tests. * The requesed Checkmk version has been installed * Virtual environments have been prepared Change-Id: I89f29f026fcfa3d81a65f32437e1649758b6e121 Commit: 04ea4897701dac7137425bcb6ff76a7241a9ca5d https://github.com/tribe29/checkmk/commit/04ea4897701dac7137425bcb6ff76a724… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M buildscripts/infrastructure/build-nodes/centos-6/Dockerfile M buildscripts/infrastructure/build-nodes/centos-7/Dockerfile M buildscripts/infrastructure/build-nodes/centos-8/Dockerfile M buildscripts/infrastructure/build-nodes/debian-10/Dockerfile M buildscripts/infrastructure/build-nodes/debian-8/Dockerfile M buildscripts/infrastructure/build-nodes/debian-9/Dockerfile A buildscripts/infrastructure/build-nodes/scripts/install-freetds.sh M buildscripts/infrastructure/build-nodes/scripts/install-gnu-toolchain.sh A buildscripts/infrastructure/build-nodes/scripts/install-nodejs.sh M buildscripts/infrastructure/build-nodes/scripts/install-openssl.sh A buildscripts/infrastructure/build-nodes/scripts/install-pipenv.sh M buildscripts/infrastructure/build-nodes/scripts/install-python.sh A buildscripts/infrastructure/build-nodes/scripts/install-python2.sh M buildscripts/infrastructure/build-nodes/sles-12sp1/Dockerfile M buildscripts/infrastructure/build-nodes/sles-12sp2/Dockerfile M buildscripts/infrastructure/build-nodes/sles-12sp3/Dockerfile M buildscripts/infrastructure/build-nodes/sles-12sp4/Dockerfile M buildscripts/infrastructure/build-nodes/sles-15/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-14.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-16.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-18.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-19.04/Dockerfile M buildscripts/infrastructure/build-nodes/ubuntu-19.10/Dockerfile M tests/testlib/containers.py M tests/testlib/utils.py M tests/testlib/version.py Log Message: ----------- Make integration tests use dev containers instead of OS containers * Add standard nodejs, pipenv and python 2 to dev containers * The integration tests need a our standard Python virtual environments. To be able to build them, we need our standard tool chain. For example older distros don't have Python 3 in the version we need. To reduce the complexity we now build our own standard Python version for This may be more important in the future, when working on incremental builds. * Also restructuring the Dockerfiles a bit to have more single steps that can be cached by docker Todo: The package lists of the different containers have a lot of redundancies. We should work on centralizing them somehow. Change-Id: Ifced4e968aad3daefaaed1050d570f5cf5296dbf Commit: f5f7505dcc0e832113e60efccbf29cc981bca95e https://github.com/tribe29/checkmk/commit/f5f7505dcc0e832113e60efccbf29cc98… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M tests/testlib/containers.py Log Message: ----------- Improve logging Change-Id: Icda18298778928871a8ab7ab09674d10b643daf6 Commit: 901ba6a2d0a0b13efe3b69c690e07ce8d3c7e737 https://github.com/tribe29/checkmk/commit/901ba6a2d0a0b13efe3b69c690e07ce8d… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M tests/testlib/containers.py Log Message: ----------- Check that locally available images are up-to-date Communicate with the registry (if reachable) and check whether or not the local and remote image differ. In case they do, the image from the registry is fetched. Change-Id: Ibdb83ca90a8f6311382469c295d890a219fe3cac Commit: 59b046920a1715d9099e5734400b7f4e1c6371da https://github.com/tribe29/checkmk/commit/59b046920a1715d9099e5734400b7f4e1… Author: Lars Michelsen <lm(a)tribe29.com> Date: 2020-03-07 (Sat, 07 Mar 2020) Changed paths: M tests-py3/docker/test_docker.py M tests-py3/scripts/install-cmk.py M tests-py3/scripts/run-dockerized.py M tests/testlib/containers.py M tests/testlib/site.py M tests/testlib/version.py Log Message: ----------- Extend package installation logic * Move out of testlib to dedicated script * Add support for all supported distros Change-Id: I92b9f73ee65396d6734498da98cbf4a1898abb64 Compare: https://github.com/tribe29/checkmk/compare/b27b6a7ea68f...59b046920a17