With OS-8653 proving the concept, we can make the SmartOS build deliver sha256sums.txt alongside md5sums.txt.
Dan McDonald commented on 2025-07-31T11:58:53.140-0400:
Given other deliverables ship md5sums.txt as well, we may want to move this issue into TOOLS and generalize it.
Also, given some of the size of these artifacts and (relatively) small quantity of entries, a fast, parallel version of digest(1) that just uses libmd and threads directly may speed along the build enough to notice. SmartOS uses sum -x md5 which is 32-bit and jumps through hoops. A 2GiB file (in tmpfs) takes over 4sec with sum -x md5, and a half second less with digest -a md5.
Dan McDonald commented on 2025-07-31T16:57:46.543-0400:
To generalize it as a TOOLS project means we’d need to update the eng submodule on a lot of repos. The bits-upload.sh script of eng will require some additional work and unless it gets a semi-drastic rewrite, it may not be able to use any fast-parallel digest.
Dan McDonald commented on 2025-08-01T11:07:57.809-0400:
Tested with local build:
smartos-build-2(~/smartos-live)[0]% git branch
* OS-8679
OS-8680
master
smartos-build-2(~/smartos-live)[0]% ptime gmake smartos-publish
# Publish SmartOS platform 20250801T051636Z images
mkdir -p /export/home/danmcd/smartos-live/output/bits/platform
cp output/platform-20250801T051636Z/root.password \
/export/home/danmcd/smartos-live/output/bits/platform/SINGLE_USER_ROOT_PASSWORD.txt
cp output-iso/platform-20250801T051636Z.iso \
/export/home/danmcd/smartos-live/output/bits/platform/smartos-20250801T051636Z.iso
cp output-usb/platform-20250801T051636Z.usb.gz \
/export/home/danmcd/smartos-live/output/bits/platform/smartos-20250801T051636Z-USB.img.gz
cp output-vmware/smartos-20250801T051636Z.vmwarevm.tar.gz \
/export/home/danmcd/smartos-live/output/bits/platform
(cd /export/home/danmcd/smartos-live/output/bits/platform && \
/export/home/danmcd/smartos-live/tools/smartos-index 20250801T051636Z > index.html)
(cd /export/home/danmcd/smartos-live/output/bits/platform && /export/home/danmcd/smartos-live/tools/generate-sums *)
real 5.686201584
user 5.720009185
sys 1.814173517
smartos-build-2(~/smartos-live)[0]% diff -ru output/bits*
Common subdirectories: output/bits/platform and output/bits-old/platform
diff -ru output/bits/platform/index.html output/bits-old/platform/index.html
--- output/bits/platform/index.html Fri Aug 1 10:50:25 2025
+++ output/bits-old/platform/index.html Fri Aug 1 10:32:42 2025
@@ -6,5 +6,4 @@
<a href="smartos-20250801T051636Z.iso">smartos-20250801T051636Z.iso</a><br>
<a href="smartos-20250801T051636Z.vmwarevm.tar.gz">smartos-20250801T051636Z.vmwarevm.tar.gz</a><br>
<a href="md5sums.txt">md5sums.txt</a>
-<a href="sha256sums.txt">sha256sums.txt</a>
</body></html>
diff -ru output/bits/platform/md5sums.txt output/bits-old/platform/md5sums.txt
--- output/bits/platform/md5sums.txt Fri Aug 1 10:50:29 2025
+++ output/bits-old/platform/md5sums.txt Fri Aug 1 10:32:44 2025
@@ -1,5 +1,5 @@
4f180c5d52c56becd87b73e76f376642 SINGLE_USER_ROOT_PASSWORD.txt
-7a998431645d021a74f060ca46614b22 index.html
+293c22b561b8727128e86888dde8e8e5 index.html
b51873d5c10ce80a56e2aab6207f5004 smartos-20250801T051636Z-USB.img.gz
fc2bced25f73b2903ccaed7c93c3a290 smartos-20250801T051636Z.iso
683fca116f6e971ab9cbd78ea1626a76 smartos-20250801T051636Z.vmwarevm.tar.gz
Only in output/bits/platform: sha256sums.txt
smartos-build-2(~/smartos-live)[1]% Kicking off Jenkins next to see what /Joyent_Dev/public/builds/platform/... results look like.
Dan McDonald commented on 2025-08-01T17:07:38.983-0400:
This change does NOT affect /Joyent_Dev/public/builds/platform/... at all. Those md5sums.txt files are generated in eng and that’s a whole TOOLS- ticket.
Testing it with a release is interesting, and our success condition. It only affects /Joyent_Dev/public/SmartOS/<BUILDSTAMP>. I’m imagining an early release build (will be buildstamped `20250806` for sure), inspecting the results, and THEN respinning the build at/after 0000UTC like normal, modulo any actual bad results.
Dan McDonald commented on 2025-08-04T14:25:35.346-0400:
Once this is IA'ed I will NOT push it until RC has been declared. Once that happens I will spin an early release. After the spin I will inspect the deliverables for both sums files that are both correct, and use piadm(8) to test the sums for the ISO files.
Dan McDonald commented on 2025-08-04T14:37:16.990-0400:
Quick verification of generated checksum files. Use pre-generated single-user root password. bits-old is before this fix. bits-smartos is this fix SmartOS only, bits is Triton-and-SmartOS.
smartos-build-2(~/smartos-live)[0]% cat output/bits*/platform/SINGLE_USER_ROOT_PASSWORD.txt
ueh9Goji4jajiefu
ueh9Goji4jajiefu
ueh9Goji4jajiefu
smartos-build-2(~/smartos-live)[0]% grep SINGLE_USER_ROOT_PASSWORD.txt output/bits*/platform/md5sums.txt
output/bits-old/platform/md5sums.txt:4f180c5d52c56becd87b73e76f376642 SINGLE_USER_ROOT_PASSWORD.txt
output/bits-smartos/platform/md5sums.txt:4f180c5d52c56becd87b73e76f376642 SINGLE_USER_ROOT_PASSWORD.txt
output/bits/platform/md5sums.txt:4f180c5d52c56becd87b73e76f376642 SINGLE_USER_ROOT_PASSWORD.txt
smartos-build-2(~/smartos-live)[0]% grep SINGLE_USER_ROOT_PASSWORD.txt output/bits*/platform/sha256sums.txt
output/bits-smartos/platform/sha256sums.txt:1a12f0d78485028ea35339587ae8bfabcb6c294873e6a1528da59281db339fc3 SINGLE_USER_ROOT_PASSWORD.txt
output/bits/platform/sha256sums.txt:1a12f0d78485028ea35339587ae8bfabcb6c294873e6a1528da59281db339fc3 SINGLE_USER_ROOT_PASSWORD.txt
smartos-build-2(~/smartos-live)[0]% Dan McDonald commented on 2025-08-05T16:59:08.252-0400:
This issue will not be marked as Resolved until post-integration/pre-release testing of an early release branch demonstrates working checksums files.
Dan McDonald commented on 2025-08-06T13:31:24.103-0400:
Using last release (old method plus manually-computed sha256sums):
kebe(~)[2]% /tmp/verify_smartos_sums.bash 20250724T001011Z
Lines match:
12f00ff447098e789dcf762df37d2d7a SINGLE_USER_ROOT_PASSWORD.txt
Oh no! Two lines inequal for 20250724T001011Z:
download: ac66e5690bba5b2a81e058af9685b64a build.log
local: e12e603b8073a8262f033d7c0918111d build.log
Lines match:
d0a0d82c81fe981dd0783c5572ca3b50 changelog.txt
Lines match:
9dcbe54ec3c8ee66abb533315df9e39e configure-projects
Lines match:
908bb1aa988a5344b0ca0812d3087379 gitstatus.json
Lines match:
aaa82928ec3c6634534deb3a7b2ece6a index.html
Lines match:
a83043888a5f142c7c6fd12467c29afc platform-release-20250724-20250724T001011Z.tgz
Lines match:
b429822c79a454d1d33913e01d675d8f smartos-20250724T001011Z-USB.img.gz
Lines match:
8965f5b772707b4ee3c9227ac5603580 smartos-20250724T001011Z.iso
Lines match:
52352f6fb9c74f8205d9dcf9e124c7cd smartos-20250724T001011Z.vmwarevm.tar.gz
Lines match:
0cd3c4d942a5bcbbd77e15640c865dc0 tests-release-20250724-20250724T001011Z.tgz
Lines match:
91f14b422a6ab52f16dacffd7ba285d0cbce96ee5b2c7dc1ba0bce86be99d150 SINGLE_USER_ROOT_PASSWORD.txt
Oh no! Two lines inequal for 20250724T001011Z:
download: ca1fbb82457429e4a87f811c00b3d91a257b03418dda9ce618a7249ed25bfb8f build.log
local: 4371b77857630a82a9dcebd95842fda93214433e41f19626b85e3623e5747454 build.log
Lines match:
408f7603274d69dc685373788676297f9623d6da87c6522d31dbff846a0000b6 changelog.txt
Lines match:
dd27851c5f45465dfd99816a75a4c51e54c7c1f5add867f794358cca1d095dd5 configure-projects
Lines match:
cac5d70a6f0b699f365c6da82e0d00da2cc24b2434ad9d206427fa9094b869d0 gitstatus.json
Lines match:
3c35f23ace241108cc6a9831e37440398c002d9ed81f736e10dd4a58c87a8336 index.html
Lines match:
53afe9ed63cff7f3b026b038b584909ef5262b9e203aabf5b9cd768ee449017c platform-release-20250724-20250724T001011Z.tgz
Lines match:
50796911ffe716e93a8342753dc1633c7d5cc1bc1dc1d2fea0d28db113e77474 smartos-20250724T001011Z-USB.img.gz
Lines match:
064f12b226b8769bd3a11ce83eb77d456660ba32f35a45dcace8729692e47d46 smartos-20250724T001011Z.iso
Lines match:
6cc58bd6295f0214774366223760c35249b1f6afaeab0f77a266e604f8e2e5e3 smartos-20250724T001011Z.vmwarevm.tar.gz
Lines match:
07e10229f47b7479c952d8be2fa583d1c802f688e355c73147043484051998ce tests-release-20250724-20250724T001011Z.tgzThere’s only a discrepancy in build.log. Odd that both the manually computed sha256 and the Jenkins computed md5 deviate. I expect this is because our checksumming actions occur while still generating build.log. The pre-release release build will be telling here.
Dan McDonald commented on 2025-08-06T13:41:18.377-0400:
The pre-release release build checks out the same way the old release does:
kebe(~)[0]% /tmp/verify_smartos_sums.bash 20250806T135059Z
Lines match:
363faee25f180efc4cfee19b34788004 SINGLE_USER_ROOT_PASSWORD.txt
Oh no! Two lines inequal for 20250806T135059Z:
download: 5bd0032f41fceadee87b37d08fe1144e build.log
local: 22cf56d0bda5b004da9082688b25e4b9 build.log
Lines match:
bfc2e70c54de883308de3a0d4a23ca4f changelog.txt
Lines match:
4d3b77460ccbe6e81a3b850eead53fcb configure-projects
Lines match:
f6be9224f9ff7bb8c2673e31ee06890a gitstatus.json
Lines match:
b2e7f726971c03097670d2433134bde1 index.html
Lines match:
00c7331be80edc02e88a5fe25f33a1f8 platform-release-20250807-20250806T135059Z.tgz
Lines match:
4d849eda12d134904049848fe3cf207a smartos-20250806T135059Z-USB.img.gz
Lines match:
3486da1f5d74d2b59aaaea3031cdc6b5 smartos-20250806T135059Z.iso
Lines match:
52774a2c25172496f7e23fb646af7f68 smartos-20250806T135059Z.vmwarevm.tar.gz
Lines match:
f056645e91025cf635e1717c317004b3 tests-release-20250807-20250806T135059Z.tgz
Lines match:
e8f39f3a9bc1e3755a2283b3f6ca078cbe2a4a3d90276b1abbcb74744c6de60b SINGLE_USER_ROOT_PASSWORD.txt
Oh no! Two lines inequal for 20250806T135059Z:
download: 6f25f646969e24aaf79fa68ddb3f916f59db1ed69eba1e89164f3229a4398562 build.log
local: f786c40fc251c5955f12f57748206a5216790e7704f13908bf2510c188826756 build.log
Lines match:
d176153220c577a943afcd98f2f206f6892d3068ae4cc5c10074a0c9754d52b8 changelog.txt
Lines match:
03ac73053c664529f4a15223edbc5194d04c312b2448f378f2ae96cfc13f8d46 configure-projects
Lines match:
08e1522bd745c59883039d031f55c7e4229bf03a4950b6982c74c6e8c23dc537 gitstatus.json
Lines match:
ec11dcd5699e4b024db81a29c0e35f09b83f42cbfb8034a86ce51bf2b6a65a56 index.html
Lines match:
a190698f1ac134bcff7e3024d4879d94be98268b0b11bdaf4ba102e039733b91 platform-release-20250807-20250806T135059Z.tgz
Lines match:
655e48fa34a00ae301b28de8931db6dca18b08d9e63fa10725872176db18bb0b smartos-20250806T135059Z-USB.img.gz
Lines match:
fbc32b18b849562ef50f5b0ccf855c17fe31cf3c78db62887f55d7777e7516d1 smartos-20250806T135059Z.iso
Lines match:
0e43c4eb95911f1bd41b87a77ab95accf894a5e9d82b21ac5ae2458d5f9885c3 smartos-20250806T135059Z.vmwarevm.tar.gz
Lines match:
28fa4d1b913935093cc9874a5ce80585798a2fbcf468dca652a23be5b0e31eb5 tests-release-20250807-20250806T135059Z.tgzDan McDonald commented on 2025-08-06T13:41:47.589-0400:
Script used is attached here.
Dan McDonald commented on 2025-08-06T13:45:19.272-0400:
And finally, piadm(8) works as expected:
MD5:
[root@smartos-bios ~]# PIADM_DIGEST_ALGORITHM=md5 piadm -v install latest
Version 1 of /var/piadm/piadm.conf
The following file contents have been configured:
PIADM_CONFIG_VERSION=1
Selecting lone boot pool zones by default.
Downloading latest SmartOS ISO
######################################################################### 100.0%
published_checksum: 3486da1f5d74d2b59aaaea3031cdc6b5
local_checksum: 3486da1f5d74d2b59aaaea3031cdc6b5
Installing PI 20250806T135059Z
Removing old ./os/ directory
Creating new ./os/ directory
Including Platform Image 20250806T135059Z
Including Platform Image 20250710T000436Z
Including Platform Image 20250626T191621Z
Including Platform Image 20250612T001439Z
Including Platform Image 20250529T000820Z
Including Platform Image 20250515T001218Z
Including Platform Image 20250501T000434Z
Including Platform Image 20250417T003816Z
Including Platform Image 20250403T000540Z
Including Platform Image 20250320T000454Z
Including Platform Image 20250306T000316Z
Including Platform Image 20250220T074131Z
Including Platform Image 20250206T001102Z
Including Platform Image 20250124T134639Z
Including Platform Image 20250109T000743Z
[root@smartos-bios ~]#
SHA256:
[root@alder-lake ~]# PIADM_DIGEST_ALGORITHM=sha256 piadm -v install latest
Version 1 of /var/piadm/piadm.conf
The following file contents have been configured:
PIADM_CONFIG_VERSION=1
Selecting lone boot pool zones by default.
Downloading latest SmartOS ISO
######################################################################### 100.0%
published_checksum: fbc32b18b849562ef50f5b0ccf855c17fe31cf3c78db62887f55d7777e7516d1
local_checksum: fbc32b18b849562ef50f5b0ccf855c17fe31cf3c78db62887f55d7777e7516d1
Installing PI 20250806T135059Z
Removing old ./os/ directory
Creating new ./os/ directory
Including Platform Image 20250806T135059Z
Including Platform Image 20250710T000436Z
Including Platform Image 20250626T191621Z
Including Platform Image 20250612T001439Z
Including Platform Image 20250529T000820Z
Including Platform Image 20250515T001218Z
Including Platform Image 20250501T000434Z
Including Platform Image 20250417T003816Z
Including Platform Image 20250403T000540Z
Including Platform Image 20250320T000454Z
Including Platform Image 20250306T000316Z
Including Platform Image 20250220T074131Z
Including Platform Image 20250206T001102Z
Including Platform Image 20250124T134639Z
Including Platform Image 20250109T000743Z
[root@alder-lake ~]#