For users having non-English, and especially non-qwerty layouts, using the host
shell can be very awkward. There was no option to change the keymaps as they
haven't been installed in the OS, and the persistence couldn't have been
achieved because of read-only /etc.
With upstream patch merged in #4224, we have an option to put
/etc/vconsole.conf to a writable location and use the same approach as in the
timezone PR. This is needed because even if we only bind-mounted the file from
the overlay directory, the Systemd services which start early will still refer
to the inode on the read-only FS. Also, gzip is required as current version of
kbd in Buildroot (v2.6.4) always compresses the keymaps using gzip. We can get
rid of this after we bump to kbd v2.9.0 [1] or newer. The overall bloat in
local build of the OS is slightly over 1 MiB, so it is acceptable.
With these changes, the `localectl set-keymap` command can be used to use any
available keymap from the installed `kbd` package (refer to `localectl
list-keymaps` for complete lists) and persist it between reboots.
[1] https://github.com/legionus/kbd/releases/tag/v2.9.0Fixes#1775
To make system timezone configurable, we need to have /etc/localtime
writable, and it must be possible to atomically create a symlink from
this place, which means the whole parent folder must be writable. We
don't have /etc writable and can't use the usual bind mount for this.
Latest Systemd v258 has patch that allows setting an environment
variable that sets where the localtime should be written. This can be
persisted in the overlay partition, with a symlink from /etc/localtime
leading there, finally pointing to the actual zoneinfo file. If the
symlink doesn't exist, create it by hassos-overlay script (it's not
really needed as UTC is the default, but Systemd does the same if you
change from non-UTC timezone back to UTC).
Also disable BR2_TARGET_LOCALTIME, so /etc/localtime and /etc/timezone
(the latter is only informative and non-standard) are not written by the
tzdata package build.
When creating OVA image, the CPU is slacking at the end of the build because it
is creating three ZIP archives, each one on a single CPU only. As we're
creating only single-entry archives, we can use pigz to use all cores.
The actual speedup on my machine (16C/32T) reflects the number of cores - it
takes around 2 seconds instead of 1 minute.
Update of OpenSSL in OS 12.2 from 1.1.1 to 3.2 changed the output of `openssl
sha256` command. It seems that some hypervisors don't like this and fail if
it's not plain "SHA256".
Fixes#3654
As we don't have the info utility in HAOS, it's worthless to preserve info
pages. While there are currently some files in /share/info (coming from GRUB2
tools install), /usr/share/info was added pre-emptively.
Because the OTA hooks interact with GRUB environment using grub-editenv, we
have BR2_TARGET_GRUB2_INSTALL_TOOLS enabled. However, that brings massive bloat
of files that are never used in HAOS, as it also installs many other binaries,
GRUB modules and translations.
As it's not possible to configure what gets installed in grub2 package, remove
the undesired files in the post-build function. This brings savings of ~8.5MB
of space in the root partition.
Removal of the e2scrub binary is not needed anymore, as it's not installed and
only BR2_PACKAGE_E2FSPROGS_E2IMAGE is enabled. Moreover, it's been probably
wrong since the very beginning, as the TARGET_DIR prefix was missing, possibly
leading to removal of the binary from the host/builder.
* Use name.sh functions for paths in genimage
Paths for images generated outside of genimage were not used in genimage
definitions. Use them as the single source of truth.
Images generated by genimage itself (e.g. kernel.img) don't need to use those
functions, so remove the unused ones.
* Use EROFS instead of SquashFS for root FS
* Enabled EROFS in common kernel fragment
* RootFS image switched to EROFS with options to get decent compression
* rootfstype removed from kernel command line
* Get size of correct FS image in GH build summary
* Use Genimage for declarative image layout instead of s[fg]disk and dd
* Change partition type to hybrid for M1, M1S and Green
This is what it really is, so just make sure only one "fix" function is
called.
* Change efi BOOT_SYS to gpt
There is no reason to have separate efi and boot sys, since all boards
that use efi also use grub as the loader.
* Change BOOT_SYS to more explanatory PARTITION_TABLE_TYPE
* Add units to DISK_SIZE
* Add forced-primary patch and use it in MBR images
* Avoid disabling SC2155, remove old comments
* Copy Odroid-m1 config for new odroid-m1s board
* config: Adjust names and paths for odroid-m1s
* configs: Use rk3566 blobs for ATF
* set correct fdt in uboot.ush
* Add linux patches with Odroid-m1s devicetree
Synced from Hardkernel unofficial 6.1 tree
ae33b44557/arch/arm64/boot/dts/rockchip/rk3566-odroid-m1s.dts
With additional cleanup and fixes for mainline linux
* Add Odroid M1S to Github actions
* uboot: Patch boot order to set SD Card first
* Create u-boot placeholder partion for odroid-m1s also
* Switch u-boot to full odroid-m1s config
* cherry-pick emmc stability improvements
* Generalise u-boot to use ${devtype} instead of hardcoded mmc
* Remove deprecated snps, reset options from device tree
* re-enable uboot ethernet
Generate the certificate only once and make it available. The preferred
option that doesn't generate warnings would be to use secrets in the
repository config, in that case no certificate is generated or archived.
There is bunch of kernel config options that are not propagated
correctly to the kernel configuration after fragments are merged
and processed by Kconfig. Current Buildroot tools are not good at
discovering these - while we cleaned up most inconsistencies by using
linux-diff-config and output from the merge_config.sh script, there
are still options that were removed or get a different value than
intended because of dependencies, etc.
This commit adds a Python script that is using Kconfiglib to parse
current kernel's Kconfig files and the generated .config and compare
the requested values from individual kernel config fragments. The
script can be used manually by running `make linux-check-dotconfig`
from the buildroot directory (with path to BR2_EXTERNAL directory set)
and it's called also from the CI, where it generates Github Workflow
warning annotations when some of the values are not present or when set
incorrectly.
The kconfiglib.py is checked-in to the repo as well, because the library
is currently abandoned on PyPI and packaged version has a bug that causes
errors parsing Kconfigs in newer Linux versions, fixed in outstanding
pull request ulfalizer/Kconfiglib#119 - so version from this PR is used
here.
If pypi/support#2526 is ever resolved, we could remove it from our repo
and use pip for installing the package as a requirement during build
of the build container.
There is no sanity check when creating OS images, so when some of the
partitions gets too big, part of its data may get overwritten by the
following partition, resulting in corrupted image. Add checks for the
defined partition sizes and bail out if they're too big.
* Generate self-signed certificates for development
To simplify development generate a self-signed certificate on first
build. Also make sure that the self-signed certificate is being added
the RAUC keyring so that manual updates can be performed.
* Add self-signed certificat independent of deployment type
* Add a warning when building with self-signed certificate
* Adjust Home Assistant versioning to prepare for new release strategy
With OS 11 we'll create rc pre-releases which will get directly pushed
to the beta channel. In contrast, release builds will get directly
pushed to the stable channel.
Similar to Home Assistant Core we'll create bump commits for all stable
and beta releases. This makes sure that the source code matches the
built binaries for all releases.
The development build will get a generated version. To avoid issues
with the new rc builds the dev build version will get injected on source
level now.
* Apply suggestions from code review
* Add ODROID-M1 board support
* Add Rockchip kernel config for ODROID-M1
Kernel defconfig for Rockchip is based on Armbian kernel defconfig
from config/kernel/linux-rk3568-odroid-edge.config (git hash
95c829f9e664).
* Add U-Boot/Kernel patches
* Add Rockchip blob support
Add package which provides Rockchip TPL and ATF firmware binaries.
* Use latest U-Boot for ODROID-M1
* Fix Rockchip blob support
* Update defconfig
* Use GPT by default
* Create uboot partition to support non-recovery boot
* Enable eMMC boot in U-Boot SPL
* Drop unnecessary mmc device selection
Distro boot already activates the right mmc device. The extra selection
seems to actually cause problems for eMMC boot.
* Make sure driver for eMMC is built-in
* Use odroid-m1 as Supervisor machine
* Add ODROID-M1 to CI pipeline and issue template
* Bump to Linux 6.1.16
Currently the only board supporting GPT partition table and SPL is the
ASUS Tinker board. Its Rockchip boot loader is stored at LBA 0x40 (64)
which is well past the last LBA of a regular GPT partition table which
is at LBA 33). Therefor a custom GPT main partition table location (via
sgdisk -j, --adjust-main-table=sector) is not necessary.
Technically we could copy anything after LBA 34 from the SPL image, but
since we don't support a board which needs that space for its SPL let's
stick with the well aligned Rockchip start at LBA 64.
Note: To preserve the layout we still add the SPL size to the regular
offset. Technically we could start the boot partition at LBA 16384, but
this would mean a different partition table compared to before and
different offset of subsequent partitions compared to other GPT
platforms.
* Support custom sized SPL/raw boot region
This is required for Rockchip which by default stores the U-Boot FIT
image at the 8MiB offset.
* Ignore shellcheck warning
* Update config for Buildroot 2023.02
* Use Buildroot's version of the rtl8821cu package
Buildroot provides a newer driver for the RTL8821CU based chipsets
provided by https://github.com/morrownr/8821cu-20210118.
* Pass argument when verifying partition table
This also avoids running into a segmentation fault in the current
version of sgdisk.
* Remove obsolte GRUB2/NetworkManager patches
* Bump buildroot
* buildroot 90aa1a6daa...4832525e6c (4596):
> package/runc: add support for CGroup device permission updates
> package/network-manager: fix build with -Dmodem_manager=false
> package/dbus-broker: bump to release 33
> package/iptables: Allow to use iptables with nf_tables backend
> package/brcmfmac_sdio-firmware-rpi: bump to latest version
> package/linux-firmware: Deploy fewer Intel WiFi 22000 series variants
> package/linux-firmware: Add more Intel WiFi 22000 series variants
> package/linux-firmware: Add Broadcom BNX2 firmware
> package/rpi-firmware: bump version to 1.20230106
> Update for 2023.02-rc2
* Use Ubuntu 22.04 for CI checks
* Bump xe-guest-utilities to 7.33.0
* Remove unnecessary shellcheck ignore for xe-guest-utilities
* Address new buildroot check-packages issues
* Drop default NetworkManager configuration
NetworkManager will automatically connect using the global defaults.
Also Supervisor today will create a profiles once the user configures
the network explicitly.
* Create system-connection directory
* Add tempio host package
tempio is a template helper using Go's template engine and sprig
functions.
* Use tempio to generate rauc manifest
* Use tempio to generate rauc system.conf
* Use shell functions for install hooks
* Use post-install hook to initialize GRUB2 bootloader env
Unfortunately the boot name to be updated (RAUC_SLOT_BOOTNAME) is not
available when updating the "boot" slot. Instead, initialize the boot
slot in a kernel post-install slot.
* Add AArch64/ARM64 EFI boot support (for QEMU and some boards)
* Allow GRUB to load cmdline.txt-like
* Enable qcow2/vmdk disk images
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Use LSI Logic SCSI controller in vmdk descriptor as well
For some reason, the vmdk disk format's descriptor contains the
controller type as well. By default, qemu-img sets it to "ide", which
seems not optimal especially for VMware's ESXi. Set adapter type to
commonly supported "lsilogic".
* Move ova image generation to hdd-image.sh
* Bump buildroot
* buildroot 3c5f87185d...5ffdf6ccc5 (1):
> package/e2fsprogs: Create y2038 capable file systems by default
* Use inode size of 256 bytes for overlayfs
By default older versions of mkfs.ext4 create file systems with inode
size of 128 bytes. This does not allow for 64-bit timestamps, which
leads to y2038 compatibility warnings. Use 256 bytes inodes.
* Use skopeo to download container images
Separate container download from image build. This will allow to share
the downloaded images between multiple builds.
We won't store the Supervisor container with the version tag, just with
the latest tag. This allows to simplify the procedure a bit. It seems
there is no downside to this approach.
* Use official Docker in Docker images to build data partition
Instead of building our own Debian based image let's use the official
Docker in Docker image. This avoids building an image for the hassio
data partition and speeds up build as well.
This calls mount commands using sudo to mount the data partition as part
of the buildroot build now. This is not much different from before as
mount has been called as root inside the container, essentially equates
to the same "isolation" level.
* Use image digest as part of the file name
The landing page has no version information in the tag. To avoid
potentially source caching issues, use the digest as part of the file
name.
Other compression methods remove the original image file at compression.
Add the -m (move) command to zip to do the same when compressing with
zip. This saves some space in the builds image/release directory.
* Add squashfs with LZ4 and LZO compression to Barebox
* Add squashfs with LZO compression to U-Boot
* Use squashfs for Linux kernel partition
Generate a squashfs image with LZO compression for the Linux kernel
partition. Adjust the boot scripts to be file system independent commands
to boot from squashfs.
In the past file system extents have been deactivated to get better
performance in U-Boot. However, the performance issue has been addressed
with commit d5aee659f217 ("fs: ext4: cache extent data") in U-Boot. The
performance should be equal to regular files using no extents.
Enabling extents has an advantage however: Files are stored more
efficently, especially relatively large files like a kernel image. The
impact is not all that big (~100KiB), but worthwhile nonetheless.
* Start ha-cli on tty1 instead of a getty
Instead of starting a getty start the ha-cli directly. This will show
the banner right on startup with the important information such as IP
address of the instance or the URL to reach it.
* Use default shell as root shell instead of HA CLI
Instead of using the ha-cli.sh script as login shell use the regular
shell. Amongst other things, this allows to run VS Code devcontainers
remotely via SSH or using scp. The HA CLI is still available using the
`ha` command.
* Fix issue with latest shellcheck version
The latest shellcheck versions use a new error number for non-POSIX
string replacement. Change to ignore this new error number.
* Ignore shellcheck issue about not following sourced files
Newer shellcheck versions also warn when shellcheck does not follow
sourcing of files with known path:
Not following: ./meta was not specified as input (see shellcheck -x).
We check those files separately so ignore this error for the two scripts
affected.
Virtual Disk images are often used on Windows and/or Mac platforms where
xz is not a widely known file ending and also not supported by dafault.
Use zip which is much better known.
Keep using xz for boards since those are not meant to be extracted by
users but directly used in Etcher. Also keep using xz for qcow2, since
qcow2 is mostly used on Linux platforms where xz is available by default
and zip usually needs an extra package.
Use sparse files instead of files written full of zeros. This speeds up
the image generation process significantly. It also makes sure that
virtual disk image formats are minimal in size.
Note: qemu-img automatically generates sparse files when detecting a
block full of zeros. But this is applied on the write side, after image
convertion: The disk image format itself still thinks the whole image
is allocated, leading to larger image than necessary. Also some output
format seem to regonize chunks of zero and create sparse files themself.
With this change, the raw source image file is a sparse file. This is
regocnized by qemu-img at read time (see block/file-posix.c), and leads
to "native" sparse files in the output format.
Some numbers
- qcow2 1.8G -> 862M (same on-disk size)
- vdi 15G -> 888M (same on-disk size)
- vhdx 30G -> 1.1G (918M -> 861M on-disk size)
- vmdk 1.8G -> 866M (about the same on-disk size)
Obviously this also affects the compressed size. But because there are
still lots of zeros, the difference in compressed size is not that big.
* Change HASSOS_ID from hassos to haos
Use a rauc install-check hook to make this update compatible with OS
releases using hassos in the compatible string.
* Use home-assistant as organization in CPE_NAME
Align with Home Assistant core which uses home-assistant with a dash as
organization in CPE_NAME.
The e2scrub utilities only make sense on system which use LVM. They
come with e2fsprogs and can't be disabled currently. Drop them manually
in our post-build script.
* Add development build version part to version number
Add third part in the version number to indicate development builds.
Generate a default version number based on the date, e.g.
"5.6.dev20201124".
* Add GitHub Action workflow for development builds
Add another GitHub workflow for development builds. Make it triggered
only for now. The version number is generated by the workflow and
passed to all builds to make sure all builds have the same development
build version.
* Add documentation
* Add resolved.conf to disable stub resolver and DNSSEC
There are Add-Ons which try to bind port 53 on all interfaces including
127.0.0.53. Disable the stub resolver to make them continue working. We
don't need the resolver currently anyway.
Also disable DNSSEC to make sure the baords can access a NTP time server
even when their time is incorrect (since DNSSEC validation may fail).
This is a known chicken-egg problem with systemd-resolved/systemd-timesyncd
and might be addressed in a future version, with what we can reenable
DNSSEC:
https://github.com/systemd/systemd/issues/5873
* Make sure resolve gets added only once to nsswitch.conf
Only add resolve to nsswitch.conf if not already present.
* Update buildroot-patches for 2020.11-rc1 buildroot
* Update buildroot to 2020.11-rc1
Signed-off-by: Stefan Agner <stefan@agner.ch>
* Don't rely on sfdisk --list-free output
The --list-free (-F) argument does not allow machine readable mode. And
it seems that the output format changes over time (different spacing,
using size postfixes instead of raw blocks).
Use sfdisk json output and calculate free partition space ourselfs. This
works for 2.35 and 2.36 and is more robust since we rely on output which
is meant for scripts to parse.
* Migrate defconfigs for Buildroot 2020.11-rc1
In particular, rename BR2_TARGET_UBOOT_BOOT_SCRIPT(_SOURCE) to
BR2_PACKAGE_HOST_UBOOT_TOOLS_BOOT_SCRIPT(_SOURCE).
* Rebase/remove systemd patches for systemd 246
* Drop apparmor/libapparmor from buildroot-external
* hassos-persists: use /run as directory for lockfiles
The U-Boot tools use /var/lock by default which is not created any more
by systemd by default (it is under tmpfiles legacy.conf, which we no
longer install).
* Disable systemd-update-done.service
The service is not suited for pure read-only systems. In particular the
service needs to be able to write a file in /etc and /var. Remove the
service. Note: This is a static service and cannot be removed using
systemd-preset.
* Disable apparmor.service for now
The service loads all default profiles. Some might actually cause
problems. E.g. the profile for ping seems not to match our setup for
/etc/resolv.conf:
[85503.634653] audit: type=1400 audit(1605286002.684:236): apparmor="DENIED" operation="open" profile="ping" name="/run/resolv.conf" pid=27585 comm="ping" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Drop AVAHI and use systemd-resolved to announce hostname via mDNS
and LLMNR. Also continue to offer the _workstation._tcp.local service
since it is used by the CoreDNS mDNS plug-in.
Currently the Microsoft Reserved Partition GUID is used for this FAT32
formatted partition. This GUID is a rather Microsoft Windows specific
GUID and not commonly used on Linux.
On Linux systems partitions of this type do not get automatically
mounted (see /usr/lib/udev/rules.d/80-udisks2.rules). However, since
this partition contains some files user commonly need to adjust
(config.txt, cmdline.txt) it would be good if the partition does get
mounted.
Use Microsoft Basic Data instead, which is used by default for FAT32
partition (even by Linux partitioning tools such as gparted). Tested
on ASUS Tinker Board and RPi4.
Aligning partitions (and hence file system structures) to higher level
then 512 byte sectors is common practise and highly recommended for flash
backed block devices. It makes sure that the underlaying flash translation
layer (FTL) does not amplify writes due to missalignment of its erase
block size. Use a 1MiB boundary which is what a modern fdisk is doing.
Before this change:
# fdisk /dev/mmcblk0
Welcome to fdisk (util-linux 2.35.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/mmcblk0: 14.57 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x48617373
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 * 16384 65537 49154 24M c W95 FAT32 (LBA)
/dev/mmcblk0p2 65539 1228814 1163276 568M 5 Extended
/dev/mmcblk0p3 1228816 1425425 196610 96M 83 Linux
/dev/mmcblk0p4 1425427 30535679 29110253 13.9G 83 Linux
/dev/mmcblk0p5 65540 114693 49154 24M 83 Linux
/dev/mmcblk0p6 114695 638984 524290 256M 83 Linux
/dev/mmcblk0p7 638986 688139 49154 24M 83 Linux
/dev/mmcblk0p8 688141 1212430 524290 256M 83 Linux
/dev/mmcblk0p9 1212432 1228814 16383 8M 83 Linux
After this change:
# fdisk /dev/mmcblk0
Welcome to fdisk (util-linux 2.35.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/mmcblk0: 14.57 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x48617373
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 * 16384 65535 49152 24M c W95 FAT32 (LBA)
/dev/mmcblk0p2 65536 1239039 1173504 573M 5 Extended
/dev/mmcblk0p3 1241088 1437695 196608 96M 83 Linux
/dev/mmcblk0p4 1439744 30535679 29095936 13.9G 83 Linux
/dev/mmcblk0p5 67584 116735 49152 24M 83 Linux
/dev/mmcblk0p6 118784 643071 524288 256M 83 Linux
/dev/mmcblk0p7 645120 694271 49152 24M 83 Linux
/dev/mmcblk0p8 696320 1220607 524288 256M 83 Linux
/dev/mmcblk0p9 1222656 1239039 16384 8M 83 Linux
See also:
https://unix.stackexchange.com/questions/248939/how-to-achieve-optimal-alignment-for-emmc-partitionhttp://3gfp.com/wp/2014/07/formatting-sd-cards-for-speed-and-lifetime/
