GNU bug report logs - #61765
custom toolchain blog post

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 61765 in the body.
You can then email your comments to 61765 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
To: guix-patches <at> gnu.org
Subject: custom toolchain blog post
Date: Fri, 24 Feb 2023 13:51:56 -0500

Here is a rough draft of my toolchain post.
I hope it can make an interesting contribution to the Guix literature.

- Mitchell


From 4f6c43091ffd67cdbc5f041e496f61bc8a06070e Mon Sep 17 00:00:00 2001
From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
Date: Fri, 24 Feb 2023 13:02:05 -0500
Subject: [PATCH] website: Add custom toolchain blog post

* website/posts/custom-toolchains-with-guix.md: New file.
---
 website/posts/custom-toolchains-with-guix.md | 557 +++++++++++++++++++
 1 file changed, 557 insertions(+)
 create mode 100644 website/posts/custom-toolchains-with-guix.md

diff --git a/website/posts/custom-toolchains-with-guix.md b/website/posts/custom-toolchains-with-guix.md
new file mode 100644
index 0000000..f73f1ab
--- /dev/null
+++ b/website/posts/custom-toolchains-with-guix.md
@@ -0,0 +1,557 @@
+# Table of Contents
+
+1.  [Overview](#org2633a51)
+2.  [Anatomy of a toolchain](#orgc440e9e)
+3.  [Bootstrapping a Toolchain](#orgd42b6c3)
+4.  [Defining the Packages](#org55042c5)
+	1.  [Binutils](#org67da1ec)
+	2.  [GCC sans libc](#org82d6f83)
+	3.  [Newlib(-nano)](#orgf6bafbc)
+	4.  [Complete toolchain](#org052f2a2)
+5.  [Integrating with Zephyr Build System](#orgc3f87f4)
+	1.  [Testing](#org9f3c314)
+
+All code is available at [guix-zephyr](https://github.com/paperclip4465/guix-zephyr) channel.
+
+
+<a id="org2633a51"></a>
+
+# Overview
+
+In order to deploy embedded software using Guix we first need to teach Guix
+how to build it. Since Guix bootstraps everything this means we must teach Guix
+how to build our toolchain.
+
+The [Zephyr Project](https://zephyrproject.org) uses its own fork of GCC with custom configs for
+the architectures supported by the project.
+
+
+<a id="orgc440e9e"></a>
+
+# Anatomy of a toolchain
+
+Toolchains are responsible for taking high level descriptions of programs
+and lowering them down to a series of equivalent machine instructions.
+This process involves more than just a compiler. The compiler uses the `binutils`
+to manipulate it's internal representation down to a given architecture.
+It also needs the use of the C standard library as well as a few other libraries
+needed for some compiler optimizations.
+
+The C library provides the interface to the underlying kernel. System calls like `write`
+and `read` are provided by `Glibc` on most Linux distributions.
+
+In embedded systems smaller implementations like `newlib` and `newlib-nano` are used.
+
+
+<a id="orgd42b6c3"></a>
+
+# Bootstrapping a Toolchain
+
+In order to compile GCC we need a C library that's been compiled for
+our target architecture. How can we cross compile our C library if we
+need our C library to build a cross compiler? The solution is to build
+a simpler compiler that doesn't require the C library to function.
+It will not be capable of as many optimizations and it will be very slow,
+however it will be able to build the C libraries as well as the complete version
+of GCC.
+
+In order to build the simpler compiler we need to compile the `binutils` to
+work with our target architecture.
+The `binutils` can be bootstrapped with our host GCC and have no target dependencies.
+
+[For more information read this.](https://crosstool-ng.github.io/docs/toolchain-construction/)
+
+Doesn't sound so bad right? It isn't&#x2026; in theory.
+However internet forums since time immemorial have been
+littered with the laments of those who came before.
+From incorrect versions of `ISL` to the wrong C library being linked
+or the host linker being used, etc.
+The one commonality between all of these issues is the environment.
+Building a cross toolchain is difficult because isolating build
+environments is difficult.
+
+In fact as of `v0.14.2` the zephyr SDK repository took down the build
+instructions and posted a sign that read
+"Building this is too complicated, don't worry about it."<sup><a id="fnr.1" class="footref" href="#fn.1" role="doc-backlink">1</a></sup>
+
+We will neatly side step all of these problems and not
+risk destroying or polluting our host system with garbage
+by using Guix to manage our environments for us.
+
+Our toolchain only requires the first pass compiler because
+newlib(-nano) is statically linked and introduced to the toolchain
+by normal package composition.
+
+
+<a id="org55042c5"></a>
+
+# Defining the Packages
+
+All of the base packages are defined in `zephyr/packages/zephyr.scm`.
+Zephyr modules are defined in `zephyr/packages/zephyr-xyz.scm`, following
+the pattern of other module systems implemented by Guix.
+
+
+<a id="org67da1ec"></a>
+
+## Binutils
+
+First thing we need to build is the `arm-zephyr-eabi` binutils.
+This is very easy in Guix.
+
+	(define-module (zephyr packages zephyr)
+	  #:use-module (guix packages))
+
+	(define-public arm-zephyr-eabi-binutils
+	  (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
+		(package
+		  (inherit xbinutils)
+		  (name "arm-zephyr-eabi-binutils")
+		  (version "2.38")
+		  (source
+		   (origin (method git-fetch)
+				   (uri (git-reference
+						 (url "https://github.com/zephyrproject-rtos/binutils-gdb")
+						 (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
+				   (file-name (git-file-name name version))
+				   (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
+		  (arguments
+		   `(#:tests? #f
+			 ,@(substitute-keyword-arguments (package-arguments xbinutils)
+				 ((#:configure-flags flags)
+				  `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
+		  (native-inputs
+		   (append
+			(list texinfo
+				  bison
+				  flex
+				  gmp
+				  dejagnu)
+			(package-native-inputs xbinutils)))
+		  (home-page "https://zephyrproject.org")
+		  (synopsis "binutils for zephyr RTOS"))))
+
+The function `cross-binutils` returns a package which has been
+configured for the given gnu triplet.  We simply inherit that package
+and replace the source.
+The zephyr build system expects the binutils to be prefixed with
+`arm-zephyr-eabi-` which is accomplished by adding another flag to the
+`#:configure-flags` argument.
+
+We can test our package definition using the `-L` flag with `guix build`
+to add our packages.
+
+	guix build -L guix-zephyr zephyr-binutils
+
+	/gnu/store/...-zephyr-binutils-2.38
+
+This directory contains the results of `make install`.
+
+
+<a id="org82d6f83"></a>
+
+## GCC sans libc
+
+This one is a bit more involved. Don't be afraid!
+This version of GCC wants ISL version 0.15. It's easy enough
+to make that happen. Inherit the current version of ISL and swap
+out the source and update the version. For most packages the build process doesn't
+change that much between versions.
+
+	(define-public isl-0.15
+	  (package
+		(inherit isl)
+		(version "0.15")
+		(source (origin
+				  (method url-fetch)
+				  (uri (list (string-append "mirror://sourceforge/libisl/isl-"
+											version ".tar.gz")))
+				  (sha256
+				   (base32
+					"11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))
+
+Like the binutils, there is a function for creating cross-gcc packages.
+This one accepts keywords specifying which binutils and libc to use.
+If libc isn't given (like here), gcc is configured with many options disabled
+to facilitate being built without libc. Therefore we need to add the extra options
+we want <sup><a id="fnr.2" class="footref" href="#fn.2" role="doc-backlink">2</a></sup>
+
+	(define-public gcc-arm-zephyr-eabi-12
+	  (let ((xgcc (cross-gcc "arm-zephyr-eabi"
+							 #:xbinutils zephyr-binutils)))
+		(package
+		  (inherit xgcc)
+		  (version "12.1.0")
+		  (source (origin (method git-fetch)
+						  (uri (git-reference
+								(url "https://github.com/zephyrproject-rtos/gcc")
+								(commit "0218469df050c33479a1d5be3e5239ac0eb351bf")))
+						  (file-name (git-file-name (package-name xgcc) version))
+						  (sha256
+						   (base32 "1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
+						  (patches
+						   (search-patches "gcc-12-cross-environment-variables.patch"
+										   "gcc-cross-gxx-include-dir.patch"))))
+		  (native-inputs
+		   (modify-inputs (package-native-inputs xgcc)
+			 ;; Get rid of stock ISL
+			 (delete "isl")
+			 ;; Add additional dependencies that xgcc doesn't have
+			 ;; including our special ISL
+			 (prepend flex
+					  perl
+					  python-3
+					  gmp
+					  isl-0.15
+					  texinfo
+					  python
+					  mpc
+					  mpfr
+					  zlib)))
+		  (arguments
+		   (substitute-keyword-arguments (package-arguments xgcc)
+			 ((#:phases phases)
+			  `(modify-phases ,phases
+				 (add-after 'unpack 'fix-genmultilib
+				   (lambda _
+					 (substitute* "gcc/genmultilib"
+					   (("#!/bin/sh") (string-append "#!" (which "sh"))))
+					 #t))
+
+				 (add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
+				   (lambda* (#:key inputs #:allow-other-keys)
+					 (let ((gcc (assoc-ref inputs  "gcc")))
+					   ;; Remove the default compiler from CPLUS_INCLUDE_PATH to
+					   ;; prevent header conflict with the GCC from native-inputs.
+					   (setenv "CPLUS_INCLUDE_PATH"
+							   (string-join
+								(delete (string-append gcc "/include/c++")
+										(string-split (getenv "CPLUS_INCLUDE_PATH")
+													  #\:))
+								":"))
+					   (format #t
+							   "environment variable `CPLUS_INCLUDE_PATH' changed to ~a~%"
+							   (getenv "CPLUS_INCLUDE_PATH"))
+					   #t)))))
+
+			 ((#:configure-flags flags)
+			  ;; The configure flags are largely identical to the flags used by the
+			  ;; "GCC ARM embedded" project.
+			  `(append (list "--enable-multilib"
+							 "--with-newlib"
+							 "--with-multilib-list=rmprofile"
+							 "--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
+							 "--enable-plugins"
+							 "--disable-decimal-float"
+							 "--disable-libffi"
+							 "--disable-libgomp"
+							 "--disable-libmudflap"
+							 "--disable-libquadmath"
+							 "--disable-libssp"
+							 "--disable-libstdcxx-pch"
+							 "--disable-nls"
+							 "--disable-shared"
+							 "--disable-threads"
+							 "--disable-tls"
+							 "--with-gnu-ld"
+							 "--with-gnu-as"
+							 "--enable-initfini-array")
+					   (delete "--disable-multilib" ,flags)))))
+		  (native-search-paths
+		   (list (search-path-specification
+				  (variable "CROSS_C_INCLUDE_PATH")
+				  (files '("arm-zephyr-eabi/include")))
+				 (search-path-specification
+				  (variable "CROSS_CPLUS_INCLUDE_PATH")
+				  (files '("arm-zephyr-eabi/include"
+						   "arm-zephyr-eabi/c++"
+						   "arm-zephyr-eabi/c++/arm-zephyr-eabi")))
+				 (search-path-specification
+				  (variable "CROSS_LIBRARY_PATH")
+				  (files '("arm-zephyr-eabi/lib")))))
+		  (home-page "https://zephyrproject.org")
+		  (synopsis "GCC for zephyr RTOS"))))
+
+This GCC can be built like so.
+
+	guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi
+
+	/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
+	/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0
+
+Great! We now have our stage-1 compiler.
+
+
+<a id="orgf6bafbc"></a>
+
+## Newlib(-nano)
+
+The newlib package package is quite straight forward (relatively).
+It is mostly adding in the relevent configuration flags and patching
+the files the `patch-shebangs` phase missed.
+
+	  (define-public zephyr-newlib
+	  (package
+		(name "zephyr-newlib")
+		(version "3.3")
+		(source (origin
+				  (method git-fetch)
+				  (uri (git-reference
+						(url "https://github.com/zephyrproject-rtos/newlib-cygwin")
+						(commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
+				  (sha256
+				   (base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
+		(build-system gnu-build-system)
+		(arguments
+		 `(#:out-of-source? #t
+		   #:configure-flags '("--target=arm-zephyr-eabi"
+							   "--enable-newlib-io-long-long"
+							   "--enable-newlib-io-float"
+							   "--enable-newlib-io-c99-formats"
+							   "--enable-newlib-retargetable-locking"
+							   "--enable-newlib-lite-exit"
+							   "--enable-newlib-multithread"
+							   "--enable-newlib-register-fini"
+							   "--enable-newlib-extra-sections"
+							   "--disable-newlib-wide-orient"
+							   "--disable-newlib-fseek-optimization"
+							   "--disable-newlib-supplied-syscalls"
+							   "--disable-newlib-target-optspace"
+							   "--disable-nls")
+		   #:phases
+		   (modify-phases %standard-phases
+			 (add-after 'unpack 'fix-references-to-/bin/sh
+			   (lambda _
+				 (substitute* '("libgloss/arm/cpu-init/Makefile.in"
+								"libgloss/arm/Makefile.in"
+								"libgloss/libnosys/Makefile.in"
+								"libgloss/Makefile.in")
+				   (("/bin/sh") (which "sh")))
+				 #t)))))
+		(native-inputs
+		 `(("xbinutils" ,zephyr-binutils)
+		   ("xgcc" ,gcc-arm-zephyr-eabi-12)
+		   ("texinfo" ,texinfo)))
+		(home-page "https://www.sourceware.org/newlib/")
+		(synopsis "C library for use on embedded systems")
+		(description "Newlib is a C library intended for use on embedded
+	systems.  It is a conglomeration of several library parts that are easily
+	usable on embedded products.")
+		(license (license:non-copyleft
+				  "https://www.sourceware.org/newlib/COPYING.NEWLIB"))))
+
+And the build.
+
+	guix build -L guix-zephyr zephyr-newlib
+
+	/gnu/store/...-zephyr-newlib-3.3
+
+
+<a id="org052f2a2"></a>
+
+## Complete toolchain<sup><a id="fnr.3" class="footref" href="#fn.3" role="doc-backlink">3</a></sup>
+
+Now that we've got the individual tools it's time to create our complete toolchain.
+For this we need to do some package transformations.
+Because these transformations are going to have to be done for every combination of
+gcc/newlib it is best to create a function which we can reuse for every version
+of the SDK.
+
+	(define (arm-zephyr-eabi-toolchain xgcc newlib version)
+	  "Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C
+	library variant NEWLIB."
+	  (let ((newlib-with-xgcc (package (inherit newlib)
+									   (native-inputs
+										(alist-replace "xgcc" (list xgcc)
+													   (package-native-inputs newlib))))))
+		(package
+		  (name (string-append "arm-zephyr-eabi"
+							   (if (string=? (package-name newlib-with-xgcc)
+											 "newlib-nano")
+								   "-nano" "")
+							   "-toolchain"))
+		  (version version)
+		  (source #f)
+		  (build-system trivial-build-system)
+		  (arguments
+		   '(#:modules ((guix build union)
+						(guix build utils))
+			 #:builder
+			 (begin
+			   (use-modules (ice-9 match)
+							(guix build union)
+							(guix build utils))
+			   (let ((out (assoc-ref %outputs "out")))
+				 (mkdir-p out)
+				 (match %build-inputs
+				   (((names . directories) ...)
+					(union-build (string-append out "/arm-zephyr-eabi")
+								 directories)
+					#t))))))
+		  (inputs
+		   `(("binutils" ,zephyr-binutils)
+			 ("gcc" ,xgcc)
+			 ("newlib" ,newlib-with-xgcc)))
+		  (synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
+		  (description "This package provides a complete GCC tool chain for ARM
+	bare metal development with zephyr rtos.  This includes the GCC arm-zephyr-eabi cross compiler
+	and newlib (or newlib-nano) as the C library.  The supported programming
+	language is C.")
+		  (home-page (package-home-page xgcc))
+		  (license (package-license xgcc)))))
+
+This function creates a special package which consists of the toolchain in a special directory hierarchy, i.e `arm-zephyr-eabi/`.
+Our complete toolchain definition looks like this.
+
+	(define-public arm-zephyr-eabi-toolchain-0.15.0
+	  (arm-zephyr-eabi-toolchain
+	   gcc-arm-zephyr-eabi-12
+	   zephyr-newlib
+	   "0.15.0"))
+
+To build:
+
+	guix build -L guix-zephyr arm-zephyr-eabi-toolchain
+
+	/gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0
+
+
+<a id="orgc3f87f4"></a>
+
+# Integrating with Zephyr Build System
+
+Zephyr uses CMake as it's build system. It contains numerous CMake
+files in both the so-called `ZEPHYR_BASE`, the zephyr source code
+repository, as well as a handful in the SDK
+which help select the correct toolchain for a given board.
+
+There are standard locations the build system will look for the
+SDK. We are not using any of them.  Our SDK lives in the store,
+immutable forever. According to [this](https://docs.zephyrproject.org/latest/develop/west/without-west.html) the variable
+`ZEPHYR_SDK_INSTALL_DIR` needs to point to our custom spot.
+
+We also need to grab the cmake files from the [repository](https://github.com/zephyrproject-rtos/sdk-ng) and create a
+file `sdk_version` which contains the version string `ZEPHYR_BASE`
+uses to find a compatible SDK.
+
+Along with the SDK proper we need to include a number of python
+packages required by the build system<sup><a id="fnr.4" class="footref" href="#fn.4" role="doc-backlink">4</a></sup>.
+
+	(define-public zephyr-sdk
+	  (package
+		(name "zephyr-sdk")
+		(version "0.15.0")
+		(home-page "https://zephyrproject.org")
+		(source (origin (method git-fetch)
+						(uri (git-reference
+							  (url "https://github.com/zephyrproject-rtos/sdk-ng")
+							  (commit "v0.15.0")))
+						(file-name (git-file-name name version))
+						(sha256 (base32 "04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
+		(build-system trivial-build-system)
+		(arguments
+		 `(#:modules ((guix build union)
+					  (guix build utils))
+		   #:builder
+		   (begin
+			 (use-modules (guix build union)
+						  (ice-9 match)
+						  (guix build utils))
+			 (let* ((out (assoc-ref %outputs "out"))
+					(cmake-scripts (string-append (assoc-ref %build-inputs "source")
+												  "/cmake"))
+					(sdk-out (string-append out "/zephyr-sdk-0.15.0")))
+			   (mkdir-p out)
+
+			   (match (assoc-remove! %build-inputs "source")
+				 (((names . directories) ...)
+				  (union-build sdk-out directories)))
+
+			   (copy-recursively cmake-scripts
+								 (string-append sdk-out "/cmake"))
+
+			   (with-directory-excursion sdk-out
+				 (call-with-output-file "sdk_version"
+				   (lambda (p)
+					 (format p "0.15.0")))
+				 #t)))))
+		(propagated-inputs
+		 (list
+		  arm-zephyr-eabi-toolchain-0.15.0
+		  zephyr-binutils
+		  python-3
+		  python-pyelftools
+		  python-pykwalify
+		  python-pyyaml
+		  python-packaging
+		  dtc))
+		(native-search-paths
+		 (list (search-path-specification
+				(variable "ZEPHYR_SDK_INSTALL_DIR")
+				(files '("")))))
+		(synopsis "SDK for zephyrRTOS")
+		(description "zephyr-sdk contains bundles a complete gcc toolchain as well
+		as host tools like dtc, openocd, qemu, and required python packages.")
+		(license license:apsl2)))
+
+
+<a id="org9f3c314"></a>
+
+## Testing
+
+In order to test we will need an environment with the SDK installed.
+We can take advantage of `guix shell` to avoid installing test
+packages into our home environment. This way if it causes problems we
+can just exit the shell and try again.
+
+	guix shell -L guix-zephyr zephyr-sdk cmake ninja git
+
+`ZEPHYR_BASE` can be cloned into a temporary workspace to test our
+toolchain functionality.<sup><a id="fnr.5" class="footref" href="#fn.5" role="doc-backlink">5</a></sup>
+
+	mkdir /tmp/zephyr-project
+	cd /tmp/zephyr-project
+	git clone https://github.com/zephyrproject-rtos/zephyr
+	export ZEPHYR_BASE=/tmp/zephyr-project/zephyr
+
+In order to build for the test board (k64f in this case) we need to
+get a hold of the vendor Hardware Abstraction Layers and CMSIS.<sup><a id="fnr.6" class="footref" href="#fn.6" role="doc-backlink">6</a></sup>
+
+	git clone https://github.com/zephyrproject-rtos/hal_nxp &&
+	git clone https://github.com/zephyrproject-rtos/cmsis
+
+To inform the build system about this module we pass it in with
+`-DZEPHYR_MODULES=` which is a semicolon separated list of paths
+containing a module.yml file.
+
+To build the hello world sample we use the following incantation.
+
+	cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
+		  -GNinja \
+		  -DBOARD=frdm_k64f \
+		  -DBUILD_VERSION=3.1.0 \
+		  -DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
+		&& ninja -Cbuild
+
+If everything is set up correctly we will end up with a `./build`
+directory with all our build artifacts. The SDK is installed correctly!
+
+
+# Footnotes
+
+<sup><a id="fn.1" href="#fnr.1">1</a></sup> I'm paraphrasing, but [not by much](https://github.com/zephyrproject-rtos/sdk-ng/tree/v0.14.2#build-process).
+
+<sup><a id="fn.2" href="#fnr.2">2</a></sup> I got them from the SDK configuration scripts on the [sdk github](https://github.com/zephyrproject-rtos/sdk-ng) as well as the
+commits to use for each of the tools.
+
+<sup><a id="fn.3" href="#fnr.3">3</a></sup> *Mostly* complete. libstdc++ does not build because
+\`arm-zephyr-eabi\` is not \`arm-none-eabi\` so a dynamic link check is
+performed/failed. I cannot figure out how crosstool-ng handles this.
+
+<sup><a id="fn.4" href="#fnr.4">4</a></sup> This is no longer the case. Now python packages are provided by the zephyr package
+which was not available when this was being developed.
+
+<sup><a id="fn.5" href="#fnr.5">5</a></sup> For now. Eventually we will need to create a package for `zephyr-base` that
+our guix zephyr-build-system can use.
+
+<sup><a id="fn.6" href="#fnr.6">6</a></sup> These will also need to become guix packages to allow the build system to compose modules.
-- 
2.39.1

Message #8 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
Cc: 61765 <at> debbugs.gnu.org
Subject: Re: bug#61765: custom toolchain blog post
Date: Mon, 27 Feb 2023 13:50:17 +0100

Hello Mitchell,

Mitchell Schmeisser <mitchellschmeisser <at> librem.one> skribis:

> Here is a rough draft of my toolchain post.
> I hope it can make an interesting contribution to the Guix literature.

Yay, definitely!

>>From 4f6c43091ffd67cdbc5f041e496f61bc8a06070e Mon Sep 17 00:00:00 2001
> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
> Date: Fri, 24 Feb 2023 13:02:05 -0500
> Subject: [PATCH] website: Add custom toolchain blog post
>
> * website/posts/custom-toolchains-with-guix.md: New file.

This looks great to me!  It’s useful insight for anyone who might want
to add a cross-compilation target to Guix or simply learn how this is
implemented.

> +++ b/website/posts/custom-toolchains-with-guix.md
> @@ -0,0 +1,557 @@
> +# Table of Contents
> +
> +1.  [Overview](#org2633a51)
> +2.  [Anatomy of a toolchain](#orgc440e9e)
> +3.  [Bootstrapping a Toolchain](#orgd42b6c3)
> +4.  [Defining the Packages](#org55042c5)
> +	1.  [Binutils](#org67da1ec)
> +	2.  [GCC sans libc](#org82d6f83)
> +	3.  [Newlib(-nano)](#orgf6bafbc)
> +	4.  [Complete toolchain](#org052f2a2)
> +5.  [Integrating with Zephyr Build System](#orgc3f87f4)
> +	1.  [Testing](#org9f3c314)
> +
> +All code is available at [guix-zephyr](https://github.com/paperclip4465/guix-zephyr) channel.
> +
> +
> +<a id="org2633a51"></a>

You can remove the table of contents and all the HTML snippets that
pandoc added—I don’t think that works as expected with Haunt/CommonMark.

> +# Overview

You can drop the heading.

> +In order to deploy embedded software using Guix we first need to teach Guix
> +how to build it. Since Guix bootstraps everything this means we must teach Guix
> +how to build our toolchain.
> +
> +The [Zephyr Project](https://zephyrproject.org) uses its own fork of GCC with custom configs for
> +the architectures supported by the project.

We want blog posts to be widely accessible so I would recommend
providing more context in the intro.  In particular, I’d suggest:

  1. Mentioning that Guix supports cross-compilation (not everyone is
     aware of that), with a link to
     <https://guix.gnu.org/en/manual/en/html_node/Cross_002dCompilation.html>.

  2. Adding a couple of sentences saying what Zephyr is (I didn’t know
     about it before :-)).

  3. Saying a few words as to why Zephyr uses a GCC fork.

  4. Clearly stating that you added support for cross-compilation to
     Zephyr with Guix in a channel, linking to said channel, and that
     this is what the remainder of the post will describe.

You can check out posts at <https://guix.gnu.org/blog> for inspiration.

> +# Anatomy of a toolchain
> +
> +Toolchains are responsible for taking high level descriptions of programs
> +and lowering them down to a series of equivalent machine instructions.
> +This process involves more than just a compiler. The compiler uses the `binutils`
> +to manipulate it's internal representation down to a given architecture.
> +It also needs the use of the C standard library as well as a few other libraries
> +needed for some compiler optimizations.
> +
> +The C library provides the interface to the underlying kernel. System calls like `write`
> +and `read` are provided by `Glibc` on most Linux distributions.
>
> +In embedded systems smaller implementations like `newlib` and `newlib-nano` are used.

I’d suggest not using fixed-width font (backquotes) for proper nouns;
you can write “glibc” or “the GNU C Library (glibc)”, “Binutils” or “the
GNU Binary Utilities (Binutils)”, etc.

> +First thing we need to build is the `arm-zephyr-eabi` binutils.
> +This is very easy in Guix.
> +
> +	(define-module (zephyr packages zephyr)
> +	  #:use-module (guix packages))
> +
> +	(define-public arm-zephyr-eabi-binutils
> +	  (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
> +		(package
> +		  (inherit xbinutils)
> +		  (name "arm-zephyr-eabi-binutils")
> +		  (version "2.38")
> +		  (source
> +		   (origin (method git-fetch)
> +				   (uri (git-reference
> +						 (url "https://github.com/zephyrproject-rtos/binutils-gdb")
> +						 (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
> +				   (file-name (git-file-name name version))
> +				   (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
> +		  (arguments
> +		   `(#:tests? #f
> +			 ,@(substitute-keyword-arguments (package-arguments xbinutils)
> +				 ((#:configure-flags flags)
> +				  `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
> +		  (native-inputs
> +		   (append
> +			(list texinfo
> +				  bison
> +				  flex
> +				  gmp
> +				  dejagnu)
> +			(package-native-inputs xbinutils)))
> +		  (home-page "https://zephyrproject.org")
> +		  (synopsis "binutils for zephyr RTOS"))))

Code snippets should be written like this, without leading indentation:

  ```scheme
  (define …)
  ```

This will enable syntax highlighting.

> +We can test our package definition using the `-L` flag with `guix build`
> +to add our packages.
> +
> +	guix build -L guix-zephyr zephyr-binutils
> +
> +	/gnu/store/...-zephyr-binutils-2.38

Likewise:

  ```
  guix build foo
  ```

> +# Integrating with Zephyr Build System
> +
> +Zephyr uses CMake as it's build system. It contains numerous CMake

s/it's/its/

One thing that’s not clear to me: with this in place, can you do “guix
build --target=arm-zephyr-eabi hello”, for instance?  If not, what’s
missing to support it?

It would be great if you could finish with a short conclusion stating,
for instance, the key takeaway message, lessons learned, and/or your
thoughts on how this could benefit others in the broader community.

I wonder if it would be worth mentioning
<https://guix.gnu.org/manual/en/html_node/Platforms.html> too, and how
one would go about adding a  module.  WDYT?

Could you send an updated patch?

Thanks for contributing this article!

Ludo’.

Message #13 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
To: Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#61765: custom toolchain blog post
Date: Mon, 27 Feb 2023 10:35:43 -0500

Message #16 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
Cc: 61765 <at> debbugs.gnu.org
Subject: Re: bug#61765: custom toolchain blog post
Date: Sat, 11 Mar 2023 13:11:55 +0100

Hi Mitchel,

(Please keep the issue Cc’d.)

Apologies for the delay!

Mitchell Schmeisser <mitchellschmeisser <at> librem.one> skribis:

>> One thing that’s not clear to me: with this in place, can you do “guix
>> build --target=arm-zephyr-eabi hello”, for instance?  If not, what’s
>> missing to support it?
>
> You cannot. I do not know how to describe it in a succinct way but
> suffice to say the applications need to know they are targeting Zephyr
> when they are written. The application will include a `prj.conf` which
> is analogous to a custom defconfig in the Linux kernel.
> Either in this file, or somewhere in the CMakeLists.txt a `BOARD`
> variable is set to specify a specific board definition.
> These board definitions contain information about the architecture and
> the CMake scripts themselves pick the toolchain.
>
> It's not that it's impossible to implement something like `guix build
> --target=arm-zephyr-eabi k64f-hello-world` but the k64f-hello-world
> would be written in such a way that the target is implicit in the
> package.

OK.  To put it differently, a typical POSIX program won’t work on
Zephyr; programs have to target the Zephyr interfaces, right?

> The way I envision the `--target/system` flags being used in this
> context is `guix build --target=arm-linux-gnueabihf k64f-hello-world`
> which will still produce the correct firmware but will allow the
> firmware to be staged to another machine which will be responsible for
> the final deployment over some transport.

Or rather:

  guix build --target=arm-zephyr-eabi k64f-hello-world

?

>> I wonder if it would be worth mentioning
>> <https://guix.gnu.org/manual/en/html_node/Platforms.html> too, and how
>> one would go about adding a  module.  WDYT?
>
> I considered trying to add Zephyr platforms but I'm not sure it's worth
> the effort.
> In addition to the patch to the website I attached another post(in org)
> which describes how I integrated this toolchain into the Guix
> infrastructure to allow defining firmware packages.
> Maybe there will be additional information in there which can help you
> understand where I'm going with all of this.
>
> There will be a part 3 (and possibly more) about how to practically use
> this stuff in a real project.

Woow.  :-)

> From 0920ec7d951354c94c3da277d58e54b587522622 Mon Sep 17 00:00:00 2001
> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
> Date: Mon, 27 Feb 2023 10:20:32 -0500
> Subject: [PATCH] website: Add toolchain blog post
>
> website/blog/custom-toolchains-with-guix.md: New file

I pushed it under the drafts directory for now, to leave others a bit
more time to comment before we publish.  I followed up with a commit
editing things a bit, mostly fixing typographical issues,
spelling/capitalization, code formatting, and removing tabs.

  https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/drafts/custom-toolchains-with-guix.md

(BTW, I made slight modifications to some of the code snippets to!  One
package was using (append (list …) (package-native-inputs …)), which
really works “by chance” I guess; you should use ‘modify-inputs’
instead.)

Let me know if anything’s amiss!

Thanks,
Ludo’.

Message #19 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 61765 <at> debbugs.gnu.org
Subject: Re: bug#61765: custom toolchain blog post
Date: Sat, 11 Mar 2023 11:50:28 -0500

[Message part 1 (text/plain, inline)]

Hi Ludo,
> 
> OK.  To put it differently, a typical POSIX program won’t work on
> Zephyr; programs have to target the Zephyr interfaces, right?

This is mostly correct. I think a lot of my conceptual difficulties come from the fact that west/zephyr try to tackle the issues of software deployment and dependency management at the project level. What I mean to say is that in the rest of the software world we have things like auto tools or pkg-config to help with component composition but Zephyr tries to do all composition through several thousands of lines of cmake. Every bit as complex as the kernel kconfig but even more so since modules can be scattered across the file system.
One of the selling points of zephyr is that your application is abstracted from the hardware and can run on multiple boards without modification. However to make this happen you need to do a lot of work on the application side. 
Below you can see an example. You have to provide the proper kernel config for every target you want to support.
https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/net/mqtt_publisher

Using guix there may be a more elegant solution to these problems but maybe not.

Thank you for all your feedback,
Mitchell


> On Mar 11, 2023, at 7:12 AM, Ludovic Courtès <ludo <at> gnu.org> wrote:
> 
> Hi Mitchel,
> 
> (Please keep the issue Cc’d.)
> 
> Apologies for the delay!
> 
> Mitchell Schmeisser <mitchellschmeisser <at> librem.one> skribis:
> 
>>> One thing that’s not clear to me: with this in place, can you do “guix
>>> build --target=arm-zephyr-eabi hello”, for instance?  If not, what’s
>>> missing to support it?
>> 
>> You cannot. I do not know how to describe it in a succinct way but
>> suffice to say the applications need to know they are targeting Zephyr
>> when they are written. The application will include a `prj.conf` which
>> is analogous to a custom defconfig in the Linux kernel.
>> Either in this file, or somewhere in the CMakeLists.txt a `BOARD`
>> variable is set to specify a specific board definition.
>> These board definitions contain information about the architecture and
>> the CMake scripts themselves pick the toolchain.
>> 
>> It's not that it's impossible to implement something like `guix build
>> --target=arm-zephyr-eabi k64f-hello-world` but the k64f-hello-world
>> would be written in such a way that the target is implicit in the
>> package.
> 
> OK.  To put it differently, a typical POSIX program won’t work on
> Zephyr; programs have to target the Zephyr interfaces, right?
> 
>> The way I envision the `--target/system` flags being used in this
>> context is `guix build --target=arm-linux-gnueabihf k64f-hello-world`
>> which will still produce the correct firmware but will allow the
>> firmware to be staged to another machine which will be responsible for
>> the final deployment over some transport.
> 
> Or rather:
> 
>  guix build --target=arm-zephyr-eabi k64f-hello-world
> 
> ?
> 
>>> I wonder if it would be worth mentioning
>>> <https://guix.gnu.org/manual/en/html_node/Platforms.html> too, and how
>>> one would go about adding a  module.  WDYT?
>> 
>> I considered trying to add Zephyr platforms but I'm not sure it's worth
>> the effort.
>> In addition to the patch to the website I attached another post(in org)
>> which describes how I integrated this toolchain into the Guix
>> infrastructure to allow defining firmware packages.
>> Maybe there will be additional information in there which can help you
>> understand where I'm going with all of this.
>> 
>> There will be a part 3 (and possibly more) about how to practically use
>> this stuff in a real project.
> 
> Woow.  :-)
> 
>> From 0920ec7d951354c94c3da277d58e54b587522622 Mon Sep 17 00:00:00 2001
>> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
>> Date: Mon, 27 Feb 2023 10:20:32 -0500
>> Subject: [PATCH] website: Add toolchain blog post
>> 
>> website/blog/custom-toolchains-with-guix.md: New file
> 
> I pushed it under the drafts directory for now, to leave others a bit
> more time to comment before we publish.  I followed up with a commit
> editing things a bit, mostly fixing typographical issues,
> spelling/capitalization, code formatting, and removing tabs.
> 
>  https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/drafts/custom-toolchains-with-guix.md
> 
> (BTW, I made slight modifications to some of the code snippets to!  One
> package was using (append (list …) (package-native-inputs …)), which
> really works “by chance” I guess; you should use ‘modify-inputs’
> instead.)
> 
> Let me know if anything’s amiss!
> 
> Thanks,
> Ludo’.

[Message part 2 (text/html, inline)]

Message #22 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 61765 <at> debbugs.gnu.org
Subject: Re: bug#61765: custom toolchain blog post
Date: Mon, 13 Mar 2023 09:52:32 -0400

[Message part 1 (text/plain, inline)]

Here are two patches with minor fixes.

Sorry about the code formatting... Does anyone have any tips for getting 
emacs to format scheme code in markdown?


Thanks,

Mitchell


On 3/11/23 07:11, Ludovic Courtès wrote:
> Hi Mitchel,
>
> (Please keep the issue Cc’d.)
>
> Apologies for the delay!
>
> Mitchell Schmeisser <mitchellschmeisser <at> librem.one> skribis:
>
>>> One thing that’s not clear to me: with this in place, can you do “guix
>>> build --target=arm-zephyr-eabi hello”, for instance?  If not, what’s
>>> missing to support it?
>> You cannot. I do not know how to describe it in a succinct way but
>> suffice to say the applications need to know they are targeting Zephyr
>> when they are written. The application will include a `prj.conf` which
>> is analogous to a custom defconfig in the Linux kernel.
>> Either in this file, or somewhere in the CMakeLists.txt a `BOARD`
>> variable is set to specify a specific board definition.
>> These board definitions contain information about the architecture and
>> the CMake scripts themselves pick the toolchain.
>>
>> It's not that it's impossible to implement something like `guix build
>> --target=arm-zephyr-eabi k64f-hello-world` but the k64f-hello-world
>> would be written in such a way that the target is implicit in the
>> package.
> OK.  To put it differently, a typical POSIX program won’t work on
> Zephyr; programs have to target the Zephyr interfaces, right?
>
>> The way I envision the `--target/system` flags being used in this
>> context is `guix build --target=arm-linux-gnueabihf k64f-hello-world`
>> which will still produce the correct firmware but will allow the
>> firmware to be staged to another machine which will be responsible for
>> the final deployment over some transport.
> Or rather:
>
>    guix build --target=arm-zephyr-eabi k64f-hello-world
>
> ?
>
>>> I wonder if it would be worth mentioning
>>> <https://guix.gnu.org/manual/en/html_node/Platforms.html> too, and how
>>> one would go about adding a  module.  WDYT?
>> I considered trying to add Zephyr platforms but I'm not sure it's worth
>> the effort.
>> In addition to the patch to the website I attached another post(in org)
>> which describes how I integrated this toolchain into the Guix
>> infrastructure to allow defining firmware packages.
>> Maybe there will be additional information in there which can help you
>> understand where I'm going with all of this.
>>
>> There will be a part 3 (and possibly more) about how to practically use
>> this stuff in a real project.
> Woow.  :-)
>
>>  From 0920ec7d951354c94c3da277d58e54b587522622 Mon Sep 17 00:00:00 2001
>> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
>> Date: Mon, 27 Feb 2023 10:20:32 -0500
>> Subject: [PATCH] website: Add toolchain blog post
>>
>> website/blog/custom-toolchains-with-guix.md: New file
> I pushed it under the drafts directory for now, to leave others a bit
> more time to comment before we publish.  I followed up with a commit
> editing things a bit, mostly fixing typographical issues,
> spelling/capitalization, code formatting, and removing tabs.
>
>    https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/drafts/custom-toolchains-with-guix.md
>
> (BTW, I made slight modifications to some of the code snippets to!  One
> package was using (append (list …) (package-native-inputs …)), which
> really works “by chance” I guess; you should use ‘modify-inputs’
> instead.)
>
> Let me know if anything’s amiss!
>
> Thanks,
> Ludo’.

[0001-website-custom-toolchains-with-guix-Code-fix.patch (text/x-patch, attachment)]

[0002-website-custom-toolchains-with-guix-Update-reference.patch (text/x-patch, attachment)]

Message #25 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
Cc: 61765 <at> debbugs.gnu.org
Subject: Re: bug#61765: custom toolchain blog post
Date: Wed, 15 Mar 2023 15:45:43 +0100

Hi,

Mitchell Schmeisser <mitchellschmeisser <at> librem.one> skribis:

> Sorry about the code formatting... Does anyone have any tips for
> getting emacs to format scheme code in markdown?

No tips, other than making sure to untabify.  :-)

> From 0f6c28345a51e20004bc16b73bda1c0e5ccb7f4c Mon Sep 17 00:00:00 2001
> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
> Date: Mon, 13 Mar 2023 09:36:36 -0400
> Subject: [PATCH 1/2] website: custom-toolchains-with-guix: Code fix
>
> * website/drafts/custom-toolchains-with-guix.md: Removed unnecessary
> native-inputs from gcc-arm-zephyr-eabi-toolchain code block.

[...]

> From 25a65c6ace2a3df43d6c3c6d6e23062a51419025 Mon Sep 17 00:00:00 2001
> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
> Date: Mon, 13 Mar 2023 09:39:22 -0400
> Subject: [PATCH 2/2] website: custom-toolchains-with-guix: Update reference
>  URL
>
> * website/drafts/custom-toolchains-with-guix.md: Changed url from
> "latest" to 3.1 which was the current version at the time of writting.

Applied, and finally published:

  https://guix.gnu.org/en/blog/2023/building-toolchains-with-guix/

Thank you!

Ludo’.

Message #30 received at 61765 <at> debbugs.gnu.org (full text, mbox):

From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 61765 <at> debbugs.gnu.org
Subject: Re: bug#61765: custom toolchain blog post
Date: Wed, 15 Mar 2023 21:55:21 -0400

Ah I’m so honored, thank you for the encouragement!

-Mitchell 

> On Mar 15, 2023, at 10:46 AM, Ludovic Courtès <ludo <at> gnu.org> wrote:
> 
> Hi,
> 
> Mitchell Schmeisser <mitchellschmeisser <at> librem.one> skribis:
> 
>> Sorry about the code formatting... Does anyone have any tips for
>> getting emacs to format scheme code in markdown?
> 
> No tips, other than making sure to untabify.  :-)
> 
>> From 0f6c28345a51e20004bc16b73bda1c0e5ccb7f4c Mon Sep 17 00:00:00 2001
>> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
>> Date: Mon, 13 Mar 2023 09:36:36 -0400
>> Subject: [PATCH 1/2] website: custom-toolchains-with-guix: Code fix
>> 
>> * website/drafts/custom-toolchains-with-guix.md: Removed unnecessary
>> native-inputs from gcc-arm-zephyr-eabi-toolchain code block.
> 
> [...]
> 
>> From 25a65c6ace2a3df43d6c3c6d6e23062a51419025 Mon Sep 17 00:00:00 2001
>> From: Mitchell Schmeisser <mitchellschmeisser <at> librem.one>
>> Date: Mon, 13 Mar 2023 09:39:22 -0400
>> Subject: [PATCH 2/2] website: custom-toolchains-with-guix: Update reference
>> URL
>> 
>> * website/drafts/custom-toolchains-with-guix.md: Changed url from
>> "latest" to 3.1 which was the current version at the time of writting.
> 
> Applied, and finally published:
> 
>  https://guix.gnu.org/en/blog/2023/building-toolchains-with-guix/
> 
> Thank you!
> 
> Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.