Install base Raspbian from repository, not using an image

Question

Raspbian has no installer, so how can I install it by starting with debootstrap? How it works with Debian can you find here: install Debian with debootstrap and the package can be found here: debootstrap for Raspbian.

Is there anyone who knows how to fit this together?

Answer 1

━━━ Setup using a Linux computer with intel processor ━━━

This installation uses another computer with an intel processor. If you want to run the installation on a Raspberry Pi then look at the other answer here: Installation using a Raspberry Pi. If possible, I recommend that installation.

We have to run debootstrap with an armhf architecture on a PC with intel processor. This is only possible with an emulator. Qemu is a program that can do this. So we have to install that together with debootstrap and additional needed packet binfmt-support ¹. I use Debian GNU/Linux 9.4 (stretch) arch x86_64. Others that have debootstrap and qemu should also work:

pc ~$ sudo apt install binfmt-support qemu qemu-user-static debootstrap

We also need the raspbian.public.key to check integrety of the raspbian repository. So we download and import it into the apt keyring /etc/apt/trusted.gpg ²:

pc ~$ wget http://archive.raspbian.org/raspbian.public.key -O - | sudo apt-key add -q

Now we prepare the SD Card we want to install the bootstraped Raspbian. For reference I use Raspbian Stretch Light 2018-06-27. Download and flash it to the SD Card. Start it with your RasPi, login and poweroff. This seems to be necessary to intitalize default boot sequence.

Then put it into the card reader on the PC. For my example it is attached to /dev/sdb. Format the second partition with ext4 and mount that partition:

pc ~$ sudo -Es
pc ~# mkfs.ext4 -FL rootfs /dev/sdb2
pc ~# mkdir /mnt/sdb2
pc ~# mount /dev/sdb2 /mnt/sdb2

Now we install a basic Raspbian from debootstrap with Qemu:

pc ~# qemu-debootstrap --keyring=/etc/apt/trusted.gpg --arch armhf stretch /mnt/sdb2/ http://mirrordirector.raspbian.org/raspbian/

When finished prepare installation for booting. We have to create a fstab:

pc ~# cat > /mnt/sdb2/etc/fstab <<EOF
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
EOF

Clear root password for first login. DON'T FORGET to set a password immediately after first login into the new pibootstrap!

pc ~# sed -i 's/^root:x:/root::/' /mnt/sdb2/etc/passwd
pc ~# sed -i 's/^root:*:.*$/root::::/' /mnt/sdb2/etc/shadow

pc ~# umount /dev/sdb2
pc ~# e2fsck -f /dev/sdb2
pc ~# exit
pc ~$

Put the SD Card in the RasPi and boot.

We have a clean tiny but working Raspbian consuming just 276 MB of diskspace. This is a good startpoint to build clear test environments or to learn how it works step by step. Maybe you will first make an account for a normal user and configure network connection, so you can use apt to install software packages.

Setup basic Raspbian for systemd

Login as root without password. Then excute:

rpi ~# passwd

Then do:

rpi ~# mkdir -p /var/log/journal
rpi ~# systemd-tmpfiles --create --prefix=/var/log/journal # ignore warning

rpi ~# apt --autoremove purge ifupdown
rpi ~# cat > /etc/systemd/network/04-eth.network <<EOF
[Match]
Name=e*
[Network]
DHCP=yes
EOF

rpi ~# systemctl enable systemd-networkd.service
rpi ~# systemctl enable systemd-resolved.service
rpi ~# ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
rpi ~# systemctl start systemd-networkd.service
rpi ~# systemctl start systemd-resolved.service

Copy /etc/apt/sources.list and /etc/apt/sources.list.d/raspi.list from a default Raspbian installation. Then:

rpi ~# echo 'APT::InstallRecommends "false";' > /etc/apt/apt.conf.d/99install-recommends
rpi ~# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553
rpi ~# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010
rpi ~# apt update
rpi ~# apt install dirmngr
rpi ~# apt full-upgrade

rpi ~# apt install libraspberrypi-bin raspberrypi-kernel raspberrypi-sys-mods
rpi ~# apt install sudo bash-completion lvm2 dbus firmware-brcm80211 
rpi ~# adduser pi
rpi ~# useradd --groups sudo,adm pi

add to sudoers:
# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) NOPASSWD: ALL

rpi ~# #change hostname
rpi ~# sed -i 's/.*/raspberrypi/' /etc/hostname
rpi ~# echo "127.0.1.1       raspberrypi" >> /etc/hosts

Reboot.

references:
[1] EmDebian CrossDebootstrap
[2] man apt-key
[3] man debootstrap

Answer 2

━━━ Installation using a Raspberry Pi ━━━

This installation uses a Raspberry Pi for setup. If you want to use another computer then look at the other answer here: Setup using a Linux computer with intel processor.

You need two SD Cards, one to boot the RasPi and one attached with your SD Card reader on an USB port of the RasPi. I will try to present two installation paths because they are similar and have many common setups. First I will install Raspbian on two primary partitions as it is done on the default image Raspbian Buster. This is less error prone for newbies or to get quick a running system. On the other path I will use logical volumes because I love to use its nice features for easy backup, restore from snapshots, desaster recovery and others. I labled the setups with COMMON, PRIMPART (primary partitions) and LVM. If you want to use primary partitions just follow only parts labled with COMMON and PRIMPART, ignore parts with LVM. Installing logical volumes only use COMMON and LVM.

I will partition and format a virgin SD Card so I'm able to configure it to just what I like. First I prepare the master boot record (MBR), then partition the SD Card and format it. I will do it on the Raspberry Pi so we do not have problems running software for the ARM processor to be used for installation and we can chroot into the new installation and prepare needed things. it is recommended to use an up to date Raspbian version to start the RasPi for installation because we copy some configuration files. For reference I use Raspbian Buster Lite udated with sudo apt update && sudo apt full-upgrade && sudo reboot. I assume you have the internet connection already running.

---

COMMON:
Boot the RasPi and after login create a default msdos partition table on the empty SD Card. I assume it is attached to /dev/sda:

rpi ~$ sudo -Es
rpi ~# parted /dev/sda mktable msdos

Now we have made a master boot record (MBR), the very first sector with 512 bytes. It also contains a very small boot sequence at the beginning. You can check it with:

rpi ~# hd -v -n512 /dev/sda

Partition the SD Card with sizes what you like. I will use a 16 GB card:

rpi ~# parted /dev/sda mkpart primary fat32 8192s 132MB
rpi ~# parted /dev/sda mkpart primary 132MB 100%

The boot sequence in the MBR is not used by a Raspberry Pi so we will delete it with:

rpi ~# dd if=/dev/zero of=/dev/sda bs=1 count=440 conv=notrunc

This is not really required but I do it to be on the save side because the MBR from the original Raspbian image is also nullified on this sequence. Then we format the first partition with fat32 and make a mount point.

rpi ~# mkfs.vfat -vF 32 -n BOOT /dev/sda1
rpi ~# mkdir /mnt/p2

---

PRIMPART:
Format the second partition with ext4 and mount it:

rpi ~# mkfs.ext4 -vL rootfs /dev/sda2
rpi ~# mount /dev/sda2 /mnt/p2

---

LVM:
Setup logical volumes in the second partition, format it with ext4 and mount it:

rpi ~# parted /dev/sda set 2 lvm on
rpi ~# pvcreate /dev/sda2  # define physical volume
rpi ~# vgcreate rpi_vg /dev/sda2  # create volume group using physical volume
rpi ~# lvcreate rpi_vg --name root_lv --size 8G  # create logical volume in volume group

# format the logical volume and mount it
rpi ~# mkfs.ext4 -vL rootfs /dev/mapper/rpi_vg-root_lv
rpi ~# mount /dev/mapper/rpi_vg-root_lv /mnt/p2

---

COMMON:
Mount the boot partition:

rpi ~# mkdir /mnt/p2/boot
rpi ~# mount /dev/sda1 /mnt/p2/boot/

We install the debootstrap installation program and execute it:

rpi ~# apt install debootstrap
rpi ~# /usr/sbin/debootstrap --arch armhf buster /mnt/p2 http://raspbian.raspberrypi.org/raspbian

Now we have a tiny but complete Debian/Raspbian operating system on the new SD Card. We have to do some essential setups. Copy the default repository sources list files:

rpi ~# cp /etc/apt/sources.list /mnt/p2/etc/apt/
rpi ~# cp /etc/apt/sources.list.d/raspi.list /mnt/p2/etc/apt/sources.list.d/

---

PRIMPART:
First set the kernel boot parameter with boot/cmdline.txt:

rpi ~# echo 'console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait' > /mnt/p2/boot/cmdline.txt

You can use console=serial0,115200 instead of console=tty1 if you are using the serial debug console like me. If you don't like Predictable Network Interface Names you can append net.ifnames=0 to cmdline.txt. To make the new SD Card bootable we have to set /etc/fstab:

rpi ~# cat > /mnt/p2/etc/fstab <<EOF
/dev/mmcblk0p1  /boot    vfat    defaults            0    2
/dev/mmcblk0p2  /        ext4    defaults,noatime    0    1
EOF

---

LVM:
First set the kernel boot parameter with boot/cmdline.txt:

rpi ~# echo 'console=tty1 root=/dev/mapper/rpi_vg-root_lv rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait' > /mnt/p2/boot/cmdline.txt

You can use console=serial0,115200 instead of console=tty1 if you are using the serial debug console like me. If you don't like Predictable Network Interface Names you can append net.ifnames=0 to cmdline.txt. To make the new SD Card bootable we have to set /etc/fstab:

rpi ~# cat > /mnt/p2/etc/fstab <<EOF
/dev/mmcblk0p1                /boot    vfat    defaults            0    2
/dev/mapper/rpi_vg-root_lv    /        ext4    defaults,noatime    0    1
EOF

---

COMMON:
Now we chroot into the new tiny installation and install the kernel and firmware:

rpi -# mount --bind /dev/pts /mnt/p2/dev/pts
rpi ~# LANG=C.UTF-8 chroot /mnt/p2 /bin/bash
root@raspberrypi:/# export TERM=linux

apt claims a signing key so we have to get it first:

root@raspberrypi:/# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 82B129927FA3303E

The only user on the new installation is root but its password isn't set, so we will not be able to log in after booting the new SD Card. So first set the root password:

root@raspberrypi:/# passwd root

Now we install the kernel and firmware:

root@raspberrypi:/# apt install dialog
root@raspberrypi:/# apt update
root@raspberrypi:/# apt full-upgrade
root@raspberrypi:/# echo raspi4 > /etc/hostname  # or what you want to name your RasPi
root@raspberrypi:/# apt install raspberrypi-bootloader
root@raspberrypi:/# apt install libraspberrypi-bin
root@raspberrypi:/# apt install raspberrypi-archive-keyring raspberrypi-sys-mods
root@raspberrypi:/# apt install sudo
# WiFi driver
root@raspberrypi:/# apt install --no-install-recommends firmware-brcm80211 wireless-regdb crda

Only if you want to use the serial debug console, like me, then enable it:

root@raspberrypy:/# echo 'enable_uart=1' >> /boot/config.txt

---

PRIMPART:
Exit chroot:

root@raspberrypi:/# exit
rpi ~#

---

LVM:
For the lvm we need to load the driver on early stage boot to be able to access the root volume. So we have to use an init ramdisk and install lvm driver:

root@raspberrypi:/# apt install initramfs-tools
root@raspberrypi:/# apt install lvm2

# I only want to load listed modules
root@raspberrypi:/# sed -i 's/^MODULES=most/MODULES=list/' /etc/initramfs-tools/initramfs.conf

In /boot/config.txt define to load the ramdisk depending on the Raspberry Pi version. First look what modules have to be loaded, for example:

root@Raspberrypi:/# ls /lib/modules
4.19.97+  4.19.97-v7+  4.19.97-v7l+  4.19.97-v8+

Then at the top of /boot/config.txt insert these lines:

[pi3]
initramfs initrd.img-4.19.97-v7+ followkernel
[pi4]
initramfs initrd.img-4.19.97-v7l+ followkernel
# arm_64bit=1
# initramfs initrd.img-4.19.97-v8+ followkernel
[all]

Finally create the needed init ramdisks and exit chroot. You can ignore warnings:

root@Raspberrypi:/# mkinitramfs -o /boot/initrd.img-4.19.97-v7+ 4.19.97-v7+
root@Raspberrypi:/# mkinitramfs -o /boot/initrd.img-4.19.97-v7l+ 4.19.97-v7l+
root@raspberrypi:/# exit
rpi ~#

Don't forget to update the ramdisk version and recreate it if you update kernel or firmware, otherwise it may be possible that you cannot boot anymore. But then you can mount the SD Card to your RasPi like this, chroot into it and recreate the ramdisk.

---

COMMON:
Clean up and poweroff:

rpi ~# 
rpi ~# umount /mnt/p2/boot
rpi ~# umount /mnt/p2/dev/pts
rpi ~# umount /mnt/p2
rpi ~# rmdir /mnt/p2
rpi ~# systemctl poweroff

Now change the SD Card and boot the new installation. Login with root.

We have a clean tiny but working Raspbian with usable apt package manager. It consumes just 356 MB of diskspace for the root partition. This is a good starting point to build custom environments or to learn how it works step by step. Maybe you will first make an account for a normal user and configure network connection, so you can use apt to install software packages.

references:
(1) How to make an image file from scratch
(2) Easy backups and snapshots of a running system with LVM