Move Raspbian installation from 1st gen to 4th gen Raspi (SD card failing)

Question

I have a Raspbian Buster install on my Raspberry Pi Model B (1st gen). The Raspi had problems with freezes and booting so I figured, it's time for an upgrade and bought a Raspberry Pi 4. Turned out, the old Raspi is ok, just the SD card is starting to fail. So, I'd like to move my Raspbian install to the new SD card anyway.

The problem is, the old SD card is failing and I cannot dd the partition from the old to the new SD card (the copying speed gets very low at some point and file system errors pop up). So I did a clean new Raspbian install using the RaspberryPi Imager (nice simple tool btw.) and would like to:

1. Export a list of explicitly installed packages from the old to the new install. I have SSH access to the old install, it mostly freezes only around once a week, so I can use apt or any other CLI tool. In the Arch Linux universe I would do a "pacman -Qe" and get a list of all explicitly installed packages, but on Raspbian I'm a bit stuck with this. Web search gives mostly rather old and cumbersome results. Is there any reliable way to get a list of explicitly installed packages which is easy to reuse for installing them on the new Raspbian install?

2. Copy data, system and user configuration. I thought about copying /var/www, /etc, and /home from the old install to the new one. I put a lot of effort into configuring stuff (webserver with Let's Encrypt and a DynDNS IP update script, local print server, local file sharing via NFS, Syncthing, git repository etc). Would like to just copy anything user created and not bother setting everything up again. Is this likely gonna work?

Thanks for any input!

Answer 1

I have modified my apthist script to show only user installed packages.

#!/bin/bash
#Print apt-get history EXCEPT for upgrades
# 2017-08-06
# 2020-10-07    Include packages installed by packagekit
# 2020-12-04    delete lines containing 'apt upgrade' and preceding line

for logf in $(ls /var/log/apt/history.log.*.gz | sort -rV) ; \
 do zcat $logf | grep -E -A 1 "Start-Date:|Commandline:" | sed -e '/Requested-By:/d' ; done \
  | tac | sed -e '/^--/d' -e '/apt .*upgrade/{N;d;}' | tac

# Include most recent
grep -E  -A 1  "Start-Date:|Commandline:" /var/log/apt/history.log | sed -e '/Requested-By:/d' \
  | tac | sed -e '/^--/d' -e '/apt .*upgrade/{N;d;}' | tac

1. Below is a script I use to list apt history. This is still handy if you are interested in upgrades.

#!/bin/bash
#Print apt-get history
# 2020-10-07    Include packages installed by packagekit
for logf in $(ls /var/log/apt/history.log.*.gz | sort -rV) ; do zcat $logf | grep -E -A 1 "Start-Date:|Commandline:" | sed -e '/Requested-By:/d' ; done
# Include most recent
grep -E  -A 1  "Start-Date:|Commandline:" /var/log/apt/history.log | sed -e '/Requested-By:/d'

This produces a more manageable list which is easily edited to remove the many "apt upgrade' etc and leave a simple list.

There are some limitations;

1. only the last 12 months are shown (because the logs are rotated monthly and only the 12 most recent are kept) but this can be extended by editing /etc/logrotate.d/apt

The following script uses the history produced by the above and filters to show User history, excluding updates.

#! /bin/sh
# 2020-10-07    Include packages installed by packagekit
# delete lines containing 'apt upgrade' and preceding line
# useful for finding apt history EXCEPT for upgrades
# run on output of apthist.sh

cat $1 | tac | sed -e '/^--/d' -e '/apt .*upgrade/{N;d;}' | tac

2. You SHOULD do regular backups.
   Backup image of SD Card produces a small customised image that can be used to create new SD Cards.
   This is a file based backup, so should be immune to SD Card problems.

3. You may also be interested in the following. I use this to synchronise my latest updated OS to a SD Card which contains an installed Raspberry Pi OS. You could use this to populate a freshly installed OS. This does not copy /home, but you could uncomment the line to include it.

#!/bin/bash
# script to synchronise Pi files to SD Card
# 2019-08-12
# 2020-09-30
# 2020-10-23    copy /home/*

# This script synchronises necessary files from running OS on Pi to SD Card
# SD Card (already formatted with partitions containing OS) in SD Card reader on Pi

# NOTE does not copy cmdline.txt or /etc/fstab
# NOTE does not copy /home/*

# NOTE file rsync EXCLUDE_FILE is required
# use 'rsync-dup-exclude.txt' as we WANT logs & .bash_history to be copied
EXCLUDE_FILE='rsync-dup-exclude.txt'

echo $EXCLUDE_FILE
BOOT_MOUNT='/mnt/SDA1'
ROOT_MOUNT='/mnt/SDA2'

# Check/create Mount Points
if [ ! -e $BOOT_MOUNT ]; then
    mkdir $BOOT_MOUNT
fi
if [ ! -e $ROOT_MOUNT ]; then
    mkdir $ROOT_MOUNT
fi
    echo $BOOT_MOUNT  $ROOT_MOUNT
# Mount Partitions
if ! $(mountpoint -q $BOOT_MOUNT); then
    mount /dev/sda1 $BOOT_MOUNT # mount partition containing boot files
fi
if ! $(mountpoint -q $ROOT_MOUNT); then
    mount /dev/sda2 $ROOT_MOUNT # mount root partition containing OS files
fi

if $(mountpoint -q $BOOT_MOUNT) && $(mountpoint -q $ROOT_MOUNT); then
    echo $BOOT_MOUNT  $ROOT_MOUNT
    echo "Synchronise boot files to SD Card"
    rsync -a --delete-during --exclude='cmdline.txt' --exclude='/SD?' --exclude='/SD??' /boot/ $BOOT_MOUNT/
    echo "Synchronise OS files to SD Card"
$ROOT_MOUNT/
#   rsync -axH --progress --delete-during --exclude='/etc/fstab' --exclude='/home/*' --exclude='WolframEngine' --exclude='scratch2' --exclude-from=$EXCLUDE_FILE / $ROOT_MOUNT/
    rsync -axH --progress --delete-during --exclude='/etc/fstab' --exclude='WolframEngine' --exclude='scratch2' --exclude-from=$EXCLUDE_FILE / $ROOT_MOUNT/
    umount $BOOT_MOUNT
    umount $ROOT_MOUNT
else
    echo "Backup Card not available or not writable"
fi

This needs to file rsync-dup-exclude.txt to exclude unnecessary files.

/proc/*
/sys/*
/dev/*
/boot/*
/tmp/*
/run/*
/mnt/*/*
/var/cache/*

/boot/cmdline.txt
/etc/fstab

.Trashes
._.Trashes
.fseventsd
.Spotlight-V100
.DS_Store
.AppleDesktop
.AppleDB
Network Trash Folder
Temporary Items

.cache
/etc/fake-hwclock.data
/var/lib/rpimonitor/stat/
/etc/hostname
/etc/hosts