Copy existing Raspbian installation to smaller SD card

Question

Is it possible to copy an existing and configured Raspbian installation to a smaller SD card?

When I first installed Raspbian I only had a 32 GB card at hand which obviously is bigger than needed.

Answer 1

In this answer, I demonstrate what to do step-by-step for people to understand the logic behind the solution and to be able to apply steps in their other problems.

But firstly, it should be stated that it is a generic (not raspi specific) problem to migrate filesystems from an SD card to a smaller (but big enough for data) SD card.

Requirements

A laptop with a micro SD card reader and Linux (I prefer Ubuntu) running on it.

Abbreviations

PIBOX      : Raspberry Pi which is used
SD_CARD_A  : 8GB micro SD card which is used on PIBOX and on which Raspbian-lite (the OS) is installed
SD_CARD_B  : 2GB micro SD card which will be used on PIBOX and on which Raspbian-lite (the OS) will be installed

Partitions of SD_CARD_A

While PIBOX is running, we list the partitions (unnecessary system partitions are not being displayed here).

root@pibox:~# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      7.3G  1.1G  5.9G  16% /
/dev/mmcblk0p1 vfat       63M   21M   43M  33% /boot

There are 2 partitions on SD_CARD_A as / and /boot. Even 2GB is not used in total.

Backup SD_CARD_A

After we shutdown and halt PIBOX, we take SD_CARD_A out from PIBOX board and put it into the card reader of our laptop.

Partitions of SD_CARD_A are automatically mounted to our system as /dev/sdc1 and /dev/sdc2.

root@mylaptop:~# df -Th
Filesystem                    Type      Size  Used Avail Use% Mounted on
/dev/sdb2                     ext4       22G   13G  7.9G  63% /
/dev/sdb1                     vfat      197M  2.6M  195M   2% /boot/efi
/dev/sda8                     ext4       66G   11G   52G  17% /home
/dev/sdc1                     vfat       63M   21M   43M  33% /media/some_user_name/boot
/dev/sdc2                     ext4      7.3G  1.1G  5.9G  16% /media/some_user_name/some_uuid_serial

We unmount those partitions from our system to operate on them successfully.

root@mylaptop:~# umount /dev/sdc1
root@mylaptop:~# umount /dev/sdc2

We display the device info of SD_CARD_A in details for confirmations in next steps.

root@mylaptop:~# fdisk -l /dev/sdc
Disk /dev/sdc: 7969 MB, 7969177600 bytes
246 heads, 62 sectors/track, 1020 cylinders, total 15564800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2019f6d8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            8192      137215       64512    c  W95 FAT32 (LBA)
/dev/sdc2          137216    15564799     7713792   83  Linux

Above you can see that SD_CARD_A has a capacity of 8GB.

We clone SD_CARD_A into pibox.img file.

root@mylaptop:~# dd bs=4MB if=/dev/sdc of=pibox.img
1992+1 records in
1992+1 records out
7969177600 bytes (8.0 GB) copied, 416.582 s, 19.1 MB/s

Check the size of copied bytes, it is equal to the value we got by fdisk -l /dev/sdc command.

Linux loopback module

Linux has a module called loopback which provides us to handle a file as a block device.

We load loopback module.

root@mylaptop:~# modprobe loop

We find an unused loopback device path.

root@mylaptop:~# losetup -f /dev/loop0

Now, we create a loopback device for pibox.img file.

root@mylaptop:~# losetup /dev/loop0 pibox.img

We trigger kernel about partition changes.

root@mylaptop:~# partprobe /dev/loop0

We confirm if previous operations are successful.

root@mylaptop:~# losetup /dev/loop0
/dev/loop0: [0806]:69 (/root/pibox.img)

We display the loopback device info in details to compare it with SD_CARD_A.

root@mylaptop:~# fdisk -l /dev/loop0
Disk /dev/loop0: 7969 MB, 7969177600 bytes
255 heads, 63 sectors/track, 968 cylinders, total 15564800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2019f6d8

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1            8192      137215       64512    c  W95 FAT32 (LBA)
/dev/loop0p2          137216    15564799     7713792   83  Linux

Above you can see that the loopback device size (= 7969177600 bytes) and partitions are same with SD_CARD_A's.

Basic Maths

From now on, we will be focused on the partition /dev/loop0p2. Let's name it THE_PARTITION.

The block size is 512 bytes (as printed on the line starting with Units = sectors .....)

THE_PARTITION starts from block 137216 and ends at block 15564799 which means that it has the size of 15427584 blocks (= 15564799 − 137216 + 1).

So, the size of THE_PARTITION in bytes is 7898923008 bytes (= 512 * 15427584).

To fit THE_PARTITION in SD_CARD_B, we want it to have a new size of 3710940 blocks or in other words 1900001280 bytes (= 512 * 3710940).

So, the new end block number is 3848155 calculated by start block number (=137216) + size in blocks (=3710940) - 1.

Filesystem vs. Partition

There are 2 operations which should not be mistaken for each other.

• Resizing filesystem. We will shrink the file system on THE_PARTITION by setting its size to 3710940 blocks.
• Resizing partition. We will shrink THE_PARTITION by setting its end block number to 3848155.

Shrinking filesystem

Before shrinking the file system, it should be marked as clean by e2fsck.

root@mylaptop:~# e2fsck -f /dev/loop0p2
e2fsck 1.42.9 (4-Feb-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0p2: 41175/475776 files (0.2% non-contiguous), 309183/1928448 blocks

We shrink the file system with resize2fs.

root@mylaptop:~# resize2fs /dev/loop0p2 3710940s
resize2fs 1.42.9 (4-Feb-2014)
Resizing the filesystem on /dev/loop0p2 to 463867 (4k) blocks.
The filesystem on /dev/loop0p2 is now 463867 blocks long.

Shrinking partition

We learn what THE_PARTITION number is with parted.

root@mylaptop:~# parted /dev/loop0
GNU Parted 2.3
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: Loopback device (loop)
Disk /dev/loop0: 7969MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  70.3MB  66.1MB  primary  fat16        lba
 2      70.3MB  7969MB  7899MB  primary  ext4

(parted) quit

We shrink THE_PARTITION with parted.

root@mylaptop:~# parted /dev/loop0 unit s resizepart 2 3848155
Warning: Shrinking a partition can cause data loss, are you sure you want to continue?
Yes/No? Yes  

We are done with the loopback device. We detach it.

root@mylaptop:~# losetup -d /dev/loop0

Truncating image file

We verify the new partition table.

root@mylaptop:~# fdisk -l pibox.img 

Disk pibox.img: 7969 MB, 7969177600 bytes
255 heads, 63 sectors/track, 968 cylinders, total 15564800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2019f6d8

    Device Boot      Start         End      Blocks   Id  System
pibox.img1            8192      137215       64512    c  W95 FAT32 (LBA)
pibox.img2          137216     3848155     1855470   83  Linux

In the output, it is clearly seen that the end block number of THE_PARTITION is decreased from 15564799 to 3848155.

The last block that we use is 3848155. The block numbering start from 0. So, we have 3848155 + 1 blocks in total and the new size of pibox.img file should be 1970255872 bytes (= (3848155 + 1) * 512).

We truncate pibox.img file.

root@mylaptop:~# truncate --size=1970255872 pibox.img

We verify the new size of pibox.img file.

root@mylaptop:~# ls -l pibox.img 
-rw-r--r-- 1 root root 1970255872 Oct 13 21:53 pibox.img

Creating SD_CARD_B

We put SD_CARD_B into the card reader of our laptop. Partitions of SD_CARD_B are automatically mounted to our system as /dev/sdc1 and /dev/sdc2.

root@mylaptop:~# df -Th
Filesystem                    Type      Size  Used Avail Use% Mounted on
/dev/sdb2                     ext4       22G   13G  7.9G  63% /
/dev/sdb1                     vfat      197M  2.6M  195M   2% /boot/efi
/dev/sda8                     ext4       66G   11G   52G  17% /home
/dev/sdc1                     vfat       63M   21M   43M  33% /media/some_user_name/boot
/dev/sdc2                     ext4      1.8G  1.6G   59M  97% /media/some_user_name/some_uuid_serial

Above you can see that SD_CARD_B has a capacity of 2GB.

We unmount those partitions from our system to operate on SD_CARD_B successfully.

root@mylaptop:~# umount /dev/sdc1
root@mylaptop:~# umount /dev/sdc2

We clone pibox.img file into SD_CARD_B.

root@mylaptop:~# dd bs=4MB if=pibox.img of=/dev/sdc
492+1 records in
492+1 records out
1970255872 bytes (2.0 GB) copied, 646.967 s, 3.0 MB/s

Check the size of copied bytes, it is equal to the value we got by ls -l pibox.img command.

Booting PIBOX

After we take SD_CARD_B out from our laptop and put it into PIBOX board, we boot the system and login into PIBOX console.

We list the partitions (some other unnecessary system partitions are not being displayed here).

root@pibox:~# df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      1.8G  1.1G  601M  64% /
/dev/mmcblk0p1 vfat       63M   21M   43M  33% /boot

Answer 2

When you used dd if=/dev/sdx of=/path/to/image bs=1M , /dev/sdx refers to the entire "disk", so the image would always be the size of the whole card.

Instead, you'd need to use dd if=/dev/sdxn ... where n is the partition number.

You'll probably need to do this twice - once for the /boot partition and once for the / partition.

Then you'd need to create partitions on the new card that are at least as big as those two original ones, to dd the content back to.

Answer 3

Use something like parted (partition editor) to shrink the primary partition to a smaller size and then use a tool like Clonezilla to copy from the now smaller partition to your new card. You will likely have to do this on another computer, though.

Answer 4

1. Create an image of the card using one of the already mentioned methods - How do I backup my Raspberry Pi?

2. Use the script at http://sirlagz.net/2013/03/10/script-automatic-rpi-image-downsizer/ to downsize the image

3. Restore the downsized image on a new smaller card

Answer 5

Rpi's SD Card copier is a great tool you can use to create copy of your cards,it can create a copy of bigger SD card to smaller one.(as long as it has enough space to store all your files)

Example Raspbian in 32 GB memory card can be copied to 16 GB,but it should not use storage more than 16 GB.

link for this feature : link for this update

Answer 6

I've been using rsync to copy filesystems from one disk to another for a while now, with no hiccups. The advantage of using rsync is that it's copying the content of the filesystem, rather than doing a block-level copy of the device; as a result, it really doesn't care what size the target and source drives are, so long as the target drive has enough space to hold the data.

So here's how I would do it:

1. Create a new raspbian installation on your desired new, smaller SD card.
2. Boot into the new installation and expand the filesystem to fill the whole disk. Shutdown the pi.
3. Now mount the new and old cards and use rsync -avx oldFilesystem newFilesystem to copy/overwrite the filesystem on the new card with the filesystem from the old card.
4. Finally, boot into your new system, and run rpi-update to make sure that your firmware is consistent and up to date.

After this, your new card should have a perfectly functional Raspbian system installed on it.

Answer 7

I created a shell script to backup and restore all the data on a SD Card. It first deletes some data (corresponding to my project) and shrinks the partition to the minimum size so the image is as big as the data on the SD card is. As an addition, the script creates a *.zip file of the image. After restoring the created image to another SD Card the partition will be enlarged to maximum size. The script uses the commands which are mentioned in the other answers. As this is my fist shell script with this size it took me hours to create it and it is not perfect jet. Especially I don't know how to handle the return values of resize2fs and fdisk so the user has to type in the values I need. Are there any ideas to fix that? I hope this script helps someone else. Feel free to edit and improve it.

"Usage:
    <skriptname> -b <path>                  create backup of SC Card (dev/mmcblk0) to file <path>/JJJJ-MM-DD_HHMM.img
    <skriptname> -r <path>/FILENAME.img     restore an exitsting image (<path>/FILENAME.img) to the SD Card (dev/mmcblk0) 
    <skriptname> -r <path>/FILENAME.zip     unzip and restore an exitsting image (<path>/FILENAME.zip) to the SD Card (dev/mmcblk0)
    <skriptname> -h                         show this hlep

here it is:

#!/bin/bash 

# check if the user is root
if (( $EUID != 0 )); then
  echo "This script requires root privileges please run as root"
  exit
fi


while getopts ":b:r:h" opt; do
  case $opt in
    b)
      mode="backup"
      OUTPATH=$OPTARG
      ;;
    r)
      mode="restore"
      DIRFILENAME=$OPTARG
      ;;
    h)
      mode="help"
      ;;
    \?)
      echo "Invalid option: -$OPTARG. Use -h for help" >&2
      exit 1
      ;;
    :)
      echo "Option -$OPTARG requires an argument. Use -h for help" >&2
      exit 1
      ;;
  esac
done
# no option
if [ $OPTIND == 1 ]
then
  echo "$(basename "$0") needs an option! Use -h for help"
  exit 1
fi


myMount(){
  # create mountpoint if not existing
  if [ ! -d /tmp/sd2/ ] ; then
    mkdir /tmp/sd2
  fi

  # mount partition
  mount -v -t ext4 /dev/mmcblk0p2 /tmp/sd2
  err=$?
  if [ $err != 0 ]; then
    echo "mount failed error: $err"
    exit 1
  fi
}

myUmount(){
  cd /home/ # otherwise umount will fail
  # fuser -vm /tmp/sd2/

  # umount partition
  umount -v /tmp/sd2
  err=$?
  if [ $err != 0 ]; then
    echo "umount failed error: $err"
    exit 1
  fi
}

myEnlarge(){
  echo "enlarge partition..."
  # enlarge partition is not posible with fdisk -> delete and recreate it
  (
  echo d # delete partition
  echo 2 # patition number
  echo n # add a new partition
  echo p # primary partition
  echo 2 # partition number
  echo   # first sector (accept default: varies)
  echo   # last sector (accept default: varies)
  echo w # write changes
  ) | fdisk /dev/mmcblk0

  echo "\n check filesystem... "
  e2fsck -f -v -C 0 /dev/mmcblk0p2

  # enlarge filesystem to maxsize
  resize2fs -p /dev/mmcblk0p2
}

case "$mode" in
"help")
  echo "Usage:
    $(basename "$0") -b <path>                  create backup of SC Card (dev/mmcblk0) to file <path>/JJJJ-MM-DD_HHMM.img
    $(basename "$0") -r <path>/FILENAME.img     restore an exitsting image (<path>/FILENAME.img) to the SD Card (dev/mmcblk0) 
    $(basename "$0") -r <path>/FILENAME.zip     unzip and restore an exitsting image (<path>/FILENAME.zip) to the SD Card (dev/mmcblk0)
    $(basename "$0") -h                         show this hlep
--------------------------------
Adrian Zeitler, Germany 2017"
  ;;
"backup")  ####################################### backup ####################################### 
  echo "an image of the SD Card (/dev/mmcblk0) whitch is as smal as possible will be created to $OUTPATH."
  # ------------------  delete some data --------------------

  echo "Do you want to delete tempfiles? [y/n]" 
  read delfiles

  if [ "$delfiles" = "y" ]
    then
      echo "Delete tempfiles..."

      myMount

      # remove some data
      cd /tmp/sd2/home/alarm/
      rm -v -f hagelbeere.db
      rm -v -f HAILcam.log
      rm -v -f HAILcam.log.1
      rm -v -f test.jpg

      myUmount

    elif [ "$delfiles" = "n" ]
      then
    echo "I don't delete anything."
    else
    echo "Sorry, I didn't understand."
    exit 1
  fi


  # --------------------------------------------------------------
  # shrink partition 2 to minimum size

  echo "check file system... "
  e2fsck -f -v -C 0 /dev/mmcblk0p2
  err=$?
  if [ $err != 0 ]; then
    echo "file system check failed, error: $err"
    exit 1
  fi

  echo "shrink filesystem of partition 2 to minimum size..."
  resize2fs -p -M /dev/mmcblk0p2
  err=$?
  if [ $err != 0 ]; then
    echo "resize2fs failed, error: $err"
    exit 1
  fi
  # --> Das Dateisystem auf /dev/mmcblk0p2 ist nun 692365 Blöcke groß.

  echo "Please tell me the new filesystem size displayed above:"
  read size
  # from resize2fs blocksize, fdisk wants sector: sector = block * 8
  size=$(( $size*8 ))

  # shrink partition is not posible with fdisk -> delete and recreate it
  (
  echo d # delete partition
  echo 2 # patition number
  echo n # add a new partition
  echo p # primary partition
  echo 2 # partition number
  echo   # first sector (accept default: varies)
  echo +$size  # last sector
  echo w # write changes
  ) | fdisk /dev/mmcblk0
  err=$?
  if [ $err != 0 ]; then
    echo "fdisk failed, error: $err"
    exit 1
  fi


  # --------------------------------------------------------------

  # fill unused space with zeros
  echo "Do you want to fill unused space with zeros? [y/n]" 
  read fillzeros


  if [ "$fillzeros" = "y" ]
    then
      echo "Copy zeros. This will end up with an error. But this is ok."

      myMount    

      dd if=/dev/zero | pv | dd of=/tmp/sd2/nullen.datei conv=noerror,notrunc,sync bs=10240
      # exits with error -> this is normal

      # dlelete zeros
      rm -v -f /tmp/sd2/nullen.datei
      sync

      myUmount

    elif [ "$fillzeros" = "n" ]
      then
    echo "I don't delete anything."
    else
    echo "Sorry, I didn't understand."
    exit 1
  fi

  # --------------------------------------------------------------

  # find out end of partition
  fdisk -l /dev/mmcblk0
  echo "Please tell me the end of mmcblk0p2 displayed above."
  read count



  DATE=$(date +"%Y-%m-%d_%H%M")
  IMGFILENAME=$DATE.img 
  echo "Do you want to create image with filename $OUTPATH$IMGFILENAME? [y/n]"
  read answer
  if [ "$answer" = "y" ]
  then
    echo "Do you want to create a *.zip file of the created image? [y/n]"
    read zip
    echo "Do you want to enlarge partition 2 to maxsize after image creation? [y/n]"
    read enlarge

    echo "create image..."
    cd $OUTPATH
    # create image with dd, stop at and of partition
    # count=N   copy only N input blocks
    # bs=BYTES  read and write up to BYTES bytes at a time = block size
    # pv    show status
    dd if=/dev/mmcblk0 | pv -s $(( $count*512 )) | dd of=$IMGFILENAME bs=512 count=$count
    err=$?
    if [ $err != 0 ]; then
      echo "dd failed error: $err"
      exit 1
    fi

    # --------------------------------------------------------------
    # create zip file
    # or like this:
    # sudo dd if=/dev/sdX | pv |gzip > /pfad/zur/datei.img.gz
    if [ "$zip" = "y" ]
    then
      echo "create zip file..."
      zip $DATE.zip $IMGFILENAME
    fi
    # --------------------------------------------------------------
  fi

  # --------------------------------------------------------------
  # enlarge partition 2

  if [ "$enlarge" = "y" ]
  then
    myEnlarge
  fi

  ;; #end case mode backup
"restore")  ####################################### restore ####################################### 
  #chek if image exists
  if [[ -s "$DIRFILENAME" ]]
  then
    # check if file is an image or zip file
    if [[ $DIRFILENAME =~ \.img$ ]]
    then
      IMGFILENAME=$(basename "$DIRFILENAME")
    elif [[ $DIRFILENAME =~ \.zip$ ]]
    then
      ZIPFILENAME=$(basename "$DIRFILENAME")
    else
      echo "Not the right file format. I accept *.img and *.zip"
      exit 1
    fi
  else
    echo "Image file does not exist."
    exit 1
  fi
  echo "the file $DIRFILENAME will be restored to the SD Card /dev/mmcblk0"

  #change to the path of the imagefile
  SOURCEPATH=$(dirname "$DIRFILENAME")
  cd $SOURCEPATH


  if [ "$ZIPFILENAME" != "" ]
  then
    echo "unzip file"
    # change file extention form zip zu img
    l=$(( ${#ZIPFILENAME}-3 ))
    IMGFILENAME="${ZIPFILENAME:0:l}img"
    unzip $ZIPFILENAME
  fi

  echo "Do you realy want to restore $SOURCEPATH/$IMGFILENAME to the SD card /dev/mmcblk0? 
  Warning: all data on the device /dev/mmcblk0 will be lost! [y/n]"
  read answer
  if [ "$answer" = "y" ]
  then
    echo "Do you want to enlarge partition 2 to maxsize after restoring? [y/n]"
    read enlarge
    echo "restore image..."
    filesize=$(wc -c <"$IMGFILENAME")
    echo "Filesize = $filesize Byte"
    dd if=$IMGFILENAME | pv -s $filesize | dd of=/dev/mmcblk0 bs=512
    err=$?
    if [ $err != 0 ]; then
      echo "dd failed error: $err"
      exit 1
    fi
  fi

  # --------------------------------------------------------------
  # enlarge partition 2
  if [ "$enlarge" = "y" ]
  then
    myEnlarge
  fi

  ;; #end case mode restore
esac

Answer 8

The easiest solution I found was to do a backup of the original larger card using the dd commands outlined above and then restore the image to the smaller card using something like piwriter. dd may work as well... not sure. PiWriter returned an error since it ran out of room, but since the image didn't contain any actual data beyond the size of the smaller card it was just truncating empty sectors. I'm not sure what the implications of this are... partition may need checking or repair but I can verify that it worked when I put it into the Pi.

Answer 9

I use an old version of win32diskimager-RELEASE-0.1-r15-win32 to read the image, it creates an image of 4GB even from a 8GB SD card an then writes the image with the latest version of win32diskimager.

I use the older version because the old one will skip every error.

Answer 10

A simple way to do this to take an the image of the 32G SD card and reduce its size using the PiShrink script (which uses resize2fs and other tools). The PiShrink script needs to be run on a Linux based machine (which could be a Pi with access to the image file):

sudo pishrink.sh 32sdcard.img

The command has some options - some of which can help with further reducing the size of the image (e.g. -p "Remove logs, apt archives, dhcp leases and ssh hostkeys").

Answer 11

Look for a program called rpi-clone by Bill Wilson(?). It's on GitHub. You can run it on your pi and it can create a clone on another device, which can be a different size. It will even change the part-uuid's for you. It will not copy stuff that's no use copying, and you can exclude folders that you don't want to copy over. Later on you can update the clone and it will copy only what has changed.