Queztaz/Creating an Offline Debian Mirror Repository

Creating an Offline Debian Mirror Repository

Preamble

Directions for creating a fully functional offline Debian mirror with most, but not all, packages available.

Creating the offline mirror

Overview

Install xr09's mirror script
Download Packages to folder
Generate package index list
Use the mirror

Mirror Script

Clone this repo: https://github.com/xr09/debian-mini-repo/blob/master/cron-debmirror

Edit the script and change the REPOS_DIR variable in the mirror.sh file. This should be the local path that will store the packages.

Edit the debmirror.conf file and set the host to a Debian mirror (i.e "ftp.debian.org"). I'd suggest using a third party mirror to mindful of Debian's bandwidth costs.

Edit the debmirror.conf file path in the mirror.sh script. The script will expect to find the file in /opt/ otherwise.

Sync the Mirror Repo to a local folder

Run the mirror script. This will begin downloading all the Debian packages.

./mirror.sh

Generate a package list

Everytime you re-synchronize your mirror, run this command to recreate the package.gz index file. Apt won't work without this.

The Packages.gz file must be located in the root folder of your offline mirror.

dpkg-scanpackages . /dev/null | gzip -9 > Packages.gz

Using the mirror

Add the mirror to your sources.list file. Be mindful of keeping remote repositories in this list. For stability I would suggest only leaving the offline repo in your sources file unless otherwise needed.

vim /etc/apt/sources.list

deb [trusted=yes] file:/mnt/Debian /

The file repository will be live at this point

apt update
apt install ranger

Source Code

I modified the mirror.sh script to use a SCRIPT_DIR variable. The debmirror.conf file has debian as the host. The crontab file is unchanged.

Mirror.sh

#!/usr/bin/env bash

SCRIPT_DIR=$(dirname "$(realpath "$1")")
REPOS_DIR="/mnt/Debian/"

debmirror --method=http --config-file=$SCRIPT_DIR/debmirror.conf --nosource $REPOS_DIR \
--ignore-release-gpg --nosource --no-check-gpg --postcleanup --allow-dist-rename \
--root=debian --rsync-extra=none -d stable   \
--exclude-deb-section='(games|debug|news|zope|electronics|comm|gnustep|haskell|ocaml|hamradio|gnu-r|science)' \
          --exclude='(/i19n/Translation-.*\.bz2)' \
          --exclude='(kfreebsd-.*)' \
          --include='(/python[1-9.]*-doc)' \
          --exclude='(.*java.*doc.*)' \
          --exclude='(.*debian.*reference.*)' \
          --exclude='(.*-dbg.*)' \
          --exclude='(/cairo-dock.*)' \
          --exclude='(/.*-doc)' \
          --include='(/(sqlite4|ansible|coffescript|geany|libjs|nginx|php|postgres|pypy|python-(django|flask|jinja|sql|werkzeug)).*doc)' \
          --include='(/.*(xorg|ruby).*doc.*)' \
      --include='(/firebird.*-doc)' \
      --include='(/python[1-9.]*-doc)' \
      --exclude='(openjdk.*doc.*)' \
          --exclude='(typo4-src.*)' \
          --exclude='(gdb-source.*)' \
          --exclude='(w4af-console.*)' \
          --exclude='(/calligra.*)' \
          --exclude='(/mono/.*)' \
          --exclude='(/texlive-(extra|lang)/.*)' \
          --exclude='(/atlas/.*)' \
          --exclude='(/childsplay-alphabet-sounds.*)' \
          --exclude='(/aegis/.*)' \
          --exclude='(/abinit/.*)' \
          --exclude='(/ardour.*/.*)' \
          --exclude='(/amsn/.*)' \
          --exclude='(/aolserver5/.*)' \
          --exclude='(/ant/.*)' \
          --exclude='(/adios/.*)' \
          --exclude='(/blender/.*)' \
          --exclude='(/bullet/.*)' \
          --exclude='(/visp/.*)' \
          --exclude='(/ampache-themes/.*)' \
          --exclude='(/alpine/.*)' \
          --exclude='(/antlr3/.*)' \
          --exclude='(/aspectj/.*)' \
          --exclude='(/advi/.*)' \
          --exclude='(/anjuta/.*)' \
          --exclude='(/amule/.*)' \
          --exclude='(/afterstep/.*)' \
          --exclude='(/asymptote/.*)' \
          --exclude='(/allegro4.4/.*)' \
          --exclude='(/ampache-themes/.*)' \
          --exclude='(/aspectc++/.*)' \
          --exclude='(/amarok/.*)' \
          --exclude='(/basemap/.*)' \
          --exclude='(/brise/.*)' \
          --exclude='(/boost1.54/.*)' \
          --exclude='(/root-system/.*)' \
          --exclude='(/c++-annotations/.*)' \
          --exclude='(/asio/.*)' \
          --exclude='(/wordnet/.*)' \
          --exclude='(/gcc-4.6/.*)' \
          --exclude='(/wine-gecko-1.4/.*)' \
          --exclude='(/ace/.*)' \
          --exclude='(/digikam/.*)' \
          --exclude='(/e17/.*)' \
          --exclude='(/abiword/.*)' \
          --exclude='(/maxima/.*)' \
          --exclude='(/mapnik/.*)' \
          --exclude='(libpython.*testsuite.*)' \
          --exclude='(/musixtex/.*)' \
          --exclude='(/magics\+\+/.*)' \
          --exclude='(ada-reference.*)' \
          --exclude='(pyxplot-doc.*)' \
          --exclude='(gnumach.*)' \
          --exclude='(/wims.*)' \
          --exclude='(/webgen.*)' \
          --exclude='(/libtao.*)' \
          --exclude='(/tao.*)' \
          --exclude='(/activiz.*)' \
          --exclude='(/agda.*)' \
          --exclude='(/debian-edu.*)' \
          --exclude='(/dff.*)' \
          --exclude='(/elastix.*)' \
          --exclude='(/fonts-ipamj.*)' \
          --exclude='(/fonts-baekmuk.*)' \
          --exclude='(/fonts-hanazono.*)' \
          --exclude='(/fonts-horai.*)' \
          --exclude='(/fonts-ipa.*)' \
          --exclude='(/fonts-kiloji.*)' \
          --exclude='(/fonts-kou.*)' \
          --exclude='(/fonts-nanum.*)' \
          --exclude='(/fonts-nanum.*)' \
          --exclude='(/fonts-takao.*)' \
          --exclude='(/fonts-yozvox.*)' \
          --exclude='(/fonts-mikachan.*)' \
          --exclude='(/fp-.*)' \
          --exclude='(/gnucash.*)' \
          --exclude='(/gnumed.*)' \
          --exclude='(/grass.*)' \
          --exclude='(/iceape.*)' \
          --exclude='(/instalation-guide-.*)' \
          --exclude='(/ooohg.*)' \
          --exclude='(/libosl-doc.*)' \
          --exclude='(/libreoffice-help.*)' \
          --exclude='(/ns2.*)' \
          --exclude='(/open-axiom.*)' \
          --exclude='(/openclipart.*)' \
          --exclude='(/paraview.*)' \
          --exclude='(/sofa-.*)' \
          --exclude='(/tagainijisho-dic-de.*)' \
          --exclude='(/texmacs.*)' \
          --exclude='(/lilypond.*)' \
          --exclude='(/vtk.*)' \
          --exclude='(/elmer.*)' \
          --exclude='(/felix.*)' \
          --exclude='(/debian-installer-6.0-)' \
          --exclude='(/evolution.*)'\
          --exclude='(/gimp-help-.*)'\
          --include='(/gimp-help-common.*)'\
          --include='(/gimp-help-en.*)'\
          --exclude='(/mbrola-de.*)'\
          --exclude='(/mgltools.*)'\
          --exclude='(/mecab.*)'\
          --exclude='(/context.*)'\
          --exclude='(/cuneiform.*)'\
          --exclude='(/celestia.*)'\
          --exclude='(/cclib.*)'\
          --exclude='(/nvidia-visual.*)'\
          --exclude='(/nvidia-cuda-toolkit.*)'\
          --exclude='(/nvidia-cuda-doc.*)'\
          --exclude='(/scribus-doc.*)'\
          --exclude='(/selfhtml.*)'\
          --exclude='(/etoys.*)'\
          --exclude='(/gmap.*)'\
          --exclude='(/icc-profiles.*)'\
          --exclude='(/picon.*)'\
          --exclude='(/fsl.*)'\
          --include='(/fslint.*)'\
          --exclude='(/cain.*)'\
          --exclude='(/cmuc.*)'\
          --exclude='(/cm-super.*)'\
          --exclude='(/jmol.*)'\
          --exclude='(/libjmol.*)'\
          --exclude='(/mypaint.*)'\
          --exclude='(/mysql-source.*)'\
          --exclude='(/makehuman.*)'\
          --exclude='(/mayavi2.*)'\
          --exclude='(/ovito.*)'\
          --exclude='(/opencascade.*)'\
          --exclude='(/libopencascade.*)'\
          --exclude='(/oce-draw.*)'\
          --exclude='(/liboce.*)'\
          --exclude='(/sunpinyin.*)'\
          --exclude='(/openscenegraph.*)'\
          --exclude='(/libopenscenegraph.*)'\
          --exclude='(/opendnssec.*)'\
          --exclude='(/openturns.*)'\
          --exclude='(/libopenturns.*)'\
          --exclude='(/volview.*)'\
          --exclude='(/libbvx.*)'\
          --exclude='(/libvibrant.*)'\
          --exclude='(/ncbi.*)'\
          --exclude='(/libncbi.*)'\
          --exclude='(/nwchem.*)'\
          --exclude='(/libwine-gecko.*)'\
          --exclude='(/whitedune.*)'\
          --exclude='(/xemacs21.*)'\
          --exclude='(/acl2.*)'\
          --exclude='(/axiom.*)'\
          --exclude='(/open-axiom.*)'\
          --exclude='(/scilab.*)'\
          --exclude='(/swe-.*)'\
          --exclude='(/libswe.*)'\
          --exclude='(/sitplus.*)'\
          --exclude='(/racket.*)'\
          --exclude='(/radiance.*)'\
          --exclude='(/regina-normal.*)'\
          --exclude='(/quantlib.*)'\
          --exclude='(/ruby-feedtools-doc.*)'\
          --exclude='(/ruby-activeldap-doc.*)'\
          --exclude='(/kde-l10n-.*)'\
          --exclude='(/quantum-espresso.*)'\
          --exclude='(/emacs23.*)'\
          --exclude='(/emboss.*)'\
          --exclude='(/jemboss.*)'\
          --exclude='(/lammps.*)'\
          --exclude='(/lazarus.*)'\
          --exclude='(/lcl.*)'\
          --exclude='(/installation-guide.*)'\
          --exclude='(/ibus-table-chinese.*)'\
          --exclude='(/tuxpaint.*)'\
          --exclude='(/tesseract.*)'\
          --exclude='(/tagainjisho.*)'\
          --exclude='(/fluid-soundfont.*)'\
          --exclude='(/freepats.*)'\
          --exclude='(/ferret.*)'\
          --exclude='(/festvox.*)'\
          --exclude='(/freecad.*)'\
          --exclude='(/festvox-.*)'\
          --exclude='(/festival.*)'\
          --exclude='(/frama-c.*)'\
          --exclude='(/fonts-cwtex.*)'\
          --exclude='(/freefem.*)'\
          --exclude='(/fonts-unfonts.*)'\
          --exclude='(/biomaj.*)'\
          --exclude='(/doc-linux-.*)'\
          --include='(/doc-linux-html.*)'\
          --include='(/doc-linux-text.*)'\
          --exclude='(/digikam-doc.*)'\
          --exclude='(/dotlrn.*)'\
          --exclude='(/gfxboot.*)'\
          --exclude='(/gcc-4.*-source.*)'\
          --exclude='(/gmt.*)'\
          --exclude='(/games-thumbnails.*)'\
          --exclude='(/pari-.*)'\
          --exclude='(/libpari.*)'\
          --exclude='(/eglib-source.*)'\
          --exclude='(/expeyes.*)'\
          --exclude='(/k3d.*)'\
          --exclude='(/gcompris/.*)'\
          --exclude='(/geotranz/.*)'\
          --exclude='(/linux-source-.*)'\
          --exclude='(/sweethome3d.*)'\
          --exclude='(/unidic-mecab/.*)'\
          --exclude='(/eclipse.*/.*)'\
          --exclude='(/insighttoolkit4/.*)'\
          --exclude='(/gap-tomlib/.*)'\
          --exclude='(/ko.tex-unfonts/.*)'\
          --exclude='(/openjdk-[0-9]-source.*)'\
          --exclude='(/openvrml/.*)'\
          --exclude='(/coq/.*)'\
          --exclude='(/mozc/.*)'\
          --exclude='(/norwegian/.*)'\
          --exclude='(/nuvola-icon-theme.*)'\
          --exclude='(/kiten/.*)'\
          --exclude='(/ding/.*)'\
          --exclude='(/live-manual/.*)'\
          --exclude='(/scratch/.*)'\
          --exclude='(/freevo/.*)'\
          --exclude='(/pinyin-database/.*)'\
          --exclude='(/invesalius/.*)'\
          --exclude='(/hydrogen-drumkits/.*)'\
          --exclude='(/guitarix/.*)'\
          --exclude='(/stardict-xmlittre/.*)'\
          --exclude='(/kmymoney.*)'\
          --exclude='(/pleiades.*)'\
          --exclude='(/kstars-data-extra-tycho2.*)'\
          --exclude='(/gcj.*)'\
          --exclude='(/libgjc.*)'\
          --exclude='(/latex-cjk.*)'\
          --exclude='(/scala.*)'\
          --exclude='(/groovy.*)'\
          --exclude='(/coinor-.*)' \
          --exclude='(/iceweasel-l10n-.*)' \
          --exclude='(/debian-installer-netboot-images.*)'

debmirror.conf

# Output options
$verbose=1;
$progress=1;
$debug=0;

$host="ftp.debian.org";
$user="anonymous";
$passwd="anonymous@";
$remoteroot="debian";
$download_method="http";
@sections="main,contrib,non-free";
@arches="amd64";

$omit_suite_symlinks=0;
$skippackages=0;

$i18n=0;
$getcontents=0;
$do_source=0;
$max_batch=0;

# Save mirror state between runs; value sets validity of cache in days
$state_cache_days=0;

# Security/Sanity options
$ignore_release_gpg=1;
$ignore_release=0;
$check_md5sums=0;
$ignore_small_errors=1;

# Cleanup
$cleanup=1;
$post_cleanup=1;

# Locking options
$timeout=300;

# FTP/HTTP options
$passive=1;

# set proxy if you need it
#$proxy="http://10.0.0.1:3128";

# Dry run
$dry_run=0;

# Don't keep diff files but use them
$diff_mode="use";

# The config file must return true or perl complains.
# Always copy this.
1;

cron-debmirror

# place this file at /etc/cron.d/debmirror


# change the paths if you need to

# mirror every day at 5am
0 5 * * * root /PATH/TO/mirror.sh >> /var/log/debmirror-debian.log 2>&1