| View previous topic :: View next topic |
| Author |
Message |
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
 Posted: Sat May 02, 2015 3:41 pm Post subject: |
 |
|
I misposted this (checked it in my records, the screencasts and packet dumps: mea cupla) on:
Posted: Fri Apr 17, 2015 1:16 am
https://forums.gentoo.org/viewtopic-t-999436.html#7733646
(and I'm vacating it from there)
---
There are, currently, some strangely-induced, possibly by phpBB, possibly by other causes, spaces, and not exactly plain blanks, in the script above....
Anyway, unfinished, but probably better way to study this case for which the Air-Gapped Install is the remedy, is from:
A Schmoog Intrusion
http://www.croatiafidelis.hr/foss/cap/cap-140516_SchI/
and also, how I made the video that you should be able to see it there in HTML5, is what you can, soon, finishing the upload of files (oh but I'm never sure under my dear censors in Croatia...)... hopefully finishing the upload, so also this might be of some worth:
EDIT: Should all be uploaded properly, pgp-signed and all.
A Demo with a Few Tips on Simple Video Manipulation
http://www.croatiafidelis.hr/foss/cap/cap-140516_SchI/howto.php
Cheers! |
|
| Back to top |
|
 |
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Sat May 02, 2015 4:05 pm Post subject: |
|
|
I would really like to point readers here to a Gentoo Tip that made my day today.
It is in indirect relation with the issues here, and I don't care that some would question that relation.
It's someone got lots of newbies to install Larry the Big Guy Oracle's Java, and I don't like that.
People building Air-Gapped can have lots of their defences hollowed out or worse by intrusive programs like Larry's, who took over also Mysql and some other FOSS stuff...
So this is the tip:
How to avoid Oracle's JAVA
https://forums.gentoo.org/viewtopic-t-1015568.html
Thanks, my folks! |
|
| Back to top |
|
|
miroR
l33t
Joined: 05 Mar 2008
Posts: 826
|
| Posted: Mon Jan 25, 2016 9:55 am Post subject: |
|
|
I have been using this method that I have, with a lot of struggling and imperfections on my part, successfully put together, and explained, in this topic that you are reading (
Air-Gapped Gentoo Install, Tentative
https://forums.gentoo.org/viewtopic-t-987268.html
)
.
There are a few additional sub-methods that ought to be followed, to have your own mirror updated without too many unnecessary packages. And also these methods relieve the servers you download from.
For true Air-Gapped, you need a local mirror only as complete as you might in some probability need. You don't need all the 180GB or more that it be.
It's easy to rsync-download with some packages set in an exclusion list and in such way not downloaded. Which ones depends on your personal preferences.
E.g. I don't like anything Google, anything at all, and when I can, I live without Schmoog the Schmoogle. So one of my choices will generally be to put all the chrom* in the exclusion list to give to rsync on the command line.
But I'll show it to you with the (primitive) scripts that I will use this morning as I update my Gentoo mirror.
Surely you need to choose your own mirror, unless you live, like me, in relative proximity of Germany, in which case you could probably use the scripts literally.
But whichever the way, please don't even try doing it unless you do it at your own responsability. I give no warranties. I only say that these methods work for me.
So, if you need to, consult the Gentoo mirror list and replace 'de-mirror.org' with the mirror more appropriate for your side of the world.
In the bottom command, the file files-exclude.ls-1 is what I already put in the list the last time I updated my local mirror. The ls-1 (that's a '1', one) is there because it is a listing just like what you get when you:
Here is the command to list the remote mirror and save the listing into a timestamped file:
| Code: |
#!/bin/bash
rsync -nav --exclude-from=files-exclude.ls-1 rsync://de-mirror.org/gentoo/distfiles/ 2>&1 | tee /mnt/sde1/rsync_de-mirror.org-nav_`date +%y%m%d_%H%M`.txt
|
See 'man rsync' (and 'man date'). What the above does, is it gives you the listing, in a file like this:
(Pls. notice the string 'receiving incremental file list' in it. We'll need that later.)
| Code: |
$ cat rsync_de-mirror.org-nav_160125_0648.txt >> /Cmn/mr/Gen_160125_Air-Gap_rsync_mirror.txt
_ ___ _ ___ _ _ ____ _____
/ |( _ ) / | |_ _|_ __ | |_ ___ _ __ _ __ ___| |_ / ___|| ____|
| |/ _ \/\ | | || '_ \| __/ _ \ '__| '_ \ / _ \ __| \___ \| _|
| | (_> < | | || | | | || __/ | | | | | __/ |_ ___) | |___
|_|\___/\/_| |___|_| |_|\__\___|_| |_| |_|\___|\__| |____/|_____|
Welcome to mirror.eu.oneandone.net, a service provided by 1&1 Internet SE
--------------------------------------------------------------------------
This mirror is available via HTTP, FTP and RSYNC at:
* <http://mirror.eu.oneandone.net/>
* <ftp://mirror.eu.oneandone.net/>
* <rsync://mirror.eu.oneandone.net/>
... [ 44 lines snipped here ] ...
receiving incremental file list
drwxr-xr-x 3,325,952 2016/01/25 05:52:01 .
-rw-r--r-- 104,051 2014/11/29 14:52:55 0.3.tar.gz
-rw-r--r-- 1,555,942 2013/06/21 21:50:20 0.8-b3.tar.gz
-rw-r--r-- 174,494 2013/01/28 09:51:39 0.9.2.tar.gz
-rw-r--r-- 736 2015/03/20 16:13:40 0001-x86-Put-COPY3_IF_LT-under-HAVE_6REGS.patch.gz
-rw-r--r-- 161,749 2013/05/30 16:37:46 010FC8BD229B7F68C8C4D5BDE399475373096601-non-schema.jar
-rw-r--r-- 359,397 2015/02/11 15:02:16 01CD242F06F6F7E4E61C9E05ABBE07318E501D51-org.eclipse.mylyn.wikitext.core_1.9.0.20131007-2055_nosignature.jar
-rw-r--r-- 24,444 2015/10/04 22:58:52 03-infinality-2.6-2015.10.04.patch.xz
-rw-r--r-- 27,056 2015/11/29 15:49:40 03-infinality-2.6.2-2015.11.28.patch.xz
-rw-r--r-- 6,424 2014/01/19 12:05:52 034-0010-module-setup.sh-add-comments.patch.bz2
... [ 74687 lines snipped here ]...
-rw-r--r-- 339,898 2003/03/24 17:22:04 zssh-1.5a.tgz
-rw-r--r-- 344,964 2003/09/24 05:39:17 zssh-1.5c.tgz
-rw-r--r-- 245,592 2010/09/20 14:36:12 zsync-0.6.2.tar.bz2
-rw-r--r-- 195,927 2012/07/08 09:09:33 zukini-20120806.zip
-rw-r--r-- 220,148 2013/12/17 14:43:38 zukitwo-2013.12.10.tar.xz
-rw-r--r-- 436,590 2015/04/28 01:03:34 zukitwo-2014.10.22.zip
-rw-r--r-- 582,120 2015/06/01 09:10:01 zuluCrypt-4.7.6.tar.bz2
-rw-r--r-- 550,309 2015/09/01 06:56:52 zuluCrypt-4.7.7.tar.bz2
-rw-r--r-- 769,209 2006/05/31 06:09:39 zvbi-0.2.22.tar.bz2
-rw-r--r-- 1,047,761 2013/08/28 17:06:14 zvbi-0.2.35.tar.bz2
-rw-r--r-- 15,256,377 2014/02/03 02:04:30 zygrib-6.2.3.tgz
-rw-r--r-- 15,287,496 2015/03/25 03:11:23 zygrib-7.0.0.tgz
-rw-r--r-- 517,620 2013/05/21 15:20:18 zygrib-cities_0-300.txt.gz
-rw-r--r-- 495,119 2013/05/21 15:20:18 zygrib-cities_1k-3k.txt.gz
-rw-r--r-- 512,733 2013/05/21 15:20:19 zygrib-cities_300-1k.txt.gz
-rw-r--r-- 851 2013/06/01 10:34:55 zygrib-icon.png
-rw-r--r-- 105,735,981 2013/05/21 15:52:01 zygrib-maps2.4.tgz
-rw-r--r-- 4,185,453 2015/02/26 01:38:10 zynaddsubfx-2.5.0.tar.gz
-rw-r--r-- 4,214,568 2015/07/04 21:17:16 zynaddsubfx-2.5.1.tar.gz
-rw-r--r-- 4,572,973 2015/11/14 04:01:41 zynaddsubfx-2.5.2.tar.gz
-rw-r--r-- 654,842 2007/03/18 18:19:10 zziplib-0.13.49.tar.bz2
-rw-r--r-- 685,418 2010/12/30 10:01:48 zziplib-0.13.60.tar.bz2
-rw-r--r-- 685,770 2012/03/12 00:57:44 zziplib-0.13.62.tar.bz2
-rw-r--r-- 11,370 2010/03/02 21:10:14 zzuf-0.13-zzcat-zzat-rename.patch.bz2
-rw-r--r-- 461,498 2010/01/31 13:30:00 zzuf-0.13.tar.gz
sent 2,753 bytes received 1,995,000 bytes 363,227.82 bytes/sec
total size is 182,543,535,057 speedup is 91,374.43 (DRY RUN)
|
It is neither: not too heavy on the server, nor such long downloading time, to take out of the above huge listing just those that are larger than 100,000,000 that is: which are of size greater than 100MB, and pick among those just what you still do need, and delete from there (i.e. exempt them from the files-exclude.ls-1 of the files not to be downloaded). And to let rsync update whichever smaller files, whether you need them or not (It's some 75,000 files! Who wants to pick from every single one among that many?)... And anyway only the big ones really take time to download!
| Code: |
$ grep -A1000000 'receiving incremental file list' rsync_de-mirror.org-nav_160125_0648.txt | wc -l
74726
$ grep -A1000000 'receiving incremental file list' rsync_de-mirror.org-nav_160125_0648.txt | head -74723 | tail -74721 > rsync_de-mirror.org-nav_160125_0648.txt.ls-l
$
|
That's 'ls-l' (lowercase L), for 'long listing', just like in 'ls -l'.
I hope you remember the string we used here. And now notice that 74726-3=74723. And then 74723-2=74721. See 'man head' and 'man tail' if unclear.
That gets us the long listing of files to be updated without the talk in top and bottom.
Out of all the listing, we are really interested only in lines like (near the bottom of that listing):
| Code: |
-rw-r--r-- 105,735,981 2013/05/21 15:52:01 zygrib-maps2.4.tgz
|
And this:
| Code: |
cat rsync_de-mirror.org-nav_160125_0648.txt.ls-l | egrep '[0-9]{3},[0-9]{3},[0-9]{3}' > rsync_de-mirror.org-nav_160125_0648.txt.ls-l-S
|
will make such a list for us. "-S" is for size. See 'man egrep'.
| Code: |
$ cat rsync_de-mirror.org-nav_160125_0648.txt.ls-l-S
-rw-r--r-- 571,228,108 2015/03/13 23:05:02 0ad-0.0.18-alpha-unix-data.tar.xz
-rw-r--r-- 102,215,680 2010/12/20 18:54:40 6.2.2.0-TIV-TSMBAC-LinuxX86.tar
... [ 38 lines snipped here ] ...
-rw-r--r-- 155,616,187 2015/01/13 00:36:39 amd-catalyst-omega-14.12-linux-run-installers.zip
-rw-r--r-- 1,023,118,640 2015/11/20 03:56:27 amd64-debug-libreoffice-5.0.3.2.tar.xz
-rw-r--r-- 475,584,701 2015/11/20 04:10:43 amd64-debug-libreoffice-gnome-java-5.0.3.2.xd3
-rw-r--r-- 258,628,239 2015/06/04 19:00:00 android-studio-ide-141.1980579-linux.zip
-rw-r--r-- 100,065,044 2014/10/18 13:25:34 appliance-1.28.1.tar.xz
-rw-r--r-- 207,172,456 2008/07/12 17:10:22 axiom-may2008-src.tgz
-rw-r--r-- 129,694,711 2013/08/17 22:59:22 basemap-1.0.7.tar.gz
-rw-r--r-- 123,217,291 2013/07/05 15:52:22 bndlib-2.1.0.tar.gz
-rw-r--r-- 138,928,992 2014/07/04 09:11:50 calligra-2.8.5.tar.xz
-rw-r--r-- 138,966,248 2014/12/04 15:52:31 calligra-2.8.7.tar.xz
-rw-r--r-- 194,239,580 2015/08/19 18:56:06 calligra-2.9.6.tar.xz
-rw-r--r-- 194,348,264 2015/08/29 22:55:16 calligra-2.9.7.tar.xz
-rw-r--r-- 125,697,465 2014/12/19 18:41:04 charm-6.6.1.tar.gz
-rw-r--r-- 432,281,684 2016/01/20 21:25:25 chromium-48.0.2564.82.tar.xz
-rw-r--r-- 452,249,352 2016/01/16 09:27:34 chromium-49.0.2623.0.tar.xz
-rw-r--r-- 122,543,145 2006/07/21 03:54:46 cpma-mappack-full.zip
... [ 20 lines snipped here ] ...
-rw-r--r-- 202,964,824 2015/03/11 17:18:41 fglrx-installer_15.200.orig.tar.gz
-rw-r--r-- 146,418,413 2011/01/10 23:01:55 fillets-ng-data-1.0.0.tar.gz
-rw-r--r-- 181,383,793 2015/11/02 14:39:36 firefox-38.4.0esr.source.tar.bz2
-rw-r--r-- 181,371,970 2015/12/15 17:13:06 firefox-38.5.0esr.source.tar.bz2
-rw-r--r-- 175,414,296 2015/12/14 20:29:06 firefox-43.0.source.tar.xz
-rw-r--r-- 134,835,922 2008/11/30 22:14:31 fluid-soundfont_3.1.tar.gz
... [ 40 lines snipped here ] ...
-rw-r--r-- 123,706,294 2007/02/23 11:08:17 legends_linux-0.4.1.42.run
-rw-r--r-- 123,964,866 2007/09/17 02:08:37 legends_linux-0.4.1.43.run
-rw-r--r-- 164,742,068 2015/10/27 22:34:09 libreoffice-4.4.6.3.tar.xz
-rw-r--r-- 167,009,360 2015/10/24 20:26:06 libreoffice-5.0.3.2.tar.xz
-rw-r--r-- 167,305,516 2015/12/13 14:09:27 libreoffice-5.0.4.2.tar.xz
-rw-r--r-- 117,081,330 2009/08/28 01:44:03 libstdcxx-39.tar.gz
-rw-r--r-- 113,132,379 2015/04/07 20:31:04 libvpx-testdata-1.4.0.tar.bz2
... [ 105 lines snipped here ] ...
-rw-r--r-- 159,265,180 2015/06/19 08:26:32 xmind-portable-3.5.3.201506180105.zip
-rw-r--r-- 984,854,761 2015/10/08 10:50:15 xonotic-0.8.1.zip
-rw-r--r-- 105,735,981 2013/05/21 15:52:01 zygrib-maps2.4.tgz
|
Do you see the chromium there, which I would only be interested to install if I needed to see how my pages work (if I had time to write web pages)? But otherwise I tell everybody: Dillo is the safest, but really minimal browser, incomplete. And Mozilla seems to have truly started to care for the privacy of its users, unless I've mistaken to have started trusting them. But I do trust them again!, and so: Firefox I recommend. See why here (warning: useless and, at least some, false accusations against me there is best for the reader to skip; go for the information provided in the links therefrom, to what Mozilla developers told me):
More non-Decryptables (from Mozilla Cloud)
https://forums.gentoo.org/viewtopic-t-1034140.html#7847998
.
And so, e. g. firefox-<...> I do need to update. Also libreoffice-<...> I need (except I don't need the amd64-debug-libreoffice-<...> debugging packages).
| Code: |
$ cp -iav rsync_de-mirror.org-nav_160125_0648.txt.ls-l-S rsync_de-mirror.org-nav_160125_0648.txt.ls-l-Sr
|
('r' in <...>-Sr is for real. Meaning we'll really use that one, not the one without r. Have to make your abbreviations on some mnemonic principle.)
So after the libreoffice-<...> and firefox-<...> I manually removed from that listing in the copied file rsync_de-mirror.org-nav_160125_0648.txt.ls-l-Sr, see:
| Code: |
$ diff rsync_de-mirror.org-nav_160125_0648.txt.ls-l-S rsync_de-mirror.org-nav_160125_0648.txt.ls-l-Sr
79,81d78
< -rw-r--r-- 181,383,793 2015/11/02 14:39:36 firefox-38.4.0esr.source.tar.bz2
< -rw-r--r-- 181,371,970 2015/12/15 17:13:06 firefox-38.5.0esr.source.tar.bz2
< -rw-r--r-- 175,414,296 2015/12/14 20:29:06 firefox-43.0.source.tar.xz
125,127d121
< -rw-r--r-- 164,742,068 2015/10/27 22:34:09 libreoffice-4.4.6.3.tar.xz
< -rw-r--r-- 167,009,360 2015/10/24 20:26:06 libreoffice-5.0.3.2.tar.xz
< -rw-r--r-- 167,305,516 2015/12/13 14:09:27 libreoffice-5.0.4.2.tar.xz
|
we, next, awk for the column $5 of each line (the fifth column contains the name of the file.
See 'man awk'. We can awk that straight into the already existing (in my case) or new (but will exist the next time if you start using this Aid-Gapped method), files-exclude.ls-1:
| Code: |
$ cat rsync_de-mirror.org-nav_160125_0648.txt.ls-l-Sr | awk '{ print $5 }' >> files-exclude.ls-1
|
And now I can start updating my system and it'll take much less then it would if I hadn't done that exclusion list. And I will still very probably have all that I need for my minimalist [*], but pretty safe, and cloneable[**], Air-Gapped Gentoo system.
I use a script like:
| Code: |
#!/bin/bash
rsync -nav --exclude-from=files-exclude.ls-1 rsync://de-mirror.org/gentoo/distfiles/ distfiles/ 2>&1 | tee /mnt/sde1/rsync_de-mirror.org-nav_`date +%y%m%d_%H%M`.txt
echo "Enter for real rsync'ing"
read FAKE
rsync -av --exclude-from=files-exclude.ls-1 rsync://de-mirror.org/gentoo/distfiles/ distfiles/ 2>&1 | tee /mnt/sde1/rsync_de-mirror.org-av_`date +%y%m%d_%H%M`.txt
|
(The read FAKE is just to give me time to check how the script fared, and hit Enter; it doesn't read anything. Well it could, but I don't use it for reading anything  .)
And once I'm done, I use the little used nowadays, but probably the safest way to transfer data when they are not too sizeable: BluRay data discs. I use them to copy the new files over into another mirror which is always offline, and which my real Air-Gapped system accesses, which is the system that I update and rebuild, and which I clone off. Where I'm writing and posting this from is always a cloned system, never the Air-Gapped master.
And the Air-Gapped master of mine is not courious about internet, it couldn't even care to go there. It cares only for my security, and it gets everything from its clones anyway .
And just one more thing. In case there are any problems with the scripts, see if you have, by error of some kind, in the system, or wherever that it be from, some:
phpBB Strange White Space problem
https://forums.gentoo.org/viewtopic-t-1032010.html
Cheers!
---
[*] The best way to describe it is: minimalist, because I don't use a system with systemd's best friend: dbus. See:
Uninstalling dbus and *kits (to Unfacilitate Remote Seats)
https://forums.gentoo.org/viewtopic-t-992146.html
And also see how among the people that understand the horror unleashed by systemd against the true Unix nature of FOSS Linux, there is a growing interest for such minimal, no-dbus no-poetterware installation at:
Upgrade to Devuan and minimalism
https://git.devuan.org/dev1fanboy/Upgrade-Install-Devuan/wikis/Upgrade-to-Devuan-and-minimalism
[**] See:
Postfix smtp/TLS, Bkp/Cloning Mthd, Censorship/Intrusion
https://forums.gentoo.org/viewtopic-t-999436.html#7613044 |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
|
| Back to top |
|
|
gentoo-freak
n00b
Joined: 11 Sep 2009
Posts: 16
|
| Posted: Thu Apr 21, 2016 1:00 pm Post subject: Indeed |
|
|
#JFR |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
