How to use rsync on a Mac / macOS to sync files with a locally mounted FAT32 disk, and deal sensibly with Unicode characters, umlauts, accents, etc.

So, there’s a lot of discussion of the above problem; you are on a Mac, and you are trying to “rsync” files from a source to a destination, and it complains about umlauts, complex quote characters, or other Unicode.

If you are like me, you’ll waste time trying to work out what encodings and codepages are applicable to the FAT32 filesystem, assuming that that’s the correct approach; it turns out not. Some of them will also claim that FAT32 has native code-page support, but that the Long File Name support apparently requires UTF-16, or perhaps UCS2.

You’ll also run into people earnestly explaining that Mac HFS requires everything to be put into “fully” UTF-8 decomposed form before being turned into the filename, and that this is the problem. But then they quit on giving you a solution.

You may even find people talking about solving a similar problem on Mac-to-Linux systems (eg: NAS) – or in passing you will learn that time resolution on FAT32 is about 2 seconds, so you can’t sensibly use time-comparison without adding some kind of time-fuzz – but neither of these will solve the filename issue.

You are probably also annoyed by half a dozen blogposts by geeks who hack around this with Perl, or who attack the questioner to say WHY HAVEN’T YOU JUST REFORMATTED THE DISK TO BE <ext4, zfs, ntfs> YOU ENORMOUS BABOON? (tip: because the camera still has to read it, too)

The actual answer – the one you were looking for – is really simple.

  1. brew install rsync # because the macOS rsync is old
  2. rsync -av –iconv=utf-8-mac,utf-8-mac –size-only /from/dir/ /to/fat32dir/

It seems to basically boil down to convincing rsync to decompose the filenames that it [previously created and] reads back from the FAT32 filesystem, before comparing them to what it can see in the HFS filesystem.

So, tell it on both sides that it has to use “utf-8-mac”, thereby levelling the filename-comparison playing field. Suddenly everything works fast again. It might be possible to tune it even further, to simplify more, but I’m so pleased with this that I just want to share.

Leave a comment if this worked for you and you found it / appreciated the writeup.

Installing a mkcert certificate into Tor Browser

Update #3: 3 July 2019

  • Now you have to disable OCSP too

Update #2: 20 June 2019

  • With recent updates to TorBrowser, the storage of local certificate roots is somewhat more complex, but can be performed as follows.
  • Be aware: whilst you are configured like this, some privacy-protective aspects of TorBrowser are reduced or switched-off; ensure that you restore your browser to defaults before using it for privacy-protecting purposes.

Installation part 1

  • open about:config
  • click “I accept the risk!”
  • search for “security.nocertdb” in the box provided
    • if the “value” field says “default”/”true”, then:
      • double-click on it to make it “modified”/”false”
  • search for “browser.privatebrowsing.autostart” in the box provided
    • if the “value” field says “default”/”true”, then:
      • double-click on it to make it “modified”/”false”
  • search for “security.ssl.enable_ocsp_stapling” in the box provided
    • if the “value” field says “default”/”true”, then:
      • double-click on it to make it “modified”/”false”
  • dismiss the about:config tab
  • IMPORTANT: NOW RESTART TOR BROWSER

Installation part 2

  • open Menu > TorBrowser > Preferences > Privacy & Security
  • scroll down to Security > Certificates
  • uncheck Query OCSP responder servers to confirm the current validity of certificates
  • click “View Certificates”
  • select “Authorities” tab
  • click “Import”, select your “rootCA.pem” file, click “Open”
    • Popup: ensure that “Trust this CA to identify websites” is ticked/enabled
  • click “Ok”
  • check that “mkcert development CA” now appears in the list of authorities
  • navigate to the target URL

Uninstalling

  • When you are eventually finished with your certificate
    • Uninstall/remove the certificate, using the same dialogues
    • Reverse the about:config changes which you performed above
    • IMPORTANT: RESTART TOR BROWSER

Dark Web vs: Deep Web – 2018 edition

There’s a lot of confusion out there re: dark web vs: deep web — and it’s not going to get any better; I feel that the only way for humanity to move forward on this matter is for both terms to die, or at least diffuse until they are meaningless in much the same way that “cyber” means “network security related” and “cloud” means “other people’s (rented) computers“.

Nonetheless, the question of technical definition still remains, and this is what I hew to:

  • deep web: traditionally: that part of the web which is not findable via popular search engines such as Google; this may be because the content is held behind authentication, or because indexing has been requested-disabled via the “robots.txt” mechanism. The former, of course, includes almost the entirety of Facebook, not to mention your personal email accounts, your bank accounts, and so forth. People who like to hype-up the “deep web” generally neglect to point out these mundanities.
  • dark web: traditionally: that part of the web which is not accessible without “special software”, although the definition of “special” is apt to change with time because at one point a “Graphical Web Browser” like Mosaic was considered to be “special”, since the web at that point was entirely text-based. Generally the speaker is referring to Tor Onion Networking (formerly “Hidden Services”) when they want to talk about the “dark web”, and when they do so it is invariably to scare the listener with tales of evil dark-web websites such as Facebook, ProPublica, and the New York Times, from all of which one may easily purchase drugs and guns by sending them something called “Bitcoin”.
  • deep dark web: accessing your password-protected Facebook account via the Facebook “Onion Site”. Usage: intentionally silly.
  • intellectual dark web: the speaker/writer is alluding to a cabal of intellectual whose opinions he/she does not agree with, or hopes that the reader will be fearful of, in order to promote the lectures/clickbait that the speaker/writer is promoting.

This latter is particularly interesting, because it is a recent (2018?) innovation, and hopefully suggests that “deep web” is already on the path towards post-modern meaninglessness.

Any speaker or writer who illustrates either the “deep web” or the “dark web” with a picture of an iceberg, should not be considered credible.

Irony is a dish best served cold

 

Test Image

National Loyalty Card III – China takes it seriously:

https://www.wsj.com/articles/in-sign-of-resistance-chinese-balk-at-using-apps-to-snitch-on-neighbors-1514566110

The “Safe Zhejiang” app enables users to notify authorities of problems ranging from leaky drains and domestic disputes to traffic violations and illegal publications, in text or photographic form, as long as the informants reveal their location and identity.

In exchange, they get perks including discounts at upmarket coffee shops and coupons for taxi-hailing and music-streaming services, as well as for the Alipay online-payment system, run by the financial affiliate of local tech giant Alibaba Group Holding Ltd.

Previously:

Random Snippets of AberMUD-1 Source Code in B

Rebooting Dropsafe: March 6th 2017

Notes:

Well, today would have been Dad’s 98th birthday. I’ll drink a small toast to him, later.

Project Status:

Since I don’t really have a system for posting here at the moment, I think I’ll start with a brief list of everything that I think I am currently doing:

  • Raspberry Pi Rig: currently:
    • 6x RPi 3b in a rack, as a general compute surface
    • 1x RPi 3b in a fan-case, as rig controller/master
    • 1x RPi 3b in a passive case, as backup
  • Mana Project
    • RPi 1 for port of Mana; have finally obtained second wifi adapter
  • Soldering Project
    • RPi ZeroW + Pimoroni “Scroll Phat” dot-matrix display
    • Build & then use for
  • Half-A-Gig Onion
    • On hold, awaiting bandwidth.
  • EOTK
    • Stable, works, is remarkably quick
    • Goal is to ship stable & feature-complete 1.x version then start on 2.0
      • 2.x version will move more rewriting to Lua to reduce dependencies
  • Resurrecting old GPU Rig
    • Inherited 3x Radeon 5870 from JC
    • Unfortunately, Ubuntu stopped loving Radeon in v14 Trusty Tahir
    • Open-source Radeon drivers do not cut the mustard
      • Therefore: adventures in legacy linux and stale proprietary driver code
  • Java Experiments
    • Stalled by EOTK but that’s okay, it needed some thinking-time
  • Garden
    • Total mess.
    • Need to spend a few days sawing and stacking lumber for next winter, to dry out over the summer
    • Also need to push(@compost, shift(@compost)) which will be hard work
  • Work
    • Can wait until after summer.

Setting up on dropsafezeahmyho.onion using EOTK

https://dropsafe.dropsafezeahmyho.onion/

And into its 16th year…

…testing a new server.