Here’s why Apple’s new child safety features are so controversial

Technical Concerns

Apple's new child safety features have sparked significant debate due to their technical and policy implications. Riana Pfefferkorn highlights concerns about the transparency and accuracy of Apple's image analysis, noting the lack of sufficient documentation for thorough evaluation 1. She explains that Apple's system involves on-device hashing of images uploaded to iCloud Photos, comparing them against a database of known child sexual abuse material (CSAM) 1. This process, however, raises questions about oversight and the potential for misuse, as the system's inner workings remain largely opaque 2.

What they're doing is, again, this will be done on the client baked into the operating system and deployed for every iPhone running iOS 15 once that comes out around the world.

--- Riana Pfefferkorn

The complexity of Apple's approach, which includes human review of flagged images, adds another layer of controversy, especially given the potential for collateral damage and privacy concerns 3.

   

Scanning Controversies

The scanning methods employed by Apple have led to widespread controversy, particularly regarding potential government demands. Nilay Patel points out that consumers expect some level of scanning when data is uploaded to the cloud, but Apple's method is seen as overly complex and potentially expansive 4. Riana Pfefferkorn expresses concern that governments might pressure Apple to extend this technology beyond CSAM to other types of content, such as terrorist material or even politically sensitive images 4.

My prediction is that the next thing that Apple will be pressured to do will be to deploy the same thing for GIFCT as they are currently doing for the NCMEc database of hashes of CSAM.

--- Riana Pfefferkorn

The potential for misuse is further complicated by the lack of transparency in the databases used, which could include content beyond universally agreed-upon illegal material 5.