Abstract: Around the world, legislators and regulators are scrutinising the power accumulated by firms selling and orchestrating digital infrastructures from operating systems and hardware to cloud services and social networks. New digital rights and obligations are emerging, such as in the EU's proposed Digital Markets Act, Digital Services Act and Data Act. At the same time, more countries than ever have passed comprehensive privacy laws in the face of evolving threats, whilst new digital rights, such as interoperability, portability or repair, are emerging. States have been blindsided by how difficult it is to exercise power and influence over decisions of predominant digital infrastructures, even in state-centric areas such as public health. Seen as a whole, these are thought to bring a range of tensions between challenges like privacy, security, competition, content moderation, and democratic control. In this talk, I will examine these trade-offs in the context of regulation and practice. Who is involved in framing this debate, and to what ends? How should we openly and rigorously think about such tensions? What kind of knowledge and institutions do we need to navigate these waters? At this important junction, I hope to give the audience a model of where we are facing today, so we can better decide together on what kind of digital societies we might want, and how to work together on bringing it into being and maintaining it.

Biography: Michael Veale is Associate Professor of Digital Rights and Regulation at the Faculty of Laws, University College London. He researches and publishes across disciplinary venues, focussing on the intersection of technology, policy and society around issues raised by machine learning, privacy-enhancing technologies, online tracking and data rights. His research is regularly cited by journalists, policymakers, courts and regulators, and he has authored a number of policy reports for organisations such as the European Commission, the Commonwealth Secretariat, the Royal Society, and the Law Society of England and Wales.

[Paper]
[Artifact]

Cyber deception is emerging as a promising approach to defending networks and systems against attackers and data thieves. However, despite being relatively cheap to deploy, the generation of realistic content at scale is very costly, due to the fact that rich, interactive deceptive technologies are largely hand-crafted. With recent improvements in Machine Learning, we now have the opportunity to bring scale and automation to the creation of realistic and enticing simulated content. In this work, we propose a framework to automate the generation of email and instant messaging-style group communications at scale. Such messaging platforms within organisations contain a lot of valuable information inside private communications and document attachments, making them an enticing target for an adversary. We address two key aspects of simulating this type of system: modelling when and with whom participants communicate, and generating topical, multi-party text to populate simulated conversation threads. We present the LogNormMix-Net Temporal Point Process as an approach to the first of these, building upon the intensity-free modeling approach of Shchur et al. to create a generative model for unicast and multi-cast communications. We demonstrate the use of fine-tuned, pre-trained language models to generate convincing multi-party conversation threads. A live email server is simulated by uniting our LogNormMix-Net TPP (to generate the communication timestamp, sender and recipients) with the language model, which generates the contents of the multi-party email threads. We evaluate the generated content with respect to a number of realism-based properties, that encourage a model to learn to generate content that will engage the attention of an adversary to achieve a deception outcome.

[Artifact]

[Paper]


An enterprise today deploys multiple security middleboxes such as firewalls, IDS, IPS, etc. in its network to collect different kinds of events related to threats and attacks. These events are streamed into a SIEM (Security Information and Event Management) system for analysts to investigate and respond quickly with appropriate actions. However, the number of events collected for a single enterprise can easily run into hundreds of thousands per day, much more than what analysts can investigate under a given budget constraint (time). In this work, we look into the problem of prioritizing suspicious events or anomalies to analysts for further investigation. We develop SIERRA, a system that processes event logs from multiple and diverse middleboxes to detect and rank anomalous activities. SIERRA takes an unsupervised approach and therefore has no dependence on ground truth data. Different from other works, SIERRA defines contexts, that help it to provide visual explanations of highly-ranked anomalous points to analysts, despite employing unsupervised models. We evaluate SIERRA using months of logs from multiple security middleboxes of an enterprise network. The evaluations demonstrate the capability of SIERRA to detect top anomalies in a network while outperforming naive application of existing anomaly detection algorithms as well as a state-of-the-art SIEM-based anomaly detection solution.

[Artifact]

[Artifact]

Abstract: Since the disclosure of Spectre attacks in 2018, both academia and industry have made considerable efforts to defend against all variants of transient execution attacks, the class of attacks to which Spectre belongs to. This presentation will give an update on the state of the art in 2022 to answer the title question (spoiler alert: the answer is NO) and present some of the remaining open problems in the area of transient execution attacks.

Biography: Tamara Rezk is a research director at Inria Sophia Antipolis, France. Her main research interests are in the field of system security. She has supervised Ph.D. students on topics such as static and dynamic security analyses, web security, formal methods for security, secure compilation, and provable cryptography. She currently focuses her research on defenses and principled methods to deal with microarchitectural attacks, and web application attacks.

[Paper]
[Artifact]

An encrypted search algorithm (ESA) allows a user to encrypt its data while preserving the ability to search over it. As all practical solutions leak some information, cryptanalysis plays an important role in the area of encrypted search. Starting with the work of Islam et al. (NDSS'12), many attacks have been proposed that exploit different leakage profiles under various assumptions. While these attacks improve our understanding of leakage, it can sometimes be difficult to draw definite conclusions about their practical performance. This is due to several reasons, including a lack of open-source implementations (which are needed to reproduce results), empirical evaluations that are conducted on restricted datasets, and in some cases reliance on relatively strong assumptions that can significantly affect accuracy. In this work, we address these limitations. First, we design and implement LEAKER, an open-source framework that evaluates the major leakage attacks against any dataset and that we hope will serve the community as a common way to evaluate leakage attacks. We identify new real-world datasets that capture different use cases for ESAs and, for the first time, include real-world user queries. Finally, we use LEAKER to systematically evaluate known attacks on our datasets, uncovering sometimes unexpected properties that increase or diminish accuracy. Our evaluation shows that some attacks work better on real-world data than previously thought and that others perform worse.

[Paper]
Privacy is a huge concern for cryptocurrencies and blockchains as most of these systems log everything in the clear. This has resulted in several academic and industrial initiatives to address privacy. Starting with the UTXO model of Bitcoin, initial works brought confidentiality and anonymity to payments. Recent works have expanded to support more generalized forms of private computation. Such solutions tend to be highly involved as they rely on advanced cryptographic primitives and creative techniques to handle issues related to dealing with private records (e.g. concurrency and double spending). This situation makes it hard to comprehend the current state-of-the-art, much less build on top of it. To address these challenges, we develop a systematization of knowledge for privacy-preserving solutions in blockchain. To the best of our knowledge, our work is the first of its kind. After motivating design challenges, we devise two systematization frameworks---the first as a stepping stone to the second---and use them to study the state-of-the-art. For our first framework, we study the zero-knowledge proof systems used in surveyed solutions, based on their key features and limitations. Our second is for blockchain privacy-preserving solutions; we define several dimensions to categorize the surveyed schemes and, in doing so, identify two major paradigms employed to achieve private computation. We go on to provide insights to guide solutions' adoption and development. Finally, we touch upon challenges related to limited functionality, practicality, and accommodating new developments.

[Artifact]

[Paper]
[Artifact]

Computing the count of distinct elements in large data sets is a common task but naive approaches are memory-expensive. The HyperLogLog (HLL) algorithm (Flajolet et al., 2007) estimates a data set’s cardinality while using significantly less memory than a naive approach, at the cost of some accuracy. This trade-off makes the HLL algorithm very attractive for a wide range of applications such as database management and network monitoring, where an exact count may not be needed. The HLL algorithm and variants of it are implemented in systems such as Redis and Google Big Query. Recently, the HLL algorithm has started to be proposed for use in scenarios where the inputs may be adversarially generated, for example counting social network users or detection of network scanning attacks. This prompts an examination of the performance of the HLL algorithm in the face of adversarial inputs. We show that in such a setting, the HLL algorithm’s estimate of cardinality can be exponentially bad: when an adversary has access to the internals of the HLL algorithm and has some flexibility in choosing what inputs will be recorded, it can manipulate the cardinality estimate to be exponentially smaller than the true cardinality. We study both the original HLL algorithm and a more modern version of it (Ertl, 2017) that is used in Redis. We present experimental results confirming our theoretical analysis. Finally, we consider attack prevention: we show how to modify HLL in a simple way that provably prevents cardinality estimate manipulation attacks.

[Artifact]

[Artifact]

[Artifact]

[Artifact]

[Artifact]

[Paper]
[Artifact]

Program debloating aims to enhance the performance and reduce the attack surface of bloated applications. Several techniques have been recently proposed to specialize programs. These approaches are either based on unsound strategies or demanding techniques, leading to unsafe results or a high overhead debloating process. In this paper, we address these limitations by applying partial-evaluation principles to generate specialized applications. Our approach relies on a simple observation that an application typically consists of configuration logic, followed by the main logic of the program. The configuration logic specifies what functionality in the main logic should be executed. LMCAS performs partial interpretation to capture a precise program state of the configuration logic based on the supplied inputs. LMCAS then applies partial-evaluation optimizations to generate a specialized program by propagating the constants in the captured partial state, eliminating unwanted code, and preserving the desired functionalities. Our evaluation of LMCAS on commonly used benchmarks and real-world applications shows that it successfully removes unwanted features while preserving the functionality and robustness of the deblated programs, runs faster than prior tools, and reduces the attack surface of specialized programs. LMCAS runs 1500x, 4.6x, and 1.2x faster than the state-of-the-art debloating tools CHISEL, RAZOR, and OCCAM, respectively; achieves 25% reduction in the binary size; reduces the attack surface of code-reuse attacks by removing 51.7% of the total gadgets and eliminating 83% of known CVE vulnerabilities

[Artifact]

[Paper]
[Artifact]

Recent advances in fuzz testing have introduced several forms of feedback mechanisms, motivated by the fact that for a large range of programs and libraries, edge- coverage alone is insufficient to reveal complicated bugs. In- spired by this line of research, we examined existing program representations looking for a match between expressiveness of the structure and adaptability to the context of fuzz testing. In particular, we believe that data dependency graphs (DDGs) represent a good candidate for this task, as the set of information embedded by this data structure is potentially useful to find vulnerable constructs by stressing combinations of def-use pairs that would be difficult for a traditional fuzzer to trigger. Since some portions of the dependency graph overlap with the control flow of the program, it is possible to reduce the additional instrumentation to cover only “interesting” data-flow dependencies, those that help the fuzzer to visit the code in a distinct way compared to standard methodologies. To test these observations, in this paper we propose DDFuzz, a new approach that rewards the fuzzer not only with code coverage information, but also when new edges in the data dependency graph are hit. Our results show that the adoption of data dependency instrumentation in coverage-guided fuzzing is a promising solution that can help to discover bugs that would otherwise remain unexplored by standard coverage approaches. This is demonstrated by the 72 different vulnerabilities that our data-dependency driven approach can identify when executed on 38 target programs from three different datasets.

[Paper]
[Artifact]

Machine unlearning is the process through which a deployed machine learning model is made to forget about some of its training data points. While naively retraining the model from scratch is an option, it is almost always associated with large computational overheads for deep learning models. Thus, several approaches to approximately unlearn have been proposed along with corresponding metrics that formalize what it means for a model to forget about a data point. In this work, we first taxonomize approaches and metrics of approximate unlearning. As a result, we identify verification error, i.e., the L2 difference between the weights of an approximately unlearned and a naively retrained model, as an approximate unlearning metric that should be optimized for as it subsumes a large class of other metrics. We theoretically analyze the canonical training algorithm, stochastic gradient descent (SGD), to surface the variables which are relevant to reducing the verification error of approximate unlearning for SGD. From this analysis, we first derive an easy-to-compute proxy for verification error (termed unlearning error). The analysis also informs the design of a new training objective penalty that limits the overall change in weights during SGD and as a result facilitates approximate unlearning with lower verification error. We validate our theoretical work through an empirical evaluation on learning with CIFAR-10, CIFAR-100, and IMDB sentiment analysis.

[Artifact]

[Artifact]

[Artifact]

[Artifact]

[Paper]
Consumer IoT devices may suffer malware attacks, and be recruited into botnets or worse. There is evidence that generic advice to device owners to address IoT malware can be successful, but this does not account for emerging forms of persistent IoT malware. Less is known about persistent malware, which resides on persistent storage, requiring targeted manual effort to remove it. This paper presents a field study on the removal of persistent IoT malware by consumers. We partnered with an ISP to contrast remediation times of 760 customers across three malware categories: Windows malware, non-persistent IoT malware, and persistent IoT malware. We also contacted ISP customers identified as having persistent IoT malware on their network-attached storage devices, specifically QSnatch. We found that persistent IoT malware exhibits a mean infection duration many times higher than Windows or Mirai malware; QSnatch has a survival probability of 30% after 180 days, whereby most if not all other observed malware types have been removed. For interviewed device users, QSnatch infections lasted longer, so are apparently more difficult to get rid of, yet participants did not report experiencing difficulty in following notification instructions. We see two factors driving this paradoxical finding: First, most users reported having high technical competency. Also, we found evidence of planning behavior for these tasks and the need for multiple notifications. Our findings demonstrate the critical nature of interventions from outside for persistent malware, since automatic scan of an AV tool or a power cycle, like we are used to for Windows malware and Mirai infections, will not solve persistent IoT malware infections.

[Paper]
[Artifact]

Direct-to-consumer genetic testing services are gaining momentum: As of today, companies such as 23andMe or AncestryDNA have already attracted 26 million customers. These services raise privacy concerns, exacerbated by the fact that their customers can then share their genomic data on platforms such as GEDmatch. Notwithstanding their right to learn about their genetic background or to share their genomic data, it is paramount that individuals realize that such a behavior damages their relatives’ privacy (i.e., kin genomic privacy). In this paper, we present KGP Meter, a new online tool that provides means for raising awareness in the general public about the privacy risks of genomic data sharing. Our tool features various properties that makes it highly interactive, privacy-preserving (i.e., not requiring access to the actual genomic data), and user-friendly. It explores possible configurations in an optimized way and combines well-established graphical models with an entropy-based metric to compute kin genomic privacy scores. Our experiments show that KGP Meter is very responsive. We design and implement an interface that enables users to draw their family trees and indicate which of their relatives’ genomes are known, and that communicates the resulting privacy scores to the users. We then analyze the usage of the tool and survey users to better understand users’ perceptions towards these risks and evaluate our tool. We observe that most of them find the privacy score worrisome, and that the large majority of them find KGP Meter useful.

[Paper]
[Artifact]

CAPTCHAs are designed to prevent malicious bot programs from abusing websites. Most online service providers deploy audio CAPTCHAs as an alternative to text and image CAPTCHAs for visually impaired users. However, prior research investigating the security of audio CAPTCHAs found them highly vulnerable to automated attacks using Automatic Speech Recognition (ASR) systems. To improve the robustness of audio CAPTCHAs against automated abuses, we present the design and implementation of an audio adversarial CAPTCHA (aaeCAPTCHA) system in this paper. The aaeCAPTCHA system exploits audio adversarial examples as CAPTCHAs to prevent the ASR systems from automatically solving them. Furthermore, we conducted a rigorous security evaluation of our new audio CAPTCHA design against five state-of-the-art DNN-based ASR systems and three commercial Speech-to-Text (STT) services. Our experimental evaluations demonstrate that aaeCAPTCHA is highly secure against these speech recognition technologies, even when the attacker has complete knowledge of the current attacks against audio adversarial examples. We also conducted a usability evaluation of the proof-of-concept implementation of the aaeCAPTCHA scheme. Our results show that it achieves high robustness at a moderate usability cost compared to normal audio CAPTCHAs. Finally, our extensive analysis highlights that aaeCAPTCHA can significantly enhance the security and robustness of traditional audio CAPTCHA systems while maintaining similar usability.

[Paper]
[Artifact]

Information flow control and dynamic policies is a difficult relationship yet to be fully understood. While dynamic policies are a natural choice in many real-world applications that downgrade and upgrade the sensitivity of information, understanding the meaning of security in this setting is challenging. In this paper we revisit the knowledge-based security conditions to reinstate a simple and intuitive security condition for dynamic policies: A program is secure if at any point during the execution the attacker’s knowledge is in accordance with the active security policy at that execution point. Our key observation is the new notion of policy consistency to prevent policy changes whenever an attacker is already in possession of the information that the new policy intends to protect. We use this notion to study a range of realistic attackers including the perfect recall attacker, bounded attackers, and forgetful attackers, and their relationship. Importantly, our new security condition provides a clean connection between the dynamic policy and the underlying attacker model independently of the specific use case. We illustrate this by considering the different facets of dynamic policies in our framework. On the verification side, we design and implement DYNCOVER, a tool for checking dynamic information-flow policies for Java programs via symbolic execution and SMT solving. Our verification operates by first extracting a graph of program dependencies and then visiting the graph to check dynamic policies for a range of attackers. We evaluate the effectiveness and efficiency of DYNCOVER on a benchmark of use cases from the literature and designed by ourselves, as well as the case study of a social network. The results show that DYNCOVER can analyze small but intricate programs indicating that it can help verify security-critical parts of Java applications. We release DYNCOVER publicly to support open science and encourage researchers to explore the topic further.

[Paper]
[Artifact]

To prevent applications from leaking users’ private data to attackers, researchers have developed runtime information flow control (IFC) mechanisms. Most existing approaches are either based on taint tracking or multi-execution, and the same technique is used to protect the entire application. However, today’s applications are typically composed of multiple components from heterogenous and unequally trusted sources. The goal of this paper is to develop a framework to enable the flexible composition of IFC enforcement mechanisms. More concretely, we focus on reactive programs, which is an abstract model for event-driven programs including web and mobile applications. We formalize the semantics of existing IFC enforcement mechanisms with well-defined interfaces for composition, define knowledge-based security guarantees that can precisely quantify the effect of implicit leaks from taint tracking, and prove sound all composed systems that we instantiate the framework with. We identify requirements for future enforcement mechanisms to be securely composed in our framework. Finally, we implement a prototype in OCaml and compare the effects of different compositions.

[Artifact]

[Paper]
[Artifact]

Memory safety in complex applications implemented in unsafe programming languages such as C/C++ is still an unresolved problem in practice. Many different types of defenses have been proposed in the past to mitigate this problem. The most promising next step is a tighter integration of the hardware and software level: modern mitigation techniques are either accelerated using hardware extensions or implemented in the hardware by extensions of the ISA. In particular, memory tagging, as proposed by ARM or SPARC, promises to solve many issues for practical memory safety. Unfortunately, Intel x86-64, which represents the most important ISA for both the desktop and server domain, lacks support for hardware-accelerated memory tagging, so memory tagging is not considered practical for this platform. In this paper, we present the design and implementation of an efficient, software-only pointer tagging scheme for Intel x86-64 based on a novel metadata embedding scheme. The basic idea is to alias multiple virtual pages to one physical page so that we can efficiently embed tag bits into a pointer. Furthermore, we introduce several optimizations that significantly reduce the performance impact of this approach to memory tagging. Based on this scheme, we propose a novel use-after-free mitigation scheme, called xTag, that offers better performance and strong security properties compared to state-of-the-art methods. We also show how double-free vulnerabilities can be mitigated. Our approach is highly compatible, allowing pointers to be passed back and forth between instrumented and non-instrumented code without losing metadata, and it is even compatible with inline assembly. We conclude that building exploit mitigation mechanisms on top of our memory tagging scheme is feasible on Intel x86-64, as demonstrated by the effective prevention of use-after-free bugs in the Firefox web browser.

[Artifact]

[Artifact]

[Artifact]

[Paper]
[Artifact]

The Android operating system is currently the most popular mobile operating system in the world. Android is based on Linux and therefore inherits its features including its Inter-Process Communication (IPC) mechanisms. These mechanisms are used by processes to communicate with one another and are extensively used in Android. While Android-specific IPC mechanisms have been studied extensively, Unix domain sockets have not been examined comprehensively, despite playing a crucial role in the IPC of highly privileged system daemons. In this paper, we propose SAUSAGE, an efficient novel static analysis framework to study the security properties of these sockets. SAUSAGE considers access control policies implemented in the Android security model, as well as authentication checks implemented by the daemon binaries. It is a fully static analysis framework, specifically designed to analyze Unix domain socket usage in Android system daemons, at scale. We use this framework to analyze 200 Android images across eight popular smartphone vendors spanning Android versions 7-9. As a result, we uncover multiple access control misconfigurations and insecure authentication checks. Our notable findings include a permission bypass in highly privileged Qualcomm system daemons and an unprotected socket that allows an untrusted app to set the scheduling priority of other processes running on the system, despite the implementation of mandatory SELinux policies. Ultimately, the results of our analysis are worrisome; all vendors except the Android Open Source Project (AOSP) have access control issues, allowing an untrusted app to communicate to highly privileged daemons through Unix domain sockets introduced by hardware manufacturer or vendor customization.

[Paper]
[Artifact]

Verifiable Dynamic Searchable Symmetric Encryption (VDSSE) enables users to securely outsource databases (document sets) to cloud servers and perform searches and updates. The verifiability property prevents users from accepting incorrect search results returned by a malicious server. However, we discover that the community currently only focuses on preventing malicious behavior from the server but ignores incorrect updates from the client, which are very likely to happen since there is no record on the client to check. Indeed most existing VDSSE schemes are not sufficient to tolerate incorrect updates from the client. For instance, deleting a nonexistent keyword-identifier pair can break their correctness and soundness. In this paper, we demonstrate the vulnerabilities of a type of existing VDSSE schemes that fail them to ensure correctness and soundness properties on incorrect updates. We propose an efficient fault-tolerant solution that can consider any DSSE scheme as a black-box and make them into a fault-tolerant VDSSE in the malicious model. Forward privacy is an important property of DSSE that prevents the server from linking an update operation to previous search queries. Our approach can also make any forward secure DSSE scheme into a fault-tolerant VDSSE without breaking the forward security guarantee. In this work, we take FAST [1] (TDSC 2020), a forward secure DSSE, as an example, implement a prototype of our solution, and evaluate its performance. Even when compared with the previous fastest forward private construction that does not support fault tolerance, the experiments show that our construction saves 9× client storage and has better search and update efficiency.

[Artifact]

[Paper]
In recent years, oblivious pseudorandom functions (OPRFs) have become a ubiquitous primitive used in cryptographic protocols and privacy-preserving technologies. The growing interest in OPRFs, both theoretical and applied, has produced a vast number of different constructions and functionality variations. In this paper, we provide a systematic overview of how to build and use OPRFs. We first categorize existing OPRFs into essentially four families based on their underlying PRF (Naor-Reingold, Dodis-Yampolskiy, Hashed Diffie-Hellman, and generic constructions). This categorization allows us to give a unified presentation of all oblivious evaluation methods in the literature, and to understand which properties OPRFs can (or cannot) have. We further demonstrate the theoretical and practical power of OPRFs by visualizing them in the landscape of cryptographic primitives, and by providing a comprehensive overview of how OPRFs are leveraged for improving the privacy of internet users. Our work systematizes 15 years of research on OPRFs and provides inspiration for new OPRF constructions and applications thereof.

[Paper]
[Artifact]

Motivated by the rise of quantum computers, existing public-key cryptosystems are expected to be replaced by post-quantum schemes in the next decade in billions of devices. To facilitate the transition, NIST is running a standardization process which is currently in its final Round. Only three digital signature schemes are left in the competition, among which Dilithium and Falcon are the ones based on lattices. Classical fault attacks on signature schemes make use of pairs of faulty and correct signatures to recover the secret key which only works on deterministic schemes. To counter such attacks, Dilithium offers a randomized version which makes each signature unique, even when signing identical messages. In this work, we introduce a novel Signature Correction Attack which not only applies to the deterministic version but also to the randomized version of Dilithium and is effective even on constant-time implementations using AVX2 instructions. The Signature Correction Attack exploits the mathematical structure of Dilithium to recover the secret key bits by using faulty signatures and the public-key. It can work for any fault mechanism which can induce single bit-flips. For demonstration, we are using Rowhammer induced faults. Thus, our attack does not require any physical access or special privileges, and hence could be also implemented on shared cloud servers. We perform a thorough classical and quantum security analysis of Dilithium and successfully recover 1,851 bits out of 3,072 bits of secret key s1 for security level 2. The lattice strength against quantum attackers is reduced from 2128 to 281 while the strength against classical attackers is reduced from 2141 to 289. Hence, the Signature Correction Attack may be employed to achieve a practical attack on Dilithium (security level 2) as proposed in Round 3 of the NIST post-quantum standardization process.

[Paper]
[Artifact]

This paper investigates an adversary’s ease of attack in generating adversarial examples for real-world scenarios. We address three key requirements for practical attacks for the real-world: 1) automatically constraining the size and shape of the attack so it can be applied with stickers, 2) transform-robustness, i.e., robustness of a attack to environmental physical variations such as viewpoint and lighting changes, and 3) supporting attacks in not only whitebox, but also black-box hard-label scenarios, so that the adversary can attack proprietary models. In this work, we propose GRAPHITE, an efficient and general framework for generating attacks that satisfy the above three key requirements. GRAPHITE takes advantage of transform-robustness, a metric based on expectation over transforms (EoT), to automatically generate small masks and optimize with gradient-free optimization. GRAPHITE is also flexible as it can easily trade-off transform-robustness, perturbation size, and query count in black-box settings. On a GTSRB model in a hard-label black-box setting, we are able to find attacks on all possible 1,806 victim-target class pairs with averages of 77.8% transform-robustness, perturbation size of 16.63% of the victim images, and 126K queries per pair. For digital-only attacks where achieving transform-robustness is not a requirement, GRAPHITE is able to find successful small-patch attacks with an average of only 566 queries for 92.2% of victim-target pairs. GRAPHITE is also able to find successful attacks using perturbations that modify small areas of the input image against PatchGuard, a recently proposed defense against patch-based attacks.

[Paper]
[Artifact]

Neural backdoors represent one primary threat to the security of deep learning systems. The intensive research has produced a plethora of backdoor attacks/defenses, resulting in a constant arms race. However, due to the lack of evaluation benchmarks, many critical questions remain under-explored: (i) what are the strengths and limitations of different attacks/defenses? (ii) what are the best practices to operate them? and (iii) how can the existing attacks/defenses be further improved? To bridge this gap, we design and implement TROJANZOO, the first open-source platform for evaluating neural backdoor attacks/defenses in a unified, holistic, and practical manner. Thus far, focusing on the computer vision domain, it has incorporated 8 representative attacks, 14 state-of-the-art defenses, 6 attack performance metrics, 10 defense utility metrics, as well as rich tools for in-depth analysis of the attack-defense interactions. Leveraging TROJANZOO, we conduct a systematic study on the existing attacks/defenses, unveiling their complex design spectrum: both manifest intricate trade-offs among multiple desiderata (e.g., the effectiveness, evasiveness, and transferability of attacks). We further explore improving the existing attacks/defenses, leading to a number of interesting findings: (i) one-pixel triggers often suffice; (ii) training from scratch often outperforms perturbing benign models to craft trojan models; (iii) optimizing triggers and trojan models jointly greatly improves both attack effectiveness and evasiveness; (iv) individual defenses can often be evaded by adaptive attacks; and (v) exploiting model interpretability significantly improves defense robustness. We envision that TROJANZOO will serve as a valuable platform to facilitate future research on neural backdoors.

[Paper]
[Artifact]

Machine learning (ML) has made tremendous progress during the past decade and is being adopted in various critical real-world applications. However, recent research has shown that ML models are vulnerable to multiple security and privacy attacks. In particular, backdoor attacks against ML models have recently raised a lot of awareness. A successful backdoor attack can cause severe consequences, such as allowing an adversary to bypass critical authentication systems. Current backdooring techniques rely on adding static triggers (with fixed patterns and locations) on ML model inputs which are prone to detection by the current backdoor detection mechanisms. In this paper, we propose the first class of dynamic backdooring techniques against deep neural networks (DNN), namely Random Backdoor, Backdoor Generating Network (BaN), and conditional Backdoor Generating Network (c-BaN). Triggers generated by our techniques can have random patterns and locations, which reduce the efficacy of the current backdoor detection mechanisms. In particular, BaN and c-BaN based on a novel generative network are the first two schemes that algorithmically generate triggers. Moreover, c-BaN is the first conditional backdooring technique that given a target label, it can generate a target-specific trigger. Both BaN and c-BaN are essentially a general framework which renders the adversary the flexibility for further customizing backdoor attacks. We extensively evaluate our techniques on three benchmark datasets: MNIST, CelebA, and CIFAR-10. Our techniques achieve almost perfect attack performance on backdoored data with a negligible utility loss. We further show that our techniques can bypass current state-of-the-art defense mechanisms against backdoor attacks, including ABS, Februus, MNTD, Neural Cleanse, and STRIP.

[Artifact]