Targeted Collection.

Contextually relevant Twitter data are collected using the Twitter API based on keywords, accounts, languages, and spatiotemporal filters specified by us. For example, during the 2017 French presidential election, keywords include the leading candidates, #Macron and #LePen, and French election-related issues, including hostile narratives expected to harm specific candidates, e.g., voter abstention (17) and unsubstantiated allegations (6, 18). Because specific narratives and influential IO accounts are discovered subsequently, they offer additional cues to either broaden or refocus the collection. In the analysis described in the preceding subsection, 28 million Twitter posts and nearly 1 million accounts potentially relevant to the 2017 French presidential election were collected over a 30-d period preceding the election, and a total of nearly 800 million tweets and information on 13 million distinct accounts were collected.

Narrative Detection.

Narratives are automatically generated from the targeted Twitter data using a topic modeling algorithm (19). First, accounts whose tweets contain keywords relevant to the subject or exhibit predefined, heuristic behavioral patterns within a relevant time period are identified. Second, content from these accounts is passed to a topic modeling algorithm, and all topics are represented by a collection or bag of words. Third, interesting topics are identified manually. Fourth, tweets that match these topics above a predefined threshold are selected. Fifth, a narrative network is constructed with vertices defined by accounts whose content matches the selected narrative and edges defined by retweets between these accounts. In the case of the 2017 French elections, the relevant keywords are “Macron,” “leaks,” “election,” and “France”; the languages used in topic modeling are English and French.

IO Account Classification.

Developing an automated IO classifier is challenging because the number of actual examples is necessarily small, and the behavior and content of these accounts can change over both time and scenario. Semisupervised classifiers trained using heuristic rules can address these challenges by augmenting limited truth data with additional accounts that match IO heuristics within a target narrative that does not necessarily appear in the training data. This approach has the additional intrinsic benefits of preventing the classifier from being overfitted to a specific training set or narrative and provides a path to adapt the classifier to future narratives. IO account classifier design is implemented using this semisupervised machine-learning approach built with the open source libraries scikit-learn and Snorkel (20, 21) and soft labeling functions based on heuristics of IO account metadata, content, and behavior. The feature space comprises behavioral characteristics, profile characteristics, languages used, and the 1- and 2-grams used in tweets; full details are provided in SI Appendix, section C, and feature importances are illustrated in Results (see Fig. 4). Our design approach to semisupervised learning is to develop labeling functions only for behavioral and profile characteristics, not narrative- or content-specific features, with the expectation that such labeling functions are transferable across multiple scenarios.

The classifier is trained and tested using sampled accounts representing both IO-related and general narratives. The approach is designed to prevent overfitting by labeling these sampled accounts using semisupervised Snorkel heuristics. These accounts are collected as described in Targeted Collection. There are four categories of training and testing data: known IO accounts (known positives from the publicly available Twitter elections integrity dataset) (9), known non-IO accounts (known negatives composed of mainstream media accounts), Snorkel-labeled positive accounts (heuristic positives), and Snorkel-labeled negative accounts (heuristic negatives). All known positives are eligible to be in the training set, whether they are manually operated troll IO accounts or automated bots. Because we include samples that represent both accounts engaging in the IO campaign and general accounts, our training approach is weakly dependent upon the IO campaign in two ways. First, only known IO accounts whose content includes languages relevant to the IO campaign are used. Second, a significant fraction (e.g., $$ 67\%$$) of Snorkel-labeled accounts (either positive or negative) must be participants in the narrative. Finally, we establish confidence that we are not overfitting by using cross-validation to compute classifier performance. Additionally, overfitting is observed without using Snorkel heuristics (SI Appendix, section C.2), supporting the claim that semisupervised learning is a necessary component of our design. Dimensionality reduction and classifier algorithm selection are performed by optimizing precision-recall performance over a broad set of dimensionality reduction approaches, classifiers, and parameters (SI Appendix, section C). In Results, dimensionality reduction is performed with Extra-Trees (ET) (20) and the classifier is the Random Forest (RF) algorithm (20). Ensemble tree classifiers learn the complex concepts of IO account behaviors and characteristics without overfitting to the training data through a collection of decision trees, each representing a simple concept using only a few features.

Network Discovery.

The narrative network—a social network of participants involved in discussing and propagating a specific narrative—is constructed from their observed pattern of interactions. In Results, narrative networks are constructed using retweets. Narrative networks and their pattern of influence are represented as graphs whose edges represent strength of interactions. The (directed) influence from (account) vertex $$ v_i$$ to vertex $$ v_j$$ is denoted by the weighted edge $$ a_{ij}$$. For simplicity in the sequel, network vertex $$ v_i$$ is also referred to as $$ i$$. The influence network is represented by the adjacency matrix $$ \mathit{A}=\left(a_{ij}\right)$$. Because actual influence is not directly observable, the influence network is modeled as a random variable with Poisson distribution parameterized by the observed evidence of influence. Specifically, influence $$ a_{ij}$$ is modeled with prior distribution $$ a_{ij}\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left(\text{frequency of interactions from}i\text{to}j\right)$$, as counts of interactive influence in real-world networks. Observations of past interactions or influence on a subset of edges can be used to estimate the rates on the missing edges through inference on a network model that captures realistic characteristics such as sparsity, varying vertex degrees, and community structure (22).

Impact Estimation.

Impact estimation is based on a method that quantifies each account’s unique causal contribution to the overall narrative propagation over the entire network. It accounts for social confounders (e.g., community membership, popularity) and disentangles their effects from the causal estimation. This approach is based on the network potential outcome framework (23), itself based upon Rubin’s causal framework (24). Mathematical details are provided in SI Appendix, section D.

The fundamental quantity is the network potential outcome of each vertex, denoted $$ Y_i\left(\mathit{Z},\mathit{A}\right)$$, under exposure to the narrative from the source vector $$ \mathit{Z}$$ via the influence network $$ \mathit{A}$$. Precisely, $$ \mathit{Z}$$ is a binary $$ N$$ vector of narrative sources (a.k.a. treatment vector). In this study, vertices are user accounts, edges represent influence as described in Network Discovery, and the potential outcomes are the number of tweets in the narrative. The influence network is an important part of the treatment exposure mechanism. An accounts exposure to the narrative is determined by both the sources and exposures to them delivered through the influence network. The impact $$ {\zeta }_j$$ of each vertex $$ j$$ on the overall narrative propagation is defined using network potential outcome differentials averaged over the entire network:

It is impossible to observe the outcomes at each vertex with both exposure conditions under source vectors $$ {\mathit{z}}_{j+}$$ and $$ {\mathit{z}}_{j-}$$; therefore, the missing potential outcomes must be estimated, which can be accomplished using a model. After estimating the model parameters from the observed outcomes and vertex covariates, missing potential outcomes in the causal estimand $$ {\zeta }_j$$ can be imputed using the fitted model. Potential outcomes are modeled using a Poisson generalized linear mixed model (GLMM) with the canonical log-link function and linear predictor coefficients $$ \left(\tau ,\mathit{\gamma },\mathit{\beta },\mu \right)$$ corresponding to the source indicator $$ Z_i$$, $$ n$$-hop exposure vector $$ s_i^{\left(n\right)}$$, the covariate vector $$ {\mathit{x}}_i$$, and the baseline outcome. The covariate vector $$ {\mathit{x}}_i$$ includes the potential social confounders such as popularity and community membership. These confounders are accounted for through covariate adjustment to disentangle actual causal impact from effects of homophily (birds of a feather flock together) and vertex degree on outcomes, by meeting the key unconfounded influence network assumption 2 in SI Appendix, Assumption 2. For correctness and rigor, imputation of the missing potential outcomes is designed to meet the unconfoundedness assumptions that lead to an ignorable treatment exposure mechanism under network interference, detailed in SI Appendix, section D.4. The GLMM for the potential outcomes is

Targeted Collection.

The targeted collection for the 2017 French presidential election includes $$ \mathrm{28,896,185}$$ potentially relevant tweets and $$ \mathrm{999,883}$$ distinct accounts, all collected over a 30-d period preceding the election on 7 May 2017. Targeted collections for several other IO scenarios were also collected during 2017, resulting in a dataset with $$ \mathrm{782,678,201}$$ tweets and $$ \mathrm{12,723,995}$$ distinct accounts. Of these, there are $$ \mathrm{3,151}$$ known IO accounts that posted in English or French (SI Appendix, Fig. S1). The great majority of accounts are unrelated to both these known IO accounts and the French election, but will provide negative examples for IO classifier training.

Narrative Detection.

Narratives immediately preceding the election are generated automatically by dividing this broad content into language groups; restricting the content and time period to election-related posts within 1 wk preceding the election’s media blackout date of 5 May 2017; and filtering accounts based on interaction with non-French media sites pushing narratives expected to harm specific candidates. Topic modeling (19) is applied to the separate English and French language corpora, and the resulting topics are inspected by us to identify relevant narratives. Two such narratives are illustrated in Fig. 2 by the most frequent word and emoji usage appearing in tweets included within the topic. From the English corpus ($$ N=\mathrm{152,203}$$), 15 topics are generated and from the French ($$ N=\mathrm{1,070,158}$$), 30 topics. These automatically generated topics correspond closely to allegations claimed to be spread by WikiLeaks, candidate Marine Le Pen, and others (14, 25). The role of bots in spreading allegations using the #MacronLeaks hashtag narrative is studied by Ferrara (26). Two topics, one in English and one in French, pertaining to unsubstantiated financial allegations (Fig. 2) will be used to identify accounts involved in spreading these narratives, independent of whether the accounts were used for narrative detection.

Fig. 2.

Word clouds associated with the “offshore accounts” topic in English (Top) and French (Bottom). Topics are selected from those generated from the English corpus ($$ N=\mathrm{152,203}$$) and the French corpus ($$ N=\mathrm{1,070,158}$$) as described in SI Appendix.

IO Account Classification.

Twitter published IO truth data containing $$ \mathrm{50,230}$$ known IO account identities and their multimedia content (9). IO accounts represent a tiny fraction of all Twitter accounts; e.g., Twitter’s $$ \mathrm{50,230}$$ known IO accounts are only 0.02% of its 330 million active monthly users, and bots are estimated to comprise 9 to 15% of all Twitter accounts (27). This dataset is used along with heuristic rules to train a semisupervised classifier (21) using the approach described in Methodology, and the rulesets are detailed in SI Appendix, section C. The classifier is trained and tested using content from these sources: known IO accounts that have tweeted on any topic at least once in either English or French, 20 known non-IO mainstream media accounts, and Snorkel-labeled accounts randomly drawn from our dataset, such that $$ 67\%$$ are topically relevant and $$ 33\%$$ are topically neutral. There are $$ \mathrm{3,151}$$ known IO accounts from Twitter’s dataset in our dataset that have tweeted in English or French. To account for a possible upper bound of up to 15% bots (IO related or not), we randomly select $$ \mathrm{15,000}$$ presumptively false examples with $$ \mathrm{5,000}$$ each of three false classes: topically relevant accounts that tweeted at least once in English, at least once in French, and topically neutral, randomly chosen from the general dataset (SI Appendix, Fig. S1).

Precision-recall performance.

Precision-recall (P-R) performance of the classifier is computed via cross-validation using the same dataset with a $$ 90:10$$ split, averaged over 20 rounds (Fig. 3). Because the number of known cases is limited, weak supervision from the heuristic Snorkel labeling functions is used to identify true examples before cross-validation (21). All training is performed with Snorkel-labeled data, and cross-validation is performed both using Snorkel-labeled data (solid curves in Fig. 3) and omitting Snorkel positives (dashed curves). Sensitivity to language-specific training data is also computed by restricting topically relevant, false examples to specific languages. All classifiers exhibit comparably strong performance, and small differences in relative performance are consistent with our expectations. All classifiers detect IO accounts with 96% precision and 79% recall at a nominal operating point, 96% area-under-the P-R curve (AUPRC), and 8% equal-error rate (EER). The English-only and French-only classifiers perform slightly better (nominally 0 to 3%) than the combined language model, consistent with the expectation that models with greater specificity outperform less-specific models. This strong classifier performance will be combined with additional inferences—narrative networks, community structure, and impact estimation—to identify potentially influential IO accounts involved in spreading particular narratives.

Fig. 3.

P-R classifier performance with $$ \mathrm{3,151}$$ known IO accounts (9) and language-specific training data: combined English and French (red curves$$ \hspace{0.17em}$$, $$ N=\mathrm{17,999}$$), English only (green curves$$ \hspace{0.17em}$$, $$ N=\mathrm{13,155}$$), and French only (blue curves$$ \hspace{0.17em}$$, $$ N=\mathrm{13,159}$$), cross-validated using a $$ 90:10$$ split and all data (solid curves) and Snorkel positives omitted (dashed curves).

With this dataset, the original feature space has dimension $$ \mathrm{1,896,163}$$: 17 behavior and profile features, 61 languages, and $$ \mathrm{1,896,085}\hspace{0.17em}1$$- and 2-grams. SI Appendix, section C lists the behavior, profile features, and language features. Grid search over feature dimensionality is used to identify the best feature set for dimensionality reduction: 10 behavioral and profile features, 30 language features, and $$ 500\hspace{0.17em}1$$- and 2-grams. The most important features used by the classifier are illustrated by the relative sizes of feature names appearing in the word cloud of Fig. 4. Note that the most important features for IO account classification are independent of topic and pertain instead to account behavioral characteristics and frequency of languages other than English or French. This topic independence suggests the potential applicability of the classifier to other IO narratives. Furthermore, the diversity of behavioral features suggests robustness against future changes of any single behavior.

Fig. 4.

The 100 most important features in the French and English combined classifier, represented by relative size in a word cloud.

Classifier performance comparisons.

Several online bot classifiers are used to report upon and study influence campaigns (3, 26, 29), notably Botometer (formerly BotOrNot) (30) and Bot Sentinel (31). Indeed, Rauchfleisch and Kaiser (32) assert based on several influential papers that analyze online political influence that Despite the differences between general, automated bot activity and the combination of troll and bot accounts used for IO campaigns, comparing the classifier performance between these different classifiers is important because it is widespread practice to use such bot classifiers for insight into IO campaigns. Therefore, we compare the P-R performance of our IO classifier to both Botometer and Bot Sentinel. This comparison is complicated by three factors: 1) Neither Botometer nor Bot Sentinel has published classifier performance against known IO accounts; 2) neither project has posted open source code; and 3) known IO accounts are immediately suspended, which prevents post hoc analysis with these online tools.

Therefore, a proxy for known IO accounts must be used for performance comparisons. We use the observation that there exists strong correlation between likely IO accounts in our narrative network and membership in specific, distinguishable communities independently computed using an MCMC-based blockmodel (28). Community membership of accounts in the French language narrative network (Fig. 2) is illustrated in Fig. 5. Five distinct communities are detected, three of which are identified to have promoted Macron allegation narratives. The other two narratives promote pro-Macron and pro-abstention narratives. Accounts in this narrative network are classified on a 0 to 1 scale of their similarity to known IO accounts, shown in Fig. 6. Comparing Figs. 5 and 6 shows that the great majority of accounts in the “Macron allegation” communities are classified as highly similar to known IO accounts and, conversely, the great majority of accounts in the pro-Macron and pro-abstention communities are classified as highly dissimilar to known IO accounts. This visual comparison is quantified by the account histogram illustrated in SI Appendix, Fig. S9.

Fig. 5.

Community membership in the French narrative network (Fig. 2). Colors show inferred membership from a blockmodel (28).

Fig. 6.

Classifier scores over the French narrative network (Fig. 2). The 0 to 1 score range indicates increasing similarity to known IO accounts.

Using membership in these Macron allegation communities as a proxy for known IO accounts, P-R performance is computed for our IO classifier, Botometer, and Bot Sentinel (Fig. 7). Note that Botometer’s performance in Fig. 7 at a nominal 50% recall is 56% precision, which is very close to the 50% Botometer precision performance shown by Rauchfleisch and Kaiser (32) using a distinctly different dataset and truthing methodology (ref. 32, figure 4, “all”). Given this narrative network and truth proxy, both Botometer and Bot Sentinel perform nominally at random chance of 63% precision, the fraction of presumptive IO accounts. Our IO classifier has precision performance of 82 to 85% over recalls of range 20 to 80%, which exceeds random chance performance by 19 to 22%. These results are also qualitatively consistent with known issues of false positives and false negatives in bot detectors (32), although some performance differences are also likely caused by the intended design of Botometer or Bot Sentinel, which is to detect general bot activity, rather than the specific IO behavior on which our classifier is trained.

Fig. 7.

Classifier performance comparison: Snorkel-based classifier (red curve$$ \hspace{0.17em}$$, $$ N=415$$) vs. Botometer (green curve$$ \hspace{0.17em}$$, $$ N=289$$) vs. Bot Sentinel (blue curve$$ \hspace{0.17em}$$, $$ N=288$$), given proxy, community-based truth. The dashed gray horizontal line at 63% is the fraction of presumptively true examples in the community-based proxy for known IO accounts and therefore represents random chance precision performance.

Network Discovery.

Tweets that match topics of Fig. 2 are extracted from the collected Twitter data. French language tweets made in the week leading up to the blackout period, 28 April through 5 May 2017, are checked for similarity to the French language topic. To ensure the inclusion of tweets on the #MacronLeaks data dump (10, 14, 26), which occurred on the eve of the French media blackout, English language tweets from 29 April through 7 May 2017 are compared to the English topic. In total, the French topic network consists of $$ \mathrm{6,927}$$ accounts and the English topic network consists of $$ \mathrm{1,897}$$ accounts. For visual clarity, network figures are generated on the most active accounts, 459 in the French and 752 in the English networks.

Impact Estimation.

Estimation on the causal impact of each account in propagating the narrative is performed by computing the estimand in Eq. 1, considering each account as the source. Unlike existing propagation methods on network topology (33), causal inference accounts also for the observed counts from each account to capture how each source contributes to the subsequent tweets made by other accounts. Results demonstrate this method’s advantage over traditional impact statistics based on activity count and network topology alone.

Impact estimation and IO classification on the English narrative network (Fig. 2) are demonstrated in Fig. 8. Graph vertices are Twitter accounts sized by the causal impact score (i.e., posterior mean of the causal estimand) and colored by the IO classifier using the same scale as Fig. 6. Redness indicates account behavior and content like known IO accounts, whereas blueness indicates the opposite. This graph layout reveals two major communities involved in narrative propagation of unsubstantiated financial allegations during the French election. The large community at the top left comprises many accounts whose behavior and content are consistent with known IO accounts. The relatively smaller community at the bottom right includes many mainstream media accounts that publish reports on this narrative. Within this mainstream journalism-focused community, the media accounts AP, ABC, RT, and Reuters are among the most impactful, consistent with expectation. The most remarkable result, however, is that known IO accounts are among the most impactful among the large community of IO-like accounts. The existence of these IO accounts was known previously (9, 12, 13), but not their impact in spreading specific IO narratives. Also note the impactful IO accounts (i.e., the large red vertices) in the upper community that appear to target many benign accounts (i.e., the white and blue vertices).

Fig. 8.

Impact network (accounts sized by impact) colored by IO classifier score on the English narrative network (Fig. 2). Known IO accounts are highlighted in triangles. Image credits: Twitter/JackPosobiec, Twitter/RT_America, Twitter/Pamela_Moore13, Twitter/TEN_GOP.

A comparison between the causal impact and traditional impact statistics is provided in Table 1 on several representative and/or noteworthy accounts highlighted in Fig. 8. The prominent @RT_America, a major Russian media outlet, and @JackPosobiec, a widely reported (14) account in spreading this narrative, corroborate our estimate of their very high causal impact scores. This is also consistent with their early participation in this narrative, high tweet counts, high number of followers, and large PageRank centralities. Conversely, @User1 and @User2 have low impact statistics and also receive low causal impact scores as relatively nonimpactful accounts. It is often possible to interpret why accounts were impactful. E.g., @JackPosobiec was one of the earliest participants and has been reported as a key source in pushing the related #MacronLeaks narrative (14, 26) (SI Appendix, Fig. S10). In that same narrative, another impactful account @UserB serves as the initial bridge from the English subnetwork into the predominantly French-speaking subnetwork (SI Appendix, section E).

Table 1.

Comparison of impact statistics between accounts on the English network: tweets (T), retweets (RT), followers (F), first tweet time on 28 April, PageRank centrality (PR), and causal impact (CI)

Screen name	T	RT	F	1st time	PR	CI*
@RT_America	39	8	386,000	12:00	2,706	1.55
@JackPosobiec	28	123	23,000	01:54	4,690	1.43
@User1†	8	0	1,400	22:53	44	0.14
@User2†	12	15	19,000	12:27	151	0.41
@Pamela_Moore13	10	31	56,000	18:46	97	1.65
@TEN_GOP	12	42	112,000	22:15	191	1.38

* Estimate of the causal estimand in Eq. 1.

† Anonymized screen names of currently active accounts.

Known IO account (9101112–13) @Pamela_Moore13’s involvement in this narrative illustrates the relative strength of the causal impact estimates in identifying relevant IO accounts. @Pamela_Moore13 stands out as one of the most prominent accounts spreading this narrative. Yet “her” other impact statistics (T, RT, F, PR) are not distinctive and comparable in value to the not-impactful account @User2. Additionally, known IO accounts @TEN_GOP and @TXCowboysRawk (9, 12, 13, 15) and Sputnik writer @CassandraRules (34) all stand out for their relatively high causal impact and IO account classifier scores (Fig. 9). Causal impact estimation is shown to find high-impact accounts that do not stand out using traditional impact statistics. This estimation is accomplished by considering how the narrative propagates over the influence network, and its utility is demonstrated using data from known IO accounts on known IO narratives. Additional impact estimation results are provided in SI Appendix, section E.

Fig. 9.

Impact versus classifier score, English narrative network (Fig. 2). Known IO accounts run by the Internet Research Agency (IRA) (9, 1112–13, 15) are highlighted. Image credits: Twitter/Pamela_Moore13, Twitter/TEN_GOP, Twitter/CassandraRules, Twitter/TXCowboysRawk.

Influential IO Account Detection.

The outcome of the automated framework proposed in this article is the identification of influential IO accounts in spreading IO narratives. This is accomplished by combining IO classifier scores with IO impact scores for a specific narrative (Fig. 9). Accounts whose behavior and content appear like known IO accounts and whose impact in spreading an IO narrative is relatively high are of potential interest. Such accounts appear in the upper right side of the scatterplot illustrated in Fig. 9. Partial validation of this approach is provided by the known IO accounts discussed above. Many other accounts in the upper right side of Fig. 9 have since been suspended by Twitter, and some at the time of writing are actively spreading conspiracy theories about the 2020 coronavirus pandemic (35). These currently active accounts participate in IO-aligned narratives across multiple geo-political regions and topics, and no matter their authenticity, their content is used hundreds of times by known IO accounts (9) (SI Appendix, section F). Also note that this approach identifies both managed IO accounts [e.g., @Pamela_Moore13, @TEN_GOP, and @TXCowboysRawk (9, 12, 13, 15)] as well as accounts of real individuals [@JackPosobiec and @CassandraRules (14, 34)] involved in the spread of IO narratives. As an effective tool for situational awareness, the framework in this article can alert social media platform providers and the public of influential IO accounts and networks and the content they spread.