Home Resume Projects Hobbies PGP Key Experience Essays Guides Blog

Politically Incorrect (4Chan) Ethnogaphy

This page contains some teasers of the project I'm working on. Below I will show some screenshots, general plans, and resources that will contribute to the project. Likewise, I will share my current "blockages".

Notes and Thoughts

I think I’ll overall need to consult with people on how to approach this massive of a data analysis. I’ve done smaller web-scraping analyses before, but this is an enormous project.

A) Use Apache Spark for Pol Research because it is designed to handle very large data sets. I don’t have the RAM to do everything at once. However, I do have the CSV files for doing one-at-a-time analysis that I can export, and later compile for total analysis in R.

B) If 4chan closes, it will cause a lot of social problems because Pennebaker showed that writing about your problems, which is tantamount to 4chan posting, reduces behavioral upsets.

C) Perhaps I can put the CSV files into a PostgreSQL database and have R access that as a source instead of CSV?

D) I will probably have to ignore time as a factor and I can fakename dates as something like “per million rows, this was the most common topic” which is arbitrary but big enough to form a pattern since I have ~1200 files.

E) Possible Solutions: 1) Perhaps my GPU is stronger than my CPU and I can use that?; 2) CORS between server and desktop?; 3) After I write the code based on samples, maybe I can rent a big server through a tech company?; 4) Ask around on the R or JupyterLab forums to ask what the best approach to huge data analysis is.

F) What could this analysis show me? 1) Current declared beliefs can act as a filter for how current popular beliefs gained a foothold. There’s research on that, anyway. The number is disputed, but roughly 10% to 25% of a population needs to radically hold a belief to shift others into believing; 2) How does text content change over time? What persists over time? How can this be leveraged for understanding online culture?

G) Need to figure out how to make an R based survey. Maybe hostable on github or something for free? I don’t want these guys having my IP or anything.

H) Ask people to fill out surveys on beliefs and cross that data with text analysis?

Workflow for Archive-Only Data Gathering

Above is the process used for downloading the 4plebs archive that is stored on The Internet Archive. wget is a package commonly found on all Linux systems. The total tar archive downloaded was 42.4 Gb, and after running tar xf on the file, the expanded data resulted in 130 Gb. Unfortunately, this is one of the blockages I was speaking about before. I’m unable to input 130 Gb of data into R-Studio (Posit). For me to contain that data in my memory (RAM), I’d need to exceed 130 Gb of RAM.

Though, there is another work around which is that I break up the data stored in CSV format. Below, you can see some code that I used to split up the enormous CSV file of 130 Gb into 1266 files with 1,000,000 obersations each. This project would roughly contain 1.3 trillion observations that would need to be tokenized.

#!/bin/bash
split -l 1000000 -d —additional-suffix=.csv pol.csv pol
# Use split to..
# choose lines start-1,000,000
# -d add a number at the end
# --additional-suffix = ass .csv suffix
# use pol.csv
# name the new file pol{<number>}.csv

The screenshot below displays some sample output from one of the CSV files which would still need to be cleaned up. Using the tidyverse I am able to remove the numbers, and symbols, so that I can reduce my total tokens for analysis.

Ethnography Notes: Rstudio Videos, and Texts

https://tensorflow.rstudio.com/tutorials/keras/text_classification.html

https://bookdown.org/yihui/rmarkdown-cookbook/eng-bash.html

psyquil