Conversations About the Internet #5: Anonymous Facebook Employee

Facebook employees know better than most the value of privacy.

This past summer Facebook relocated from University Avenue in Palo Alto, CA — where several buildings fan out along the downtown strip — to a new central office in Stanford Research Park. A good friend and two-year veteran of Facebook invited me to check out the new space. When I arrived, a security guard handed me a non-disclosure contract to fill out, a requirement to enter the building. “Just making sure you’re not a Twitter spy,” he said. I can therefore not describe the tour my friend gave, though photos of the new space abound on the Internet. Afterwards, we went out for a drink at the Dutch Goose, a bar popular with techies and Stanford graduate students, where most of this conversation took place. Though forthcoming, my friend was anxious to preserve her anonymity; Facebook employees, after all, know better than most the value of privacy. As she is not permitted to divulge company secrets, and would like to remain employed, her name has been omitted from this interview. It provides an interesting snapshot of the inner workings and culture of Facebook in the summer of 2009.

The Rumpus: On your servers, do you save everything ever entered into Facebook at any time, whether or not it’s been deleted, untagged, and so forth?

Facebook Employee: That is essentially correct at this moment. The only reason we’re changing that is for performance reasons. When you make any sort of interaction on Facebook — upload a photo, click on somebody’s profile, update your status, change your profile information —

Rumpus: When you say “click on somebody’s profile,” you mean you save our viewing history?

Employee: That’s right. How do you think we know who your best friends are? But that’s public knowledge; we’ve explicitly stated that we record that. If you look in your type-ahead search, and you press “A,” or just one letter, a list of your best friends shows up. It’s no longer organized alphabetically, but by the person you interact with most, your “best friends,” or at least those whom we have concluded you are best friends with.

Rumpus: In other words, the person you stalk the most.

Employee: No, it’s more than just that. It’s also messages, file posts, photos you’re tagged in with them, as well as your viewing of their profile and all of that. Essentially, we judge how good of a friend they are to you.

Rumpus: When did Facebook make this change?

Employee: That was actually fairly recently, sometime in the last three months. But other than that, we definitely store snapshots, which is basically a picture of all the data on all of our servers. I want to say we do that every hour, of every day of every week of every month.

Rumpus: So this is every viewable screen?

Employee: It’s way more than that: it’s every viewable screen, with all the data behind every screen. So when we store your photos, we have six versions of your photos. We don’t store the original: we make six different versions on the photo uploader and upload those six versions.

Rumpus: And the difference between them would be sizing, certain areas are zoomed –

Employee: Exactly. Different sizes for the news feed, your profile pic, enlargement.

Rumpus: And these reside on servers in your office?

Employee: No, not in our office. Absolutely not. We have four data centers around the world. There’s one in Santa Clara, one in San Francisco, one in New York and one in London. And in each of those, there are approximately five to eight thousand servers. Each co-location of our servers has essentially the same data on it.

Rumpus: And how many users are you up to now?

Employee: That I can disclose publicly? Two hundred to two hundred twenty million.

Rumpus: And actually?

Employee: That’s just active users. As far as total accounts, including those that are potentially fake, disabled and whatnot, we’re over three hundred million. The two hundred twenty million are users who have logged on and done something with the site in the last thirty days.

Rumpus: You said they’re changing the policy of keeping all information.

Employee: No. They’re never changing that policy. We still keep all information. What I was referring to, is that if anything, we’re going to start deleting more photos for performance reasons. We are the largest photo distributor in the world.

Rumpus: Really? Is that obvious?

Employee: I don’t know the exact figures off the top of my head, but I want to say upwards of a trillion photos, and then think about six copies of each. This is the epitome of a needle in a haystack. When we need to load a webpage in half a second, we need to go and find upwards of a thousand photos — think about your newsfeed — in one get [snaps], and instantaneously. It’s hard to do.

Rumpus: You’ve previously mentioned a master password, which you no longer use.

Employee: I’m not sure when exactly it was deprecated, but we did have a master password at one point where you could type in any user’s user ID, and then the password. I’m not going to give you the exact password, but with upper and lower case, symbols, numbers, all of the above, it spelled out ‘Chuck Norris,’ more or less. It was pretty fantastic.

Rumpus: This was accessible by any Facebook employee?

Employee: Technically, yes. But it was pretty much limited to the original engineers, who were basically the only people who knew about it. It wasn’t as if random people in Human Resources were using this password to log into profiles. It was made and designed for engineering reasons. But it was there, and any employee could find it if they knew where to look.

I should also say that it was only available internally. If I were to log in from a high school or library, I couldn’t use it. You had to be in the Facebook office, using the Facebook ISP.

Rumpus: Do you think Facebook employees ever abused the privilege of having universal access?

Employee: I know it has happened in the past, because at least two people have been fired for it that I know of.

Rumpus: What did they do?

Employee: I know one of them went in and manipulated some other person’s data, changed their religious views or something like that. I don’t remember exactly what it was, but he got reported, got found out, got fired.

Rumpus: Have you ever logged in to anyone’s account?

Employee: I have. For engineering reasons.

Rumpus: Have you ever done it outside of professional reasons?

Employee: I will say, when I first started working there, yes. I used it to view other people’s profiles which I didn’t have permission to visit. I never manipulated their data in any way; however, I did abuse the profile viewing permission at several initial points when I started at Facebook.

Rumpus: How about reading their messages?

Employee: Never individually like that. I would mostly just look at profiles.

Rumpus: Would you suppose that Facebook employees might read people’s messages?

Employee: See, the thing is — and I don’t know how much you know about it — it’s all stored in a database on the backend. Literally everything. Your messages are stored in a database, whether deleted or not. So we can just query the database, and easily look at it without every logging into your account. That’s what most people don’t understand.

Rumpus: So the master password is basically irrelevant.

Employee: Yeah.

Rumpus: It’s just for style.

Employee: Right. But it’s no longer in use. Like I alluded to, we’ve cracked down on this lately, but it has been replaced by a pretty cool tool. If I visited your profile, for example, on our closed network, there’s a ‘switch login’ button. I literally just click it, explain why I’m logging in as you, click ‘OK,’ and I’m you. You can do it as long as you have an explanation, because you’d better be able to back it up. For example, if you’re investigating a compromised account, you have to actually be able to log into that account.

Rumpus: Are your managers really on your ass about it every time you log in as someone else?

Employee: No, but if it comes up, you’d better be able to justify it. Or you will be fired.

Rumpus: I would imagine they take this—

Employee: Pretty seriously. I don’t really fuck around, at all.

Rumpus: They invented a Chief Officer position for it, Chris Kelly, right?

Employee: Chief Privacy Officer Chris Kelly, correct. Running for Attorney General of California.

Rumpus: Is that a standard position at Silicon Valley web companies?

Employee: I think it’s becoming more of a standard officer position, especially with Web 2.0, 3.0, where the model is basically get as much information out there as you can. Obviously, someone needs to step back and make sure there is some information privacy here, or at least as much as we can put in place.

Rumpus: Facebook was probably a big trendsetter in that regard, right?

Employee: In my opinion, we’ve always provided the most nitty-gritty user privacy settings from the beginning. There’s no other site out there that’s this customizable.

Rumpus: Would you like to give your take on the last few rounds of fuck ups, Facebook Beacon, and the recent Terms of Service controversy?

Employee: It’s really hard to judge exactly the way users are going to react. We just didn’t have a good enough beta-testing system in place. When you have a group of twenty engineers working on a project, they think it’s the most beautiful, immaculate thing in the world, and then they build it, and a project manager approves it. Initially, when that was the case, we just pushed it, and if users didn’t like it we pulled it back. That was just our philosophy, one of trial and error. Whereas now we’ve started running psychological analysis, starting to…

Rumpus: Oh really?

Employee: Fuck yeah. Are you kidding me? We do eye-tracking to see where your eyes move while you browse Facebook.

Rumpus: What do you mean by “eye-tracking”?

Employee: For example, when we want to introduce new features, like when we streamlined the browsing of photo albums, you know, where you can click ‘next’ above the photo, and the page stays the same except you get the next photo? We did tests on that, and actually found out it increased the number of page views by 77%, essentially because we were reducing 77% of the page load, and therefore it was loading faster, and thus generating more clicks. We not only reduced our bandwidth, and how much we have to pay for our Internet, but we made the site faster and increased the clicks-per-minute, which is what we’re truly interested in.

Rumpus: So in what other ways do you track behavior, that isn’t necessarily obvious to users?

Employee: We track everything. Every photo you view, every person you’re tagged with, every wall-post you make, and so forth.

Rumpus: So maybe you know about this, maybe you don’t. There’s a paradox with international expansion, because obviously all internet companies aspire to a worldwide market, but as service enters countries without great infrastructure, such as 3rd-world countries, the companies have to provide the infrastructure and the countries don’t actually produce any (or much) ad revenue.

Employee: I don’t know anything about that, actually. The one comment I would make about that, is that we’ve definitely tried to continue expanding to 3rd- world countries. Take Iran — well, Iran is not a 3rd world country — but when the Iranian elections came up, and then the disputes, we found out they were using Facebook as a tool to organize themselves and expose their qualms and discontent with the government. So publicly we translated the entire site into Farsi within 36 hours. It was our second right-to-left language, which was actually really difficult for us. Literally the entire site is flipped in a mirror. The fact that we did it in thirty-six hours — they hired twenty some-odd translators, and engineers worked around the clock to get it rolled out — was pretty fucking phenomenal. We had at least three times as many user registrations per day the first day it was out, and it has been growing. So we’re definitely still serious about foreign outreach. And the thing is, we have such a gigantic market share in the larger sections of Europe, in Australia, in Mexico, in the States and Canada, and that’s where 99.9% of our ad revenue is and probably will be always — or at least will be the next five, ten years. So the fact that we’re breaching into these other markets mostly means just allowing family and friends to connect even more deeply, which is really our ultimate goal.

Rumpus: What’s the creepiest Facebook interaction you have had?

Employee: Well, the weirdest one I’ve ever seen was one I was able to investigate, one of the situations which required me to log into other accounts. This guy had emailed my friend at school a very very odd message, pertaining to the name ‘Caitlin,’ which is her name, and ‘poop.’ It was literally one of the creepiest things I’ve ever seen: a two-page message about the name ‘Caitlin’ and its semantic relation to ‘poop.’ We found out that he had actually sent it to the first two hundred Caitlins he found on Facebook search.

Rumpus: That’s weird.

Employee: Really weird. Out of nowhere, no reasoning. He started sending it twenty times a day, to different Caitlins, for three weeks or so.

Rumpus: What’s the most bizarre?

Employee: I found a fake account created from Berkeley that used the profile picture and information from the brother of one of my very good friends. We looked up the guy who created the original profile, and he had never ever heard of him, never ever met him, obviously had never seen him. But this guy had evidently added him as a friend, and sadly he accepted it, but literally stole all of this guy’s information, created a fake account, and was communicating with himself from the fake account. He was writing on his wall and posting back to the “other person’s” wall. We found out the guy actually had about fifteen fake accounts that he created, stealing other users’ pictures and information to create the accounts, and was actually communicating back and forth with himself. Just to try to make himself appear cool, I guess?

Rumpus: That’s a really sad display of humanity.

Employee: Yeah. That is the most bizarre encounter that comes to mind. Those two are the big instances I’ve seen that made me say, “What the hell is going on?”

Rumpus: So tell me about the engineers.

Employee: They’re weird, and smart as balls. For example, this guy right now is single-handedly rewriting, essentially, the entire site. Our site is coded, I’d say, 90% in PHP. All the front end — everything you see — is generated via a language called PHP. He is creating HPHP, Hyper-PHP, which means he’s literally rewriting the entire language. There’s this distinction in coding between a scripted language and a compiled language. PHP is an example of a scripted language. The computer or browser reads the program like a script, from top to bottom, and executes it in that order: anything you declare at the bottom cannot be referenced at the top. But with a compiled language, the program you write is compiled into an executable file. It doesn’t have to read the program from beginning to end in order to execute commands. It’s much faster that way. So this engineer is converting the site from one that runs on a scripted language to one that runs on a compiled language. However, if you went to go talk to him about basketball, you would probably have the most awkward conversation you’d have with a human being in your entire life. You just can’t talk to these people on a normal level. If you wanted to talk about basketball, talk about graph theory. Then he’d get it. And there’s a lot of people like that. But by golly, they can do their jobs.

Rumpus: So what will be the net effect of running the site on Hyper PHP?

Employee: We’re going to reduce our CPU usage on our servers by 80%, so practically, users will just see this as a faster site. Pages will load in one fifth of the time that they used to.

Rumpus: When’s it coming out?

Employee: When it’s done. Next couple of months, ideally.

Rumpus: So where do these geeks come from?

Employee: I would say at least 70% of Facebook engineers are from Harvard and Stanford.

Rumpus: Wow. I know Zuckerberg went Harvard, what’s the Stanford connection? I mean other than just Palo Alto.

Employee: I don’t think there’s any question that Stanford is the number one CS department in the world.

Rumpus: Stanford engineers invented Silicon Valley.

Employee: They did.

Rumpus: How has the recent move affected the company?

Employee: Facebook just moved offices to Stanford Research Park, which is where the original HP was started. Before it was kind of sprawled out. We had seven or eight offices downtown.

Rumpus: Any changes in atmosphere after the move?

Employee: It was just nice to have everyone in one office. Before, any meetings that happened were inconvenient for most people. I mean, engineering was split up into three offices. It was a pain. Now there’s more unity, more ease of communication. Everything feels more internal. It’s super-friendly. I think the coolest thing about the work environment is the trust. They don’t care what, where, how, when, as long as you get your shit done. If you want to work at a bar, the ball game, a park, the roof, they don’t give a fuck. Just get your shit done. Hence I was able to ditch work, come have two pitchers with you, and I will literally be able to go back and get my work done. And it goes a long way. Because I know I can get these things done. I know I’m going to have to go back. And I may be there until ten or eleven tonight.

Rumpus: I’m sorry we drank all these beers.

Employee: It’s the trust deal. We’re able to do that. We don’t have to worry. We can put our personal lives first, as long as we get our work done.

Translate :

The Rumpus :
Dalam server facebook, apa kamu menyimpan semua data yang masuk jika pengguna menguploadnya ? semisal foto, messages, baik yang masih ada maupun yang sudah di hapus ?

Facebook Employee :
Itu sangat benar, semua aktivitas yang kamu dilakukan di facebook, mulai dari membalas surat orang, upload foto, mengecek profil orang lain, dan menulis di wall orang lain, semuanya terdapat di server kami dan bisa kami lacak.

The Rumpus :
Ketika kamu bilang, “ mengecek profil orang lain “. Apa itu maksudnya kalian di kantor pusat facebook tau profil siapa saja yang kami klik ?

Facebook Employee :
Tentu saja ! Menurut mu, darimana kami bisa tau siapa saja teman baikmu dan siapa saja yang akan muncul di News Feed maupun Live Feed mu ? Ketika kamu login ke halaman facebookmu, mungkin kamu akan menyadari bahwa berita dari teman – teman terdekatmu saja yang muncul. Teman yang hanya kamu iseng2 add, mungkin tidak pernah kamu lihat perubahan statusnya. Kami tahu siapa saja teman terdekatmu di Facebook dari cara ini, sehingga kami bisa memilih informasi dari teman mana saja yang bisa kami tampilkan untuk anda.

The Rumpus :
Berapa banyak pengguna Facebook sekarang ?

Facebook Employee :
Yang bisa saya bocorkan ke anda secara public ? Sekitar 220 juta orang seluruh dunia.

The Rumpus :
Namun sebenarnya berapa ?

Facebook Employee :
Itu hanya user yang aktif 220 juta. Sebenarnya total pengguna FB di dunia ada sekitar 300 juta akun. Semua sekaligus dengan akun yang tidak aktif, palsu, cloning, dsb. 220 juta itu adalah angka pengguna Facebook yang masih aktif melakukan sesuatu terhadap profildia dalam 30 hari terakhir.

The Rumpus :
Kamu pernah menyinggung tentang “ Master Password “ yang sekarang katanya sudah tidak kamu gunakan ?

FB Employee :
Saya sudah lupa tepatnya kapan itu diresmikan, namun memang dulu kami punya master password untuk semua akun facebook di dunia. Jadi anda tinggal mengetikkan username yang anda inginkan, lalu passwordnnya dengan master password itu lagi, dan anda bisa masuk ke profil itu. Saya tidak akan memberitahu password itu kepada anda, namun saya bisa member informasi dan gambaran mengenai password itu, bahwa dengan menggabungkan semua tombol yang ada di keyboard, baik itu symbol, angka, tanda baca dan sebagainya, akan terlihat dan terbaca seperti “ Chuck Norris “. Itu master passwordnya ! Sangat kreatif memang !haha

The Rumpus :
Password itu bisa diakses oleh semua pegawai Facebook?

FB Employee :
Secara teknis iya, namun sebenarnya password itu hanya beredar di Bagian teknisi Facebook. Jadi tidak semua orang yang bekerja di kantor Facebook bisa menggunakan dan tahu tentang adanya master password itu. Sebenarnya master password itu diadakan untuk keperluan teknis jika ada masalah dengan server dan profil seseorang. Intinya password itu digunakan untuk keperluan kerja, bukan untuk main – main. Dan lagi, master password itu hanya bisa digunakan secara internal. Jika saya menggunakan master password itu di rumah saya, atau di Starbucks, password itu tidak akan jalann. Harus dilakukan di kantor facebook dengan ISP Facebook.

The Rumpus :
Menurut kamu apakah ada pegawai FB yang menyalahgunahi hak master password tersebut ?

FB Employee :
Saya tahu itu pernah terjadi di masa lalu. Ada 2 orang yang melakukan malpraktek dengan master password tersebut, dan mereka akhirnya dipecat.

The Rumpus :
Apa yang mereka lakukan memang ?

FB Employee :
Saya tidak tahu persis apa saja yang mereka lakukan. Namun yang pasti salah satu dari mereka masuk ke profil seseorang, mengganti agama dari orang tersebut. Orang tersebut melapor, dan pegawai tersebut ketahuan, dan akhirnya ia dipecat.

Rumpus :
Apakan kamu sendiri pernah masuk ke profil orang lain ?

FB Employee :
Pernah, untuk kepentingan kerja teknisi.

The Rumpus :
Pernahkan kamu masuk ke profil orang lain diluar kepentingan kerja ?

Facebook Employee :
Saya akan jujur, saya pernah ketika saya baru bekerja di Facebook awal – awal. Namun saya tidak memanipulasi data mereka, saya hanya masuk ke profil yang tidak bisa saya masuki. Namun ya, saya mengakui bahwa saya dulu pernah menyalahgunai hak teknisi saya untuk masuk ke profil orang lain.

The Rumpus :
Bagaimana dengan membaca inbox mereka ?

Facebook Employee :
Tidak pernah. Saya hanya melihat profil dan foto mereka. Tidak pernah melihat surat orang lain saya.

The Rumpus :
Menurut kamu, apakah para pegawai facebook suka membaca inbox orang lain ?

Facebook Employee :
Masalahnya adalah, dan mungkin tidak semua orang di luar IT tahu ini. Bahwa semua data yang kamu masukkan ke profil Facebook kamu itu akan tersimpan dan server dan database kami. Jadi, jika kami ingin membaca surat pribadi anda dengan selingkuhan anda. Kami tinggal melakukan pemanggilan query di database dan surat anda akan keluar. Itu yang tidak dipahami oleh kebanyakan orang di luar bidang IT.

The Rumpus :
Saya membayangkan bahwa mereka mengganggap hal ini merupakan sesuatu yang...

Facebook Employee :
Sangat serius ! Saya tidak pernah berani main – main dengan masalah privacy profil orang lain ini.

The Rumpus :
Apa interaksi Facebook yang paling aneh/seram yang pernah kamu temui ?

Facebook Employee :
Yang paling aneh dan pernah saya investigasi bahkan membuat saya mesti masuk ke profil orang tersebut adalah. Seorang yang mengaku bernama “ Caitlin “ mengirim surat kepada 200 orang pertama di Facebook yang memiliki nama belakang Caitlin dan menulis surat yang berisikan mengenai betapa jelaknya nama Caitlin dan betapa nama Caitlin itu memiliki korelasi dengan kata “Poop”.

The Rumpus :
Itu aneh.

Facebook Employee :
Sangat aneh. Dia terus menerus mengirim messages itu kepada seluruh Caitlin yang ada di dunia 20 menit sekali dalam tempo sekitar 3 minggu.

The Rumpus :
Apa yang paling aneh ?

Facebook Employee :
Saya menemukan sebuah profil palsu dari Berkeley. Dimana orang tersebut membuat profil dengan menggunakan nama dan foto dari orang lain. Ia sendiri mempunyai profil asli dengan nama dan foto – foto dia. Namun, dia membuat beberapa profil palsu dengan nama dan foto orang lain. Yang anehnya, semua profil palsu itu dia add sebagai teman di profil asli dia ! Dan lalu, dengan profil palsunya itu, ia menulis wall ke profil dia yang asli, Begitu juga sebaliknya. Mungkin agar ia bisa terlihat popular memiliki banyak interaksi dan wall2an dengan para cewe keren ataupun cowo tampan. Menyedihkan memang.

The Rumpus :
Itu merupakan cerminan hidup yang menyedihkan menurut saya.

FB Employee :
Betul sekali. 2 kasus tersebut membuat saya ke titik berkata, “ Apa yang terjadi dengan dunia ini ? “

The Rumpus :
Coba kamu ceritakan tentang Teknisi di Facebook.

FB Employee :
Mereka aneh, dan sangat – sangat pintar. Di sini yang saya maksud adalah sangat – sangat pintar. Tidak pintar biasa. See, semua halaman yang ada di facebook itu 90 persen itu menggunakan bahasa PHP. Bahasa PHP merupakan bahasa pemrograman yang “server side”, sehingga lebih memudahkan dalam hal pembuatan halaman dan loadinnya. Nah, teknisi kami, sekarang sampai bisa membuat bahasa pemrograman baru dari Facebook. Mereka sampai menciptkan yang namanya Hyper PHP. Jadi, secara kasarnya, mereka membuat website dengan cara baru yang mereka reka sendiri. Gila bukan. Namun, jangan pernah membiacarakan masalah umum dengan mereka jika anda tidak berada di level mereka. Jika anda membicarakan mengenai pertandingan basket hari minggu kemarin , anda mungkin akan mengalami perbincangan paling aneh sepanjang hidup anda. Anda hanya tidak bisa membicarakan kehidupan dan perbincangan normal dengan mereka. Jika anda ingin membicarakan basket, bicarakan “ Graph Teorynya “, Maka mereka baru mengerti. Memang tidak banyak orang seperti mereka, namun yang penting mereka bisa menyelesaikan kerja yang diberikan tepat waktu. Itu yang paling penting.

The Rumpus :
Jadi, darimana saja para kutu buku – kutu buku itu kalian dapatkan ?

FB Employee :
Saya bisa bilang bahwa 70 persen dari teknisi Facebook lulusan Harvard atau Stanford.

The Rumpus :
Wow, saya tahu bahwa Zuckerberg ( Empunya Facebook ) berasal dari Harvard, tapi mengapa Stanford ? Apa hubungannya ?

FB Employee :
Saya rasa Stanford sudah tidak perlu dipertanyakan lagi merupakan Departemen CS nomor 1 di dunia sekarang

The Rumpus :
Teknisi Stanford menemukan SIllicon Valley ( NB : TS juga tidak tahu Sillicon Valley thu apa. Hehe )

FB Employee :
Yep. Betul.

The Rumpus :
Terima kasih atas waktu anda. Mohon maaf waktu anda terbuang untuk minum – minum beer dan berbincang sekarang seperti ini.

FB Employee :
Tidak masalah. Selama saya bisa mengerjakan tanggung jawab saya. Kita masih bisa tetap mendahulukan personal life kita sebelum kerja jika kita orangnnya tanggung jawab.